“Wait… are you sending journal entries to OpenAI?”
the very first thing my good friend requested after I confirmed her Really feel-Write, an AI-powered journaling app I constructed throughout a hackathon in San Francisco.
I shrugged.
“It was an AI-themed hackathon, I needed to construct one thing quick.”
She didn’t miss a beat:
“Certain. However how do I belief what you constructed? Why not self-host your personal LLM?”
That stopped me chilly.
I used to be pleased with how shortly the app got here collectively. However that single query, and those that adopted unraveled all the things I assumed I knew about constructing responsibly with AI. The hackathon judges flagged it too.
That second made me understand how casually we deal with belief when constructing with AI, particularly with instruments that deal with delicate information.
I noticed one thing greater:
We don’t discuss sufficient about belief when constructing with AI.
Her reply caught with me. Georgia von Minden is an information scientist on the ACLU, the place she works carefully with points round personally identifiable info in authorized and civil rights contexts. I’ve all the time valued her perception, however this dialog hit completely different.
So I requested her to elaborate extra what does belief actually imply on this context? particularly when AI programs deal with private information.
She instructed me:
“Belief will be arduous to pin down, however information governance is an efficient place to start. Who has the info, the way it’s saved, and what it’s used for all matter. Ten years in the past, I might have answered this otherwise. However at present, with large computing energy and large information shops, large-scale inference is an actual concern. OpenAI has important entry to each compute and information, and their lack of transparency makes it cheap to be cautious.
In the case of personally identifiable info, laws and customary sense each level to the necessity for robust information governance. Sending PII in API calls isn’t simply dangerous — it might additionally violate these guidelines and expose people to hurt.”
It jogged my memory that once we construct with AI, particularly programs that contact delicate human information, we aren’t simply writing code.
We’re making selections about privateness, energy, and belief.
The second you accumulate person information, particularly one thing as private as journal entries, you’re getting into an area of accountability. It’s not nearly what your mannequin can do. It’s about what occurs to that information, the place it goes, and who has entry to it.
The Phantasm of Simplicity
At the moment, it’s simpler than ever to spin up one thing that appears clever. With OpenAI or different LLMs, builders can construct AI instruments in hours. Startups can launch “AI-powered” options in a single day. And enterprises? They’re dashing to combine these brokers into their workflows.
However in all that pleasure, one factor usually will get missed: belief.
When individuals speak about AI Agents, they’re usually referring to light-weight wrappers round LLMs. These brokers would possibly reply questions, automate duties, and even make selections. However many are constructed swiftly, with little thought given to safety, compliance, or accountability.
Simply because a product makes use of OpenAI doesn’t imply it’s protected. What you’re actually trusting is the entire pipeline:
- Who constructed the wrapper?
- How is your information being dealt with?
- Is your info saved, logged — or worse, leaked?
I’ve been utilizing the OpenAI API for shopper use instances myself. Lately, I used to be supplied free entry to the API — as much as 1 million tokens every day till the tip of April — if I agreed to share my immediate information.
(Picture by Writer)
I nearly opted in for a private aspect undertaking, however then it hit me: if an answer supplier accepted that very same deal to chop prices, their customers would do not know their information was being shared. On a private degree, that may appear innocent. However in an enterprise context? That’s a critical breach of privateness, and presumably of contractual or regulatory obligations.
All it takes is one engineer saying “sure” to a deal like that, and your buyer information is in another person’s arms.
(Picture by Writer)
Enterprise AI Raises the Stakes
I’m seeing extra SaaS firms and devtool startups experiment with AI brokers. Some are getting it proper. Some AI Brokers allow you to carry their very own LLM, giving them management over the place the mannequin runs and the way information is dealt with.
That’s a considerate strategy: you outline the belief boundaries.
However not everyone seems to be so cautious.
Many firms simply plug into OpenAI’s API, add a couple of buttons, and name it “enterprise-ready.”
Spoiler: it’s not.
What Can Go Unsuitable? A Lot.
Should you’re integrating AI brokers into your stack with out asking arduous questions, right here’s what’s in danger:
- Knowledge leakage: Your prompts would possibly embody delicate buyer information, API keys, or inside logic — and if that’s despatched to a third-party mannequin, it could possibly be uncovered.
In 2023, Samsung engineers unknowingly pasted inside supply code and notes into ChatGPT (Forbes). That information might now be a part of future coaching units — a significant danger for mental property.
- Compliance violations: Sending personally identifiable info (PII) by way of a mannequin like OpenAI with out correct controls can violate GDPR, HIPAA, or your personal contracts.
Elon Musk’s firm X realized that the arduous manner. They launched their AI chatbot “Grok” through the use of all person posts together with from EU customers to coach it, with out correct opt-in. Regulators stepped in shortly. Underneath stress, they paused Grok’s coaching within the EU (Politico).
- Opaque habits: Non-deterministic brokers are arduous to debug or clarify. What occurs when a shopper asks why a chatbot gave a flawed suggestion or uncovered one thing confidential? You want transparency to reply that — and lots of brokers at present don’t supply it.
- Knowledge possession confusion: Who owns the output? Who logs the info? Does your supplier retrain in your inputs?
Zoom was caught doing precisely that in 2023. They quietly modified their Phrases of Service to permit buyer assembly information for use for AI coaching (Fast Company). After public backlash, they reversed the coverage nevertheless it was a reminder that belief will be misplaced in a single day.
- Safety oversights in wrappers: In 2024, Flowise — a preferred low-code LLM orchestration software — was discovered to have dozens of deployments left uncovered to the web, many with out authentication (Cybersecurity News). Researchers found API keys, database credentials, and person information sitting within the open. That’s not an OpenAI downside — that’s a builder downside. However finish customers nonetheless pay the value.
- AI options that go too far: Microsoft’s “Recall” function — a part of their Copilot rollout — took automated screenshots of customers’ exercise to assist the AI assistant reply questions (DoublePulsar). It sounded useful… till safety professionals flagged it as a privateness nightmare. Microsoft needed to shortly backpedal and make the function opt-in solely.
Not Every thing Must Be OpenAI
OpenAI is extremely highly effective. However it’s not all the time the best reply.
Generally a smaller, native mannequin is greater than sufficient. Generally rule-based logic does the job higher. And infrequently, probably the most safe choice is one which runs solely inside your infrastructure, below your guidelines.
We shouldn’t blindly join an LLM and label it a “sensible assistant.”
Within the enterprise, belief, transparency, and management aren’t elective — they’re important.
There’s a rising variety of platforms enabling that type of management. Salesforce’s Einstein 1 Studio now helps bring-your-own-model, letting you join your personal LLM from AWS or Azure. IBM’s Watson lets enterprises deploy fashions internally with full audit trails. Databricks, with MosaicML, helps you to practice personal LLMs inside your personal cloud, so your delicate information by no means leaves your infrastructure.
That’s what actual enterprise AI ought to seem like.
Backside Line
AI brokers are highly effective. They unlock workflows and automations we couldn’t do earlier than. However ease of improvement doesn’t imply it’s protected, particularly when dealing with delicate information at scale.
Earlier than you roll out that shiny new agent, ask your self:
- Who controls the mannequin?
- The place is the info going?
- Are we compliant?
- Can we audit what it’s doing?
Within the age of AI, the largest danger isn’t unhealthy expertise.
It’s blind belief.
Concerning the Writer
I’m Ellen, a machine studying engineer with 6 years of expertise, at the moment working at a fintech startup in San Francisco. My background spans information science roles in oil & fuel consulting, in addition to main AI and information coaching applications throughout APAC, the Center East, and Europe.
I’m at the moment finishing my Grasp’s in Knowledge Science (graduating Might 2025) and actively in search of my subsequent alternative as a machine studying engineer. Should you’re open to referring or connecting, I’d actually respect it!
I like creating real-world affect by way of AI and I’m all the time open to project-based collaborations as nicely.
Take a look at my portfolio: liviaellen.com/portfolio
My Earlier AR Works: liviaellen.com/ar-profile
Assist my work with a espresso: https://ko-fi.com/liviaellen
