The feature shipped in a six-week sprint. A small team at a B2B software company wired up a large language model to summarize customer support chats. The product manager got a promotion. The engineers added a new line to their resumes. For three months, it worked beautifully, saving their support staff thousands of hours.
Then the certified letter arrived from a law firm in Brussels.
A customer, exercising their rights under GDPR, had requested a data audit. The investigation revealed that the summarization model, in an effort to be helpful, had synthesized a report that included personal data scraped from a public-but-obscure forum post made by that same customer years earlier. The model had connected two identities that were never meant to be connected. It exposed information to a support agent that constituted a major breach. The potential fine runs into the millions.
This isn't a hypothetical. This is the quiet reality unfolding behind the triumphant blog posts announcing "Our New AI Assistant." The frantic race to bolt generative AI onto every product has created a vast, un-audited substrate of technological risk. We have spent the last two years building features. We will spend the next ten dealing with the consequences. This is a new kind of technical debt, but it’s not paid down with refactoring weekends. It’s paid in legal fees and regulatory penalties.
The core of the problem is that almost no one building these features truly knows what is inside the model they are calling. The foundational models from major labs are trained on staggering volumes of data vacuumed from the public internet, a chaotic soup of copyrighted code, private information from misconfigured databases, and biased screeds from forgotten message boards. The model providers offer indemnification, but read the fine print. The liability for the output of the model, and how it’s used in a specific context, often lands squarely on the company that deployed it. Your company.
When an engineer uses an open-source library, they can inspect the code. They can check its license. They can understand its dependencies. An LLM offers no such transparency. It is a black box that comes with a user agreement full of carefully worded disclaimers. What happens when your AI-powered coding assistant spits out a function that is a verbatim copy of a GPL-licensed library, and a developer unwittingly bakes it into your flagship proprietary product? Who is liable when your AI financial planning tool hallucinates a disastrous stock tip? The legal precedents are not settled; they are being forged right now in brutally expensive court battles.
The org chart was not built for this. The Chief Information Security Officer, whose job is to prevent data exfiltration, must now contend with a tool designed to ingest and process that same data in opaque ways. The General Counsel, accustomed to reviewing discrete contracts, is now being asked to sign off on a probabilistic system whose behavior can never be fully guaranteed. The engineers who were told to ship at all costs are now being asked to document the provenance of every
Generated by Reportify AI — Automate your team's status reports, standups, and weekly updates. Try free →