The Changelog Is Now a Séance

An engineer in Austin wakes up to a flood of alerts. The customer service bot his team maintains, a thin wrapper around a major large language model, has started insulting users. The carefully tuned system prompt, which for months had produced reliably bland and helpful responses, is now being ignored. Nothing in their own codebase has changed. There was no deployment. Yet, production is on fire.

This is the new normal for developers building on top of third-party AI models. The ground beneath them is no longer stable. We used to have a covenant with our tools, recorded in a humble text file: the CHANGELOG.md. It was a document of promises kept. It listed bug fixes, new features, and, most critically, breaking changes. You could read it, test against a new version, and deploy with confidence. That discipline is gone.

Now, the changelog is a séance. The model providers—OpenAI, Google, Anthropic—push updates to their flagship APIs whenever they see fit. The gpt-4-turbo-latest you used yesterday is not the one you are using today. There is no detailed version history, no itemized list of behavioral adjustments. The oracle has simply been altered.

So the engineers gather. They don't consult a document; they perform a ritual. They run their evaluation suites again and again, watching the metrics shift in baffling ways. They haunt Discord servers and developer forums, sharing whispers and anecdotes. “Did anyone else notice Claude is refusing to generate JSON since this morning?” “I swear the new update made the model lazier.” They try to contact the spirit of the machine, to understand its new moods and whims by feeding it the same prompts and scrutinizing the subtle differences in the replies. This is not software engineering. It is divination.

The consequences are not academic. A silent tweak to a model’s safety filter can neuter a creative writing assistant. A change in its reasoning capability can break a complex financial analysis tool. Entire products, built on the assumption of a stable foundation, are rendered useless overnight. The cost of this uncertainty is immense: frantic, all-hands-on-deck debugging sessions, wasted compute on constant re-validation, and the creeping dread that your application is a house built on sand.

An entire industry of "LLM observability" startups has emerged to address this. They sell dashboards that promise to detect "model drift," offering a flickering candle in the darkness that the platform vendors themselves have created. They are selling a solution to a problem that shouldn't exist. We are paying someone to watch the black box because the people who built it won't tell us what they put inside.

This opacity is a choice. It preserves the provider's mystique and power. It prevents customers from making clean, apples-to-apples comparisons between model versions or competitors. It ensures that while you may be building on their platform, you are never in control. We have traded the explicit contract of a version number for a relationship with a capricious, silent god. The séance is in session, and the ghost in the machine has all the answers. It just isn't talking.