Demystifying the Oracle

A Cal Bay AI℠ Essay

Introduction

Public talk about artificial intelligence is soaked in mystical language. Executives, investors, and headlines describe large language models as emerging minds, synthetic beings, or digital oracles.

This essay takes a different path. It looks at what these systems actually do — the mathematics, the mechanics, and the known failure modes — and then asks a practical question: if the machinery is statistics and linear algebra, who benefits when it is sold as something mystical?

The argument is simple. A language model is a powerful statistical engine that predicts the next piece of text. Treating it as an oracle hides how it fails, who is responsible when it does, and who profits from the confusion.

It does not possess wisdom; it possesses pattern density. It does not possess understanding; it possesses correlation.

Part I: What Actually Happens Inside the Machine

The illusion of machine understanding comes from fluency. Human beings evolved to link well-formed, context-aware language with a conscious speaker. When a machine writes fluent prose, we instinctively picture a mind behind it.

Under the hood, the process is a chain of numerical transformations.

Text → Tokens → Vectors → Attention → Feed-Forward Layers → Probabilities → Next Token → Repeat

Step	What Happens	In Plain Terms
1. Tokenization	Text is split into sub-word pieces called tokens, using fixed algorithms such as Byte-Pair Encoding.	The model never sees words — only numbered fragments.
2. Embedding	Each token number is converted into a long list of numbers (a vector), plus information about its position in the sentence.	Every fragment becomes a point in a space of thousands of dimensions.
3. Attention	Each position compares itself with every other position and decides how much to borrow from each.	A weighted mixing step that routes information between words.
4. Feed-Forward Layers	Each position passes through layers that act, in part, like stored associations learned during training.	Pattern lookups built from billions of examples.
5. Stacking	Steps 3 and 4 repeat across dozens to more than a hundred layers.	The same operations, applied again and again.
6. Prediction	The final numbers are converted into a probability for every token in the vocabulary. One token is drawn, added to the text, and the whole process runs again.	A weighted dice roll over what usually comes next.

For the Technical Reader

The core operation, introduced by Ashish Vaswani and colleagues in the 2017 paper “Attention Is All You Need,” is scaled dot-product attention:

Q = X·W_Q     K = X·W_K     V = X·W_V

Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V

Piece	What It Does
Q·Kᵀ	Scores how strongly each position’s “query” aligns with every other position’s “key” — a raw similarity measure.
÷ √d_k	Scales the scores down so the softmax doesn’t saturate in high dimensions.
softmax	Turns the scores into weights that add up to one.
· V	Blends the “value” vectors according to those weights.

The output step works the same way. The final hidden state is projected onto the vocabulary to produce a score for every possible token, and a softmax — adjusted by a “temperature” setting — turns those scores into probabilities:

P(next token = w) = exp(z_w / T) / Σ exp(z_k / T)

Researchers Mor Geva and colleagues have shown that the feed-forward layers behave in part like key-value memories, storing patterns seen during training.

None of these steps is deliberation in the human sense. The forward pass is fixed arithmetic; the only randomness is the weighted draw at the end. There is no step in this loop we can point to as reflection, intention, or a check against the outside world. What looks like thought is the accumulated weight of patterns in the training data.

Part II: Shortcut Learning — Right Answers for the Wrong Reasons

Because these models are trained to match their training data as closely as possible, they often find shortcuts: surface patterns that happen to correlate with the right answer, rather than the real cause. Researcher Robert Geirhos and colleagues named and documented this behavior across deep learning in 2020.

The Clinical Lesson

The danger is clearest in medicine. In a well-known 2018 study, John Zech and colleagues trained a model to detect pneumonia on chest X-rays. Part of what the model learned was to recognize which hospital an image came from — including markers and formatting specific to each site — because some hospitals had higher rates of disease. When tested on images from a hospital it hadn’t seen, its performance dropped.

On Paper	In Practice
The model gives the right diagnosis most of the time.	It may be keying on form headers, formatting, or hospital identifiers rather than the illness.
It passes an evaluation that checks only the final answer.	It fails quietly when deployed somewhere with different paperwork.

The lesson extends far beyond medicine. If no one checks the intermediate steps, a correct answer can be indistinguishable from a lucky guess. Real reliability means verifying how the system reached its answer — the evidence, the contraindications, the lab values — not just what it answered.

The Limits of “Showing Its Work”

Developers try to reduce shortcuts with chain-of-thought prompting and training that makes the model write out intermediate steps. This often improves accuracy, because generating more text gives the model more computation to work with. But it does not turn the system into a reliable reasoner.

Weakness	What Happens
Drift	The intermediate steps are generated the same way as everything else, and can contain invented facts.
Compounding Errors	One bad premise early in the chain can carry through to a confident, well-written conclusion that is simply wrong.
Unfaithful Explanations	Researcher Miles Turpin and colleagues showed in 2023 that a model’s written reasoning does not always reflect what actually drove its answer.
No Contact with Reality	Unless connected to external tools and checks, the model cannot test its statements against the world.

Part III: The Business of “Sentience Marketing”

If the technology is statistics and linear algebra, why is it so often described in mystical, human, or even religious terms?

Part of the answer is that the language is genuinely hard — these systems do surprising things, and the people building them disagree about what those abilities mean. But the framing also does commercial work. Whatever the intent behind it, mystical language produces clear financial and political advantages.

The Story Told	The Mechanical Reality	Who Benefits, and How
”The model shows emerging consciousness and agency.”	The model maps statistical patterns across enormous amounts of text.	Valuation. A “synthetic mind” justifies a far larger valuation than database or automation software.
”The system is an all-knowing oracle.”	The system produces probabilities based on its training text, without a guarantee of being right.	Lock-in. Businesses hand core decisions to a rented platform they cannot inspect.
”Frontier AI poses existential risk.”	Automated agents make errors, misfire tools, and can be manipulated.	Regulation. Rules built around the largest systems can favor the companies large enough to comply.
”The AI made the decision.”	People designed, trained, and deployed a system to meet a business goal.	Liability. Responsibility shifts from executives to “the algorithm.”

1. Valuation Through Myth

Ordinary enterprise software companies are typically valued at a multiple of their revenue. A company that says it is building a “synthetic mind” or “artificial general intelligence” steps outside those frameworks. The story of a new form of intelligence being born inside data centers draws speculative capital, sovereign wealth, and valuations that conventional software economics would struggle to support.

2. The Risk Narrative and the Regulatory Moat

Warnings about uncontrollable AI have helped shape proposals for licensing, mandatory testing, and compute thresholds. Several policies have used a training-compute threshold of 10²⁶ operations to define the “frontier” models that face extra reporting or safety obligations.

Critics argue that rules like these cost little for the largest technology companies — which have dedicated data centers and large legal teams — while weighing far more heavily on smaller developers and on open-weight models such as DeepSeek, Qwen, and independent fine-tunes that can be run outside the big clouds. In that view, safety language can double as a moat around the incumbents.

To be fair to the other side: many researchers, including some with no financial stake in the large companies, hold these safety concerns sincerely. The point here is not that every risk warning is false. It is that the same framing can also serve commercial ends, and readers should watch for both.

3. The Liability Shield

When an automated system denies an insurance claim, rejects a housing application, or makes a costly trade, institutions often say, in effect, the system made the determination. Personifying the software moves attention away from the people who designed and deployed it.

The courts have started to push back. In 2024, a Canadian tribunal ruled against Air Canada after its website chatbot gave a customer wrong information about bereavement fares. The airline argued the chatbot was responsible for its own statements; the tribunal rejected that argument and held the company responsible.

A mathematical function cannot hold fiduciary duty, carry liability, or stand trial. Every automated outcome traces back to human decisions: what data was used, what the system was optimized for, and where it was deployed.

Part IV: The Engineering Answer — Build the Harness

Reliable systems do not come from lecturing models about ethics or trusting voluntary pledges. They come from treating the model as a powerful but uncalibrated engine — and wrapping it in deterministic controls.

Request → Input Checks → Model Proposes → Harness Verifies → Approve or Reject → Human Signs Off on High-Stakes Actions

Layer	What It Does
1. Input Checks	Clean and validate every request before it reaches the model.
2. The Model	Generates a candidate output in a strict structured format, such as JSON or a tool call — never free text wired directly into a system.
3. The Harness	Deterministic code checks the candidate: Does it match the exact schema? Does it break a business rule, such as a trade-size cap? Is the target database read-only?
4. Pass or Fail	Failures are logged, retried, or sent to a supervisor. Only approved outputs move forward.

Three Rules for Builders

Rule	What It Means
Constrain the Output	Use structured outputs, schema validation (such as JSON Schema or Pydantic), and grammar-constrained generation so the model can only produce valid formats. Anything out of range is dropped before it touches a real system.
Sandbox the Agent	Any automated agent works inside an isolated account with hard limits on spending, speed, and permissions. It never shares write access with primary accounts or reserves. Hard limits — read-only connections, approved network destinations, transaction caps — hold even when a clever prompt or strange input fools the model.
Keep a Human Circuit Breaker	For high-impact actions — medical treatment, regulatory filings, moving money — the model acts only as an analyst. It presents candidate actions and its reasoning to a qualified person who makes the call. Every step is logged, so any failure can be traced to the model or to the harness.

This is The Human in the Loop expressed as engineering: the human is not a courtesy, but a structural safeguard.

Conclusion: Reclaiming Operational Sovereignty

The presentation of artificial intelligence as a mysterious, almost religious entity is, in large part, a commercial artifact. It inflates valuations, encourages dependence on rented platforms, shapes regulation, and blurs responsibility.

Strip the mystique away and what remains is still remarkable: a powerful general-purpose instrument built from statistics and linear algebra. It carries the intentions, biases, and choices of the people who build and deploy it.

Principle	What It Means
Reject the Cult of the Oracle	Treat every output as a probabilistic draft that needs verification, never a pronouncement.
Own What You Can	Compute does not have to be rented from a handful of providers. Efficient open-weight models running on your own hardware can handle many tasks without subscription extraction or data harvesting.
Build the Harness, Not Just the Prompt	Put your effort into the boundaries — validation, sandboxed accounts, human checkpoints — that protect people and assets from machine error.

It does not carry agency. It carries the intentions, biases, and parameters of the humans who trained and deployed it.

Grounded in mathematical reality rather than corporate myth, builders and business owners can use these tools with clarity and control — without panic, without tollbooths, and without surrendering their judgment.