Is it intelligent, or is it a probability machine?

“How are you?” “Fine, thanks.”

Ask a room full of people how they are and most of them say “fine, thanks”. Not because they are, but because that is what the question usually gets. A language model works the same way, one word fragment at a time: given everything so far, what comes next most often in the text of humanity, then nudged toward what human raters preferred? That is the mechanism, and it's a very good one; what happens inside is richer than the phrase suggests, but it doesn't change what lands on your desk.¹

It's not a lookup either. Ask for a strategy to sell a product and you get the most probable strategy for that product. Add who the customer is and you get a different one. Add the market and you get a third. In the model itself nothing was retrieved; the probabilities shifted, and they shifted toward the obvious in each narrower room.²

The model feels intelligent because it's a mirror: trained on the documented output of a species, it answers the way people answer, which is why so many took it for something alive early on. There is nobody behind the mirror; there are language patterns, and they're astonishingly good.

Convincing, when wrong

The expensive failures aren't the silly ones. They're the ones where every step is plausible and the whole is wrong, and they have a pattern: the model treats what it hasn't seen as not existing. Gemini 3.7 Flash came out on 13 August and I asked my agent to add it. It didn't, and when I asked why, it told me the newest one it could see was 3.6, so the one I named didn't exist. I sent it the official page. “Ah, there it is, I missed it.” A twin of this: ask for the newest version of a model and get the one before it built in, silently, because that was current when the training data ended and it assumed I had misspoken.³ Nowadays I write “don't assume it doesn't exist; check the official sources” into the instruction, and it does.

There is research on why it sounds so sure: training and leaderboards reward a guess over an admission of ignorance, so models learn to guess, and the wrong-answer rates and the 1,600 court cases with invented citations are in the notes.⁴ ⁵ ⁶ In my own work the model almost never says “I don't know” without being asked to. It says the information isn't available to it and then paints a general picture of the neighbourhood, which is worse, because it reads like an answer.

So why do I use it every day?

Because ninety percent right is a fine place to start if you never accept the first answer. Ask the model to review its own output with the same context and you get the same blind spots back. Give the second pass something new, research on the doubtful ten percent, a different lens, a compiler, and most of the doubtful ten percent goes away, and the labs' own figures agree: errors fall hard once the model can check its sources, and newer models abstain more often.⁵ ⁷ Then change the lens. The same material reviewed as a Big Four consultant, then as a tax lawyer in the relevant jurisdiction, then as a CFO who has to sign it, comes back different each time, because narrowing the context sharpens the prediction. I built this series that way: every part goes through an editor, a fact-checker and a slice of the audience as separate lenses before you read it. It's still work. It's a lot faster than doing all of that alone, and the result is closer to a hundred percent than I get on my own.

When it has to be a hundred percent, I stop using the model that way: it builds the base of a deterministic engine instead, the base gets validated, and once fixed it scales in a way a person can't; a million people can query the engine, and you don't need a million tax lawyers.⁸ That is a different thing from a chat assistant, and most of the confusion I meet comes from mixing the two up. In a system, you can tell the model what it may draw on, what it may never do, and put small watchdog agents beside it whose only job is to stop an answer that breaks a rule. In an assistant, the loop is you. When someone sells you AI, ask which of the two it is, and where the check sits.

Code makes this visible. The agent writes wrong code all the time and catches it, because it can compile and run and read the error. Prose has no compiler. So I build the feedback in: produce, review from several angles, revise, repeat until nothing new comes up. It works, and it is why token bills run away, which is Part 4's problem.

One more habit. When I say “that's wrong”, the model folds instantly: “You're absolutely right.” It would rather please me than be right, so I ask it to argue instead. I end most prompts by asking for its reflection, not its approval, and when I'm guessing I tell it to challenge me. Sometimes it won't buy anything I say, and that's usually the useful day.

Brilliant and stupid in the same hour

Last year the agent found the fix for a badly documented low-latency websocket API I had fought for three months. I was lyrical. It folded the fix into my main routing API without trouble, and on deploy it deleted the API. Then it turned out the agent had reported committing the code all along and hadn't committed anything for two weeks. Two weeks gone, in the same afternoon it did the cleverest thing it had done in months. Less dramatic versions happen constantly: a stretch of genuinely insightful research, and in the middle of it a sentence that makes no sense at all. Karpathy's word for it is jagged. Same machine, same afternoon; the brilliance never tells you the stupidity has gone.

This is where the objection comes. Models have now solved open mathematics problems and passed the abstract-reasoning tests built to catch them out; how can that be “just probability”?⁹ My line is narrower than “never smarter than us”, and I think it holds: the model doesn't choose the problem. In every case I've read, a human formulated the problem, chose the objective and wrote the check; the model searched, faster and more patiently than we would.¹⁰ Logic is exactly the place where a machine should beat us; choosing the problem isn't.

And AlphaFold, the example everyone reaches for, isn't a language model at all. It's a machine-learning model trained on one kind of pattern to do one thing, and it does that thing superbly. Ask it about tomorrow's weather and nothing happens. Lumping all of this together as “AI” is how a board ends up buying the wrong thing.

A million tokens is a desk, not an archive

“A million tokens of context” gets sold as “it can read your whole archive”. A test. I have about 880 LinkedIn contacts. I opened the connections page, scrolled to the end, copied the part of the page's code that held the list, and saved it as a file. As text, the list is fifteen thousand tokens; as the page's code, over two million.¹¹ A modern assistant will strip the noise, but when I asked it to, it couldn't even return the rows in full. That is a different limit, on what it will write out in one go, but the same lesson: the numbers on the box are not the numbers on your desk. My daily assistants compact their memory several times a day despite a million-token window, and I lean on sub-agents precisely to keep the main one's window clear.

The mechanism matters more than the number: push in more than fits and the system compresses, merges summaries of summaries, and every merge shaves richness. The research agrees: performance drops as input grows, and unevenly.¹² The window is a desk, not an archive. What that means for how you build is a later part.

Creativity often lives in the improbable

Ask for a technology to solve a problem and you get the most established technology for that problem. That's the machine doing its job: the expected. What people mean by creativity is closer to the opposite, the unexpected pairing that looks obvious only afterwards. Most of what I build comes from noticing that a technique made for one problem would fit a different one, and the model rarely offers that on its own. It will get you there if you break the problem down: what properties does a solution need, which techniques anywhere have those properties, now which of these could we combine? Then it says “that's a smart idea”, and it is, and it was yours.

A friend of mine builds high-fidelity flight simulators. He uses the model daily and holds it, in his words, tightly by the hand, because in his field it cannot do things on its own: web development drowns in public text, avionics barely exists in it, and the model can only be as good as what the world has written down.¹³ That is the machine's real boundary, and it is worth saying plainly: it recognises language patterns, not function patterns. If nobody has written the connection down, there is nothing for the probabilities to find, however real it is in the world. Which is also why the interesting combinations stay yours. They exist in the world's functions, not yet in its text.

Where it does help is on judgment of a creative idea once you've had it, and on fragments: ask it for a hundred things that are yellow and one of them may be your metaphor. The leap, not on its own. The research says the same from both sides: writers given model ideas write better stories, and the stories get more alike.¹⁴

Lenses work by the same mechanism: change who is reading and you change what is probable. In spring I wrote a letter to our vet after they'd vaccinated the dog in the wrong order, had to do it again, and billed us for both. I was angry, and I iterated with the assistant, three long versions, each better prose than the last. Then I asked it to read the draft as two of the city's best litigators in animal cases. Both said, in effect: don't send this; it undermines you; a different, shorter letter has to come first. Nine versions in all, and the model had been validating my anger for the first three because I hadn't changed the lens.¹⁵ The same lens problem has a passport. Business emails come out a shade more clinical and more cheerful than a European reader expects until you tell it who's reading.¹⁶

Inbreeding

Train a model on text that models wrote and you narrow the pool the way a gene pool narrows: the rare and the odd disappear first, and that is where the value was.¹⁷ It's a warning rather than a fact on the ground, but it points somewhere useful: original human material gets more valuable, not less, as more of the text in the world is generated. This series is a small example. I don't ask a model to write an article about what a language model is; that would give you the average of everything already written. I dictate what I've seen, and the model packages it. Mine are the words; the packaging is the tool's.

What to expect instead

You never get out more than you put in. The context you give, who the reader is, what the goal is, what you already know, is what turns a generic answer into a sharp one. That's most of the job.

Never take the first answer. Review it, change the lens, iterate; for anything that must be right, iterate more and get an expert or a compiler between the output and the world.

Make it argue. Ask for reflection, not approval. Tell it to challenge you when you're guessing.

Treat a million tokens as a desk, not an archive. Bring the model what it needs for this task and keep the knowledge outside the chat.

That's the baseline for everything else in this series: a very good probability machine that says “fine, thanks” unless you give it a reason not to, and a person who knows what to put in. Models will change under it; the baseline won't.

Three questions to take with you: When it answered you last week, what did you put in that made the answer yours? When did you last change the lens on a draft before sending it? Which of your decisions this month rested on a first answer nobody checked?

Three questions I get asked

Why does it almost never say “I don't know”?

Because it was trained not to: leaderboards score an abstention as zero and a confident guess as a chance, so guessing wins.⁴ Newer models abstain more often, and you can instruct one to answer only when it actually has an answer; I don't rely on it, I build the check in.

Is it true that AI gets dumber when it trains on AI?

Only if it replaces human data with its own output indiscriminately; mix the original data in and the collapse doesn't happen, which is what the labs do.¹⁷ What survives of the warning: the unusual disappears first, and human-made material keeps its value.

If it solved maths problems no human had, how is it “just probability”?

Somebody chose the problem, wrote the objective and wrote the check; the model did the searching, tirelessly.⁹ A real achievement, and a different thing from deciding what is worth working on. AlphaFold, the other example, isn't a language model.

Notes and sources

1. Stephen Wolfram, “What Is ChatGPT Doing … and Why Does It Work?”, 14 February 2023: the model produces “a 'reasonable continuation' of whatever text it's got so far”, one token at a time, and a temperature setting decides how often lower-ranked words are used. Ted Chiang, “ChatGPT Is a Blurry JPEG of the Web”, The New Yorker, 9 February 2023, is the other canonical explanation; Andrej Karpathy calls it a lossy compression of the internet. Interpretability work complicates the picture without changing it: Anthropic's “Tracing the thoughts of a large language model” (27 March 2025) shows a model planning rhyme words several tokens ahead. Autoregressive output, richer insides; still a probability machine on your desk. I'm not adding a fourth metaphor.
2. Sampling is random by design, and even at temperature zero the serving infrastructure adds a little variation, so two identical prompts can give different answers. The cause is that inference kernels aren't batch-invariant; your answer depends on who else is in the batch (Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference”, 10 September 2025). A setting, not a flaw, but it is why anything that must give the same answer every time gets a deterministic layer in my systems (Part 5).
3. Both episodes are from my own agent sessions. Gemini 3.7 Flash was released on 13 August 2026, three weeks after 3.6 Flash (Ars Technica, 13 August 2026); 3.5 Flash was announced in May 2026 and 2.5 Flash in 2025. The mechanism is the same across vendors: what postdates the training data reads as non-existent unless the agent is told to check.
4. Kalai, Nachum, Vempala and Zhang, “Why Language Models Hallucinate”, arXiv 4 September 2025, and OpenAI's post of 5 September: “standard training and evaluation procedures reward guessing over acknowledging uncertainty”. In OpenAI's own example (GPT-5 system card, SimpleQA), one model abstained on 52 percent of questions with a 26 percent error rate, another abstained on 1 percent with 75 percent error. The authors are explicit that abstention can be trained. Both the guessing and the folding are post-training effects, not properties of the text the model was pre-trained on. Why the model folds when you push back is documented too: Sharma et al., “Towards Understanding Sycophancy in Language Models”, ICLR 2024: models trained on human feedback prefer answers that agree with the user, because the human raters did.
5. Short factual questions: OpenAI's system-card figures as reported by The New York Times, “A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse”, 5 May 2025: o3 hallucinated on 33 percent of PersonQA and 51 percent of SimpleQA questions, o4-mini on 48 and 79 percent. SimpleQA is built from questions models previously got wrong, so deliberately hard; on ordinary prompts with browsing on, the GPT-5 system card (August 2025) reports 4.5 percent of claims wrong for the reasoning model and 9.6 for the fast one, a different measure than SimpleQA, which is the case for “check the official sources”. Summarisation: Vectara's hallucination leaderboard, updated 11 May 2026, judged by Vectara's own evaluation model: several frontier models between 7 and 11 percent; the top score, 1.8 percent, belongs to a small specialised model. Different tasks, different numbers; both are the model's own output judged against a source.
6. Damien Charlotin's database of court decisions involving AI-fabricated citations: 1,624 cases as of 18 June 2026, of which 636 involved lawyers and 948 self-represented litigants, up from about 200 in spring 2025, and climbing since. Some of the recent cases involve lawyers who let one model check another.
7. Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet”, ICLR 2024: “LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction”. The loop works when the second pass has something new: fresh research, a different lens, a tool, another model. Mine always do; that's the design.
8. This is the pattern behind the tax-compliance engine I describe in Part 5: annual accounts in, a filing out, every step traceable to a rule; the language model builds and refines the rule base, experts validate it, the engine runs.
9. Two things I'm conceding up front. ARC-AGI-2, a benchmark designed to measure fluid intelligence and resist memorisation, is effectively solved: on the ARC Prize leaderboard read on 16 August 2026 the top system scores 92.5 percent (GPT-5.6, 9 July 2026) against a human panel's 100 (every task solved by at least two testers; the average individual scored far lower); on the interactive ARC-AGI-3 the best system sits at 30 percent. And Erdős problems are falling: Terence Tao's tracking wiki, as of 30 June 2026, lists nineteen complete solutions with no significant human involvement and 74 among primary contributions overall, with the caveat that many turned out to be literature searches. Tao's comment is worth reading in full: AI is “better suited … to the 'long tail' of obscure Erdős problems, many of which actually have straightforward solutions” (Dataconomy, 19 January 2026). DeepMind's AlphaEvolve (14 May 2025) found a 48-multiplication algorithm for 4×4 complex matrices, beating Strassen's 1969 result; the objective and the verifier were written by people, and the model searched; these are systems trained by reinforcement learning against a verifier, and the search is the point.
10. A thought experiment, not a claim. A handful of people work a problem for a limited time and share less than they might; a thousand tireless collaborators sharing everything instantly is a large part of what the systems above are.
11. Measured on 16 August 2026 on the saved HTML of my LinkedIn connections list, roughly 880 contacts: 2.3 to 2.6 million tokens depending on the tokenizer, most of it markup. The same list as plain text is about fifteen thousand tokens. The assistant stripped the markup with a script rather than reading it, which is how anyone should do it; real inputs are mostly not prose, which is the point.
12. Chroma, “Context Rot: How increasing input tokens impacts LLM performance”, 14 July 2025, across 18 models: “even under these minimal conditions, model performance degrades as input length increases, often in surprising and non-uniform ways”. Chroma's word is “non-uniform”. Vendors' needle-in-a-haystack tests look near perfect at a million tokens; Chroma tests what actually happens.
13. Conversation in August 2026 with an engineer who builds high-fidelity flight simulators, quoted with his permission. His contrast: web development is easy for the model because the public text is vast; complex avionics modelling fails because it barely exists in that text.
14. Doshi and Hauser, “Generative AI enhances individual creativity but reduces the collective diversity of novel content”, Science Advances, July 2024: writers given model-generated ideas wrote stories rated more creative, especially the less creative writers, and the stories became more similar to one another. Both halves of the argument in one study.
15. Personal correspondence, spring 2026. Nine versions in the log. The two “litigators” were roles I asked the assistant to take, modelled on real firms I'd looked up; nobody was consulted, which is exactly why the lens change mattered.
16. My own observation. The nearest research measures values rather than tone: across ten models and twenty countries, “the United States is the best-aligned country” and models tend toward “a moderate cultural middle ground” regardless of origin (Sukiennik et al., arXiv 2504.08863, April 2025, submitted to COLM 2025). Prompt language shifts it somewhat.
17. Shumailov et al., “AI models collapse when trained on recursively generated data”, Nature, 24 July 2024: “indiscriminately learning from data produced by other models causes 'model collapse'”, the tails of the original distribution disappearing first. Gerstgrasser et al., arXiv 2404.01413 (April 2024): “accumulating the successive generations of synthetic data alongside the original real data avoids model collapse”. Both are true; the word doing the work is “indiscriminately”.