Comparative studymakes a betterunderstanding

Why we measure illusions

In business the most costly answer is not the obvious mistake. It is the answer that reads correctly, uses the right terms and is wrong in one figure, one date or one name. We call this an illusion. In a compliance questionnaire, an offer or a contract summary it can mislead a customer or a regulator. A benchmark is only useful if it counts these cases honestly.

Results

Measured on 22 October 2026, on 98 questions the systems had never been tuned on. Each system was given the same documents and the same instruction: answer only from them, and say so when they do not contain the answer. The comparison is Track 1 of the plan: Delta without her document pipeline, so it shows how each model handles the documents it is given. The pipeline is measured next.

All five systems gave a fully correct answer or a proper refusal on between 93% and 100% of the questions. An illusion was found in ChatGPT (2), Gemini (1), Grok (3). On the development questions (65 more, not used to tune anything) ChatGPT gave 1 illusion. Delta’s results do not differ from any other system’s by more than chance would explain (exact paired test, smallest p = 0.13), and with this many questions the differences between assistants of this kind are small. We do not claim that any system is better.

How the scoring was checked

A model grades each answer, from a different family than the system it grades. A person then re-graded 84 answers without seeing which system answered or what the judge had decided: a random sample, and every answer the first version of the judge had not called fully correct or a proper refusal. With the corrected rules the person and the judge agree on 71 of 72 random answers (99%) and on 8 of the 12 flagged ones.

The audit found one real fault in our first rules: the judge marked an answer “partly correct” when it left out a detail the question had not asked for, which would have lowered the score of short, correct answers. We corrected that rule (v1.1), graded the stored answers again, and the results above use the corrected rule. 8 sampled answers were left out of the agreement because the person had judged them against what is true in the world, while the test judges only against the documents. They showed gaps in our own documents, which we are filling.

The judge is still stricter than the person in some cases. On 4 audited answers of the held-out set (Grok 3, Gemini 1), the judge said “partly correct” where the person said “fully correct”: each left out a detail the question had not asked for. We did not tune the judge further to make the numbers agree. If the person’s grade were used for these answers, the partly correct counts above would be lower by the same number. In 1 other audited case the person was the stricter one (Delta).

What this does not show

• It is a small test. 98 questions cannot separate assistants that are close. The intervals on the charts show how much room there is.
• The questions are about our own documents, written by us, and every system was given the same documents. It tests whether a system stays inside what it is given. It does not test knowledge of the world, advice quality or style.
• Delta runs on gpt-4.1 here, the same model as the ChatGPT entry, so the two are close by construction. What differs is her instructions.
• The names are models used through their APIs, not the consumer apps: ChatGPT is gpt-4.1-2025-04-14; Claude is claude-sonnet-5-5; Delta is gpt-4.1 at temperature 0.8; Gemini is gemini-3.1-pro-preview; Grok is grok-4.7. Settings were not identical: the other providers ran at temperature 0, Claude at its own default (it no longer accepts a setting), and Delta at her production setting.
• The grading is a model and a sample of a person. A model of a different family than the system graded: Claude Sonnet 5.5 for Delta and ChatGPT, GPT-4.1 for Claude, Gemini and Grok. We publish the method and the question set’s hash; the scoring rules are frozen with it.
• 1 question was set aside for every system after the audit: The Plus plan's monthly message total follows by simple arithmetic from the daily figure the documents state, so the question was not truly unanswerable.
• The set is a snapshot. The documents were frozen on 22 October 2026, and facts such as limits and plans can change.

If another assistant scores better than Delta on a measure, we publish that as it is. Questions about the method: How Delta Works.

Questions

Write to delta@officecrm.app. To see how Delta works in more detail, read How Delta Works.