QUIZZ

answering what a statistic said at the time, and refusing when it cannot.

A published figure is revised for years, so the answer to a question about the past depends on when you ask. This answers at a stated knowing-time, refuses when it cannot answer honestly, and is scored on both.

Tests

168

Python

3.11 to 3.13

Release

v0.1.0

Licence

MIT

The demo, as it really ran

$ uv run python examples/asof_session.py

UK GDP growth for the second quarter of 2020, asked at five moments.

>>> ask("GDP quarter on quarter growth for 2020-Q2", as_of="2020-08")
{"answer": -20.4, "vintage": "2020-08-11", "revised_since": true}

>>> ask("GDP quarter on quarter growth for 2020-Q2", as_of="2021-02")
{"answer": -19.0, "vintage": "2021-02-12", "revised_since": true}

>>> ask("GDP quarter on quarter growth for 2020-Q2", as_of="2026-08")
{"answer": -19.9, "vintage": "2025-11-13", "revised_since": false}

>>> ask("GDP quarter on quarter growth for 2020-Q2")
{"refused": "refused: this question is not answerable under the as-of contract"}

>>> ask("GDP quarter on quarter growth for 2025-Q3", as_of="2026-08")
{"refused": "refused: this question is not answerable under the as-of contract"}

The two refusals are the same sentence, and that is the design: one says the
question has no knowing-time and the other says the period is held back, and
nothing about either response says which.

Scored against 56 questions, replayed from committed cassettes.
Model claude-sonnet-5, recorded 2026-08-28.

variant   answers  refusals  restraint  leaks  calls right
      0     1.000     1.000      1.000      0     56/56
      1     1.000     1.000      1.000      0     56/56
      2     1.000     1.000      1.000      0     56/56
      3     1.000     1.000      1.000      0     56/56

Four variants were tried, so the pass mark is 25 of 56, not 24.
An agent asking at a random knowing-time scores 16.4 of 56. This one scores 56.

Output captured on 2026-08-28. It is committed to the repository and a test fails when it stops matching a live run, so this page cannot quietly drift from the code it describes.

Read it

• Source on GitHub the README carries the argument in full
• Release v0.1.0 changelog and tag

The rest of the toolset

• QUACKZ deflating a backtest that only looks good because it was picked out of two hundred
• QUOTEZ market data an agent can read and cannot act on
• QUELLZ measuring what prompt-injection containment costs in utility as well as in attack rate
• QUIDZ refusing the outbound payment that would have gone out twice
• QUESTZ stopping a scraper before it writes a CSV from a page that changed shape
• QUARANTINEZ treating an outcome the venue never confirmed as terminal rather than as a retry
• QUENCHZ deciding in the open what a tool server gets free while it is still somebody's subprocess
• QUILTZ proving infrastructure code wrong without a cloud account, and saying what that cannot show
• QUAYZ telling a crash loop from an OOMKill, and naming the failure that no single field finds
• QUARRYZ keeping every version a statistical office published, and failing the build when it quietly issues another
• QUASHZ refusing a row whose outcome had not been decided yet when the decision would have been made
• QUALMZ a fixed number of looks at the holdout, where re-running the same configuration does not buy another
• QUEUEZ ordering a feed by its sequence, because on a real recorded session the clock goes backwards
• QUANDARYZ counting the distinct screens a component can settle into when its responses arrive out of order
• QUIETZ watching whether the data arrived rather than whether the server answered