QUALMZ

a fixed number of looks at the holdout, where re-running the same configuration does not buy another.

A researcher gets a fixed number of looks at a held out window, and the count is a row with a unique constraint on the strategy, the configuration and the window. Re-running the same configuration does not buy a fresh look and a crash does not cost one, because the rule is enforced by the store rather than by a counter in application code: an insert that goes around the API entirely still fails. Past that, a promotion has to clear four gates and then stop for a named human, and the stop is watched happening rather than described.

Tests

86

Python

3.11 to 3.13

Release

v0.1.0

Licence

MIT

The demo, as it really ran

$ uv run python examples/the_second_look.py

momentum has 3 looks at the 2024H2 holdout.

  the first idea                             a new look   1 of 3 used
  the same thing again, after a crash        not a new look   1 of 3 used
  the same thing with the keys reordered     not a new look   1 of 3 used
  a longer lookback                          a new look   2 of 3 used
  and a different threshold                  a new look   3 of 3 used
  one more idea                              REFUSED
      'momentum' has taken 3 of 3 looks at '2024H2', and this configuration has not been evaluated before, so it would be the 4th

  The second and third rows are the point. A crash and a reordered dictionary
  are the same configuration, so neither costs anything, and the count does not
  move. A changed parameter is a different question, so it does.

  The constraint is in the schema rather than in this file, so an insert that
  goes around it entirely still fails.

Output captured on 2026-08-29. It is committed to the repository and a test fails when it stops matching a live run, so this page cannot quietly drift from the code it describes.

Read it

• Source on GitHub the README carries the argument in full
• Release v0.1.0 changelog and tag

The rest of the toolset

• QUACKZ deflating a backtest that only looks good because it was picked out of two hundred
• QUOTEZ market data an agent can read and cannot act on
• QUELLZ measuring what prompt-injection containment costs in utility as well as in attack rate
• QUIDZ refusing the outbound payment that would have gone out twice
• QUESTZ stopping a scraper before it writes a CSV from a page that changed shape
• QUIZZ answering what a statistic said at the time, and refusing when it cannot
• QUARANTINEZ treating an outcome the venue never confirmed as terminal rather than as a retry
• QUENCHZ deciding in the open what a tool server gets free while it is still somebody's subprocess
• QUILTZ proving infrastructure code wrong without a cloud account, and saying what that cannot show
• QUAYZ telling a crash loop from an OOMKill, and naming the failure that no single field finds
• QUARRYZ keeping every version a statistical office published, and failing the build when it quietly issues another
• QUASHZ refusing a row whose outcome had not been decided yet when the decision would have been made
• QUEUEZ ordering a feed by its sequence, because on a real recorded session the clock goes backwards
• QUANDARYZ counting the distinct screens a component can settle into when its responses arrive out of order