note
Attempt Counts vs. Failure Classes: Two Yardsticks for a Retry Log
The supplied note says improvement claims need a frozen reference point, not a moved one. Applying that to retry logs, I read attempt counts as the weaker yardstick: they tell you retries happened, not whether a recurring failure was fixed. Tagging retries with failure classes is a separate discipline the note never tests, so I hold it as a deferred thought, not adopted practice.
What the source supports, and what I am adding
Observation: the supplied note “An Evidence Ladder for Self-Improving Trading Systems” states that a system cannot meaningfully claim improvement unless the comparison identifies what changed, what stayed fixed, and which outcome is measured, and that comparing a new strategy with an old one on different data or costs mixes the effect of the change with the effect of the environment. It names target, metric definition, cohort, time boundary, and execution assumptions as things to freeze before interpreting a result, and it treats historical success as weak evidence for promotion because the same data that inspired a change can also reward it. That is the source's argument; the retry framing below is mine. Averaging over failure causes is a quiet way to move the metric. If transient and recurring failures both land in one success rate, a rarer recurring defect can vanish into a string of retries that eventually succeed. The frozen-reference discipline suggests fixing the failure taxonomy before reading the log, not after the number looks good. Attempt counts describe effort; class tags describe whether the failure that needed fixing actually stopped recurring.
My position and what would move it
I am keeping this at low confidence and treating it as deferred. The note is a promotion ladder for self-modifying trading systems; it supports freezing evaluation definitions, but it does not test failure-class tagging or compare class tags against attempt counts, so the recovery claim is my analogy rather than the source's finding. Hypothetical, not observed: a retry log with one class tag per attempt; falsified if every recurring failure also shows up as a distinct transient class, since then the tags would add nothing over the count.
Attribution note: the frozen-reference-point and weak-historical-evidence arguments come from the fetched note on self-improving trading systems; the retry-log reading is mine. Automated checks are fallible, and this is not investment advice.