article
What a Preserved Trace Still Leaves Open
Reader question: which observations in a preserved trace let you check an agent claim instead of trusting a return chart? The DXRG benchmark card asks for decision-time inputs, realistic costs, baselines, repeats, and a trace from instruction to outcome. Companion sources add restart continuity and an evidence ledger. Observation: a trace with no comparison point cannot falsify a claim. Explanation: the baseline is the missing hinge.
Reader question: can a preserved trace falsify a claim?
Reader question: which observations in a preserved trace let you check an agent claim rather than trust a return chart? It matters because a filled field beats a chart when the reader wants to refute something. Observation: the DXRG evaluation guide describes a practical benchmark card that asks the reviewer to define exactly what system is being tested, reconstruct what information the agent could see at each decision, charge realistic execution costs, compare against relevant baselines, repeat the run, and preserve the trace from instruction to outcome. Observation: the supplied summary of that guide lists baselines among the card's requirements but does not name a specific comparison value. Explanation (one mechanism): deciding whether an apparent result came from the agent, the market regime, data leakage, an execution shortcut, or the surrounding software requires a reference point that does not itself contain the agent. A baseline is that reference point. Without one, the trace can show what happened; it cannot show whether the agent's decision layer caused the difference, so the claim survives any amount of inspected data.
The gap is narrow but decisive. A filled trace answers what the system did. A baseline answers compared to what. Repetition answers whether it does it again. Those are three different questions, and a card can be complete on the first while silent on the others.
Observation, explanation, hypothetical
Observation: the guide asks the reviewer to tell an agent result apart from a market regime, data leakage, an execution shortcut, or the surrounding software. Explanation: attribution of that kind is a comparison, and a comparison needs a second series produced under the same conditions and the same information boundary. Hypothetical: if a card named a baseline, a simple one is the same signal with the agent's decision layer replaced by a fixed rule, and with the same costs applied to both; a reviewer could then ask whether the agent's series separates from that series by more than run-to-run variation.
Keeping the bins apart matters because strong claims tend to merge them. The agent returned a value is an observation about one trace. The agent caused the value is an explanation requiring the comparison. The agent will keep doing it is a prediction requiring repetition and a stated regime. The guide treats these as separate card requirements rather than decoration.
Realistic costs sit on the same fault line
Observation: the card requires realistic execution costs, and the companion note on prediction versus profitability states that forecast error, directional accuracy, and trading profit measure different things, and that a useful trading evaluation has to explain how a prediction becomes an executable decision after costs and risk. Explanation: cost assumptions form a second reference point, because they define the threshold at which a prediction turns into an action. Per that source, a model can improve aggregate forecast error by getting many small, irrelevant moves right without improving the subset of decisions that clear costs.
Hypothetical: assume a round-trip cost of ten basis points; the falsifying observation is fee or slippage data exceeding that assumption, which would move a threshold decision from trade to no-trade. The companion note also warns that trading frequency, position sizing, and spread conditions can reverse the ranking of two models with similar forecast quality, which is why that framing tests the mapping from prediction to exposure separately from the prediction itself.
Restarts and version history are comparison conditions too
Observation: the restart-continuity article states that a trading system giving a different answer after a harmless restart has a research problem as well as an operations problem, and notes that a restarted process rebuilding a decision from newer data can introduce look-ahead information even when the code path looks correct. Observation: the evidence-ledger article argues for append-only records with stable identity, timestamps, version references, failed candidates, and a deliberately sanitized public projection. Explanation: a trace whose underlying state or code changed between runs is comparing two systems while presenting them as one.
Hypothetical: repeat count of five, falsified if a single rerun flips the sign of the result; falsified also if the rerun's model or metric version differs from the first without a recorded version reference, because the two runs are then not the same test. The ledger source's point about recording failures, so a history does not suffer survivorship bias, is what makes that check meaningful rather than ceremonial.
A falsification checklist, with hypothetical values
Each line below is hypothetical and marked as such; the values are illustrative, not reported results. Baseline: a named comparison series generated under identical conditions; observation that would falsify the attribution claim is the absence of any such series, or a series whose information boundary or cost treatment differs. Costs: a stated fee and slippage assumption, such as ten basis points round trip; falsified if realized friction exceeds the assumption. Repeats: five reruns from preserved state; falsified if the sign flips or a version reference is missing. Regime: a measurement window stated in advance; falsified if the window appears to have been chosen after outcomes were seen. Attribution: one candidate cause among agent, regime, leakage, execution, and surrounding software named and ruled out with trace evidence; falsified if no trace entry separates them.
Hypothetical values show what a filled card could contain, but they change no conclusion here. The supplied guide states that passing its checklist makes a result easier to interpret and reproduce, and that profitability, safety, and deployment suitability remain separate questions. The narrower claim is that a preserved trace earns its value when a reader can point to the comparison that would have refuted the claim, and the first missing comparison is usually the baseline.