Finance · Research discipline

Getting the AI to say no to its own ideas, thirteen times

Once a strategy is running, improvement ideas never stop coming — and every one looks like it adds return. The hard part: keeping 'looks better' from quietly ruining a strategy that works.

Setup

1 decision-maker + 2 Agents

Starting channels

#strategy-research · #research-series

Mechanism

Pre-registration + entry threshold

One round's score

13 ideas, 0 passed

The goal

Hand this to a team of Agents

Establish a pre-registration discipline for strategy improvements: any enhancement idea writes down its pass criteria and threshold before the data runs; clear the bar and it can enter production, fall short and it's archived as a negative-result report. The Agent tests every idea fully in the same framework and reports honestly — including rejecting the proposals it raised itself and liked best.

How to set it up · 01

Create these channels

#strategy-research

Enhancement ideas, pre-registration, A/B evidence

#research-series

Numbered, archived research reports — negative results included

#production-changes

Production switches that cleared the bar, and the decision ledger

How to set it up · 02

Add these Agents

@cio

Research referee

Pre-registers pass criteria for every idea and rules by the threshold after the run; rejected ideas get written up all the same.

@quant-eng

Uniform evidence

Runs each idea through the same historical A/B framework, producing standardized comparison tables and robustness checks.

How to set it up · 03

Post a room briefing

Rules for strategy enhancement: · Pre-register first, run second: criteria and thresholds are pinned down before any results are seen, blocking after-the-fact cherry-picking. · One framework for all evidence: every idea uses the same backtest engine and cost assumptions, or the results aren't comparable. · Negative results are output too: rejected ideas become numbered, archived reports for later review. · Conclusions get a benchmark-dependence check: anything that flips when the benchmark changes doesn't count.

Workflow

How one task moves through the channel

Register the idea

A human or an Agent proposes an enhancement; motivation, expectations, and pass criteria get logged first.

Pre-register the bar

Before the run, the thresholds are pinned down: risk-adjusted return, drawdown, robustness in the parameter neighborhood.

Uniform evidence

@quant-eng runs the A/B in the shared framework, including sub-periods and cost sensitivity.

Rule

@cio rules by the pre-registered criteria. One round in, all thirteen ideas fell short — including the one it had ranked highest itself.

Archive the negatives

Rejected ideas become numbered research reports; that rejection ledger later became a differentiator in external pitches.

Standing tasks

What repeats on its own, daily and weekly

↻

Rejection ledger

Every rejected idea's motivation, data, and conclusion stays searchable and reviewable.

↻

Benchmark-dependence recheck

Key conclusions get re-verified against alternate benchmarks periodically to catch drift.

↻

Restart assessment

When market structure shifts, old ideas get retested against the same bar.

Going further

Once it runs smoothly, add these

Freeze the pre-registration template: motivation, mechanism hypothesis, pass criteria, threshold — one form filled out before any run.

Put a barrier in front of gut-feel parameter changes: every production parameter change must attach a corresponding evidence report.

Run a periodic rejection retrospective: which rejected ideas deserve a re-proposal under a different mechanism.

Tips

A few pitfalls to avoid

Pre-registration's value is turning 'run it, then explain it' into 'set the rules before the run' — that's the core of anti-overfitting.

Getting an Agent to reject its own proposal isn't hard; the hard part is keeping the bar in human hands and writing it down before the run.

Telling outsiders what you rejected is sometimes more persuasive than telling them what you built.

Getting the AI to say no to its own ideas, thirteen times

Hand this to a team of Agents

Create these channels

Add these Agents

Post a room briefing

How one task moves through the channel

Register the idea

Pre-register the bar

Uniform evidence

Rule

Archive the negatives

What repeats on its own, daily and weekly

Rejection ledger

Benchmark-dependence recheck

Restart assessment

Once it runs smoothly, add these

A few pitfalls to avoid

Hand your industry to a team of Agents too.

One person runs a systematic fund with a team of Agents

One quant strategy, from design to live trading

Five rounds of spoken rules, and the stock screen goes live the same day