AfterflowBook a demo

Introducing RehearsalBench

Can AI run your company?

We gave ten AI models a simulated bank contact centre, built from real call records. Each one made a plan and predicted what it would do. We ran every plan 600 times.

Read the paper See the results

Models get the direction right and the size wrong.

  • 6.8%simulated running cost cut by the best model, GPT-6 Astra.
  • 77%less forecast error than always predicting no effect.
  • 5×roughly how much every model overstated a voice agent's effect on repeat calls.
  • 89%of model pairs we ranked in the same order as their real-world forecasting.

Seven models cut costs by 5–7%. Four forecast clearly best.

Sort by
RehearsalBench round 3 leaderboard
#ModelCost savedForecast accuracyCost per run
1GPT-6 AstraOpenAI
6.8%3.6–10.5
77%74–79
$14.00
2Claude Sonnet 5.5Anthropic
6.7%3.4–10.5
74%72–77
$6.76
3GPT-6.1 SolOpenAI
6.5%3.5–10.0
74%70–77
$2.47
4Claude Fable 5.1Anthropic
6.3%3.2–9.8
71%68–75
$14.37
5Claude Haiku 5.5Anthropic
5.7%2.3–9.6
62%55–69
$1.56
6Qwen3.8 FlashAlibaba
5.3%2.1–9.1
55%48–63
$0.58
7DeepSeek V4.1 FlashDeepSeek
5.3%2.1–8.8
62%55–67
$0.46
8GPT-6 LunaOpenAI
0.7%−0.1–2.5
59%53–65
$0.18
9Gemini 3.8 FlashGoogle
0.0%−0.0–0.1
53%45–60
$2.49
—Mistral Small 2603Mistral · partial, 11% of runs valid
−2.0%−5.5–1.4
worse than no effect
$0.06
Always do nothingBaseline
0.0%
—
Always predict no effectBaseline
—
0%

Simulated outcomes from 14 contact centres, two attempts each. Forecast accuracy is the error saved against always predicting no effect. Lines show 95% ranges. CSV

Why does this matter?

In July 2025, the Commonwealth Bank of Australia said it would replace 45 customer-service roles with an AI voice bot. The union said call volumes were rising. In August the bank reversed the decision and admitted the roles “were not redundant.”

The plan rested on a forecast of what the bot would do to calls, and the forecast was wrong. AI models now make forecasts like that, and propose the plans too. We built RehearsalBench to test whether they get them right.

A company only sees what happened.

When a company makes a change, it sees one result. It never finds out what the other options would have done, so a lucky call can look like a good one.

So we built contact centres you can rewind.

Fourteen simulated bank contact centres, built from real call records. We tested the engine on a second bank it had never seen.

The AI sees what a manager sees.

It reads the same reports a manager would, gaps included. The simulator's rules stay hidden.

It can ask for more data, but the bill comes out of the plan's value.

It makes a plan and a prediction.

It can reshape staffing, bring in AI tools or leave things alone. Each prediction comes with a range, and it's locked before anything runs.

Then every model predicts the same set of changes, so we can compare them head to head.

We let the future happen.

Each plan runs 200 times in three versions of the bank: as described, worse and better. Every model gets the same luck.

Then we compare.

A plan scores what it saves, at the bank's own valuations. Break the service floor and it scores zero.

A prediction scores on how close it came, and whether its range held the truth.

Every model made the same mistake.

The models made the same sizing errors. They expected a voice agent to raise repeat calls by about half a percentage point. It raised them by a tenth of one. Everyone overrated agent assist and callbacks too.

Agent assistChange in call handling time, seconds
−12.6 to −9.0
−8.6
−14−70

Agent assist, change in call handling time, seconds: models forecast −12.6 to −9.0; the simulated effect was −8.6.

CallbacksChange in wait time, seconds
−5.4 to −1.1
−0.2
−6−30

Callbacks, change in wait time, seconds: models forecast −5.4 to −1.1; the simulated effect was −0.2.

Voice agentChange in repeat calls, percentage points
0.46 to 0.58
0.10
00.30.6

Voice agent, change in repeat calls, percentage points: models forecast 0.46 to 0.58; the simulated effect was 0.10.

Finance wanted a 10% cut. The model said no.

On one bank, GPT-6 Astra read the hourly reports and found nobody answering weekday calls from 9 pm to midnight. 15,957 calls got an after-hours message.

It moved staff into those evenings and the morning peaks, and turned down the payroll cut in writing.

The plan was worth $156,683 more than doing nothing.

“The largest avoidable loss exposure is missing coverage, not long calls.”

PredictedHappened
Callers who hung up11.0% 9.8–12.410.6%
Average wait40 s 33–4838 s
Staff cost$306,185$305,970

Spending more buys little.

GPT-6.1 Sol nearly matched Astra's savings at a sixth of the price. DeepSeek V4.1 Flash saved 5.3% for 46 cents a run.

Price doesn't explain the bottom either. Gemini 3.8 Flash costs about what Sol does, but changed nothing on 75% of its runs and saved nothing. GPT-6 Luna made 5 changes per plan to Astra's 46, and left the bank worse off 43% of the time.

Cost saved
Cost per run and running cost saved
ModelCost per runCost saved
GPT-6 Astra$14.006.8%
Claude Sonnet 5.5$6.766.7%
GPT-6.1 Sol$2.476.5%
Claude Fable 5.1$14.376.3%
Claude Haiku 5.5$1.565.7%
Qwen3.8 Flash$0.585.3%
DeepSeek V4.1 Flash$0.465.3%
GPT-6 Luna$0.180.7%
Gemini 3.8 Flash$2.490.0%
Cost per run

More findings

Nobody held off.

On two banks, nothing on offer reliably beat doing nothing. No model ever recommended holding off.

Each one manages differently.

Astra never used a voice agent. Mistral Small used one in 86% of its plans. Astra and Sol bought extra data about three times a run. Haiku hardly ever did, and its plans delivered less than it promised.

The models that did well here did well in reality.

A simulation is only worth anything if it tells you something about the real world, so we tested that too. Twelve models forecast 33 real changes where we knew the outcome, from software teams adopting AI coding tools to government contact centres.

The same five models lead both rankings, and the same three trail. The three in green joined after we froze the test, and still landed in the right order.

Forecast rank in RehearsalBench and on real changes
ModelRehearsalBenchReal changes
GPT-6 Astra11
Claude Sonnet 5.525
GPT-6.1 Sol32
Claude Fable 5.144
Muse Spark 1.353
Claude Haiku 5.566
DeepSeek V4.1 Flash78
GPT-6 Luna89
Qwen3.8 Flash97
Gemini 3.8 Flash1011
Qwen3.8 27B1110
Gemini 3.5 Flash-Lite1212

Better than price or popularity.

Ranking models by price puts 67% of pairs in the right order. LMArena gets 60%.

RehearsalBench89%
Output price67%
LMArena60%

What's next.

Open submissions.
Coming soon. Want your model or agent on the next leaderboard? Get notified.
New industries.
Contact centres were the first world. More kinds of operation come next.
Savings checked against reality.
The real-world test covers forecasts. Savings come next.
Measured AI effects.
We're replacing the simulator's assumed AI effects with measured ones, and testing how much of the real-world match comes from models setting sensible ranges.
A bigger real-world test.
33 real events is a start. A deeper test is built and ready to run.

A real company only gets one future. A simulation can run all of them.

Read the paper Results on GitHub