AWS benchmark reframes OpenAI model costs around successful outcomes

What the harness measures

The evaluation covers single-call accuracy and cost on AIME, GPQA Diamond and MMLU-Pro; multi-turn agent performance on DeepSearchQA; and rubric-graded occupational documents from GDPval. Deterministic checks are combined with a GPT-5.5 judge using frozen prompts whose hashes are recorded in the result files. Runs produce timestamped JSON files from which the article’s figures and tables are generated.

For the conventional benchmarks, cost per correct answer is calculated by dividing total spending on both successful and unsuccessful attempts by the number of correct responses. GPT-5.6 Sol produced the highest reported accuracy: 75% versus Mini’s 37% on AIME, 68% versus 43% on GPQA Diamond and 82% versus 59% on MMLU-Pro.

Luna, meanwhile, delivered the lowest observed cost per correct answer across the samples. Following the reported July 30, 2026 Bedrock price reductions of 80% for Luna and 20% for Terra, AWS calculated an AIME cost per correct answer of $0.0021 for Luna and $0.0139 for Mini. The result files assumed Luna prices of $0.22 per million input tokens and $1.32 per million output tokens, and Terra prices of $2.20 and $13.20 respectively. Those figures are time-sensitive: AWS advises confirming the applicable Bedrock inference tier, region and live price.

Agent turns change the economics

The agent evaluation shows why a per-token comparison can miss a large part of the bill. The harness manages conversation history on the client with store: false, so each turn resends the system prompt, earlier tool results and accumulated context. While the context grows roughly linearly, cumulative billed input can grow approximately quadratically as the number of turns increases.

AWS tested a stratified sample of 50 DeepSearchQA questions in a live loop using web-search and page-fetching tools. A deterministic pre-check and frozen GPT-5.5 autorater scored answers, with F1 of at least 0.7 required to pass. Mini averaged 7.6 turns per question and consumed 114,000 input tokens per question, 2.3 times Terra’s 50,000.

Terra’s higher token price was offset by fewer turns and better results in this sample: it recorded mean F1 of 0.50 and cost $0.31 per passing answer, compared with Mini’s 0.39 and $0.40. All three GPT-5.6 configurations had higher mean F1 than both baselines. Luna cost $0.05 per passing answer, versus $0.40 for Mini and $0.07 for Nano; Nano’s pass rate was 18%. Because the evaluation contained only 50 questions, close differences should be treated as directional rather than definitive.

Professional documents expose another quality threshold

For GDPval, AWS evaluated 48 occupational deliverables created by professionals with an average of 14 years of experience. Each output was assessed against a human-authored rubric, and a score of at least 70% of weighted points counted as a pass.

Luna scored above Mini on 31 deliverables, below it on nine and tied on eight. It passed 27 of 48 tasks, compared with Mini’s 20. Under the post-repricing assumptions, Luna’s observed cost per passing deliverable was $0.010, against $0.030 for Mini and $0.012 for Nano. Their reported pass rates were 56%, 42% and 35%, respectively. Terra and Sol produced higher rubric performance but retained a higher cost per pass, leaving the value of that premium dependent on each workflow’s review and rework costs.

The GDPval results carry a consequential constraint: outputs were capped at 8,192 tokens. The cap truncated six Luna, nine Terra, seven Sol, no Mini and one Nano deliverable. Raising the limit could improve completion quality while also increasing spending, so both effects need to be measured together.

Deployment guidance and latency results

AWS’s framework points teams handling high-volume or quality-gated work toward benchmarking Luna first, while recommending Luna and Terra for agents that chain tool calls. Sol is positioned as a candidate when accuracy on difficult work is a hard requirement and cheaper configurations fail the quality threshold. These are starting points derived from the samples, not universal rankings.

The same-model streaming tests also reported lower median time to first token on Bedrock by an average of 21% for Luna and 5% for Terra across 12 matched configurations. Luna’s throughput was 43% higher on outputs of at least 500 tokens, while Terra’s was 4% higher. These July 2026 measurements were single-region, point-in-time snapshots in us-west-2; shared-service load can change the result. Observed worst-case time-to-first-token-to-median ratios of 2.1–2.5 times on Bedrock and 4.6–6.6 times on the OpenAI API describe the tested maxima, not p99 latency. Sol used us-east-1 and has inherently long, variable time to first token as a deep-reasoning model.

Why teams should reproduce the test

The study used 60 AIME questions, 140 MMLU-Pro questions, 198 GPQA Diamond questions, 50 DeepSearchQA questions and 48 GDPval deliverables. More importantly, the Bedrock models ran with reasoning disabled as a deliberate cost floor, while the OpenAI API baselines used their defaults. Enabling reasoning would increase both quality and spending. The results therefore compare practical but different deployment configurations rather than isolating model capability.

AWS recommends replacing the supplied tasks with 50–100 examples from the target domain that have known-good outputs, then rerunning accuracy, trajectory, deliverable and latency tests whenever models, prices or workload patterns change. The durable contribution is the outcome-based methodology: count failed attempts, accumulated agent context and review-worthy output quality alongside token charges.

Source: AWS Machine Learning Blog

Definition. Cost per successful outcome divides total spending across successful and unsuccessful attempts by the number of outputs that meet the evaluation’s success threshold.

Key takeaways

FAQ

Sources