o3 scores 75.7% on ARC-AGI

2024-12-20 · Publisher : OpenAI · Domain : Artificial Intelligence

OpenAI unveils o3, its new reasoning model, which scores 75.7% on the ARC-AGI test within the competition's compute limit, and 87.5% with 172 times more compute. GPT-4o topped out at 5%.

What changes

  • 75.7% on the ARC-AGI semi-private set within the 10,000 USD compute limit, 87.5% in the maximum compute configuration.
  • A sudden leap: ARC-AGI-1 had gone from 0% (GPT-3, 2020) to only 5% (GPT-4o, 2024).
  • Other announced results: 96.7% on AIME 2024 and 25.2% on FrontierMath, where no model exceeded 2%.
  • The gain comes from compute at answer time: the longer the model reasons, the better it does, but the more it costs.
  • François Chollet, the test's creator, hails a breakthrough while arguing that o3 is not general AI.

On December 20, 2024, the last day of its year-end announcement series, OpenAI unveiled o3 and o3-mini, successors to the o1 reasoning model (the name “o2” was skipped because of the British carrier O2's trademark). The most discussed result came from ARC-AGI, a test designed by François Chollet to measure the ability to adapt to novel problems. ### What happened - The ARC Prize foundation evaluated o3 on its test sets and published the results the same day. - In the “high-efficiency” configuration, within the public leaderboard's compute limit, o3 scored 75.7% on the semi-private set. - In the “low-efficiency” configuration, with 172 times more compute, it reached 87.5%. - OpenAI also announced 96.7% on the AIME 2024 math exam, 25.2% on FrontierMath and 71.7% on SWE-bench Verified, 22.8 points above o1. ### Key figures | Measure (ARC-AGI-1) | Value | |---|---| | GPT-3 (2020) | 0% | | GPT-4o (2024) | 5% | | o3, semi-private set, high efficiency | 75.7% | | o3, semi-private set, 172x compute | 87.5% | | o3, public set, high efficiency / 172x compute | 82.8% / 91.5% | ### Why it is a turning point ARC-AGI had resisted large language models since 2019, as they failed on visual puzzles that are simple for humans. o3 showed that scaling compute at inference time, not only model size, can unlock new reasoning abilities. François Chollet called it “a genuine breakthrough, marking a qualitative shift in AI capabilities.” ### Caveats Chollet immediately added: “I don't think o3 is AGI yet. o3 still fails on some very easy tasks.” The cost per task in the maximum configuration ran into thousands of dollars. The results concern a preview: the o3 model released on April 16, 2025 is a different version, optimized for everyday use. ARC-AGI-2, published in 2025, brought scores back down to low levels. ### Key takeaways o3 marks the rise of models that “think” longer to answer better: performance becomes a matter of compute budget at usage time, with a cost-quality trade-off to manage.

Links

Tags : IA, Raisonnement, Benchmarks, OpenAI, ARC-AGI