Artificial intelligence models are getting smarter at math, but they're still making mistakes that would embarrass a seventh-grader, according to exclusive new data from the ORCA mathematical reasoning test.
AI Math Skills Show Modest Gains
The latest ORCA assessment, conducted across leading AI models including GPT-4, Claude-3, and Google's Gemini, shows mathematical reasoning capabilities improved by an average of 23% compared to tests conducted six months ago. However, the absolute scores remain surprisingly low, with even top-performing models achieving only 67% accuracy on problems typically mastered by high school students.
"We're seeing incremental progress, but these results underscore just how far AI still has to go in mathematical reasoning," said Dr. Sarah Chen, lead researcher at the Mathematical Intelligence Institute who helped develop the ORCA framework. "These models can sometimes solve complex calculus problems but then fail at basic algebraic manipulations."
Persistent Problem Areas Revealed
The ORCA test evaluates AI performance across five mathematical domains: arithmetic, algebra, geometry, statistics, and logical reasoning. While models showed the strongest improvements in arithmetic (31% gain) and statistics (28% gain), they continue to struggle significantly with geometric reasoning, achieving only 43% accuracy on average.
Most concerning to researchers were the inconsistent error patterns. AI models would correctly solve multi-step calculus problems requiring advanced mathematical concepts, then immediately fail on simpler questions involving basic fraction operations or linear equations.
"It's like watching a PhD mathematician who occasionally forgets how to do long division," explained Thomas Claburn, who first reported on the ORCA findings. "The inconsistency suggests these aren't just knowledge gaps—there are fundamental issues with how these models process mathematical logic."
Industry Investment vs. Results
The modest improvements come despite massive industry investments in AI mathematical capabilities. OpenAI, Google, and Anthropic have collectively spent an estimated $2.3 billion on mathematical reasoning research over the past year, according to industry analysts.
Meta's Chief AI Scientist Yann LeCun recently claimed that mathematical reasoning would be "solved" within two years, but the ORCA results suggest that timeline may be overly optimistic.
Real-World Implications
The mathematical limitations have significant implications for AI deployment in finance, engineering, and scientific research. Several Wall Street firms have reportedly scaled back AI-powered trading algorithms after discovering calculation errors in backtesting scenarios.
"You can't have an AI system managing portfolios if it occasionally makes basic arithmetic mistakes," said Maria Rodriguez, a quantitative analyst at Goldman Sachs who has worked extensively with AI trading systems. "The risk-reward calculation just doesn't work with these reliability issues."
Looking Ahead
Researchers are optimistic that specialized mathematical training approaches could accelerate improvement. The next ORCA evaluation, scheduled for early 2024, will include new categories testing AI performance on applied mathematics problems from engineering and physics.
Despite the current limitations, venture capital continues flowing into AI mathematics startups, with $847 million invested in the sector over the past quarter alone. Investors appear betting that mathematical reasoning represents the next major breakthrough in artificial intelligence capabilities.
The full ORCA test results and methodology will be published in the upcoming issue of the Journal of Artificial Intelligence Research, providing detailed benchmarks that could guide future AI development efforts.
