The Illusion of Thinking in Large Reasoning Models (LRM)

This episode investigates the reasoning capabilities of Large Reasoning Models (LRMs), a new generation of language models designed for complex problem-solving. The authors evaluate LRMs using controllable puzzle environments to systematically analyze how performance changes with problem complexity, unlike traditional benchmarks that often suffer from data contamination. Key findings reveal three performance regimes: standard LLMs surprisingly excel at low complexity, LRMs gain an advantage at medium complexity, and both models experience complete collapse at high complexity, often exhibiting a counter-intuitive decline in reasoning effort despite having a sufficient token budget. The analysis also examines the internal reasoning traces, uncovering patterns like "overthinking" on simpler tasks and highlighting limitations in LRMs' ability to follow explicit algorithms or maintain consistent reasoning across different puzzle types.