📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are rapidly automating key engineering tasks in AI research, with benchmarks reaching saturation. The remaining challenge is understanding how much of research itself can be automated, which may accelerate progress beyond current expectations.
Recent evidence indicates that AI systems can now automate the core engineering tasks involved in AI research, with several benchmarks approaching saturation. This development suggests that engineering may soon be fully automated, while the remaining question is how much of the actual research process can be automated, a topic still under investigation.
Multiple benchmarks measuring AI capabilities in AI R&D tasks show rapid progress. The CORE-Bench, which assesses research reproduction, has improved from 21.5% in September 2024 to 95.5% in December 2025, with one author declaring it ‘solved.’ Similarly, the MLE-Bench, evaluating Kaggle competition performance, rose from 16.9% to 64.4% over sixteen months, reaching a level comparable to mid-tier human performance. These benchmarks indicate that AI can now handle complex, friction-laden research tasks at a reliability level that diminishes the cost of reproduction and experimentation.
Additionally, advances in kernel design—such as automated GPU kernel generation and optimization—are becoming integrated into production workflows, further evidencing the maturation of engineering automation. These developments collectively suggest that the engineering aspect of AI R&D is nearing full automation, shifting the residual challenge to the research phase itself, which involves hypothesis formulation, experimentation, and discovery.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
CLAUDE AI UNLEASHED From First Prompts to Pro: The Complete Guide to Claude AI for Writing, Research, Coding, and Business (The Claude AI Mastery Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
GPU-Accelerated Computing with Python 3 and CUDA: From low-level kernels to real-world applications in scientific computing and machine learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
AI research reproduction benchmarks
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
automated AI experiment management
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications for the Future of AI Research and Development
The rapid automation of engineering tasks in AI research could significantly accelerate innovation cycles, reduce costs, and democratize access to advanced AI development. However, it also raises questions about the future role of human researchers, the potential for AI to undertake creative and hypothesis-driven research, and the strategic responses of institutions and organizations to these shifts. Understanding whether research itself can be automated remains critical, as it could fundamentally reshape the landscape of AI development in the next few years.
Recent Advances in AI R&D Capabilities
Over the past two years, multiple benchmarks and research efforts have demonstrated AI’s increasing proficiency in core engineering skills relevant to AI research. The CORE-Bench, MLE-Bench, and kernel design innovations show a pattern of rapid progress, with capabilities reaching or nearing saturation points. These developments build on prior work suggesting AI’s potential to automate significant portions of the engineering process, which historically required extensive human effort and expertise. The ongoing progress suggests a structural shift in how AI research and development are conducted, with automation becoming a central component.
“The pattern across multiple benchmarks indicates that AI is approaching full automation of core engineering tasks, shifting the residual focus to research-level activities.”
— Thorsten Meyer
Unresolved Questions About AI-Driven Research
It remains unclear how much of the research process—such as hypothesis generation, experimental design, and interpretation—can be effectively automated. While engineering tasks are nearing full automation, the structural question of whether AI can undertake creative and strategic research activities is still open. Additionally, the pace at which institutional and regulatory environments adapt to these technological shifts is uncertain, which could influence the practical deployment of fully automated research workflows.
Next Steps in Monitoring AI R&D Automation
Researchers and organizations will closely observe the continued saturation of benchmarks and the development of AI-driven research tools. Key milestones include the integration of automated research workflows into mainstream AI labs, validation of AI-generated hypotheses, and the emergence of new benchmarks specifically targeting research-level tasks. Policy and strategic responses from institutions will also shape how quickly and broadly these capabilities are adopted.
Key Questions
How close are we to fully automating AI research?
Based on current benchmark progress, engineering tasks are approaching full automation, but the extent to which research activities can be automated remains uncertain and is actively being studied.
What are the risks of automating research?
Potential risks include reduced human oversight, challenges in ensuring research quality, and ethical concerns about autonomous scientific discovery. These are subjects of ongoing debate and regulation.
Will human researchers become obsolete?
While automation may handle many engineering tasks, the creative and strategic aspects of research may still require human insight for the foreseeable future.
How might institutions adapt to these changes?
Institutions may need to develop new policies, invest in AI research tools, and redefine roles to integrate automated research workflows effectively.
Source: ThorstenMeyerAI.com
