Reproducible Robotics Benchmarks

The main goal of this program is to build a repository of reproducible evaluations that various stakeholders (governments, academia, and the public) can use to benchmark general robotics capabilities, providing a trusted means for the public to gauge progress in robotics. With this infrastructure, we can better understand and prepare for physical automation of labor and a possible intelligence explosion.

Currently, there are very few well-run, standardized benchmark methods for measuring general robotic capabilities. Leading robotics AI models are released with each company's private task lists, without third-party testing and evaluation. This leaves AI forecasters, policymakers, and the public without a reliable understanding of how capable these systems really are or how fast they're improving.

Forethought's work on the intelligence explosion notes that once robots can substitute for human labor, a faster industrial explosion could occur. This means AI could seize decisive power sooner, since most routes to takeover run through control of physical infrastructure. The intelligence curse also argues that once powerful actors no longer need human labor due to advancements in robotics capabilities, they lose the incentive to invest in their welfare. We are concerned about possible gradual disempowerment and intelligence explosions that could arise as a result of these advancements in robotics capabilities.

Without shared, reproducible benchmarks, people outside the labs have very little basis for judging how soon robots will take on real-world jobs, or how quickly the field is moving. With these benchmarks, we hope to inform the public and policymakers about the rate of progress in general robotics and help to reduce the risks it poses.

There are two main stages of this program. 1. Stage 1: Selection, mid-September to late September. Review Stage 1 proposals. Most of the work is judgement rather than writing code. You're going to support the creation of the evaluation rubric for applications received in the public call for proposals and actually evaluate them to find the top 12 finalist teams.

2. Stage 2: Support, October to December. Help selected teams reach a working benchmark by onboarding them to Inspect Robots, running office hours, checking in on teams to help them make progress, and doing whatever else is necessary to produce the highest-quality output.

Likely output: the first 10 to 12 independent robotics benchmarks that other people can re-run, plus the onboarding material for the next cohort.

Required

• Comfortable with Python and reading unfamiliar codebases
• Can explain technical things clearly in writing
• Highly agentic and proactive in solving problems

Helpful

• Experience with creating evaluations or benchmark design
• Robotics or RL background
• TA experience

You don't need hardware access or any prior robotics work.