Monitor and evaluate safety practices at frontier labs by scrutinizing public-facing outputs.
About the project
AI Lab Watch (ailabwatch.org) was founded to track safety at labs through analyzing their public-facing outputs, and to provide recommendations on developing better safety practices. The project has since gone dormant, and although successors (e.g. the FLI AI Safety Index) have conducted similar reviews, they don't offer AI Lab Watch's signature in-the-weeds scrutiny and attention to detail.
Mentees will help construct a website like AI Lab Watch, by systematically scrutinizing lab outputs (system cards, blog posts, safety frameworks, etc.) and developing a rigorous methodology to translate our analysis into a realistic safety assessment. The project will also translate our findings into an accessible and clear dashboard to inform key decision makers.
Theory of change
AI Lab Watch will inform key decision makers about the quality of safety in frontier labs, so regulation targets what is genuinely missing and relevant to mitigating x-risk. Additionally, labs are more likely to be comprehensive and detail oriented in their safety work when shortcomings are publicly documented.
Your role
Mentees will help revitalize AI Lab Watch, by systematically scrutinizing lab outputs (system cards, blog posts, safety frameworks, etc.) and developing a rigorous methodology to translate our analysis into a realistic safety assessment. The project will also translate our findings into an accessible and clear dashboard to inform key decision makers.
Prerequisites
Strong attention to detail and truthseeking. Context on AI safety threat models and frontier labs, or the ability to learn quickly. Useful skills include writing ability and experience with coding agents.
Application question(s)
Submit a Google Doc with full editing history enabled. No AI usage is permitted in writing or editing. Narrow and deep responses are preferred over broad and shallow.** Question 1:** Where are frontier labs' safety practices currently falling short? Cite your reasoning & sources. [250 words]
Question 2: See ailabwatch.org. Provide specific criticism on any of the current safety criterion. [150 words]
(Optional) Propose your own safety scorecard for evaluating labs. This can be high-level or detailed.
About the mentors
Harshul Basava
GT AISI
Harshul Basava co-directs the Georgia Tech AI Safety Initiative, directs operations for Second Look Research, and runs the DC Mini-Conference on AI Policy.
Meru Gopalan
GT AISI, XLab
Meru runs the fellowships and research at the Georgia Tech AI Safety Initiative, creates curriculum content for the Tracks program, and conducts research in algorithms for SAT solvers.