The rapid increase in AI-generated code creates a significant bottleneck in the software development lifecycle, particularly in the code review stage.
While AI accelerates code writing (the inner loop), it overwhelms the critical outer loop processes of testing, reviewing, and deploying code.
Graphite's AI agent, Diamond, acts as an always-on senior engineer within GitHub to summarize, prioritize, and review every code change.
The primary metric for a successful AI reviewer is the acceptance rate of its comments, as every high-signal comment should result in a code change.
Unlike traditional machine learning, optimizing large language models relies heavily on a robust evaluation process rather than other levers like feature engineering.
Models are evaluated against a large dataset of pull requests using two core metrics: matched comment rate for recall and unmatched comment rate for noise.
A detailed comparison reveals that models like GPT-4.0 excel at precision while Gemini has the best recall, but no single model wins on all metrics.
Testing reveals that Sonnet 4.0 generates the most noise, making it less suitable for high-signal code review compared to its predecessors.
Sonnet 3.7 is the chosen model because it provides the optimal blend of strong reasoning, high recall of important issues, and low generation of noisy comments.
The key to leveraging AI effectively is to constantly re-evaluate new models, as shown by preliminary tests on GR four which revealed significant performance gaps.
28:07Yan Cui
31:39Rainer Stropek
24:05Nimrod Kor
15:26Gregor Schumacher, Sujay Joshy & Marcel Gocke
25:17Scott Hanselman
27:07Sandra Ahlgrimm & Kevin Lewis
50:12Sam Witteveen
25:34Himanshu Vasishth, Mindaugas Mozūras, Jackie Brosamer & Lukas Pfeiffer
Jobs that call for the skills explored in this talk.
Gemma Analytics