The tool reviews pull requests, leaves inline comments, and provides a dashboard, with the talk focusing on lessons learned from building it.
The system uses GitHub webhooks, EventBridge, Lambda, Amazon Bedrock, and DynamoDB, with Clerk for auth and Stripe for payments.
Amazon Bedrock was selected for its strong security guarantees, data privacy policies, and serverless, token-based pricing model suitable for sensitive customer code.
Large context window sizes are misleading because a model's ability to reason over content collapses long before the advertised limit, forcing a one-prompt-per-file strategy.
To overcome low default API rate limits, strategies include requesting limit increases, using cross-region inference, and implementing fallbacks to other models for reliability.
The most effective cost control measure is to analyze only the changed lines in a pull request rather than the entire file, which also improves user experience.
A lightweight durable execution mechanism using checkpoints in DynamoDB prevents reprocessing files on Lambda retries, which are common due to slow LLM response times.
Hallucinations range from simple invalid JSON to complex errors like suggesting fixes for outdated libraries, which can be mitigated with RAG but at a significant cost.
24:05Nimrod Kor
31:39Rainer Stropek
26:30Simon A.T. Jiménez
04:17Robert Werner
25:17Scott Hanselman
47:35Laurie Voss, Rey Bango, Hannah Foxwell, Rizel Scarlett & Thomas Steiner
22:03Merrill Lutsky
32:26Krzysztof Cieślak
Jobs that call for the skills explored in this talk.
webLyzard
Vienna, Austria