Traditional development relies on rigorous testing, but AI features are often implemented based on intuition without a structured evaluation process.
GitHub Models integrates AI development into your repository by defining prompts, models, and parameters in a version-controlled YAML file.
The platform allows you to run multiple prompt and model variations against a test dataset to compare outputs on metrics like latency, coherence, and similarity.
Use the GitHub Models inference API or the Azure AI Inference SDK to load your version-controlled prompt files and integrate AI calls directly into your application.
The GitHub CLI extension allows you to run prompts and execute model evaluations directly from your terminal for rapid, local iteration before committing changes.
Use GitHub Actions to automate common repository tasks like generating changelogs from pull requests, triaging bug reports, or creating weekly issue summaries.
Integrate prompt evaluations into your CI/CD pipeline using GitHub Actions to automatically run tests and block pull requests that degrade model performance.
You can quickly convert existing prompt files to the GitHub Models format to gain access to powerful evaluation, comparison, and automation capabilities.
25:17Scott Hanselman
31:39Rainer Stropek
28:53Mike
24:11Andreas Erben
33:28Maxim Salnikov
32:26Krzysztof Cieślak
42:38Krzystof Czieslak
35:16Ankit Patel
Jobs that call for the skills explored in this talk.
