A practical demonstration shows how a cloud provider's content filter update can unexpectedly block access to documents, causing application failures.
Large language models use a transformer architecture to predict the next token based on probability, with instruction tuning and alignment shaping their final behavior.
Major LLM providers operate in secrecy, and the full technology stack from model weights to the API is complex, leaving developers with limited visibility and control.
Cloud provider content filters can change without notice, creating vulnerabilities, while the short lifecycle of model APIs requires constant adaptation.
LLMs are conceptually like alien intelligences with a fragile, human-like alignment layer that can be bypassed by jailbreaks exploiting internal model circuits.
Different LLM versions exhibit distinct behaviors and may ignore system prompts, as shown by a comparison between GPT-4 and a newer reasoning model.
Systematically test model behavior using evaluations, which can be automated by generating prompt variations or using pre-built cloud and open-source frameworks.
Mitigate model behavior drift by using advanced prompt engineering techniques like forcing reasoning, providing few-shot examples, and being highly explicit in instructions.
26:30Simon A.T. Jiménez
29:00Alex Soto
25:17Scott Hanselman
35:16Ankit Patel
23:24Keno Dreßel
27:11Emanuele Fabbiani
25:27Maxim Salnikov
32:26Krzysztof Cieślak
.gif?w=240&auto=compress,format)
Jobs that call for the skills explored in this talk.
