Self-hosting large language models offers developers greater privacy, cost savings, and control compared to third-party cloud AI services.
Running models locally improves the development inner loop, provides full data privacy, and allows for greater customization and control over the AI stack.
Explore different open source tools like Ollama for local development, vLLM for scalable production, and Podman AI Lab for containerized AI applications.
Navigate the vast landscape of open source models by understanding different model families, their specific use cases, and naming conventions.
Model quantization compresses LLMs to reduce their memory footprint, enabling them to run efficiently on consumer hardware like laptops with CPUs or GPUs.
Learn three key methods for connecting local models to your data: Retrieval-Augmented Generation (RAG), local code assistants, and building agentic applications.
Use Podman AI Lab to serve a local LLM and connect it to AnythingLLM to create a question-answering system over your private documents.
Integrate a self-hosted LLM with the Continue VS Code extension to create a private, offline-capable AI pair programmer for code generation and analysis.
Create an agentic application that uses a local LLM with external tools via the Model Context Protocol (MCP) to perform complex, multi-step tasks.
Self-hosting provides a powerful, private, and customizable alternative to third-party services, highlighting the growing potential of open source AI for developers.
31:25Patrick Koss
27:11Emanuele Fabbiani
31:50Cedric Clyburn & Legare Kerrison
28:38Tomislav Tipurić
26:30Simon A.T. Jiménez
30:48Roberto Carratalá
42:26Meta Atamel & Guillaume Laforge
34:21Aarno Aukia
.png?w=240&auto=compress,format)
.gif?w=240&auto=compress,format)
Jobs that call for the skills explored in this talk.
