Self-hosting LLMs provides greater control over data privacy, compliance, cost, and vendor lock-in compared to using third-party services.
A scalable LLM service requires key components for model acquisition, inference, storage, billing, security, and request routing.
Using network file storage (NFS) is crucial for reducing startup times and enabling fast horizontal scaling when deploying new model instances.
Aggregate token usage with tools like Redis before sending data to a payment provider to manage rate limits and improve system efficiency.
Prevent system abuse by implementing both request-based and token-based rate limiting, using estimations for output tokens to protect shared resources.
Bearer tokens offer a flexible solution for managing authentication and fine-grained authorization, such as restricting access to specific models.
Use tools like KServe or Knative on Kubernetes for intelligent autoscaling and canary deployments based on custom metrics like queue size.
Key strategies for success include robust rate limiting, modular design, continuous benchmarking, and using canary deployments for safe production testing.
30:04Roberto Carratalá & Cedric Clyburn
35:16Ankit Patel
31:12Daniel Töws
26:30Simon A.T. Jiménez
42:26Meta Atamel & Guillaume Laforge
27:11Emanuele Fabbiani
34:21Aarno Aukia
29:52Anshul Jindal & Martin Piercy
.png?w=240&auto=compress,format)
.png?w=240&auto=compress,format)
.gif?w=240&auto=compress,format)
Jobs that call for the skills explored in this talk.
