Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Strategies

• Comparing the trade-offs between Hosted APIs and self-hosted solutions.
• Exploring on-premises, edge, and air-gapped deployment patterns.
• Evaluating single model versus multi-model portfolio approaches.
• Analyzing cost efficiency per decision.

Hardware Sizing and Capacity Planning

• Assessing memory footprint relative to parameter count.
• Deciding between CPU-only performance and GPU acceleration.
• Planning for concurrency levels and throughput requirements.
• Allowing headroom for potential increases in task complexity or length.

Serving Stack Selection

• Reviewing options like vLLM and SGLang.
• Utilizing llama.cpp and GGUF for execution.
• Leveraging Ollama for local development scenarios.
• Selecting the optimal stack based on the specific shape of the decision workload.

Quantization While Preserving Calibration

• Understanding precision formats and their impact on memory usage.
• Measuring quality retention and potential calibration loss.
• Applying per-tensor and importance-aware quantization variants.
• Conducting acceptance tests post-quantization.

Latency Optimization

• Distinguishing between median and tail latency metrics.
• Analyzing how latency scales with context length.
• Implementing warm starts and model caching strategies.
• Batching multiple decision queries for efficiency.

Edge and Constrained Environments

• Running small decision models on CPU resources.
• Deploying on Apple Silicon and embedded systems.
• Managing offline operations and software update strategies.
• Distributing model weights in air-gapped settings.

Containerization and Scalability

• Designing Docker images optimized for model serving.
• Managing volumes and cache effectively.
• Implementing horizontal scaling and routing logic.
• Configuring resource limits and GPU sharing mechanisms.

Observability and Monitoring

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions.
• Tracking decision outcomes and escalation paths.
• Setting alerts for calibration drift.

Version Control and Rollout Management

• Implementing versioning for models and adapters.
• Using shadow and canary deployment techniques.
• Maintaining pinned model files for audit trails.
• Defining clear rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Managing network isolation and egress controls.
• Maintaining comprehensive audit logs of decisions.