感謝您提交詢問!我們的一位團隊成員將在短時間內與您聯繫。
感謝您提交預訂!我們的一位團隊成員將在短時間內與您聯繫。
課程簡介
AI Sovereignty and LLM Local Deployment
- Risks associated with cloud LLMs: data retention, training on inputs, and foreign jurisdiction.
- Ollama architecture: model server, registry, and OpenAI-compatible API.
- Comparison with vLLM, llama.cpp, and Text Generation Inference.
- Model licensing terms for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Setup
- Installing Ollama on Linux with CUDA and ROCm support.
- CPU-only fallback options and AVX/AVX2 optimization.
- Docker deployment and persistent volume mapping.
- Multi-GPU configurations and VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry: example command ollama pull llama3.
- Importing GGUF models from HuggingFace and TheBloke.
- Understanding quantization levels: trade-offs between Q4_K_M, Q5_K_M, and Q8_0.
- Model switching and limits on concurrent model loading.
Custom Modelfiles
- Writing Modelfile syntax: FROM, PARAMETER, SYSTEM, TEMPLATE.
- Tuning temperature, top_p, and repeat_penalty.
- System prompt engineering for role-specific behaviors.
- Creating and publishing custom models to the local registry.
API Integration
- OpenAI-compatible /v1/chat/completions endpoint usage.
- Streaming responses and JSON mode configuration.
- Integrating with LangChain, LlamaIndex, and custom applications.
- Authentication and rate limiting via reverse proxy.
Performance Optimization
- Context window sizing and KV cache management.
- Batch inference and parallel request handling.
- CPU thread allocation and NUMA awareness.
- Monitoring GPU utilization and memory pressure.
Security and Compliance
- Network isolation for model serving endpoints.
- Input filtering and output moderation pipelines.
- Audit logging of prompts and completions.
- Model provenance tracking and hash verification.
最低要求
- Intermediate proficiency in Linux and container administration.
- High-level understanding of machine learning concepts and transformer models.
- Familiarity with REST APIs and JSON formatting.
Target Audience
- AI engineers and developers seeking to replace cloud LLM APIs.
- Organizations with strict data sensitivity requirements that prohibit the use of cloud models.
- Government and defense teams necessitating air-gapped language model solutions.
14 小時