安全團隊的AI安全防護培訓

Module 1 — AI Systems for Security Engineers

Lab: Lab 01 — 01-Introduction

Understanding the architecture.

Topics:

• LLMs vs normal apps
• AI inference pipelines
• Prompt flow
• RAG architecture
• embeddings/vector databases
• agentic workflows
• tool calling
• AI gateways
• copilots
• MCP and agent protocols
• where WAF visibility exists
• where WAF visibility disappears

Key insight: Traditional WAFs often lose visibility after the prompt reaches the model.

Module 2 — OWASP GenAI Top 10

Lab: none — interactive recap/discussion

Core AI attack categories.

Topics:

• Prompt Injection
• Insecure Output Handling
• Training Data Poisoning
• Model DoS
• Supply Chain Vulnerabilities
• Sensitive Information Disclosure
• Excessive Agency
• Vector/Embedding Weaknesses
• Misinformation
• Unbounded Consumption

Include:

• Differences from classic OWASP
• Mapping to defensive controls (WAF, gateway, app-layer)
• Where each control helps
• Where each control fails

Module 3 — Prompt Injection Detection

Lab: Lab 02 — 02-Prompt-Injection

The “SQL injection moment” for AI.

Topics:

• Direct prompt injection
• Indirect prompt injection
• Hidden instructions
• Document-based attacks
• HTML/Markdown injection
• Jailbreak patterns
• Context override attacks
• Role confusion attacks

Detection strategies:

• Keyword heuristics
• Semantic classification
• Prompt linting
• Instruction boundary enforcement
• Allow/deny policies
• AI-aware regex patterns

Hands-on labs:

• Attack a chatbot
• Bypass naive filters
• Build layered detection

Module 4 — AI-Aware WAF Rules

Lab: Lab 03 — 03-WAF-Basics

How WAF rules evolve for AI systems.

• Topics:
• Protecting LLM endpoints
• Inference API protection
• Token-aware rate limiting
• Prompt size inspection
• AI-specific signatures
• Conversation anomaly detection
• Multi-turn abuse patterns
• Model enumeration attempts
• Inference scraping
• Denial-of-wallet protection

Examples:

• Protecting /v1/chat/completions
• Defending streaming APIs
• Blocking recursive agent calls

Module 5 — Securing RAG Pipelines

Lab: Lab 04 — 04-RAG-Security

One of the biggest new attack surfaces.

Topics:

• Vector DB threats
• Embedding poisoning
• Malicious PDFs/docs
• Retrieval manipulation
• Semantic poisoning
• Hidden instructions in documents
• Cross-document contamination
• Data exfiltration via retrieval

Defenses:

• Ingestion sanitization
• Trust scoring
• Metadata isolation
• Document provenance
• Retrieval policies
• Segmentation

Case study: “Upload a poisoned PDF and take over the AI assistant.”

Module 6 — Agentic AI Security

Lab: Lab 05 — 05-Agent-Security

Where things become dangerous.

Topics:

• Excessive agency
• Tool abuse
• API chaining
• Autonomous loops
• Permission escalation
• Memory poisoning
• Indirect tool execution
• Agent impersonation
• Credential leakage
• Multi-agent attacks

Defenses:

• Least privilege for agents
• Approval gates
• Runtime policy engines
• Sandboxing
• Scoped credentials
• Tool whitelisting
• Human-in-the-loop

This section is particularly relevant for managers, as the risks become operational and business-impacting.

Module 7 — API Security for AI

Lab: Lab 06 — 06-Denial-of-Wallet

AI systems are heavily API-dependent.

Topics:

• API gateways
• GraphQL AI risks
• MCP/API abuse
• JWT protection
• AI plugin security
• Agent authentication
• Delegated authorization
• Secret management
• Signed prompts
• API inventory for AI

Tie into: OWASP API Security Top 10

Module 8 — Detection Engineering & SOC Integration

Lab: Lab 07 — 07-Detection

Operational defense.

Topics:

• AI telemetry
• Prompt logging
• Token analytics
• Anomaly detection
• Semantic SIEM pipelines
• AI attack indicators
• Threat hunting for LLM abuse
• AI runtime observability

Examples:

• Detecting jailbreak campaigns
• Spotting automated agent abuse
• Identifying model scraping

Module 9 — Cloud WAFs and AI Security

Lab: none — interactive recap/discussion

Vendor-specific implementations.

Topics:

• AWS WAF for AI APIs
• Azure WAF
• Cloudflare AI Gateway
• API gateways
• Envoy AI filtering
• Kong AI Gateway
• NGINX AI security patterns

Comparison:

• Traditional WAF vs AI gateway vs app-layer guardrail
• Proxy-based vs semantic filtering

Module 10 — Building a Layered AI Defense

Lab: Lab 08 — 08-Layered-Defense

Important philosophical conclusion:

No single layer can secure AI (a WAF least of all, on its own).

Students build a layered model:

1. WAF
2. API gateway
3. AI gateway
4. Guardrails
5. Runtime monitoring
6. Identity/authorization
7. Sandbox
8. Human approval
9. Observability
10. Incident response

This aligns strongly with the “multi-layer security” model.

Module ↔ Lab map

Labs run in lab order, which follows module order.

The course has 10 modules but 8 labs: Modules 2 and 9 are interactive recap/discussion sessions and have no lab.

Each lab is tagged with its corresponding module throughout this outline.

• Lab 01 (Module 1)
  • Folder: 01-Introduction
  • Title: Explore an AI system — what's on the wire
• Lab 02 (Module 3)
  • Folder: 02-Prompt-Injection
  • Title: Attack a chatbot & bypass naive filtering
• Lab 03 (Module 4)
  • Folder: 03-WAF-Basics
  • Title: Build AI-aware WAF rules
• Lab 04 (Module 5)
  • Folder: 04-RAG-Security
  • Title: Poison a RAG pipeline
• Lab 05 (Module 6)
  • Folder: 05-Agent-Security
  • Title: Secure an autonomous agent
• Lab 06 (Module 7)
  • Folder: 06-Denial-of-Wallet
  • Title: Detect denial-of-wallet attacks
• Lab 07 (Module 8)
  • Folder: 07-Detection
  • Title: Monitor AI abuse patterns in logs
• Lab 08 (Module 10)
  • Folder: 08-Layered-Defense
  • Title: Build a layered AI defense architecture

Capstone

Students defend a simulated enterprise AI assistant.

Attackers attempt:

1. Prompt injection
2. Tool abuse
3. Credential theft
4. Retrieval poisoning
5. Excessive API consumption
6. Agent escalation

Teams build:

• WAF rules
• AI gateway policies
• Runtime detection
• Guardrails
• Incident response