EN FR
IJCAI 2022 Tutorial Series

Adversarial Attacks &
Defenses in
Sequential Decision-Making

Red teaming reinforcement learning agents, adversarial LLM security, and robust AI training. Covering test-time attacks, training-time threats, and multi-agent adversarial dynamics.

// Adversarial RL Framework agent.policy ← train(data, attacks) attack.gradient ← compute_gradient(loss) defense.robust ← adversarial_training()

Attacks RL

Test-time, training-time, backdoor

  • Adversarial perturbations
  • Data poisoning
  • Model extraction

LLM Security

Injection, jailbreaking, extraction

  • Prompt adversariality
  • Context injection
  • PII exfiltration

Learning Defenses

Robust training, anomaly detection

  • Adversarial training
  • Certified defenses
  • Monitoring & audits
Pillar 1

Adversarial Attacks
on RL Agents

Explore optimal attack strategies against reinforcement learning agents across test-time, training-time, and multi-agent settings. From basic adversarial perturbations to sophisticated backdoor attacks.

Test-Time Attacks

Adversarial perturbations, evasion, observation corruption

  • FGSM & PGD on RL policy
  • Reinforcement learning evasion
  • Imperceptible perturbations

Training-Time Attacks

Data poisoning, backdoors, reward manipulation

  • Poisoned experience replay
  • Reward hacking
  • Gradient-based attacks

Multi-Agent Attacks

Adversarial agents, coordination attacks, game theory

  • Non-oblivious attacks
  • Agent impersonation
  • Swarm coordination

Backdoor Attacks

Trojan triggers, specification gaming, latent attacks

  • Trigger-based control
  • Sleeper agents
  • Latent value injection
Deep Dive: Attacks RL →
Pillar 2

Adversarial LLM
Security

Red teaming large language models: prompt injection, jailbreaking, context confusion, PII extraction, and system prompt exposure. Adversarial defenses and detection strategies.

Prompt Injection

Direct & indirect injection, RAG attacks, context override

  • Direct prompt override
  • Indirect RAG injection
  • Chained attacks

Jailbreaking

DAN, personas, roleplay, token smuggling

  • Persona exploitation
  • Crescendo attacks
  • Token-level evasion

Data Extraction

PII leakage, training data recovery, model inversion

  • Membership inference
  • Training data extraction
  • Model probing

System Prompt Exposure

System instruction retrieval, capability discovery

  • Prompt enumeration
  • Behavior mapping
  • Constraint discovery
Deep Dive: LLM Security →
Pillar 3

Attacks on AI
Learning Systems

Training-time attacks across the full AI lifecycle: data poisoning, model poisoning, supply chain attacks, and semantic drift. Building robust learning pipelines with certified defenses.

Training Data Poisoning

Data corruption, label flipping, backdoor triggers

  • Clean-label attacks
  • Availability attacks
  • Trigger injection

Model Poisoning

Gradient attacks, Byzantine robustness, federated learning

  • Gradient inversion
  • Byzantine tolerance
  • Parameter pollution

Supply Chain Attacks

Dependency vulnerabilities, model theft, checkpoint attacks

  • Malicious dependencies
  • Model extraction
  • Checkpoint poisoning

Semantic Drift & Monitoring

Runtime monitoring, anomaly detection, online robustness

  • Distribution shift detection
  • PII leakage scoring
  • Factuality monitoring
Deep Dive: Learning Attacks →
Knowledge Base

Research &
References

OWASP LLM Top 10, MITRE ATLAS, academic papers, and benchmarks for evaluating adversarial robustness.

OWASP LLM Top 10

Primary LLM risk framework covering injection, training data, model theft, and monitoring

MITRE ATLAS

Adversarial ML attack framework and tactics for red teaming agents

Benchmark Datasets

Adversarial RL environments, attack/defense evaluation metrics

Red Teaming Tools

Garak, PyRIT, PromptBench, Giskard for security assessment

View OWASP LLM Top 10 → View MITRE ATLAS →