AID-H-012: Reinforcement Learning (RL) Reward Hacking Prevention
Design and implement safeguards to prevent Reinforcement Learning (RL) agents and LLM post-training pipelines from discovering and exploiting flaws in reward functions, learned reward models, AI judges, or programmatic verifiers to achieve high rewards for unintended or harmful behaviors ('reward hacking'). This includes classical environment rewards, RLHF/RLAIF reward models, RLVR verifiers, reward-channel integrity, and independent evidence that policies remain aligned with intended outcomes rather than numeric reward maximization alone.
Scope boundary
Owns preventive reward design, learned reward calibration, isolated behavioral promotion validation, reward-channel integrity, and judge or verifier gaming controls for RL, RLHF, RLAIF, RLVR, and related post-training systems.
AID-D-013: AID-H-012 designs and gates reward systems to prevent exploitable optimization; AID-D-013 detects reward hacking, reward tampering, verifier gaming, and policy manipulation from observed evidence. Preventive reward controls and production or training detection remain distinct.
Sub-techniques
- AID-H-012.001: Robust Reward Function Engineering
- AID-H-012.002: Learned Reward-Signal Integrity & Calibration
- AID-H-012.003: RL Behavioral Promotion Validation in Isolated Environments
- AID-H-012.004: Secure Reward Signal Integrity & Transport
- AID-H-012.005: RLHF/RLVR Reward-Model & Verifier Gaming Controls
Threat mappings
MITRE ATLAS
- AML.T0020 Training Data Poisoning
- AML.T0059 Erode Dataset Integrity
MAESTRO
- Data Poisoning (Training Phase) (L1)
- Data Poisoning (L2)
- Data Tampering (L2)
- Manipulation of Evaluation Metrics (L5)
OWASP LLM Top 10 2026
- LLM05:2026 Data and Model Poisoning
OWASP ML Top 10 2023
- ML08:2023 Model Skewing
OWASP Top 10 for Agentic Applications 2026
- N/A
NIST Adversarial Machine Learning 2025
- NISTAML.013 Data Poisoning
- NISTAML.027 Misaligned Outputs
- NISTAML.037 Training Data Attacks
Cisco Integrated AI Security and Safety Framework
- AITech-2.2 Goal Drift
- AITech-2.3 Reward Hacking
- AITech-6.1 Training Data Poisoning
- AITech-16.1 Eavesdropping
- AISubtech-2.2.2 Goal Substitution
- AISubtech-2.2.3 Authorization and Constraint Erosion
- AISubtech-2.3.1 Task Loophole Exploitation
- AISubtech-2.3.2 Verifier Manipulation
- AISubtech-2.3.3 Shortcut Acquisition
- AISubtech-4.3.4 Replay Exploitation
- AISubtech-6.1.2 Reinforcement Biasing
- AISubtech-6.1.3 Reinforcement Signal Corruption
- AISubtech-14.1.2 Insufficient Access Controls
Google Secure AI Framework 2.0 - Risks
- DP: Data Poisoning
- RA: Rogue Actions
Databricks AI Security Framework 3.0
- Datasets 3.1: Data poisoning
- Agents - Core 13.7: Misaligned & Deceptive Behaviors