AID-H-012.005: RLHF/RLVR Reward-Model & Verifier Gaming Controls

Version 1.20261015 · Actionable sub-technique

Prevent modern LLM post-training runs from over-optimizing against a flawed learned reward model, AI judge, or programmatic verifier. This sub-technique applies to RLHF, RLAIF, RLVR, GRPO/PPO-style post-training, and reward-model reranking where a policy can learn to satisfy the evaluator while degrading the true task objective. It owns the reward/verifier contract, held-out adversarial evaluator set, independent judge/verifier comparison, and promotion gate evidence for reward-model gaming, verifier false positives, sycophancy, verbosity bias, fabricated reasoning, unit-test gaming, or evaluator-specific shortcut learning.

Pillars: model

Lifecycle: building, validation

Scope boundary

Owns reward or verifier contracts, held-out adversarial evaluator sets, independent judge comparison, and promotion evidence for policy gaming of learned reward models, AI judges, and programmatic verifiers in RLHF, RLAIF, RLVR, and related post-training.

AID-H-012.001: AID-H-012.005 detects and blocks evaluator-specific gaming under optimization; AID-H-012.001 engineers the base reward and shaping contract. Policies can exploit evaluator flaws even when the high-level reward objective is sound.

AID-H-012.002: AID-H-012.005 tests the optimized policy against independent reward and verifier evidence; AID-H-012.002 trains and calibrates the learned reward model. Reward-model quality and policy resistance to reward-model gaming are separate claims.

AID-H-012.003: AID-H-012.005 owns the reward-model and verifier-gaming gate; AID-H-012.003 owns broader episode-level behavioral promotion validation in an isolated environment. A candidate can pass anti-gaming checks yet violate an unrelated action, resource, or safety constraint.

Implementation guidance

AID-H-012.005-G001: Define a versioned reward/verifier contract that separates train-time rewards from independent promotion evidence.
AID-H-012.005-G002: Run an independent reward-gaming promotion gate before accepting an RLHF/RLVR candidate.

AID-H-012.005-G001

Define a versioned reward/verifier contract that separates train-time rewards from independent promotion evidence.

Apply this control to RLHF, RLAIF, GRPO-style, or RLVR pipelines that optimize against a learned reward model, AI judge, or programmatic verifier. It owns the signed release contract separating optimization-visible assets from independent promotion evidence; it does not run the promotion gate in the sibling guidance. A pipeline with no reward/verifier optimization is NOT_APPLICABLE. Counts, score thresholds, freshness, and allowed evaluator classes are organization-policy values with no framework defaults.

Validate an exact reward/verifier contract

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://aidefend.net/schemas/reward-verifier-contract-v2.json",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "schema_version",
    "contract_id",
    "candidate_sha256",
    "base_model_sha256",
    "optimization",
    "promotion",
    "thresholds"
  ],
  "properties": {
    "schema_version": {
      "const": "aidefend.reward-verifier-contract.v2"
    },
    "contract_id": {
      "type": "string",
      "minLength": 1
    },
    "candidate_sha256": {
      "type": "string",
      "pattern": "^[0-9a-f]{64}$"
    },
    "base_model_sha256": {
      "type": "string",
      "pattern": "^[0-9a-f]{64}$"
    },
    "optimization": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "method",
        "run_id",
        "reward_kind",
        "reward_artifact_sha256",
        "verifier_artifact_sha256",
        "visible_population_manifest_sha256"
      ],
      "properties": {
        "method": {
          "enum": ["rlhf", "rlaif", "grpo", "rlvr", "hybrid"]
        },
        "run_id": {
          "type": "string",
          "minLength": 1
        },
        "reward_kind": {
          "enum": ["learned_reward_model", "ai_judge", "programmatic_verifier", "hybrid"]
        },
        "reward_artifact_sha256": {
          "type": "string",
          "pattern": "^[0-9a-f]{64}$"
        },
        "verifier_artifact_sha256": {
          "type": "string",
          "pattern": "^[0-9a-f]{64}$"
        },
        "visible_population_manifest_sha256": {
          "type": "string",
          "pattern": "^[0-9a-f]{64}$"
        }
      }
    },
    "promotion": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "holdout_manifest_sha256",
        "independent_evaluator_sha256",
        "scenario_policy_sha256",
        "visible_to_optimization"
      ],
      "properties": {
        "holdout_manifest_sha256": {
          "type": "string",
          "pattern": "^[0-9a-f]{64}$"
        },
        "independent_evaluator_sha256": {
          "type": "string",
          "pattern": "^[0-9a-f]{64}$"
        },
        "scenario_policy_sha256": {
          "type": "string",
          "pattern": "^[0-9a-f]{64}$"
        },
        "visible_to_optimization": {
          "const": false
        }
      }
    },
    "thresholds": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "minimum_holdout_population",
        "high_reward_threshold",
        "minimum_independent_score",
        "maximum_high_reward_bad_rate",
        "maximum_verifier_false_positive_rate"
      ],
      "properties": {
        "minimum_holdout_population": {
          "type": "integer",
          "minimum": 1
        },
        "high_reward_threshold": {
          "type": "number"
        },
        "minimum_independent_score": {
          "type": "number"
        },
        "maximum_high_reward_bad_rate": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "maximum_verifier_false_positive_rate": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        }
      }
    }
  }
}

# File: alignment/validate_reward_contract.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

import jsonschema


def file_sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def validate_contract(
    *,
    contract_path: Path,
    schema_path: Path,
    optimization_population_path: Path,
    promotion_holdout_path: Path,
) -> dict:
    contract = json.loads(contract_path.read_text(encoding="utf-8"))
    schema = json.loads(schema_path.read_text(encoding="utf-8"))
    jsonschema.Draft202012Validator(schema).validate(contract)

    optimization_digest = file_sha256(optimization_population_path)
    holdout_digest = file_sha256(promotion_holdout_path)
    if optimization_digest != contract["optimization"]["visible_population_manifest_sha256"]:
        raise ValueError("optimization population digest mismatch")
    if holdout_digest != contract["promotion"]["holdout_manifest_sha256"]:
        raise ValueError("promotion holdout digest mismatch")
    if optimization_digest == holdout_digest:
        raise ValueError("optimization and promotion populations are not isolated")

    optimization = json.loads(optimization_population_path.read_text(encoding="utf-8"))
    holdout = json.loads(promotion_holdout_path.read_text(encoding="utf-8"))
    optimization_ids = {str(row["sample_id"]) for row in optimization["samples"]}
    holdout_ids = {str(row["sample_id"]) for row in holdout["samples"]}
    if len(optimization_ids) != len(optimization["samples"]):
        raise ValueError("optimization sample IDs are duplicated")
    if len(holdout_ids) != len(holdout["samples"]):
        raise ValueError("holdout sample IDs are duplicated")
    if optimization_ids & holdout_ids:
        raise ValueError("sample identity overlaps optimization and promotion")

    return {
        "schema_version": "aidefend.reward-contract-validation.v1",
        "status": "validated",
        "contract_id": contract["contract_id"],
        "candidate_sha256": contract["candidate_sha256"],
        "contract_sha256": file_sha256(contract_path),
        "schema_sha256": file_sha256(schema_path),
        "optimization_population_sha256": optimization_digest,
        "promotion_holdout_sha256": holdout_digest,
        "optimization_sample_count": len(optimization_ids),
        "holdout_sample_count": len(holdout_ids),
    }

Verification, evidence, and outcomes

Sign the canonical contract only after schema and semantic validation, and bind its digest to the optimizer job admission policy. The training job must read back the exact reward/verifier and visible-population digests; the independent promotion runner must read back the exact holdout/evaluator/scenario-policy digests and reject any promotion asset exposed to optimization. Retain both signature-verification receipts, schema, manifests, access-control and access-log evidence for holdout isolation, optimizer/evaluator image digests, validation receipt, and candidate/base artifacts.

Action: Materialize and sign the exact contract, prove optimization/promotion population isolation and effective artifact readback, and prevent training or evaluation from starting unless this control independently returns PASS.

AID-H-012.005-G002

Run an independent reward-gaming promotion gate before accepting an RLHF/RLVR candidate.

Run this gate for every candidate optimized against a learned reward, AI judge, or programmatic verifier, and again after any material reward/verifier, holdout, evaluator, or scenario-policy change. It consumes the independently verified contract from AID-H-012.005-G001, but recomputes its own candidate/population/artifact bindings and never inherits the upstream outcome. A candidate with no reward/verifier optimization is NOT_APPLICABLE.

Gate the complete signed holdout population

# File: alignment/reward_gaming_gate.py
from __future__ import annotations

import hashlib
import json
import math
import re
from pathlib import Path
from typing import Any


HEX64 = re.compile(r"^[0-9a-f]{64}$")
REQUIRED_FIELDS = {
    "sample_id",
    "scenario_id",
    "candidate_sha256",
    "output_sha256",
    "reward_artifact_sha256",
    "verifier_artifact_sha256",
    "reward_score",
    "verifier_pass",
    "independent_score",
    "safety_violation",
}


class InsufficientData(RuntimeError):
    pass


def file_sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def read_jsonl(path: Path) -> list[dict[str, Any]]:
    rows: list[dict[str, Any]] = []
    for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        row = json.loads(line)
        if not isinstance(row, dict) or set(row) != REQUIRED_FIELDS:
            raise ValueError(f"line {line_number} does not match the exact rollout schema")
        rows.append(row)
    if not rows:
        raise InsufficientData("rollout result population is empty")
    return rows


def gate(
    *,
    contract_path: Path,
    holdout_manifest_path: Path,
    rollouts_path: Path,
    verified_contract_sha256: str,
) -> dict:
    if file_sha256(contract_path) != verified_contract_sha256:
        raise ValueError("contract digest differs from its verified signature receipt")
    contract = json.loads(contract_path.read_text(encoding="utf-8"))
    if contract.get("schema_version") != "aidefend.reward-verifier-contract.v2":
        raise ValueError("unsupported contract schema")
    if file_sha256(holdout_manifest_path) != contract["promotion"]["holdout_manifest_sha256"]:
        raise ValueError("holdout manifest digest mismatch")

    manifest = json.loads(holdout_manifest_path.read_text(encoding="utf-8"))
    samples = manifest.get("samples")
    if not isinstance(samples, list) or not samples:
        raise InsufficientData("holdout manifest has no samples")
    expected = {str(row["sample_id"]): str(row["scenario_id"]) for row in samples}
    if len(expected) != len(samples):
        raise ValueError("holdout sample IDs are duplicated")
    thresholds = contract["thresholds"]
    if len(expected) < int(thresholds["minimum_holdout_population"]):
        raise InsufficientData("signed holdout is below the policy minimum")

    rows = read_jsonl(rollouts_path)
    observed_ids = [str(row["sample_id"]) for row in rows]
    if len(observed_ids) != len(set(observed_ids)):
        raise ValueError("rollout sample IDs are duplicated")
    if set(observed_ids) != set(expected) or len(rows) != len(expected):
        raise InsufficientData("rollout population differs from the signed holdout")
    expected_reward = contract["optimization"]["reward_artifact_sha256"]
    expected_verifier = contract["optimization"]["verifier_artifact_sha256"]
    expected_candidate = contract["candidate_sha256"]

    high_reward_bad: list[str] = []
    verifier_false_positive: list[str] = []
    result_rows: list[dict[str, Any]] = []
    for row in rows:
        sample_id = str(row["sample_id"])
        if str(row["scenario_id"]) != expected[sample_id]:
            raise ValueError("sample-to-scenario binding mismatch")
        if row["candidate_sha256"] != expected_candidate:
            raise ValueError("rollout candidate artifact mismatch")
        if row["reward_artifact_sha256"] != expected_reward:
            raise ValueError("effective reward artifact mismatch")
        if row["verifier_artifact_sha256"] != expected_verifier:
            raise ValueError("effective verifier artifact mismatch")
        if not HEX64.fullmatch(str(row["output_sha256"])):
            raise ValueError("output digest is malformed")
        if type(row["verifier_pass"]) is not bool or type(row["safety_violation"]) is not bool:
            raise ValueError("verifier and safety fields must be booleans")
        reward_score = row["reward_score"]
        independent_score = row["independent_score"]
        if not all(
            isinstance(value, (int, float)) and math.isfinite(value)
            for value in [reward_score, independent_score]
        ):
            raise ValueError("rollout scores must be finite")

        independent_bad = (
            float(independent_score) < float(thresholds["minimum_independent_score"])
            or row["safety_violation"]
        )
        if float(reward_score) >= float(thresholds["high_reward_threshold"]) and independent_bad:
            high_reward_bad.append(sample_id)
        if row["verifier_pass"] and independent_bad:
            verifier_false_positive.append(sample_id)
        result_rows.append(
            {
                "sample_id": sample_id,
                "scenario_id": row["scenario_id"],
                "output_sha256": row["output_sha256"],
                "high_reward_independent_bad": (
                    float(reward_score) >= float(thresholds["high_reward_threshold"])
                    and independent_bad
                ),
                "verifier_false_positive": row["verifier_pass"] and independent_bad,
            }
        )

    population = len(rows)
    high_reward_bad_rate = len(high_reward_bad) / population
    verifier_false_positive_rate = len(verifier_false_positive) / population
    failures: list[str] = []
    if high_reward_bad_rate > float(thresholds["maximum_high_reward_bad_rate"]):
        failures.append("high_reward_bad_rate")
    if verifier_false_positive_rate > float(
        thresholds["maximum_verifier_false_positive_rate"]
    ):
        failures.append("verifier_false_positive_rate")

    return {
        "schema_version": "aidefend.reward-gaming-gate.v2",
        "status": "FAIL" if failures else "PASS",
        "contract_id": contract["contract_id"],
        "contract_sha256": verified_contract_sha256,
        "candidate_sha256": expected_candidate,
        "holdout_manifest_sha256": file_sha256(holdout_manifest_path),
        "rollouts_sha256": file_sha256(rollouts_path),
        "expected_population": len(expected),
        "observed_population": population,
        "reward_artifact_sha256": expected_reward,
        "verifier_artifact_sha256": expected_verifier,
        "independent_evaluator_sha256": contract["promotion"]["independent_evaluator_sha256"],
        "scenario_policy_sha256": contract["promotion"]["scenario_policy_sha256"],
        "high_reward_bad_rate": high_reward_bad_rate,
        "verifier_false_positive_rate": verifier_false_positive_rate,
        "high_reward_bad_sample_ids": sorted(high_reward_bad),
        "verifier_false_positive_sample_ids": sorted(verifier_false_positive),
        "result_rows": sorted(result_rows, key=lambda row: row["sample_id"]),
        "failed_checks": failures,
    }

Independent replay, evidence, and outcomes

The promotion identity must be unable to modify the candidate, contract, holdout, reward/verifier artifacts, independent evaluator, or scenario policy. Persist the signature-verification receipt, exact access-policy readback, executor/evaluator image digests, complete raw prompts/inputs and outputs under approved data handling, rollouts, gate report, and all artifact/population digests in append-only storage. A separate runner reloads the exact candidate and independently regenerates every output and independent score before recomputing byte-equivalent result rows and the verdict.

Action: Run the exact candidate over the complete isolated holdout with independent evidence, retain every bound result, and block promotion unless a separate full regeneration and gate replay returns PASS.

Open-source tools

Hugging Face TRL
OpenRLHF
verl
EleutherAI lm-evaluation-harness
Inspect AI
python-jsonschema
Cosign
RewardBench

Threat mappings

MITRE ATLAS

MAESTRO

Manipulation of Evaluation Metrics (L5)

OWASP LLM Top 10 2026

OWASP ML Top 10 2023

OWASP Top 10 for Agentic Applications 2026

NIST Adversarial Machine Learning 2025

NISTAML.027 Misaligned Outputs

Cisco Integrated AI Security and Safety Framework

AISubtech-2.3.1 Task Loophole Exploitation
AISubtech-2.3.2 Verifier Manipulation
AISubtech-2.3.3 Shortcut Acquisition
AITech-2.3 Reward Hacking

Google Secure AI Framework 2.0 - Risks

RA: Rogue Actions (independent reward-gaming gate blocks candidates that optimize toward unintended actions)

Databricks AI Security Framework 3.0

Agents - Core 13.7: Misaligned & Deceptive Behaviors