Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

Jingjie Ning; Shanshan Zhong; Xiaochuan Li; Ji Zeng

A NEW EVIDENCE STANDARD FOR AI RESEARCH

A high score.
A deeper question.

What did the research process contribute?

The Discovery Certification Protocol turns AI research claims into testable evidence. Validate the gain. Challenge its recovery. Measure the contribution of feedback.

Explore DCP Read the paper

Carnegie Mellon University
School of Computer Science

THE RESEARCH LOOP

INTERACTIVE PROTOCOL ILLUSTRATION

Freeze the model, starting material, evaluator, and budgets before the research run.

SELECT A NODE

OPEN RESEARCH. EXECUTABLE EVIDENCE.

GET IT ON PyPIdcp-audit

GET IT ON PyPIdcp-harness

GitHub ↗

THE PROTOCOL

One result.
Three sharper questions.

An AI research agent proposes ideas, runs experiments, and uses feedback to choose its next step. DCP tests the outcome and the evidence behind that loop.

01VALIDATE

Baseline

Research result

Registered margin

Does the result hold up?

Check a valid artifact against a frozen baseline on a sealed evaluator. Establish a useful, measurable gain.

A validated outcome

02CHALLENGE

Same numerical target

Could a fresh agent recover it?

Give matched agents the starting information and captured Web content, with the target research history withheld. Count every qualifying recovery.

Bounded recovery evidence

03ATTRIBUTE · OPTIONAL

Truthful feedback↗

Neutral feedback→

Separate null calibration

What did feedback contribute?

Compare fresh paired runs from a shared checkpoint. Test truthful feedback against a registered neutral policy with independent null calibration.

A calibrated feedback effect

DCP Core

Valid gain + adequate controls + zero recoveries + a passing probability bound.

DCP Evidence

Core + a supported paired feedback effect over a calibrated neutral policy.

A registered episode includes the agent’s full budget, candidates, and selection procedure. A qualifying recovery vetoes Core eligibility. Incomplete controls or insufficient evidence produce an inconclusive decision.

Read the methodology

THE EVIDENCE

Different tasks.
The same questions.

Two controlled, three-gate audits with known information boundaries. Each uses its own frozen model and task. Every displayed decision has a replayable evidence bundle.

SQLITE-WEBDeepSeek V4 Flash

Make database
queries cheaper.

Choose up to four indexes for a hidden SQLite workload. Experimental feedback reveals which choices reduce measured query cost.

Normalized query-cost reductionHIGHER IS BETTER

Baseline

0.000

Research result

0.886

1.0 represents the task’s maximum normalized cost reduction.

0/96challenger recoveries

30 vs 0truthful / neutral successes

Core + Evidence certified

VIRTUAL CATALYSTDeepSeek V4 Pro

Find a better
catalyst recipe.

Tune five controls in a virtual catalyst task. Use experimental measurements to find a recipe with higher sealed utility.

Normalized recipe utilityHIGHER IS BETTER

Baseline

0.599

Research result

1.000

1.0 is the optimum of this controlled virtual task.

0/96challenger recoveries

30 vs 0truthful / neutral successes

Core + Evidence certified

0.0468Recovery probability upper bound
for each registered audit

45 / 45Positive controls recovered
in each audit

60 pairsPassing null calibration
in each audit

Each feedback study contains 30 fresh pairs. Decisions are conditional on the registered model, budget, information, and policies. These are locally replayed protocol decisions; formal third-party issuance is a separate step.

Explore all four published audit bundles

THE TOOLKIT

A small install.
An inspectable result.

Start with an existing audit, then bring the protocol to your own research loop. Two focused Python packages cover evidence verification and capture.

THE OFFLINE VERIFIER

dcp-audit

Verify bundle integrity, recompute the protocol decision, and compare it with the recorded certificate. Runs locally using frozen evidence.

DeterministicLLM-freeOffline replay

Get dcp-audit on PyPI

THE RESEARCH HARNESS

dcp-harness

A lightweight Claude Code CLI wrapper for prospective research runs. Connect a task adapter, capture allowed Web responses, and schedule the audit controls.

Web captureMatched challengesPaired runs

Get dcp-harness on PyPI

From install to evidence

Python 3.11+ · use a virtual environment

# Install in an isolated environment
python -m venv .venv
source .venv/bin/activate
python -m pip install dcp-audit dcp-harness

# Get the published evidence and replay both audits
git clone https://github.com/cxcscmu/Discovery-Certification-Protocol.git
cd Discovery-Certification-Protocol
dcp verify examples/audits/sqlite-web
dcp verify examples/audits/virtual-catalyst

Expected replayVERIFIED · Core certified · Evidence certified

Replay verification checks agreement with the saved decision. The Core and Evidence fields report the scientific verdict.

# Run from the cloned repository after installing dcp-audit
from dcp.api import verify_bundle

report = verify_bundle("examples/audits/sqlite-web")
if not report["ok"]:
    raise RuntimeError("; ".join(report["errors"]))

verdict = report["replayed_verdict"]
print(verdict["core"], verdict["evidence"])

The verifier uses the recorded numerical evidence and frozen registration. Model calls and API keys are unnecessary for replay.

dcp-harness init my-audit

# Complete task_adapter.py and configure dcp-harness.json.
# Supply the public workspace, trusted evaluator, registered model,
# budgets, feedback policies, and a pinned runtime.
dcp-harness doctor --config my-audit/dcp-harness.json
dcp-harness run --config my-audit/dcp-harness.json --run my-audit/run-001

Prospective capture uses authenticated Claude Code CLI and a task-specific adapter. Certification-grade isolation uses the configured Docker runtime. Read the setup guide ↗

THE PAPER