Verifier
A verifier is a mechanism for checking a proposed answer, action, or final state against a specification. In an agent workflow, it might inspect a saved file or database state rather than trusting the agent’s statement that the task is complete. Verification can be deterministic, model-based, human-assisted, or a combination.
How it works
Define the requirement and identify observable evidence. A test can compare values, check constraints, execute code, or inspect an environment after actions. OSWorld illustrates execution-based task evaluation in computer environments. The verifier itself needs tests for false acceptance and false rejection.
A check that sees only part of the outcome may miss unintended side effects. Multiple checks can be needed when correctness includes both completing the task and respecting constraints.
Why it matters for licensing
Evaluation data and outcome labels are more interpretable when the verification method is documented. A recipient needs to know what a success label means, which evidence was available, and where the checker may be unreliable.
Example
Fictional example: An agent creates a draft invoice. A verifier checks the customer reference, line items, total, and draft status. It rejects an invoice that has the right total but was sent without approval.
Limitations and misconceptions
A passing check proves only what the check can establish under its assumptions. Model-based judges can be inconsistent, while rule-based checks can miss valid alternatives or hidden errors. A vulnerable verifier can become a target for reward hacking.
Questions to ask
- What evidence does the check inspect, and what does it omit?
- Have false positives, false negatives, and alternate valid outcomes been tested?
- Can the agent affect the scoring mechanism or hide side effects?