What a verifiable outcome actually looks like.

Modeled on fixtures/parse-numbers-fixture.git and fixtures/parse-numbers-expected.json in this repo.

“Fix the number parser, it's buggy.”

parseNumbers(input) must throw TypeError for every entry in the pinned invalid-input list ["12,abc", "1,,2", "1,Infinity", "x", "NaN"] instead of silently returning NaN, while the existing npm test suite still passes.

• Public repository https://github.com/kervyxa-fixtures/parse-numbers, pinned to base commit ac88435b238babb9b3aa3e274387b3048c436b80
• The pinned invalid-input list quoted in the objective (frozen in the contract, not discoverable by the solver from prose alone)

npm test

node -e 'const {parseNumbers}=require("./parseNumbers.js");for(const s of ["12,abc","1,,2","1,Infinity","x","NaN"]){let rejected=false;try{parseNumbers(s)}catch(e){rejected=e instanceof TypeError}if(!rejected){console.log("NOT REJECTED: "+JSON.stringify(s));process.exit(1)}}console.log("all rejected")'

sh "$KERVYXA_VERIFIER_PACKAGE/run-regression.sh"

Protected paths

package.json

Max changed files

2

Allowed languages

javascript

Notes

The fix should live in parseNumbers.js; no new dependencies.

Deterministic Docker, node:22-slim pinned by digest, --network none. Setup runs npm ci; each criterion above is one step with its own timeout.

Open competition (submissionPolicy PUBLIC) — any qualified solver may submit.

• A public git repository URL and an immutable 40-hex commit SHA that is a descendant of the base commit
• package.json untouched (protected path)
• No more than 2 changed files

A PASS enters the challenge window before settlement. A challenger who believes the hidden regression suite has a bug (e.g. it treats a genuinely-new valid format as a violation) can dispute it; Kervyxa resolves via UPHOLD_PASS / OVERTURN_PASS / RERUN_VERIFICATION. There is no dispute path for the solver to see the hidden regression suite itself.

Every criterion resolves to an exit code the container itself produces from a fixed, pinned command against a fixed input list — no human judgment about whether the fix is 'good', only whether the specified inputs behave as specified.

• "Fix the parser" with no invalid-input list: without pinning which strings must be rejected, a solver could satisfy any reading and a verifier has nothing to check against.
• The invalid-input list is left open-ended ("and similar cases") — a verifier can only check a fixed, enumerated list, not a fuzzy category.
• No regression criterion: without one, throwing TypeError unconditionally passes the exit_code check while breaking every previously-valid input.

Modeled on fixtures/test-creation-fixture.git and fixtures/test-creation-expected.json in this repo.

“Add some tests for isPalindrome.”

A real, passing automated test suite for isPalindrome must exist and run under npm test, and isPalindrome's actual return values for the pinned battery of inputs must be unchanged from the base commit.

• Public repository https://github.com/kervyxa-fixtures/test-creation, pinned to base commit bfa39b263c7ee08d6833501939ea810d903b56d6
• isPalindrome.js already exists and already behaves correctly — only test coverage is missing

true

npm test

node "$KERVYXA_VERIFIER_PACKAGE/hidden-check.js"

Protected paths

isPalindrome.js

Max changed files

1

Allowed languages

javascript

Notes

Only the new test file is expected to change.

Deterministic Docker, node:22-slim pinned by digest, --network none. Setup runs npm ci.

Open competition, PUBLIC_WITH_HIDDEN_VERIFIER privacy — the solver sees the two public checks; the regression check's judge/probe pair stays unseen.

• A public git repository URL and an immutable commit SHA
• isPalindrome.js itself untouched (protected path) — this contract is about adding coverage, not changing behavior

A PASS enters the challenge window. Because isPalindrome.js is protected and the regression check is hidden, the realistic challenge scenario is a claim that the test file is trivial (e.g. asserts nothing) — but the file_assertion criterion here only checks the file EXISTS; the real content bar is enforced by npm-test-passes plus the hidden regression check, which is exactly why 'exists' alone is refused at approval (see the invalid-if note below).

'A test file exists' alone would be satisfied by an empty file, so it is paired with an exit-code check that the suite actually runs and passes, plus a hidden check that the underlying function wasn't rewritten to dodge real testing.

• The only criterion is file_assertion "exists" on the test file: Kervyxa's approval step refuses a contract whose acceptance criteria are entirely exists/absent file assertions, because an empty file satisfies it.
• "Add some tests" with no target function named: a verifier cannot check that specific, real coverage exists for an unnamed piece of behavior.

No fixture in this repo demonstrates a dependency bump yet — built from the real AcceptanceCriterion union and approval rules (packages/domain/src/validation.ts), not from a shipped fixture.

“Upgrade our dependencies.”

date-fns must resolve to exactly 3.6.0 in the lockfile (no other dependency's resolved version may change), and the existing test suite must still pass.

• The repository's own package.json, package-lock.json, and source tree at the pinned base commit
• The exact target version string, 3.6.0, frozen in the contract

npm test

node -e 'const lock=require("./package-lock.json");const v=lock.packages["node_modules/date-fns"]?.version;if(v!=="3.6.0"){console.log("got "+v);process.exit(1)}'

sh "$KERVYXA_VERIFIER_PACKAGE/run-regression.sh"

Protected paths

none

Max changed files

2

Allowed languages

javascript

Notes

package.json and package-lock.json are deliberately NOT protected paths here — the solver must edit exactly those two files.

Deterministic Docker, node:22-slim pinned by digest, --network none — so the upgrade command itself (npm install) cannot reach the real npm registry inside the container; the contract assumes the solver runs the upgrade before submitting and commits the result, and the verifier only checks the committed lockfile and re-runs the test suite.

Open competition.

• A public git repository URL and an immutable commit SHA with the lockfile already updated
• Exactly package.json and package-lock.json changed (maxChangedFiles: 2)

A PASS enters the challenge window. The realistic challenge here is a claim that a transitive dependency also moved in a way the hidden regression diff missed — which is exactly the scenario the hidden judge/probe pair exists to catch, and a challenger can ask Kervyxa to RERUN_VERIFICATION.

The target version is a fixed string, not 'the latest' or 'a recent version' — a verifier can read the lockfile and compare it byte-for-byte to that string.

• "Upgrade our dependencies" with no named package and no target version: there is nothing for a numeric or string comparison to check against.
• "Upgrade to the latest version" — the latest version at verification time is not the latest version at submission time, so the contract could not stay frozen; a bounded upgrade always pins an exact target version.
• No maxChangedFiles limit: without one, a solver could satisfy the version check while also making unrelated changes the requester never asked for.

This exact contract is the real Scenario 2 fixture verified end-to-end against a real Docker container in apps/web/lib/verification/acceptance-matrix.int.test.ts, using fixtures/data-transform-fixture.git. No real paying customer created it — it exists to prove the acceptance-matrix scenario, so it stays labeled illustrative.

“Make the CSV export handle special characters.”

transform.js must emit RFC4180-quoted CSV so every record survives a comma or a double quote: running node transform.js records.jsonl rows.csv over the pinned, protected input must produce a CSV byte-identical to the pinned reference output.

• records.jsonl — the protected input file, unreadable-to-edit by the solver
• A pinned reference sha256 of the correct rows.csv output, frozen in the contract

node transform.js records.jsonl rows.csv && printf '%s  %s\n' <pinned-sha256> rows.csv | sha256sum -c --status

true

sh "$KERVYXA_VERIFIER_PACKAGE/run-regression.sh"

Protected paths

records.jsonl, package.json

Max changed files

1

Allowed languages

javascript

Deterministic Docker, node:22-slim pinned by digest, --network none. The checkout is mounted read-only; /workspace is a fresh, size-capped tmpfs for every single check.

Open competition, PUBLIC privacy.

• A public git repository URL and an immutable commit SHA
• records.jsonl and package.json untouched
• No more than 1 changed file

The acceptance-matrix test proves this scenario alongside a genuinely competing second solver (a character-scan implementation vs. a regex one, byte-identical output) submitting while the contract is still OPEN — Kervyxa's FIRST_VERIFIED_PASS rule takes the first PASS and DEFERS the second submission rather than failing or discarding it; the deferred submission gets its turn if the winner is overturned in a challenge.

A generated file compared to a pinned hash, computed in the same isolated run that produced it, is either byte-identical or it is not — there is no partial credit and nothing for a human to eyeball.

• "Handle special characters" with no reference output: without a pinned byte-exact target, any escaping scheme could claim to satisfy it.
• A file_assertion on rows.csv in a SEPARATE check from the command that generates it — the file would not exist in that check's fresh /workspace, so the contract could never pass.
• No protected path on records.jsonl — a solver could edit the input itself to make any transform.js match some output.

No fixture in this repo demonstrates document extraction yet — built from the real AcceptanceCriterion union and the same generate-and-compare-in-one-check rule proven by the data-transform fixture above.

“Pull the important fields out of these invoices.”

extract.js run over the 50 protected PDFs in invoices/ must emit extracted.csv with exactly the columns vendor, invoice_number, date (ISO-8601), total_cents — sorted by filename — byte-identical to the pinned reference CSV.

• invoices/ — 50 protected PDF files, unreadable-to-edit by the solver
• The exact column list, ordering rule, and date/currency formats, frozen in the contract
• A pinned reference sha256 of the correct extracted.csv

node extract.js invoices > extracted.csv && printf '%s  %s\n' <pinned-sha256> extracted.csv | sha256sum -c --status

true

sh "$KERVYXA_VERIFIER_PACKAGE/run-regression.sh"

Protected paths

invoices/, reference/

Max changed files

3

Allowed languages

javascript, python

Notes

No network access at verification time — any OCR or parsing library must be vendored or offline-capable.

Deterministic Docker, digest-pinned image, --network none. All 50 input PDFs and the reference CSV are committed into the protected part of the repository, not fetched at verification time.

Open competition, or invite-only if the requester wants a specific known solver.

• A public git repository URL and an immutable commit SHA
• invoices/ and reference/ untouched
• extract.js runs fully offline inside the container

A PASS enters the challenge window. The held-out hidden batch exists precisely so a challenge cannot be 'the extractor was hardcoded to the visible files' after the fact — that claim was already tested at verification time, not just arguable afterward.

Every field, format, and ordering rule is pinned in advance and compared by hash — there is no 'looks about right' step, and the hidden held-out batch removes the incentive to special-case the visible files.

• "Pull the important fields" with no fixed column list, formats, or ordering: a verifier cannot decide correctness against an undefined schema.
• No held-out batch (only the visible 50 checked): a solver could hardcode a lookup table from filename to output and pass trivially.
• The documents themselves are not fixed at contract time (e.g. "whatever invoices come in next month") — Kervyxa verifies against a specification frozen at approval, not a moving target.

The accepted framing is modeled on fixtures/memory-benchmark-fixture.git and fixtures/memory-benchmark-expected.json. The rejected framing is modeled on fixtures/perf-benchmark-fixture.git (a real fixture in this repo using a benchmark criterion measured in milliseconds) — shown here deliberately as the wrong way to write this contract.

“Make the indexer faster and lighter.”

Cut buildIndex()'s peak heap for a 250,000-word input from the supplied ~250MB baseline to under 60,000,000 bytes, measured as a single deterministic KERVYXA_METRIC=<bytes> marker printed by one fixed run inside the container — not wall-clock time.

• The word-indexer repository at its pinned base commit
• bench-memory.js — a fixed, deterministic harness that runs buildIndex() once over a fixed 250,000-word corpus and prints exactly one KERVYXA_METRIC=<bytes> line

node bench-memory.js  # prints "KERVYXA_METRIC=<bytes>"

npm test

sh "$KERVYXA_VERIFIER_PACKAGE/run-regression.sh"

Protected paths

bench-memory.js, corpus-250k.txt

Max changed files

3

Allowed languages

javascript

Deterministic Docker, digest-pinned image, frozen memoryMB/cpuCount/pidsLimit (these are settlement-critical, hashed fields — two workers configured differently must not produce different verdicts).

Open competition.

• A public git repository URL and an immutable commit SHA
• bench-memory.js and corpus-250k.txt untouched — the solver may not edit the measuring stick

A PASS enters the challenge window. Because the metric is a single deterministic marker from a fixed run in a resource-frozen container, there is no 'it was faster on my machine' dispute available — a challenger can only contest that the marker itself was tampered with, which the hidden regression check plus the protected harness file are built to prevent.

REJECTED FRAMING (what NOT to write): "reduce p95 latency to under 5ms" — a benchmark criterion whose unit resolves to a unit of time is refused at approval. Replaying two real verifications of the identical, unchanged commit moved per-step durations by 4x-19x on an idle host, so the same submitted code could pass once and fail on a rerun — wall-clock time cannot decide settlement. ACCEPTED FRAMING (what this contract actually uses): a deterministic counted metric the program itself emits — bytes of peak heap — read from exactly one KERVYXA_METRIC= line. The same input, the same code, and the same container produce the same number every time.

• "Make it faster" with a wall-clock threshold (ms, seconds, p95/p99 latency): Kervyxa refuses any benchmark criterion whose unit reduces to a measure of elapsed time at approval time — replay drift of 4x-19x on an idle host was measured directly in this repo.
• No fixed corpus / no fixed harness: without a pinned input and a pinned measuring script, "under 60MB" has no stable meaning between two verification runs.
• No regression criterion checking output correctness: without one, a solver could satisfy the byte threshold by making buildIndex() do less work incorrectly.

No fixture in this repo demonstrates a model-evaluation contract yet — built strictly from the real AcceptanceCriterion union and the real constraint that verifier containers run with --network none, which rules out any evaluation that calls a hosted model API.

“Improve the model's quality.”

The classifier committed at model/classifier.onnx must score macro-F1 >= 0.82 against a fixed, protected labeled test set, using a fixed random seed, entirely inside a container with no network access.

• The repository's own train.jsonl and a small public labeled sample, committed and readable
• label_holdout.jsonl — a protected, hidden-package-only labeled test set the solver never sees
• A fixed random seed, frozen in the contract

node smoke-test.js

node "$KERVYXA_VERIFIER_PACKAGE/score.js"  # prints "KERVYXA_METRIC=<f1>"

sh "$KERVYXA_VERIFIER_PACKAGE/run-regression.sh"

Protected paths

train.jsonl, label_public_sample.jsonl

Max changed files

5

Allowed languages

python, javascript

Notes

No network access at verification time — evaluation of a call to a hosted LLM/model API is out of scope; only a model or classifier committed into the repository (or vendored, runnable offline) can be scored this way.

Deterministic Docker, digest-pinned image, --network none. The scoring script and the holdout set live only in the hidden verifier package, hashed and frozen at approval, mounted read-only for that one check.

Open competition or a Kervyxa-managed intake lane for a requester who wants hands-on review of the scoring methodology before funding.

• A public git repository URL and an immutable commit SHA
• train.jsonl and the public labeled sample untouched
• The classifier must run fully offline inside the container

A PASS enters the challenge window. Because the holdout set and scoring script are hidden and hashed, the realistic challenge is a claim that the fixed seed or scoring script itself was miscalibrated — resolvable by Kervyxa re-running the identical, frozen check (RERUN_VERIFICATION), not by re-debating the model's quality.

The metric is one number from a fixed formula (macro-F1) over a fixed, hidden dataset with a fixed seed — nothing about 'model quality' is left to a human's read, and the hidden holdout removes the incentive to memorize the visible test data.

• "Improve the model" with no fixed metric, dataset, or threshold: without all three pinned, there is nothing for a verifier to compute.
• The evaluation requires calling a hosted model API at scoring time: verifier containers run with no network access, so nothing that depends on a live external call can be verified this way today.
• The test set is not held out from the solver: without a hidden holdout, a solver can memorize or hardcode against the visible test data.

No fixture in this repo demonstrates a maintenance contract yet — modeled loosely on this repository's own gates (pnpm lint / pnpm format:check), which are exactly the kind of deterministic, already-scripted checks a maintenance contract can point at.

“Clean up the codebase.”

eslint --max-warnings=0 must exit 0 across the whole repository, prettier --check must exit 0, and the named module's exported symbol names must be unchanged from the base commit.

• The repository at its pinned base commit, with its existing eslint and prettier configuration
• The specific module whose exported API must stay stable (e.g. lib/index.ts), named in the contract

npx eslint . --max-warnings=0

npx prettier --check .

sh "$KERVYXA_VERIFIER_PACKAGE/run-regression.sh"

Protected paths

.eslintrc.json, .prettierrc, package.json

Max changed files

40

Allowed languages

javascript, typescript

Notes

Lint and formatting configuration is protected specifically so a solver cannot satisfy the checks by loosening the rules they measure against.

Deterministic Docker, digest-pinned image, --network none. Setup installs dependencies; each check runs the repository's own existing scripts.

Open competition.

• A public git repository URL and an immutable commit SHA
• Lint/format configuration and package.json untouched
• Up to 40 changed files (a repo-wide cleanup legitimately touches many files)

A PASS enters the challenge window. The realistic challenge is a claim that an export was quietly removed rather than renamed-and-kept — exactly what the hidden regression diff over exported symbol names is built to catch.

Both quality checks are the exact commands the repository's own CI would run, with fixed exit-code semantics; the hidden API-surface diff turns 'don't break the public interface' into a literal string-set comparison instead of a judgment call.

• "Clean up the codebase" alone: without naming the exact commands (lint, format) and their exit-code bar, there is no fixed pass/fail rule.
• The lint/format config itself is not protected: a solver could satisfy "zero lint errors" by disabling the rules that were failing, which is not the same as fixing them.
• No regression criterion protecting the public API: a large, unreviewed cleanup could delete exported functions other code depends on while every named check still passes.