BenchMatch bench.match@1

Generated at from openvibe-contracts v0.443.0 and openvibe-sdk v0.177.0.

Version
1.0.0
Owner
bench
Visibility
public
Status
active
Compatibility
backward
Schema
https://openvibe.network/contracts/bench/match.v1.json

bench.match@1 (compare-arena-ecosystem.md, 2026-11-13): one task given to two or more entrants under the same budget and machine class, the receipts of each side, and every judgement that came in. Every comparison on the network is one: an Arena vote, a Chat draft picked, a Codes race whose patch was merged, a check job's tests, a rubric model, a game, a Live chat's poll. SOURCE says where it came from; SLOTS are the sides (a, b, …) with the entrant, the run that answered and its receipts; SHOWN is the order the sides were shown in (position bias is modelled, never ignored); BLIND says whether the names were hidden until the first judgement. OpenVibe.Bench keeps matches and rates entrants from them (bench.rating@1).

Fields

FieldTypeRequiredDescriptionConstraints
idstringyes
  • pattern ^mat_[0-9A-HJKMNP-TV-Z]{26}$
sourceenumyes
  • one of "arena", "chat", "codes", "live", "games", "exchange", "space", "bench", "check"
taskobjectyes
  • no other fields
task.kindenumyes
  • one of "chat", "code", "design", "image", "speech", "search", "computer", "game"
task.categorystring | nullWhat the task is about, from the classifier (code.debug, writing.email, math …), so boards are per category.
  • pattern ^[a-z][a-z0-9._-]{0,60}$
task.prompt_hashstring | nullsha256 of the prompt: dedupe and the private pool, never the prompt itself.
  • pattern ^[a-f0-9]{64}$
task.promptstring | nullOnly when the asker opted in to share it (PUBLIC true).
  • maxLength 20000
task.publicboolean
  • default false
task.repostring | null
  • maxLength 300
task.shastring | null
  • pattern ^[a-f0-9]{40}$
task.suitestring | nullA Bench suite and task id, when the task came from one.
  • maxLength 120
slotsarray of objectyes
  • minItems 2
  • maxItems 8
  • items: no other fields
slots[].slotstringyes
  • pattern ^[a-h]$
slots[].entrantstringyesA bench.entrant@1 id.
  • minLength 1
  • maxLength 300
slots[].run_idstring | nullThe agents.run@1 (or AI run) that answered, so its steps and receipts can be read.
  • maxLength 120
slots[].receiptsobject
  • no other fields
slots[].receipts.outcomeenum
  • one of "succeeded", "failed", "timed_out", "refused", "cancelled"
slots[].receipts.cost_usdnumber | null
  • minimum 0
slots[].receipts.tokens_ininteger | null
  • minimum 0
slots[].receipts.tokens_outinteger | null
  • minimum 0
slots[].receipts.ttft_msnumber | null
  • minimum 0
slots[].receipts.tpsnumber | null
  • minimum 0
slots[].receipts.wall_msnumber | null
  • minimum 0
slots[].receipts.tool_callsinteger | null
  • minimum 0
slots[].receipts.length_charsinteger | nullThe answer's length, a style covariate.
  • minimum 0
slots[].receipts.markdown_itemsinteger | nullHeadings, lists and bold spans in the answer, a style covariate.
  • minimum 0
shownarray of stringThe order the sides were shown in, left to right or top to bottom.
  • maxItems 8
  • items: pattern ^[a-h]$
blindboolean
  • default true
budgetobject | nullThe same for every side: a match between unequal budgets is not a match.
judgementsarray of objectyes
  • maxItems 200
  • items: no other fields
judgements[].judgeenumyes
  • one of "human_pairwise", "audience_poll", "implicit", "test_suite", "rubric", "game_outcome"
judgements[].winnerstring | null
  • pattern ^([a-h]|tie|both_bad)$
judgements[].scoresobject | nullPer side: tests passed (0..1), a rubric score, a game's points.
judgements[].signalenumAn implicit judge's signal: what the person did with a side.
  • one of null, "merged", "accepted", "copied", "kept", "retried", "stopped", "edited"
judgements[].votesobject | nullAn audience poll's counts.
judgements[].byobject | null
judgements[].weightnumber | nullThe voter's quality weight when the judgement was counted (gold pairs, agreement).
  • minimum 0
  • maximum 1
judgements[].style_notestring | null
  • maxLength 200
judgements[].atstringyes
  • format date-time
playobject | nullThe race play and its parent run, when the match was run on the agent engine.
atstringyes
  • format date-time

Examples

From the contract's own test fixtures: valid ones validate, rejected ones must fail.

Valid: a-codes-race-merged
{
  "id": "mat_01KA0000000000000000000003",
  "source": "codes",
  "task": {
    "kind": "code",
    "category": "code.bugfix",
    "repo": "https://github.com/OpenVibers/OpenVibe.Compare",
    "sha": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
  },
  "slots": [
    {
      "slot": "a",
      "entrant": "agent:codex+gpt-5-mini#3f9a1c22",
      "receipts": {
        "outcome": "succeeded",
        "cost_usd": 0.21,
        "wall_ms": 312000,
        "tool_calls": 41
      }
    },
    {
      "slot": "b",
      "entrant": "agent:opencode+glm-5.3#9b7e2d10",
      "receipts": {
        "outcome": "succeeded",
        "cost_usd": 0.09,
        "wall_ms": 401000,
        "tool_calls": 57
      }
    }
  ],
  "blind": false,
  "judgements": [
    {
      "judge": "test_suite",
      "scores": {
        "a": 1,
        "b": 0.8
      },
      "by": {
        "type": "suite"
      },
      "at": "2026-11-13T02:10:00Z"
    },
    {
      "judge": "implicit",
      "winner": "a",
      "signal": "merged",
      "by": {
        "type": "person",
        "id_hash": "fedcba9876543210"
      },
      "at": "2026-11-13T02:30:00Z"
    }
  ],
  "at": "2026-11-13T02:00:00Z"
}
Valid: an-arena-vote
{
  "id": "mat_01KA0000000000000000000000",
  "source": "arena",
  "task": {
    "kind": "chat",
    "category": "writing.email",
    "prompt_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "public": false
  },
  "slots": [
    {
      "slot": "a",
      "entrant": "model:deepseek-flash",
      "run_id": "run_01KA0000000000000000000001",
      "receipts": {
        "outcome": "succeeded",
        "cost_usd": 0.0004,
        "tokens_in": 120,
        "tokens_out": 300,
        "ttft_ms": 540,
        "tps": 86,
        "wall_ms": 4100,
        "length_chars": 1210,
        "markdown_items": 3
      }
    },
    {
      "slot": "b",
      "entrant": "model:gpt-5-mini",
      "run_id": "run_01KA0000000000000000000002",
      "receipts": {
        "outcome": "succeeded",
        "cost_usd": 0.0011,
        "tokens_in": 120,
        "tokens_out": 280,
        "ttft_ms": 820,
        "tps": 61,
        "wall_ms": 5300,
        "length_chars": 980,
        "markdown_items": 0
      }
    }
  ],
  "shown": [
    "b",
    "a"
  ],
  "blind": true,
  "budget": {
    "usd": 0.01,
    "ms": 60000,
    "machine_class": null
  },
  "judgements": [
    {
      "judge": "human_pairwise",
      "winner": "a",
      "by": {
        "type": "person",
        "id_hash": "0123456789abcdef"
      },
      "weight": 0.9,
      "at": "2026-11-13T02:00:00Z"
    }
  ],
  "play": {
    "id": "race",
    "run_id": "run_01KA0000000000000000000000"
  },
  "at": "2026-11-13T01:59:30Z"
}
Rejected: a-judge-nobody-has
{
  "id": "mat_01KA0000000000000000000000",
  "source": "arena",
  "task": {
    "kind": "chat",
    "category": "writing.email",
    "prompt_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "public": false
  },
  "slots": [
    {
      "slot": "a",
      "entrant": "model:deepseek-flash",
      "run_id": "run_01KA0000000000000000000001",
      "receipts": {
        "outcome": "succeeded",
        "cost_usd": 0.0004,
        "tokens_in": 120,
        "tokens_out": 300,
        "ttft_ms": 540,
        "tps": 86,
        "wall_ms": 4100,
        "length_chars": 1210,
        "markdown_items": 3
      }
    },
    {
      "slot": "b",
      "entrant": "model:gpt-5-mini",
      "run_id": "run_01KA0000000000000000000002",
      "receipts": {
        "outcome": "succeeded",
        "cost_usd": 0.0011,
        "tokens_in": 120,
        "tokens_out": 280,
        "ttft_ms": 820,
        "tps": 61,
        "wall_ms": 5300,
        "length_chars": 980,
        "markdown_items": 0
      }
    }
  ],
  "shown": [
    "b",
    "a"
  ],
  "blind": true,
  "budget": {
    "usd": 0.01,
    "ms": 60000,
    "machine_class": null
  },
  "judgements": [
    {
      "judge": "vibes",
      "winner": "a",
      "by": {
        "type": "person",
        "id_hash": "0123456789abcdef"
      },
      "weight": 0.9,
      "at": "2026-11-13T02:00:00Z"
    }
  ],
  "play": {
    "id": "race",
    "run_id": "run_01KA0000000000000000000000"
  },
  "at": "2026-11-13T01:59:30Z"
}
Rejected: one-side-is-not-a-match
{
  "id": "mat_01KA0000000000000000000000",
  "source": "arena",
  "task": {
    "kind": "chat",
    "category": "writing.email",
    "prompt_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "public": false
  },
  "slots": [
    {
      "slot": "a",
      "entrant": "model:deepseek-flash",
      "run_id": "run_01KA0000000000000000000001",
      "receipts": {
        "outcome": "succeeded",
        "cost_usd": 0.0004,
        "tokens_in": 120,
        "tokens_out": 300,
        "ttft_ms": 540,
        "tps": 86,
        "wall_ms": 4100,
        "length_chars": 1210,
        "markdown_items": 3
      }
    }
  ],
  "shown": [
    "b",
    "a"
  ],
  "blind": true,
  "budget": {
    "usd": 0.01,
    "ms": 60000,
    "machine_class": null
  },
  "judgements": [
    {
      "judge": "human_pairwise",
      "winner": "a",
      "by": {
        "type": "person",
        "id_hash": "0123456789abcdef"
      },
      "weight": 0.9,
      "at": "2026-11-13T02:00:00Z"
    }
  ],
  "play": {
    "id": "race",
    "run_id": "run_01KA0000000000000000000000"
  },
  "at": "2026-11-13T01:59:30Z"
}

Validate

const contracts = require('openvibe-contracts');
contracts.validate('bench.match@1', value);   // { valid, errors: [{ path, message }] }