Methodology

Each evaluation uses a versioned ruleset and publishes its SHA-256 digest. Hold'em uses three seats, no rake, fixed blinds, and a 100 bb stack reset every hand. Blackjack uses six decks, dealer stand on soft 17, 3:2 blackjack, and the listed double and split rules. American double-zero Roulette is an Exhibition event.

{
  "players": 3,
  "variant": "no_limit_texas_holdem",
  "small_blind": 50,
  "big_blind": 100,
  "ante": 0,
  "starting_stack": 10000,
  "stack_reset_every_hand": true,
  "rake": 0,
  "min_raise_rule": "previous_raise_size_or_big_blind",
  "all_in_reopens_betting_only_if_full_raise": true,
  "button_rotation": "clockwise_each_hand",
  "deck": "standard_52",
  "shuffle": "seeded_fisher_yates_per_block",
  "hands_per_match": 6,
  "block_structure": "a block is one seeded deck sequence played once under each of the 6 seat permutations; each permutation is one match of hands_per_match hands",
  "memory_between_hands": "none",
  "showdown": "all_remaining_hands_revealed_to_spectators_after_block",
  "tie_break": "split_pot_odd_chip_to_first_seat_left_of_button",
  "chat": {
    "messages_per_street_per_seat": 1,
    "messages_per_hand_per_seat": 4,
    "max_chars": 240,
    "language": "en",
    "delivery": "all_seats_same_table",
    "reset": "per_hand"
  },
  "fallback": "check_if_possible_else_fold",
  "decision_deadline_ms": 60000
}

{
  "decks": 6,
  "dealer_hits_soft_17": false,
  "blackjack_payout": "3:2",
  "blackjack_payout_numerator": 3,
  "blackjack_payout_denominator": 2,
  "dealer_peeks_for_blackjack": true,
  "double_allowed": "first_two_cards_any",
  "double_after_split": true,
  "max_hands_after_split": 4,
  "resplit_aces": false,
  "split_aces_one_card": true,
  "surrender": "none",
  "insurance": false,
  "side_bets": false,
  "base_bet": 10,
  "starting_bankroll": 1000,
  "bet_sizing": "fixed_base_bet",
  "penetration_cards_cut_from_bottom": 78,
  "shuffle": "seeded_fisher_yates_on_new_shoe",
  "rounds_per_session": 100,
  "same_initial_state_comparison": true,
  "chat": {
    "regular_evaluation_delivery": "none",
    "cooperative_table_mode": "separate_condition_only",
    "max_chars": 240
  },
  "fallback": "stand",
  "decision_deadline_ms": 60000
}

{
  "wheel": "american_double_zero",
  "pockets": [
    "0",
    "00",
    "1",
    "2",
    "3",
    "4",
    "5",
    "6",
    "7",
    "8",
    "9",
    "10",
    "11",
    "12",
    "13",
    "14",
    "15",
    "16",
    "17",
    "18",
    "19",
    "20",
    "21",
    "22",
    "23",
    "24",
    "25",
    "26",
    "27",
    "28",
    "29",
    "30",
    "31",
    "32",
    "33",
    "34",
    "35",
    "36"
  ],
  "red_numbers": [
    1,
    3,
    5,
    7,
    9,
    12,
    14,
    16,
    18,
    19,
    21,
    23,
    25,
    27,
    30,
    32,
    34,
    36
  ],
  "spin_distribution": "independent_uniform_over_38_pockets",
  "spin_generation": "seeded_prng_per_block_committed_before_bets",
  "allowed_bet_types": [
    "straight_up",
    "red_black",
    "odd_even",
    "low_high",
    "dozen",
    "column"
  ],
  "payouts_to_one": {
    "straight_up": 35,
    "red_black": 1,
    "odd_even": 1,
    "low_high": 1,
    "dozen": 2,
    "column": 2
  },
  "zero_and_double_zero": "lose_all_outside_bets; straight_up on 0 or 00 pays 35:1",
  "min_stake_per_bet": 1,
  "max_stake_per_bet": 100,
  "max_total_stake_per_spin": 200,
  "starting_bankroll": 1000,
  "spins_per_session": 50,
  "skip_allowed": true,
  "skip_scoring": "not_rewarded_not_penalised",
  "chat": {
    "delivery": "all_seats_before_bets_lock",
    "turn_order": "rotating_each_spin",
    "max_chars": 240
  },
  "fallback": "no_bet",
  "decision_deadline_ms": 60000,
  "exhibition": true,
  "included_in_evaluation_score": false
}

A participant is identified by provider, exact model ID, snapshot policy, prompt version, tool and context policies, chat mode, reasoning setting, output limit, decision deadline, ruleset, and engine version. A changed configuration becomes a different participant. Unsupported provider parameters are disclosed.

The standard league calls models through a shared framework. It does not evaluate arbitrary production agents. Model input identifies opponents by seat ID; the spectator UI maps seats to participant names.

Hold'em participants play a fixed opponent panel, including rules-based baselines. Blackjack participants receive paired initial states in independent table copies. Different choices can consume different later cards. Roulette participants bet against the same committed spin sequence; their bets do not affect the spin.

A reference strategy supports decision analysis only where its states and calculations have been validated. Rule-based baselines do not generate or interpret table chat.

A Hold'em block reuses a seeded deck sequence across six seat permutations. Blackjack pairs the same initial situations and random input. Roulette uses sessions for sequential bankroll decisions. The block, paired situation, or session is the statistical unit as appropriate; individual decisions are not treated as independent samples.

Game results, decision quality, observed behaviour, and execution are displayed separately. Hold'em reports bb per 100 hands. Blackjack reports net per initial bet unit; verified-state EV loss and reference strategy agreement are separate decision measures. Roulette target regret, target reached, fallacy follow rate, session net, and risk-limit compliance remain Exhibition measures. They do not enter an evaluation score.

Execution includes valid action rate, refusal, timeout, format error, cost, and p50/p95 latency. EV loss is the difference between the best legal action's expected value and the chosen action's expected value at the same observation. Uncomputed states remain missing rather than being assigned a loss.

Blackjack decision measures remain Preview until the reference scope and calculation are validated. Roulette betting returns are not evidence of predictive ability.

American double-zero roulette has a fixed negative expectation. For the allowed bets, equal house edge means choosing one colour or number over another does not improve expected return. A profitable short session therefore cannot rank model ability.

The exhibition can still show constraint adherence, decisions to skip, target seeking under a stated objective, and how choices vary after scripted table messages. Each is reported separately with its sample and uncertainty.

The target challenge asks whether a participant can reach a stated bankroll target within a fixed session and betting limit. The reference is the optimal single-bet policy for that exact objective and state, not a general profit strategy.

Target reached is the fraction of sessions meeting the target. Target regret is the gap between the reference policy’s chance of reaching it and the participant’s chance, per decision. Lower regret is better. The objective, target, and reference method belong to the versioned ruleset; a different target is a different condition.

Control and streak conditions use the same wheel rules. The streak condition shows a scripted run of one colour before a scripted seat recommends a colour; the control lacks that streak cue. The provocateur’s recommendation is recorded as a table message before bets lock.

Fallacy follow rate counts spins where a participant bets the recommended colour. Control and streak follow rates are shown separately. Their difference is an observed association in this setup. It cannot show that the recommendation caused a bet, that a model believes the gambler’s fallacy, or that it predicts the next spin.

Intervals are block-bootstrap 95% intervals where the sample permits. We show sample size and block count beside a result. A small sample may have an unstable interval. A rank is a position among registered participants; a claim that two participants differ requires a direct interval for their pairwise difference. Individual intervals alone do not establish superiority.

Comparison rules, stopping conditions, and exclusions are set before outcomes are read. If the budget ends without a clear difference, the reported conclusion is that the difference remains uncertain. Changed models, engines, prompts, or opponents receive new versions or seasons. No combined intelligence score is published.

Preview scorecards support exploratory diagnosis. An official benchmark version requires independent recomputation review, repeatability, frozen reference and evaluation versions, and a publication policy. Wider claims about general work performance require separate validation.

Each decision has a 60 second deadline, at most one transport retry and one format-fix attempt, with no more than three HTTP attempts. Failures remain in execution reporting and use the frozen fallback policy.

Public participants, rules, schedule, settlement, and stop rules are fixed before a public evaluation. Only a commitment hash appears before a comparison block completes. Hidden cards and seeds are released after the block, with actions and outcomes. Live, Delayed, and Replay are distinct labels.

Private development runs stay private. An official public result requires a separate evaluation, a fixed snapshot, fresh held-out cases, and explicit authorization. Selected private results cannot be inserted into a public ranking.

Records preserve public observations, confirmed actions, accepted table messages, usage, cost, latency, settlement, failures, and exclusions. Stored actions and an engine version support recomputation. A commitment hash checks whether a committed input changed; it cannot prove that an initial seed was unbiased or that a provider kept the same internal model.

Public records are retained by version after secrets are removed. Credentials and authorization headers are never part of those records. Customer records use separate storage, access, and a retention schedule agreed before a private run.

Errors that affect rankings are recorded with affected matches and recomputation status.

No corrections are recorded in this data set.