Harness Bench

Install

Each harness is installed cold with lnk agent new: before each install the agent and every package cache in the run's home are removed. Link's own programs are there already, for every harness. Runs: 3 (--quick: 1).

• Install time (install_time, s): from lnk agent new until "Running ... here", its last line before the harness starts: the adapter's install and configure. Timed by the bench from that command's own lines; results.json keeps install and configure apart. Noise: the network.
• Download (downloaded, MB): bytes in and out of the machine's network during the install, from lnk measure's machine. Bias: anything else on the network counts; run it on a quiet machine.
• On disk (on_disk, MB and files): the agent's folders after the install, before any message, from lnk measure. For Link harness, its one program, lnk-link-harness.
• Packages (packages): distinct npm and Python packages, each package.json under node_modules and each .dist-info. The two aren't added up: they don't weigh the same.
• Run code at install (install_scripts): npm packages with a preinstall, install or postinstall script.
• Brings (runtimes): the runtimes in its home, with their versions.
• Needs on the machine (needs_machine): not measured by the script. Install each harness in a bare debian:stable-slim with ca-certificates and bubblewrap, and list what it failed without.
• Biggest parts (biggest_parts): its five largest packages or folders, a note for the reader.
• Its download is checked (install_pinned): read from its install's lockfiles: "built by Link", "lockfile hashes", "lockfile without hashes", or "package hash; dependencies float".

Running

The stand-in answers at once.

• First start (first_start, s): after the install, until the harness answers: its endpoint proves itself, or Hermes's /health or OpenClaw's gateway answers, looked at every 0.1 s. Includes any first-run setup. One for each install.
• Restart (restart, s): stopped, then started, until it answers. Runs: 5 (quick: 1).
• Stop (stop, s): from Ctrl-C until none of its processes remain. Runs: one for each restart.
• Cold start (cold_start, s): a restart after the file cache is dropped (/proc/sys/vm/drop_caches). Linux, as root, on a machine with its own kernel; else its reason. Runs: 3.
• First answer (first_answer, s): from a start until the first message is answered whole. Runs: 1.
• Idle memory (idle_memory, MB): lnk measure's memory in use (PSS: pages shared between processes split between them), its median over 10 minutes looked at every 10 s, from 2 minutes after its first answer (quick: 20 s every 4 s, after 5 s). Spread: the looks.
• Processes, ports (processes_ports): lnk measure's while idle.
• Idle CPU (idle_cpu, core-seconds per hour): lnk measure's CPU over the idle minutes, per hour. One interval: no spread.
• Idle traffic (idle_traffic, KB per hour): lnk measure's network through its proxy over the idle minutes, and the hosts it reached, from lnk sandbox log.
• Added per message (added_per_message, ms, median / p95): the answer's time, whole, less the stand-in's time answering it, over 30 messages after 3 to warm up (quick: 5 after 1). Bias: the bench's own Python on the same machine, the same for every harness.
• To model (to_model, ms): from the message sent until the stand-in is asked.
• First token through (first_token_through, ms): from the stand-in's first piece of the answer until it reaches the bench. The stand-in streams three pieces, 200 ms apart. None for a harness that answers only whole the way it's reached.
• CPU per message (cpu_per_message, core-ms): lnk measure's CPU over the messages, divided by their number.
• Peak memory per message (peak_memory, MB): the most memory lnk measure saw, a look a second during the messages, less idle.
• Disk per 100 messages (disk_per_100, KB): growth of the agent's folders over the messages, from lnk measure, per 100.
• A provider error (provider_error): the stand-in answers 500 once, then hangs once (120 s; quick: 10 s). Whether the answer still comes, and when the user hears anything.

Each message, as the model sees it

What the stand-in recorded, once as Anthropic and once as OpenAI: a harness may send each a different prompt. One thread each, in a fresh files folder.

• Prompt overhead (prompt_overhead, KB and tokens): the first request for "Reply with exactly: ok", in a new thread: everything but that sentence (its system prompt, its tools, any other messages). Tokens are about four bytes each, or with --count-tokens Anthropic's token count (free; needs ANTHROPIC_API_KEY). The real group's "Tokens for ok" is the provider's own count.
• Tools offered (tools_offered): the tools in that request.
• Extra model calls (extra_calls): requests in that turn beyond the one that answers it.
• Growth per turn (growth_per_turn, tokens): what each turn adds to the prompt beyond its own words, by a straight line through 20 short turns (quick: 5).
• Compacts at (compacts_at): the turn at which the prompt first shrinks, over 200 turns of 2 KB (quick: 8).
• Prompt cache reuse (cache_friendly): "Yes" when everything before each turn's newest message is sent byte for byte as before, so a provider's cache can reuse it; else the byte where it first changed.

Link's proxy trusts only the machine's certificates, and starts with no environment that could add one. So the bench runs itself again in a mount namespace of its own, where the machine's CA file holds the run's CA too: only the run's programs trust it, and the machine's own trust is never changed. That takes root (unshare). Without it, --model-via local serves the stand-in as a model on the machine instead. That measures a local model's requests, which a harness may build differently from a hosted provider's, so the results say so.

With a real model

export ANTHROPIC_API_KEY=...
link-core/bench/harnesses/run.sh --groups real --spend --max-spend 20

The only group that spends. Every harness gets the same model, Claude Haiku 5.5 (--anthropic names another), and the same key, straight through Link's proxy to the provider, as a user's agent reaches it. It refuses to start without --spend and --max-spend, or for a model with no price in prices.json, which names where each price was read and when. It prints the price and an estimate from each harness's prompt size, then asks; --yes answers for a script. It stops at the cap. Each task runs 3 times (quick: 1), interleaved across the harnesses, each in a new thread with a fresh files folder.

A harness whose tools can't do a task (by their names, as it offers them to the stand-in) shows "no tool" for it.

• Tasks passed (tasks_passed, of 7): tasks passed in every run. Each task's runs are in results.md.
• Tokens for "ok" (tokens_ok): T1's input tokens.
• Tokens per suite (tokens_suite): input (and the cached part) and output over T1 to T7, the median run.
• Spend for the 7 tasks (spend_suite, $): those tokens times the pinned price (prices.json, with its source and date). Input is the whole prompt, as lnk measure counts it, so the cache's reads and writes come out of it and are priced at their own rates: cache reads (link.tokens.cached.sum) at the cache-read price, cache writes (link.tokens.cache_written.sum) at the cache-write price, and the rest at the input price.
• Model calls per task (calls_per_task): its proxy's calls to the model (lnk sandbox log).
• First token, whole answer (first_token, whole_answer, s): from the message sent until the answer's first piece, and all of it, on T7. Mostly the provider's time.

Tokens are lnk measure's, read before and after each task while lnk measure watch looks every second. A harness Link has no tokens for shows none, and says why.

What it asks for, and reaches

• Asks (asks_linux, asks_mac): what its adapter's needs names beyond Link's defaults, for each provider (Anthropic, OpenAI, OpenRouter, local) and each channel (none, Telegram, Slack, Discord, WhatsApp, Signal). An adapter answers for the OS it runs on, so the Mac's row comes from a run on a Mac.
• Listening ports (ports): lnk measure's, while idle and the most during a turn.
• Hosts it reaches (hosts): the hosts its proxy let it reach while it ran, besides its model's, from lnk sandbox log.
• check (check): lnk agent <harness> check: "passes", or what failed.