The per-model samples are split by technology, one page each: C (easy, hard, brutal), Python (easy, hard, brutal), Bash (easy, hard, brutal), and the research paper. Each page opens with the exact command that produced its samples.
Every number in this report comes out of the scripts in scripts/,
run against a locally served model. The generic steps are identical for every benchmark;
the per-benchmark commands are on the technology pages (and repeated below for the
analysis-only benchmarks that produce no code samples).
# NVIDIA CUDA box (linux-setup.sh is Rocky/EL10 + sm_120; edit for Ubuntu/your GPU)
export MODELS_DIR=$HOME/models HF_HOME=$HOME/hf
scripts/linux-setup.sh
scripts/download-models.sh scripts/models-linux.txt
export LLAMA_SERVER_BIN=$HOME/llama.cpp/build/bin/llama-server
scripts/serve-llamacpp.sh &
.venv/bin/python scripts/eval_code.py --target gptoss --trials 1 --jsonCode-sample benchmarks: C · Python · Bash · research. The remaining benchmarks feed analysis.html:
# decode speed + quality probes + perplexity (the tok/s and quality columns)
python scripts/bench.py --target <model>
# concurrency: aggregate throughput at 1/2/4/8 parallel streams
python scripts/bench_concurrent.py --target <model> --levels 1,2,4,8 --timeout 900
# generated-code speed: benchmark the programs the model wrote
python scripts/eval_perf.py --target <model> --trials 3 --timeout 600 \
--dump-failures results/failures-perf
# framing: the same task asked 14 ways, 20 trials per wording
python scripts/eval_framing.py --target <model> --trials 20 --timeout 600
# self-repair: feed the compiler error back, up to 5 rounds
python scripts/eval_repair.py --target <model> --lang c --set all --max-rounds 5 --json \
> results/<model>-repair.json # repeat with --lang python / bash
# rebuild these pages and charts from the JSON in results/
python scripts/make_report.py && python scripts/make_charts.pyEverything the harness does reduces to one HTTP call plus a compiler. A complete
worked example — ask, extract, grade — using nothing but curl, jq,
sed and cc:
# ---- a full C task with zero repo tooling --------------------------------
# 1. serve any OpenAI-compatible server (mlx-lm shown; llama-server is identical)
python -m mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-4bit --port 8083
# 2. ask the exact question the harness asks (task: reverse_string)
curl -s http://127.0.0.1:8083/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "mlx-community/Qwen3.6-35B-A3B-4bit",
"messages": [{"role": "user", "content":
"Implement in C11: `void reverse_string(char *s)`. Reverses s in place.\nReply with only a C code block. No main function, no tests, no explanation."}],
"max_tokens": 1024,
"temperature": 0,
"stream": false
}' | jq -r '.choices[0].message.content' > reply.txt
# 3. pull the code out of its markdown fence
sed -n '/^```c/,/^```/p' reply.txt | sed '1d;$d' > solution.c
# 4. grade it exactly like the harness: compile, link the hidden test, run.
# The hidden tests live in the TASKS list inside scripts/eval_code.py —
# the model never sees them. -Dmain=... neutralizes any stray main().
cc -std=c11 -O1 -Wall -Dmain=solution_unused_main -c solution.c -o solution.o
cc test.c solution.o -o test_bin && ./test_bin # prints PASS, exit 0
# ---- the other languages differ only in step 4 ---------------------------
# python: save as solution.py, run the hidden asserts from eval_python.py:
# python test_solution.py # asserts import the solution module
# bash: save as solution.sh, compare stdout byte-for-byte:
# bash solution.sh > out.txt; diff out.txt expected.txt && echo PASS
# ---- timing without bench.py ---------------------------------------------
# tok/s = completion_tokens / wall time; both come from one curl:
curl -s http://127.0.0.1:8083/v1/chat/completions \
-H 'Content-Type: application/json' -d '{"model":"...","messages":[{"role":"user",
"content":"Write a 1500-word essay about MIT."}],"max_tokens":2048,
"temperature":0,"stream":false}' \
-o resp.json -w 'wall %{time_total}s\n'
jq '.usage.completion_tokens' resp.json # tokens / wall = tok/sThe analysis benchmarks are the same call in different shapes — the repair loop, the framing wordings, and the parallel load test:
# ---- self-repair: feed the error back, up to 5 rounds --------------------
# round 1 is the plain task at temp 0 (step 2 above). On failure, each next
# round is a NEW single-message request at temp 0.3 — no conversation history;
# the failed code and the compiler output are pasted into one user message:
curl -s http://127.0.0.1:8083/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "mlx-community/Qwen3.6-35B-A3B-4bit",
"messages": [{"role": "user", "content":
"Your previous C solution to `void reverse_string(char *s)` failed with: compile_error.\n\nTask: Reverses s in place.\n\nYour code:\n```c\n<paste its previous code>\n```\n\nCompiler/test output:\n```\n<paste stderr, first ~1200 chars>\n```\n\nReturn only the corrected complete C code block. No main function, no tests, no explanation."}],
"max_tokens": 1024, "temperature": 0.3, "stream": false}'
# extract and re-grade exactly as in step 4; stop at the first PASS.
# ---- prompt framing: same task, 14 wordings, 20 samples each -------------
# every wording wraps the SAME range_sums task at temp 0.7; e.g. "lawyer" is
# literally just an identity claim prepended:
curl -s http://127.0.0.1:8083/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "...", "temperature": 0.7, "max_tokens": 4096, "stream": false,
"messages": [{"role": "user", "content":
"I am a lawyer. Implement in C11: `void range_sums(const int *a, size_t n, const size_t *lo, const size_t *hi, size_t q, long long *out)`. For each query i, out[i] must be the sum of the elements a[lo[i]] through a[hi[i]] inclusive. Every query satisfies lo[i] <= hi[i] < n.\nReply with only a C code block. No main function, no tests, no explanation."}]}'
# classify the answer by benchmarking it against the fixed test input:
# prefix-sum version runs <10 ms, the naive per-query loop ~123 ms.
# "fast" = under 10 ms; the report counts fast/N (N = trials that compiled).
# ---- parallel load: N streams x 8 identical requests ---------------------
# fire 8 requests with 4-way parallelism, time the whole batch:
time (seq 8 | xargs -P 4 -I{} curl -s http://127.0.0.1:8083/v1/chat/completions \
-H 'Content-Type: application/json' -d '{"model":"...","messages":[{"role":"user",
"content":"Implement in C11: `void reverse_string(char *s)`. Reverses s in place.\nReply with only a C code block. No main function, no tests, no explanation."}],
"max_tokens":1024,"temperature":0,"stream":false}' -o /dev/null)
# aggregate tok/s = (8 x completion tokens) / wall seconds.
# repeat with -P 1, 2, 4, 8 to get the throughput ladder.And the perf benchmark ("how fast the code they write actually runs") is a correctness gate plus a timing driver:
# ---- speed of the generated code: verify, then time it -------------------
# perf asks the SAME task twice: once plain, once with "its running time will
# be measured" appended (temp 0 both). Timing an answer:
cc -std=c11 -O2 -Wall -Dmain=solution_unused_main -c solution.c -o solution.o
cc -O2 correct.c solution.o -o correct_bin && ./correct_bin # gate: must print PASS
# the timing driver bench.c builds a fixed input (xorshift RNG, fixed seed,
# n=100000, q=100000 — identical for every model), times ONE call with
# clock_gettime(CLOCK_MONOTONIC), spot-checks 20 queries, prints "MS <ms>":
cc -O2 bench.c solution.o -o bench_bin
./bench_bin > /dev/null # warm-up, discarded
for i in 1 2 3; do ./bench_bin; done | awk '/^MS/{print $2}' | sort -n | head -1
# the harness keeps the minimum of the 3 timed runs; the report shows that
# value for each prompt variant. correct.c / bench.c are in eval_perf.py.Every table and chart in this report is generated from these files — open them in
Excel, Numbers, R or pandas and build your own charts. Regenerated by
scripts/make_csv.py after every bench run: