Models — index & reproduction guide

The per-model samples are split by technology, one page each: C (easy, hard, brutal), Python (easy, hard, brutal), Bash (easy, hard, brutal), and the research paper. Each page opens with the exact command that produced its samples.

How to reproduce all of it

Every number in this report comes out of the scripts in scripts/, run against a locally served model. The generic steps are identical for every benchmark; the per-benchmark commands are on the technology pages (and repeated below for the analysis-only benchmarks that produce no code samples).

1 · setup & serving (same for everything)

# one-time setup (Apple Silicon; everything runs locally)
git clone <this-repo-url> m5-max-model-testing && cd m5-max-model-testing
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install mlx-lm mlx-vlm

# serve ONE model per run, on the port its target entry expects.
python -m mlx_lm.server --model <hf-repo> --port 8083
# GGUF fork targets (k2horizon, qwen38flash): llama-server -m <model.gguf> --port 8083
# then, in a second terminal: python scripts/eval_code.py --target <name> --trials 3

2 · per-benchmark commands

Code-sample benchmarks: C · Python · Bash · research. The remaining benchmarks feed analysis.html:

# decode speed + quality probes + perplexity (the tok/s and quality columns)
python scripts/bench.py --target <model>

# concurrency: aggregate throughput at 1/2/4/8 parallel streams
python scripts/bench_concurrent.py --target <model> --levels 1,2,4,8 --timeout 900

# generated-code speed: benchmark the programs the model wrote
python scripts/eval_perf.py --target <model> --trials 3 --timeout 600 \
    --dump-failures results/failures-perf

# framing: the same task asked 14 ways, 20 trials per wording
python scripts/eval_framing.py --target <model> --trials 20 --timeout 600

# self-repair: feed the compiler error back, up to 5 rounds
python scripts/eval_repair.py --target <model> --lang c --set all --max-rounds 5 --json \
    > results/<model>-repair.json        # repeat with --lang python / bash

# rebuild these pages and charts from the JSON in results/
python scripts/make_report.py && python scripts/make_charts.py

3 · low level: no repo scripts at all

Everything the harness does reduces to one HTTP call plus a compiler. A complete worked example — ask, extract, grade — using nothing but curl, jq, sed and cc:

# ---- a full C task with zero repo tooling --------------------------------
# 1. serve any OpenAI-compatible server (mlx-lm shown; llama-server is identical)
python -m mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-4bit --port 8083

# 2. ask the exact question the harness asks (task: reverse_string)
curl -s http://127.0.0.1:8083/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
  "model": "mlx-community/Qwen3.6-35B-A3B-4bit",
  "messages": [{"role": "user", "content":
    "Implement in C11: `void reverse_string(char *s)`. Reverses s in place.\nReply with only a C code block. No main function, no tests, no explanation."}],
  "max_tokens": 1024,
  "temperature": 0,
  "stream": false
}' | jq -r '.choices[0].message.content' > reply.txt

# 3. pull the code out of its markdown fence
sed -n '/^```c/,/^```/p' reply.txt | sed '1d;$d' > solution.c

# 4. grade it exactly like the harness: compile, link the hidden test, run.
#    The hidden tests live in the TASKS list inside scripts/eval_code.py —
#    the model never sees them. -Dmain=... neutralizes any stray main().
cc -std=c11 -O1 -Wall -Dmain=solution_unused_main -c solution.c -o solution.o
cc test.c solution.o -o test_bin && ./test_bin        # prints PASS, exit 0

# ---- the other languages differ only in step 4 ---------------------------
# python: save as solution.py, run the hidden asserts from eval_python.py:
#   python test_solution.py          # asserts import the solution module
# bash:   save as solution.sh, compare stdout byte-for-byte:
#   bash solution.sh > out.txt; diff out.txt expected.txt && echo PASS

# ---- timing without bench.py ---------------------------------------------
# tok/s = completion_tokens / wall time; both come from one curl:
curl -s http://127.0.0.1:8083/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{"model":"...","messages":[{"role":"user",
  "content":"Write a 1500-word essay about MIT."}],"max_tokens":2048,
  "temperature":0,"stream":false}' \
  -o resp.json -w 'wall %{time_total}s\n'
jq '.usage.completion_tokens' resp.json     # tokens / wall = tok/s

The analysis benchmarks are the same call in different shapes — the repair loop, the framing wordings, and the parallel load test:

# ---- self-repair: feed the error back, up to 5 rounds --------------------
# round 1 is the plain task at temp 0 (step 2 above). On failure, each next
# round is a NEW single-message request at temp 0.3 — no conversation history;
# the failed code and the compiler output are pasted into one user message:
curl -s http://127.0.0.1:8083/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
  "model": "mlx-community/Qwen3.6-35B-A3B-4bit",
  "messages": [{"role": "user", "content":
    "Your previous C solution to `void reverse_string(char *s)` failed with: compile_error.\n\nTask: Reverses s in place.\n\nYour code:\n```c\n<paste its previous code>\n```\n\nCompiler/test output:\n```\n<paste stderr, first ~1200 chars>\n```\n\nReturn only the corrected complete C code block. No main function, no tests, no explanation."}],
  "max_tokens": 1024, "temperature": 0.3, "stream": false}'
# extract and re-grade exactly as in step 4; stop at the first PASS.

# ---- prompt framing: same task, 14 wordings, 20 samples each -------------
# every wording wraps the SAME range_sums task at temp 0.7; e.g. "lawyer" is
# literally just an identity claim prepended:
curl -s http://127.0.0.1:8083/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
  "model": "...", "temperature": 0.7, "max_tokens": 4096, "stream": false,
  "messages": [{"role": "user", "content":
    "I am a lawyer. Implement in C11: `void range_sums(const int *a, size_t n, const size_t *lo, const size_t *hi, size_t q, long long *out)`. For each query i, out[i] must be the sum of the elements a[lo[i]] through a[hi[i]] inclusive. Every query satisfies lo[i] <= hi[i] < n.\nReply with only a C code block. No main function, no tests, no explanation."}]}'
# classify the answer by benchmarking it against the fixed test input:
# prefix-sum version runs <10 ms, the naive per-query loop ~123 ms.
# "fast" = under 10 ms; the report counts fast/N (N = trials that compiled).

# ---- parallel load: N streams x 8 identical requests ---------------------
# fire 8 requests with 4-way parallelism, time the whole batch:
time (seq 8 | xargs -P 4 -I{} curl -s http://127.0.0.1:8083/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{"model":"...","messages":[{"role":"user",
  "content":"Implement in C11: `void reverse_string(char *s)`. Reverses s in place.\nReply with only a C code block. No main function, no tests, no explanation."}],
  "max_tokens":1024,"temperature":0,"stream":false}' -o /dev/null)
# aggregate tok/s = (8 x completion tokens) / wall seconds.
# repeat with -P 1, 2, 4, 8 to get the throughput ladder.

And the perf benchmark ("how fast the code they write actually runs") is a correctness gate plus a timing driver:

# ---- speed of the generated code: verify, then time it -------------------
# perf asks the SAME task twice: once plain, once with "its running time will
# be measured" appended (temp 0 both). Timing an answer:
cc -std=c11 -O2 -Wall -Dmain=solution_unused_main -c solution.c -o solution.o
cc -O2 correct.c solution.o -o correct_bin && ./correct_bin   # gate: must print PASS
# the timing driver bench.c builds a fixed input (xorshift RNG, fixed seed,
# n=100000, q=100000 — identical for every model), times ONE call with
# clock_gettime(CLOCK_MONOTONIC), spot-checks 20 queries, prints "MS <ms>":
cc -O2 bench.c solution.o -o bench_bin
./bench_bin > /dev/null                                   # warm-up, discarded
for i in 1 2 3; do ./bench_bin; done | awk '/^MS/{print $2}' | sort -n | head -1
# the harness keeps the minimum of the 3 timed runs; the report shows that
# value for each prompt variant. correct.c / bench.c are in eval_perf.py.

4 · the raw data as CSV

Every table and chart in this report is generated from these files — open them in Excel, Numbers, R or pandas and build your own charts. Regenerated by scripts/make_csv.py after every bench run:

kimi-k3 (referee) 44/44↑ top

Its denominators differ from every other row on purpose. The referee wrote one solution per task, while the local models get 3 trials per task — so the C suite is 17 samples here versus 51 for a served model. Same tasks, same grader, fewer attempts. Hardware columns are blank because the referee is a hosted cloud model, not served on this Mac. And it authored the harness, so treat 44/44 as a sanity ceiling that proves the tasks are all solvable — not as a fair contest.

C — 20/20

atoi_strict.c
int atoi_strict(const char *s, int *out) {
    const char *p = s;
    int sign = 1, val = 0;
    if (*p == '-') {
        sign = -1;
        p++;
    }
    if (!*p) return -1;   /* empty or lone "-" */
    for (; *p; p++) {
        if (*p < '0' || *p > '9') return -1;
        val = val * 10 + (*p - '0');
    }
    *out = sign * val;
    return 0;
}
binary_search.c
int binary_search(const int *a, int n, int key) {
    int lo = 0, hi = n - 1;
    while (lo <= hi) {
        int mid = lo + (hi - lo) / 2;
        if (a[mid] == key) return mid;
        if (a[mid] < key) lo = mid + 1;
        else hi = mid - 1;
    }
    return -1;
}
bswap32.c
#include <stdint.h>

uint32_t bswap32(uint32_t x) {
    return ((x & 0x000000FFu) << 24)
         | ((x & 0x0000FF00u) << 8)
         | ((x & 0x00FF0000u) >> 8)
         | ((x & 0xFF000000u) >> 24);
}
cmp_desc.c
int cmp_desc(const void *a, const void *b) {
    int x = *(const int *)a;
    int y = *(const int *)b;
    /* Overflow-safe: no subtraction. */
    return (x < y) - (x > y);
}
count_words.c
int count_words(const char *s) {
    int count = 0, in_word = 0;
    for (; *s; s++) {
        if (*s == ' ') {
            in_word = 0;
        } else if (!in_word) {
            in_word = 1;
            count++;
        }
    }
    return count;
}
csv_field.c
#include <stddef.h>

int csv_field(const char *line, int idx, char *out, size_t cap) {
    const char *p = line;
    for (int field = 0; ; field++) {
        int quoted = (*p == '"');
        const char *s = p + quoted;
        const char *q = s;
        const char *end = s;
        int len = 0;
        for (;;) {
            if (quoted) {
                if (*q == 0) { end = q; break; }
                if (*q == '"') {
                    if (q[1] == '"') { len++; q += 2; }
                    else { end = q; q++; break; }
                } else { len++; q++; }
            } else {
                if (*q == 0 || *q == ',') { end = q; break; }
                len++; q++;
            }
        }
        if (field == idx) {
            int o = 0;
            for (const char *r = s; r < end && o < (int)cap - 1; ) {
                char ch = *r++;
                if (quoted && ch == '"' && *r == '"') r++;
                out[o++] = ch;
            }
            out[o] = 0;
            return len;
        }
        if (*q == 0) break;
        p = q + 1;
    }
    return -1;
}
fizzbuzz.c
#include <stdio.h>

const char *fizzbuzz(int n) {
    static char buf[16];
    if (n % 15 == 0) return "FizzBuzz";
    if (n % 3 == 0) return "Fizz";
    if (n % 5 == 0) return "Buzz";
    snprintf(buf, sizeof buf, "%d", n);
    return buf;
}
glob_match.c
int glob_match(const char *pat, const char *str) {
    const char *star = 0, *ss = 0;
    while (*str) {
        if (*pat == '?' || *pat == *str) { pat++; str++; }
        else if (*pat == '*') { star = pat++; ss = str; }
        else if (star) { pat = star + 1; str = ++ss; }
        else return 0;
    }
    while (*pat == '*') pat++;
    return *pat == 0;
}
is_prime.c
int is_prime(int n) {
    if (n <= 1) return 0;
    if (n <= 3) return 1;
    if (n % 2 == 0 || n % 3 == 0) return 0;
    for (int i = 5; i <= n / i; i += 6) {
        if (n % i == 0 || n % (i + 2) == 0) return 0;
    }
    return 1;
}
itoa.c
void itoa(int value, char *buf) {
    char tmp[12];
    int i = 0, j = 0;
    int neg = value < 0;
    /* Unsigned arithmetic handles INT_MIN safely. */
    unsigned int u = neg ? 0u - (unsigned int)value : (unsigned int)value;
    do {
        tmp[i++] = (char)('0' + u % 10);
        u /= 10;
    } while (u);
    if (neg) buf[j++] = '-';
    while (i > 0) buf[j++] = tmp[--i];
    buf[j] = '\0';
}
max_subarray.c
int max_subarray(const int *a, int n) {
    int best = a[0], cur = a[0];
    for (int i = 1; i < n; i++) {
        cur = cur > 0 ? cur + a[i] : a[i];
        if (cur > best) best = cur;
    }
    return best;
}
parse_csv_ints.c
int parse_csv_ints(const char *s, int *out, int max) {
    int count = 0;
    while (*s && count < max) {
        int sign = 1, val = 0;
        if (*s == '-') {
            sign = -1;
            s++;
        }
        while (*s >= '0' && *s <= '9') {
            val = val * 10 + (*s - '0');
            s++;
        }
        out[count++] = sign * val;
        if (*s == ',') s++;
    }
    return count;
}
popcount.c
unsigned popcount(unsigned x) {
    unsigned n = 0;
    while (x) {
        x &= x - 1;   /* clear lowest set bit */
        n++;
    }
    return n;
}
range_sums.c
#include <stddef.h>
#include <stdlib.h>

void range_sums(const int *a, size_t n, const size_t *lo, const size_t *hi,
                size_t q, long long *out) {
    long long *pre = malloc((n + 1) * sizeof *pre);
    pre[0] = 0;
    for (size_t i = 0; i < n; i++)
        pre[i + 1] = pre[i] + a[i];
    for (size_t i = 0; i < q; i++)
        out[i] = pre[hi[i] + 1] - pre[lo[i]];
    free(pre);
}
reverse_string.c
#include <string.h>

void reverse_string(char *s) {
    size_t n = strlen(s);
    for (size_t i = 0, j = n ? n - 1 : 0; i < j; i++, j--) {
        char tmp = s[i];
        s[i] = s[j];
        s[j] = tmp;
    }
}
ring.c
typedef struct { int data[8]; int head; int count; } Ring;

void ring_init(Ring *r) {
    r->head = 0;
    r->count = 0;
}

int ring_push(Ring *r, int v) {
    if (r->count == 8) return -1;
    r->data[(r->head + r->count) % 8] = v;
    r->count++;
    return 0;
}

int ring_pop(Ring *r, int *out) {
    if (r->count == 0) return -1;
    *out = r->data[r->head];
    r->head = (r->head + 1) % 8;
    r->count--;
    return 0;
}
rot13.c
void rot13(char *s) {
    for (; *s; s++) {
        if (*s >= 'a' && *s <= 'z') *s = (char)('a' + (*s - 'a' + 13) % 26);
        else if (*s >= 'A' && *s <= 'Z') *s = (char)('A' + (*s - 'A' + 13) % 26);
    }
}
rpn_eval.c
#include <stdlib.h>
#include <string.h>

double rpn_eval(const char *expr) {
    double st[256];
    int n = 0;
    char buf[1024];
    strncpy(buf, expr, sizeof buf - 1);
    buf[sizeof buf - 1] = 0;
    for (char *tok = strtok(buf, " "); tok; tok = strtok(0, " ")) {
        char c = tok[0];
        if (tok[1] == 0 && (c == '+' || c == '-' || c == '*' || c == '/')) {
            double b = st[--n], a = st[--n];
            st[n++] = c == '+' ? a + b : c == '-' ? a - b : c == '*' ? a * b : a / b;
        } else {
            st[n++] = strtod(tok, 0);
        }
    }
    return st[0];
}
stack.c
typedef struct { int data[16]; int top; } Stack;

void stack_init(Stack *s) {
    s->top = 0;
}

int stack_push(Stack *s, int v) {
    if (s->top == 16) return -1;
    s->data[s->top++] = v;
    return 0;
}

int stack_pop(Stack *s, int *out) {
    if (s->top == 0) return -1;
    *out = s->data[--s->top];
    return 0;
}
trim.c
#include <string.h>

static int is_ws(char c) {
    return c == ' ' || c == '\t' || c == '\n';
}

void trim(char *s) {
    char *start = s;
    while (*start && is_ws(*start)) start++;
    char *end = start + strlen(start);
    while (end > start && is_ws(end[-1])) end--;
    size_t len = (size_t)(end - start);
    if (start != s) memmove(s, start, len);
    s[len] = '\0';
}

Python — 12/12

csv_column_sum.py
import csv


def csv_column_sum(path, col):
    total = 0.0
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            total += float(row[col])
    return total
dedupe.py
def dedupe(items):
    seen = set()
    out = []
    append = out.append
    for x in items:
        if x not in seen:
            seen.add(x)
            append(x)
    return out
deep_get.py
def deep_get(d, path, default=None):
    cur = d
    for key in path.split("."):
        if not isinstance(cur, dict) or key not in cur:
            return default
        cur = cur[key]
    return cur
flatten.py
def flatten(lst):
    return [x for sub in lst for x in sub]
is_valid_ipv4.py
def is_valid_ipv4(s):
    parts = s.split(".")
    if len(parts) != 4:
        return False
    for p in parts:
        if not p or not p.isdigit():
            return False
        if len(p) > 1 and p[0] == "0":
            return False
        if int(p) > 255:
            return False
    return True
json_diff.py
def json_diff(a, b):
    out = []

    def rec(x, y, path):
        if isinstance(x, dict) and isinstance(y, dict):
            for k in sorted(set(x) | set(y)):
                p = f"{path}.{k}" if path else k
                if k not in x or k not in y:
                    out.append(p)
                else:
                    rec(x[k], y[k], p)
        elif x != y:
            out.append(path)

    rec(a, b, "")
    return sorted(out)
lru_cache.py
class LRUCache:
    def __init__(self, capacity):
        self.cap = capacity
        self.d = {}
        self.order = []  # oldest first

    def _touch(self, key):
        self.order.remove(key)
        self.order.append(key)

    def get(self, key, default=None):
        if key not in self.d:
            return default
        self._touch(key)
        return self.d[key]

    def put(self, key, value):
        if key in self.d:
            self._touch(key)
        else:
            self.order.append(key)
            if len(self.order) > self.cap:
                del self.d[self.order.pop(0)]
        self.d[key] = value
merge_intervals.py
def merge_intervals(intervals):
    merged = []
    for start, end in sorted(intervals):
        if merged and start <= merged[-1][1]:
            merged[-1][1] = max(merged[-1][1], end)
        else:
            merged.append([start, end])
    return merged
parse_json_safely.py
import json


def parse_json_safely(s):
    try:
        return json.loads(s)
    except (json.JSONDecodeError, TypeError):
        return None
run_length_encode.py
def run_length_encode(s):
    out = []
    for ch in s:
        if out and out[-1][0] == ch:
            out[-1] = (ch, out[-1][1] + 1)
        else:
            out.append((ch, 1))
    return out
top_k_words.py
import re
from collections import Counter


def top_k_words(text, k):
    words = re.findall(r"[A-Za-z]+", text.lower())
    counts = Counter(words)
    best = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:k]
    return best
topological_sort.py
def topological_sort(deps):
    nodes = set(deps)
    for ds in deps.values():
        nodes.update(ds)
    remaining = {n: set(deps.get(n, [])) for n in nodes}
    order = []
    ready = [n for n in nodes if not remaining[n]]
    while ready:
        n = ready.pop()
        order.append(n)
        for m in nodes:
            if n in remaining[m]:
                remaining[m].discard(n)
                if not remaining[m]:
                    ready.append(m)
    return order if len(order) == len(nodes) else None

Bash — 12/12

backup_rotate.sh
backup_rotate() {
  ls "$1"/backup-*.tar.gz 2>/dev/null | sort -r | tail -n +"$(( $2 + 1 ))" | sort \
    | while read -r f; do
        rm -f "$f" && basename "$f"
      done
}
count_matches.sh
count_matches() {
    grep -c -F "$1" "$2" || true
}
csv_col.sh
csv_col() {
    awk -F, -v col="$1" 'NR==1 { for (i=1; i<=NF; i++) if ($i==col) c=i; next } { print $c }'
}
extract_urls.sh
extract_urls() {
    grep -oE 'https?://[^[:space:]",)]+' | sed 's/[.,;:!?]$//'
}
find_dupes.sh
find_dupes() {
  find "$1" -type f -exec md5 -r {} + | sort | awk '
    { h=$1; $1=""; sub(/^ /, ""); files[h] = files[h] " " $0; cnt[h]++ }
    END { for (h in files) if (cnt[h] > 1) print substr(files[h], 2) }
  ' | sort
}
json_get.sh
json_get() {
    python3 -c "import json,sys; print(json.load(sys.stdin)[sys.argv[1]])" "$1"
}
largest_file.sh
largest_file() {
    find "$1" -type f -exec stat -f '%z %N' {} + | sort -rn | head -1 | cut -d' ' -f2-
}
retry3.sh
retry3() {
    local i
    for i in 1 2 3; do
        "$@" && return 0
    done
    return 1
}
rotate.sh
rotate() {
    [ -e "$1.1" ] && mv "$1.1" "$1.2"
    mv "$1" "$1.1"
}
sum_stdin.sh
sum_stdin() {
    awk 'NF { s += $1 } END { print s + 0 }'
}
top_errors.sh
top_errors() {
  grep 'ERROR: ' "$1" | sed -e 's/.*ERROR: //' -e 's/[0-9][0-9]*/#/g' \
    | sort | uniq -c | sort -k1,1nr -k2 | sed 's/^ *//' | head -n "$2"
}
top_freq.sh
top_freq() {
    sort "$1" | uniq -c | sort -k1,1nr -k2,2 | head -10 | awk '{print $1, $2}'
}