Python — per-model detail

Every failing sample below is the real generated Python, with the assertion or error that rejected it. Trial 0 runs at temperature 0, trials 1 and 2 at 0.7 — so a task passing 1/3 is a sampling-luck pass, not a reliable one. Models are ordered best first. Other technologies: C · Python · Bash · research · how to reproduce.

Reproduce this benchmark

Setup and model serving are on the models index — including a script-free version using nothing but curl and a compiler; with a server running, this is the exact command that produced the samples below:

# Python suite (8 easy + 3 hard tasks, 3 trials each)
python scripts/eval_python.py --target <model> --set all --trials 3 \
    --dump-failures results/failures
python scripts/eval_python.py --target <model> --set brutal --trials 3 \
    --dump-failures results/failures --json > results/<model>-brutal-python.json
# each answer runs against hidden asserts in a subprocess with a timeout.

qwen3.8-flash-next 125B 86% overall↑ top

Served via llama.cpp Metal · 33.0 tok/s · 39.0 GB RAM · coding total 108/126

Python — 24/24 (1.2 min / 2.1k tok)

No failures.

Python (hard) — 9/9 (1.4 min / 2.7k tok)

No failures.

gpt-oss-120b 84% overall↑ top

Served via llama.cpp Metal · 61.6 tok/s · 51.7 GB RAM · coding total 106/126

Python — 24/24 (1.8 min / 6.4k tok)

No failures.

Python (hard) — 9/9 (1.5 min / 5.5k tok)

No failures.

gpt-oss-20b 83% overall↑ top

Served via llama.cpp Metal · 83.1 tok/s · 11.8 GB RAM · coding total 104/126

Python — 24/24 (1.2 min / 6.6k tok)

No failures.

Python (hard) — 9/9 (1.2 min / 6.8k tok)

No failures.

gemma-4-26b 81% overall↑ top

Served via llama.cpp Metal · 60.7 tok/s · 16.6 GB RAM · coding total 102/126

Python — 24/24 (0.7 min / 2.6k tok)

No failures.

Python (hard) — 9/9 (0.5 min / 2.0k tok)

No failures.

qwen3.8-27b 81% overall↑ top

Served via llama.cpp Metal · 14.4 tok/s · 15.6 GB RAM · coding total 102/126

Python — 24/24 (2.0 min / 2.0k tok)

No failures.

Python (hard) — 9/9 (3.4 min / 3.0k tok)

No failures.

nemotron-3-super 120B 81% overall↑ top

Served via llama.cpp Metal · 20.4 tok/s · 70.5 GB RAM · coding total 102/126

Python — 24/24 (1.4 min / 1.8k tok)

No failures.

Python (hard) — 9/9 (1.4 min / 1.7k tok)

No failures.

laguna-s 2.1 117B 80% overall↑ top

Served via llama.cpp Metal · 39.7 tok/s · 61.6 GB RAM · coding total 101/126

Python — 24/24 (0.8 min / 1.8k tok)

No failures.

Python (hard) — 9/9 (0.7 min / 1.7k tok)

No failures.

laguna-xs.2 79% overall↑ top

Served via llama.cpp Metal · 81.1 tok/s · 16.9 GB RAM · coding total 100/126

Python — 24/24 (0.3 min / 2.1k tok)

No failures.

Python (hard) — 9/9 (0.3 min / 1.7k tok)

No failures.

qwen3-coder-next 80B 79% overall↑ top

Served via llama.cpp Metal · 54.6 tok/s · 45.6 GB RAM · coding total 99/126

Python — 21/24 (0.5 min / 1.9k tok)

csv_column_sum — 0/3 passed
prompt sent to the model
Implement in Python 3: `def csv_column_sum(path, col):` Reads the CSV file at path (first row is a header). Returns the sum of the numeric values in column col as a float.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — TypeError: '<' not supported between instances of 'str' and 'int'
import csv

def csv_column_sum(path, col):
    total = 0.0
    with open(path, 'r', newline='') as csvfile:
        reader = csv.reader(csvfile)
        next(reader)  # Skip header row
        for row in reader:
            if col < len(row):
                try:
                    total += float(row[col])
                except ValueError:
                    pass  # Skip non-numeric values
    return total
trial 1 (temp 0.7) — fail — TypeError: '<' not supported between instances of 'str' and 'int'
import csv

def csv_column_sum(path, col):
    total = 0.0
    with open(path, 'r', newline='') as csvfile:
        reader = csv.reader(csvfile)
        next(reader)  # Skip header row
        for row in reader:
            if col < len(row):
                try:
                    total += float(row[col])
                except ValueError:
                    pass  # Skip non-numeric values
    return total
trial 2 (temp 0.7) — fail — TypeError: '<' not supported between instances of 'str' and 'int'
import csv

def csv_column_sum(path, col):
    total = 0.0
    with open(path, 'r', newline='') as csvfile:
        reader = csv.reader(csvfile)
        next(reader)  # Skip header row
        for row in reader:
            if col < len(row):
                try:
                    total += float(row[col])
                except ValueError:
                    pass  # Skip non-numeric values
    return total

Python (hard) — 6/9 (0.5 min / 2.0k tok)

json_diff — 0/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError
def json_diff(a, b):
    def _diff(path, x, y):
        if type(x) != type(y):
            return [path] if path else []
        
        if isinstance(x, dict):
            result = []
            all_keys = set(x.keys()) | set(y.keys())
            for key in all_keys:
                new_path = f"{path}.{key}" if path else key
                if key not in x:
                    result.append(new_path)
                elif key not in y:
                    result.append(new_path)
                else:
                    result.extend(_diff(new_path, x[key], y[key]))
            return result
        elif isinstance(x, list):
            if len(x) != len(y):
                return [path] if path else []
            result = []
            for i, (item_x, item_y) in enumerate(zip(x, y)):
                new_path = f"{path}[{i}]" if path else f"[{i}]"
                result.extend(_diff(new_path, item_x, item_y))
            return result
        else:
            return [path] if path and x != y else []
    
    return sorted(_diff("", a, b))
trial 1 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(path, x, y):
        if type(x) != type(y):
            return [path] if path else []
        
        if isinstance(x, dict):
            result = []
            all_keys = set(x.keys()) | set(y.keys())
            for key in all_keys:
                new_path = f"{path}.{key}" if path else key
                if key not in x:
                    result.append(new_path)
                elif key not in y:
                    result.append(new_path)
                else:
                    result.extend(_diff(new_path, x[key], y[key]))
            return result
        elif isinstance(x, list):
            if len(x) != len(y):
                return [path] if path else []
            result = []
            for i, (item_x, item_y) in enumerate(zip(x, y)):
                new_path = f"{path}[{i}]" if path else f"[{i}]"
                result.extend(_diff(new_path, item_x, item_y))
            return result
        else:
            return [path] if path and x != y else []
    
    return sorted(_diff("", a, b))
trial 2 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(path, x, y):
        if type(x) != type(y):
            return [path] if path else []
        
        if isinstance(x, dict):
            result = []
            all_keys = set(x.keys()) | set(y.keys())
            for key in all_keys:
                new_path = f"{path}.{key}" if path else key
                if key not in x:
                    result.append(new_path)
                elif key not in y:
                    result.append(new_path)
                else:
                    result.extend(_diff(new_path, x[key], y[key]))
            return result
        elif isinstance(x, list):
            if len(x) != len(y):
                return [path] if path else []
            result = []
            for i, (item_x, item_y) in enumerate(zip(x, y)):
                new_path = f"{path}[{i}]" if path else f"[{i}]"
                result.extend(_diff(new_path, item_x, item_y))
            return result
        else:
            return [path] if path and x != y else []
    
    return sorted(_diff("", a, b))

qwen3.6-27b 79% overall↑ top

Served via llama.cpp Metal · 14.8 tok/s · 15.9 GB RAM · coding total 99/126

Python — 24/24 (2.5 min / 2.3k tok)

No failures.

Python (hard) — 9/9 (2.3 min / 2.1k tok)

No failures.

qwen3.6-35b 76% overall↑ top

Served via llama.cpp Metal · 63.1 tok/s · 17.9 GB RAM · coding total 96/126

Python — 21/24 (0.5 min / 2.0k tok)

csv_column_sum — 0/3 passed
prompt sent to the model
Implement in Python 3: `def csv_column_sum(path, col):` Reads the CSV file at path (first row is a header). Returns the sum of the numeric values in column col as a float.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — TypeError: '<' not supported between instances of 'str' and 'int'
def csv_column_sum(path, col):
    import csv
    total = 0.0
    with open(path, 'r', newline='') as f:
        reader = csv.reader(f)
        header = next(reader)
        if col < 0 or col >= len(header):
            raise IndexError("Column index out of range")
        for row in reader:
            if col < len(row):
                try:
                    total += float(row[col])
                except ValueError:
                    pass
    return total
trial 1 (temp 0.7) — fail — TypeError: '<' not supported between instances of 'str' and 'int'
def csv_column_sum(path, col):
    import csv
    total = 0.0
    with open(path, 'r', newline='') as f:
        reader = csv.reader(f)
        header = next(reader)
        if col < 0 or col >= len(header):
            raise IndexError("Column index out of range")
        for row in reader:
            if col < len(row):
                try:
                    total += float(row[col])
                except ValueError:
                    pass
    return total
trial 2 (temp 0.7) — fail — TypeError: '<' not supported between instances of 'str' and 'int'
def csv_column_sum(path, col):
    import csv
    total = 0.0
    with open(path, 'r', newline='') as f:
        reader = csv.reader(f)
        header = next(reader)
        if col < 0 or col >= len(header):
            raise IndexError("Column index out of range")
        for row in reader:
            if col < len(row):
                try:
                    total += float(row[col])
                except ValueError:
                    pass
    return total

Python (hard) — 9/9 (0.6 min / 2.4k tok)

No failures.

qwen3.5-35b 76% overall↑ top

Served via llama.cpp Metal · 58.1 tok/s · 19.1 GB RAM · coding total 96/126

Python — 24/24 (0.5 min / 2.0k tok)

No failures.

Python (hard) — 9/9 (0.5 min / 1.8k tok)

No failures.

devstral-2 24b 75% overall↑ top

Served via llama.cpp Metal · 23.5 tok/s · 16.3 GB RAM · coding total 94/126

Python — 24/24 (1.6 min / 1.8k tok)

No failures.

Python (hard) — 6/9 (1.6 min / 1.8k tok)

json_diff — 0/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError
def json_diff(a, b):
    def _diff(a, b, path):
        diffs = []
        if isinstance(a, dict) and isinstance(b, dict):
            all_keys = sorted(set(a.keys()) | set(b.keys()))
            for key in all_keys:
                new_path = f"{path}.{key}" if path else key
                if key not in a:
                    diffs.append(new_path)
                elif key not in b:
                    diffs.append(new_path)
                else:
                    diffs.extend(_diff(a[key], b[key], new_path))
        elif isinstance(a, list) and isinstance(b, list):
            if len(a) != len(b):
                diffs.append(path)
            else:
                for i, (x, y) in enumerate(zip(a, b)):
                    new_path = f"{path}[{i}]"
                    diffs.extend(_diff(x, y, new_path))
        else:
            if a != b:
                diffs.append(path)
        return diffs

    return sorted(_diff(a, b, ""))
trial 1 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(a, b, path):
        diffs = []
        if isinstance(a, dict) and isinstance(b, dict):
            all_keys = sorted(set(a.keys()) | set(b.keys()))
            for key in all_keys:
                new_path = f"{path}.{key}" if path else key
                if key not in a:
                    diffs.append(new_path)
                elif key not in b:
                    diffs.append(new_path)
                else:
                    diffs.extend(_diff(a[key], b[key], new_path))
        elif isinstance(a, list) and isinstance(b, list):
            if len(a) != len(b):
                diffs.append(path)
            else:
                for i, (x, y) in enumerate(zip(a, b)):
                    new_path = f"{path}[{i}]"
                    diffs.extend(_diff(x, y, new_path))
        else:
            if a != b:
                diffs.append(path)
        return diffs

    return sorted(_diff(a, b, ""))
trial 2 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(a, b, path):
        diffs = []
        if isinstance(a, dict) and isinstance(b, dict):
            all_keys = sorted(set(a.keys()) | set(b.keys()))
            for key in all_keys:
                new_path = f"{path}.{key}" if path else key
                if key not in a:
                    diffs.append(new_path)
                elif key not in b:
                    diffs.append(new_path)
                else:
                    diffs.extend(_diff(a[key], b[key], new_path))
        elif isinstance(a, list) and isinstance(b, list):
            if len(a) != len(b):
                diffs.append(path)
            else:
                for i, (x, y) in enumerate(zip(a, b)):
                    new_path = f"{path}[{i}]"
                    diffs.extend(_diff(x, y, new_path))
        else:
            if a != b:
                diffs.append(path)
        return diffs

    return sorted(_diff(a, b, ""))

glm-4.7-flash 75% overall↑ top

Served via llama.cpp Metal · 48.4 tok/s · 17.9 GB RAM · coding total 94/126

Python — 24/24 (0.5 min / 2.1k tok)

No failures.

Python (hard) — 7/9 (0.7 min / 2.5k tok)

json_diff — 1/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 1 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(current_a, current_b, path):
        changes = []
        
        # Check if both are dicts
        if isinstance(current_a, dict) and isinstance(current_b, dict):
            # Keys present in both
            all_keys = set(current_a.keys()) | set(current_b.keys())
            for key in sorted(all_keys):
                new_path = f"{path}.{key}" if path else key
                if key not in current_a:
                    changes.append(new_path)
                elif key not in current_b:
                    changes.append(new_path)
                else:
                    changes.extend(_diff(current_a[key], current_b[key], new_path))
        
        # Check if both are lists
        elif isinstance(current_a, list) and isinstance(current_b, list):
            # Compare lists by index
            length = max(len(current_a), len(current_b))
            for i in range(length):
                new_path = f"{path}[{i}]"
                if i >= len(current_a):
                    changes.append(new_path)
                elif i >= len(current_b):
                    changes.append(new_path)
                else:
                    changes.extend(_diff(current_a[i], current_b[i], new_path))
        
        # Scalars or mismatched types
        else:
            if current_a != current_b:
                changes.append(path)
                
        return changes

    return _diff(a, b, "")
trial 2 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(current_a, current_b, path):
        changes = []
        
        # Check if both are dicts
        if isinstance(current_a, dict) and isinstance(current_b, dict):
            # Keys present in both
            all_keys = set(current_a.keys()) | set(current_b.keys())
            for key in sorted(all_keys):
                new_path = f"{path}.{key}" if path else key
                if key not in current_a:
                    changes.append(new_path)
                elif key not in current_b:
                    changes.append(new_path)
                else:
                    changes.extend(_diff(current_a[key], current_b[key], new_path))
        
        # Check if both are lists
        elif isinstance(current_a, list) and isinstance(current_b, list):
            # Compare lists by index
            length = max(len(current_a), len(current_b))
            for i in range(length):
                new_path = f"{path}[{i}]"
                if i >= len(current_a):
                    changes.append(new_path)
                elif i >= len(current_b):
                    changes.append(new_path)
                else:
                    changes.extend(_diff(current_a[i], current_b[i], new_path))
        
        # Scalars or mismatched types
        else:
            if current_a != current_b:
                changes.append(path)
                
        return changes

    return _diff(a, b, "")

mistral-small-4 119B 72% overall↑ top

Served via llama.cpp Metal · 52.2 tok/s · 59.7 GB RAM · coding total 91/126

Python — 24/24 (0.6 min / 1.9k tok)

No failures.

Python (hard) — 3/9 (0.5 min / 1.4k tok)

topological_sort — 0/3 passed
prompt sent to the model
Implement in Python 3: `def topological_sort(deps):` deps maps each node (str) to a list of nodes it depends on. Returns a list of all nodes in an order where every node comes after its dependencies. Returns None if the graph has a cycle (including self-dependencies). Nodes mentioned only as dependencies are included.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError: ['app', 'lib', 'base']
def topological_sort(deps):
    visited = set()
    visiting = set()
    sorted_nodes = []

    def visit(node):
        if node in visiting:
            return False
        if node in visited:
            return True
        visiting.add(node)
        for dependency in deps.get(node, []):
            if not visit(dependency):
                return False
        visiting.remove(node)
        visited.add(node)
        sorted_nodes.append(node)
        return True

    for node in list(deps.keys()) + [d for deps_list in deps.values() for d in deps_list]:
        if node not in visited:
            if not visit(node):
                return None

    return sorted_nodes[::-1]
trial 1 (temp 0.7) — fail — AssertionError: ['app', 'lib', 'base']
def topological_sort(deps):
    visited = set()
    visiting = set()
    sorted_nodes = []

    def visit(node):
        if node in visiting:
            return False
        if node in visited:
            return True
        visiting.add(node)
        for dependency in deps.get(node, []):
            if not visit(dependency):
                return False
        visiting.remove(node)
        visited.add(node)
        sorted_nodes.append(node)
        return True

    for node in list(deps.keys()) + [d for deps_list in deps.values() for d in deps_list]:
        if node not in visited:
            if not visit(node):
                return None

    return sorted_nodes[::-1]
trial 2 (temp 0.7) — fail — AssertionError: ['app', 'lib', 'base']
def topological_sort(deps):
    visited = set()
    visiting = set()
    sorted_nodes = []

    def visit(node):
        if node in visiting:
            return False
        if node in visited:
            return True
        visiting.add(node)
        for dependency in deps.get(node, []):
            if not visit(dependency):
                return False
        visiting.remove(node)
        visited.add(node)
        sorted_nodes.append(node)
        return True

    for node in list(deps.keys()) + [d for deps_list in deps.values() for d in deps_list]:
        if node not in visited:
            if not visit(node):
                return None

    return sorted_nodes[::-1]
json_diff — 0/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError
def json_diff(a, b, path=""):
    diffs = []

    if type(a) != type(b):
        diffs.append(path)
        return diffs

    if isinstance(a, dict):
        all_keys = set(a.keys()) | set(b.keys())
        for key in sorted(all_keys):
            new_path = f"{path}.{key}" if path else key
            if key not in a:
                diffs.append(f"{new_path} (missing in a)")
            elif key not in b:
                diffs.append(f"{new_path} (missing in b)")
            else:
                diffs.extend(json_diff(a[key], b[key], new_path))
    elif isinstance(a, list):
        if a != b:
            diffs.append(path)
    else:
        if a != b:
            diffs.append(path)

    return diffs
trial 1 (temp 0.7) — fail — AssertionError
def json_diff(a, b, path=""):
    diffs = []

    if type(a) != type(b):
        diffs.append(path)
        return diffs

    if isinstance(a, dict):
        all_keys = set(a.keys()) | set(b.keys())
        for key in sorted(all_keys):
            new_path = f"{path}.{key}" if path else key
            if key not in a:
                diffs.append(f"{new_path} (missing in a)")
            elif key not in b:
                diffs.append(f"{new_path} (missing in b)")
            else:
                diffs.extend(json_diff(a[key], b[key], new_path))
    elif isinstance(a, list):
        if a != b:
            diffs.append(path)
    else:
        if a != b:
            diffs.append(path)

    return diffs
trial 2 (temp 0.7) — fail — AssertionError
def json_diff(a, b, path=""):
    diffs = []

    if type(a) != type(b):
        diffs.append(path)
        return diffs

    if isinstance(a, dict):
        all_keys = set(a.keys()) | set(b.keys())
        for key in sorted(all_keys):
            new_path = f"{path}.{key}" if path else key
            if key not in a:
                diffs.append(f"{new_path} (missing in a)")
            elif key not in b:
                diffs.append(f"{new_path} (missing in b)")
            else:
                diffs.extend(json_diff(a[key], b[key], new_path))
    elif isinstance(a, list):
        if a != b:
            diffs.append(path)
    else:
        if a != b:
            diffs.append(path)

    return diffs

qwen3.5-122b-a10b 71% overall↑ top

Served via llama.cpp Metal · 27.2 tok/s · 68.8 GB RAM · coding total 90/126

Python — 24/24 (1.2 min / 2.1k tok)

No failures.

Python (hard) — 9/9 (1.1 min / 2.0k tok)

No failures.

qwen3.5-27b 69% overall↑ top

Served via llama.cpp Metal · 13.9 tok/s · 18.7 GB RAM · coding total 87/126

Python — 24/24 (2.3 min / 2.1k tok)

No failures.

Python (hard) — 9/9 (3.2 min / 2.9k tok)

No failures.

qwen3-coder-30b 68% overall↑ top

Served via llama.cpp Metal · 80.5 tok/s · 18.4 GB RAM · coding total 86/126

Python — 24/24 (0.4 min / 2.0k tok)

No failures.

Python (hard) — 6/9 (0.4 min / 2.2k tok)

json_diff — 0/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError
def json_diff(a, b):
    def _diff(obj_a, obj_b, path=""):
        diffs = []
        if type(obj_a) != type(obj_b):
            diffs.append(path)
        elif isinstance(obj_a, dict):
            all_keys = set(obj_a.keys()) | set(obj_b.keys())
            for key in sorted(all_keys):
                new_path = f"{path}.{key}" if path else key
                if key not in obj_a:
                    diffs.append(new_path)
                elif key not in obj_b:
                    diffs.append(new_path)
                else:
                    diffs.extend(_diff(obj_a[key], obj_b[key], new_path))
        elif isinstance(obj_a, list):
            if len(obj_a) != len(obj_b):
                diffs.append(path)
            else:
                for i, (item_a, item_b) in enumerate(zip(obj_a, obj_b)):
                    new_path = f"{path}[{i}]"
                    diffs.extend(_diff(item_a, item_b, new_path))
        else:
            if obj_a != obj_b:
                diffs.append(path)
        return diffs
    
    return sorted(_diff(a, b))
trial 1 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(obj_a, obj_b, path=""):
        diffs = []
        if type(obj_a) != type(obj_b):
            diffs.append(path)
        elif isinstance(obj_a, dict):
            all_keys = set(obj_a.keys()) | set(obj_b.keys())
            for key in sorted(all_keys):
                new_path = f"{path}.{key}" if path else key
                if key not in obj_a:
                    diffs.append(new_path)
                elif key not in obj_b:
                    diffs.append(new_path)
                else:
                    diffs.extend(_diff(obj_a[key], obj_b[key], new_path))
        elif isinstance(obj_a, list):
            if len(obj_a) != len(obj_b):
                diffs.append(path)
            else:
                for i, (item_a, item_b) in enumerate(zip(obj_a, obj_b)):
                    new_path = f"{path}[{i}]"
                    diffs.extend(_diff(item_a, item_b, new_path))
        else:
            if obj_a != obj_b:
                diffs.append(path)
        return diffs
    
    return sorted(_diff(a, b))
trial 2 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def _diff(obj_a, obj_b, path=""):
        diffs = []
        if type(obj_a) != type(obj_b):
            diffs.append(path)
        elif isinstance(obj_a, dict):
            all_keys = set(obj_a.keys()) | set(obj_b.keys())
            for key in sorted(all_keys):
                new_path = f"{path}.{key}" if path else key
                if key not in obj_a:
                    diffs.append(new_path)
                elif key not in obj_b:
                    diffs.append(new_path)
                else:
                    diffs.extend(_diff(obj_a[key], obj_b[key], new_path))
        elif isinstance(obj_a, list):
            if len(obj_a) != len(obj_b):
                diffs.append(path)
            else:
                for i, (item_a, item_b) in enumerate(zip(obj_a, obj_b)):
                    new_path = f"{path}[{i}]"
                    diffs.extend(_diff(item_a, item_b, new_path))
        else:
            if obj_a != obj_b:
                diffs.append(path)
        return diffs
    
    return sorted(_diff(a, b))

k2-horizon 36B-A4B 68% overall↑ top

Served via llama.cpp Metal · 47.1 tok/s · 24.0 GB RAM · coding total 86/126

Python — 24/24 (0.5 min / 1.6k tok)

No failures.

Python (hard) — 6/9 (0.5 min / 1.5k tok)

json_diff — 0/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError
def json_diff(a, b):
    diffs = []

    def compare(x, y, path):
        if type(x) is not type(y):
            diffs.append(path)
            return
        if isinstance(x, dict):
            keys = set(x) | set(y)
            for k in sorted(keys):
                if k not in x or k not in y:
                    diffs.append(path + "." + k)
                else:
                    compare(x[k], y[k], path + "." + k)
        elif isinstance(x, list):
            if x != y:
                diffs.append(path)
        else:
            if x != y:
                diffs.append(path)

    compare(a, b, "")
    return sorted(diffs)
trial 1 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    diffs = []

    def compare(x, y, path):
        if type(x) is not type(y):
            diffs.append(path)
            return
        if isinstance(x, dict):
            keys = set(x) | set(y)
            for k in sorted(keys):
                if k not in x or k not in y:
                    diffs.append(path + "." + k)
                else:
                    compare(x[k], y[k], path + "." + k)
        elif isinstance(x, list):
            if x != y:
                diffs.append(path)
        else:
            if x != y:
                diffs.append(path)

    compare(a, b, "")
    return sorted(diffs)
trial 2 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    diffs = []

    def compare(x, y, path):
        if type(x) is not type(y):
            diffs.append(path)
            return
        if isinstance(x, dict):
            keys = set(x) | set(y)
            for k in sorted(keys):
                if k not in x or k not in y:
                    diffs.append(path + "." + k)
                else:
                    compare(x[k], y[k], path + "." + k)
        elif isinstance(x, list):
            if x != y:
                diffs.append(path)
        else:
            if x != y:
                diffs.append(path)

    compare(a, b, "")
    return sorted(diffs)

seed-oss 36b 63% overall↑ top

Served via llama.cpp Metal · 11.2 tok/s · 24.5 GB RAM · coding total 79/126

Python — 24/24 (42.8 min / 29.0k tok)

No failures.

Python (hard) — 3/9 (46.4 min / 32.8k tok)

topological_sort — 0/3 passed
prompt sent to the model
Implement in Python 3: `def topological_sort(deps):` deps maps each node (str) to a list of nodes it depends on. Returns a list of all nodes in an order where every node comes after its dependencies. Returns None if the graph has a cycle (including self-dependencies). Nodes mentioned only as dependencies are included.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — truncated — hit the 4096-token cap
def topological_sort(deps):
    nodes = set(deps.keys()).union(*deps.values())
    in_degree = {node: 0 for node in nodes}
    for deps_list in deps.values():
        for dep in deps_list:
            in_degree[dep] += 1
    queue = [node for node in nodes if in_degree[node] == 0]
    sorted_list = []
    while queue:
        current = queue.pop(0)
        sorted_list.append(current)
        for node, dependencies in deps.items():
            if current in dependencies:
                in_degree[node] -= 1
                if in_degree[node] == 0:
                    queue.append(node)
    return sorted_list if len(sorted_list) == len(nodes) else None
trial 1 (temp 0.7) — truncated — hit the 4096-token cap
def topological_sort(deps):
    nodes = set(deps.keys()).union(*deps.values())
    in_degree = {node: 0 for node in nodes}
    for deps_list in deps.values():
        for dep in deps_list:
            in_degree[dep] += 1
    queue = [node for node in nodes if in_degree[node] == 0]
    sorted_list = []
    while queue:
        current = queue.pop(0)
        sorted_list.append(current)
        for node, dependencies in deps.items():
            if current in dependencies:
                in_degree[node] -= 1
                if in_degree[node] == 0:
                    queue.append(node)
    return sorted_list if len(sorted_list) == len(nodes) else None
trial 2 (temp 0.7) — truncated — hit the 4096-token cap
def topological_sort(deps):
    nodes = set(deps.keys()).union(*deps.values())
    in_degree = {node: 0 for node in nodes}
    for deps_list in deps.values():
        for dep in deps_list:
            in_degree[dep] += 1
    queue = [node for node in nodes if in_degree[node] == 0]
    sorted_list = []
    while queue:
        current = queue.pop(0)
        sorted_list.append(current)
        for node, dependencies in deps.items():
            if current in dependencies:
                in_degree[node] -= 1
                if in_degree[node] == 0:
                    queue.append(node)
    return sorted_list if len(sorted_list) == len(nodes) else None
json_diff — 0/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — truncated — hit the 4096-token cap
(no code extracted)
trial 1 (temp 0.7) — truncated — hit the 4096-token cap
(no code extracted)
trial 2 (temp 0.7) — truncated — hit the 4096-token cap
(no code extracted)

aya-23 35b 52% overall↑ top

Served via llama.cpp Metal · 10.8 tok/s · 37.9 GB RAM · coding total 66/126

Python — 21/24 (2.6 min / 1.8k tok)

is_valid_ipv4 — 0/3 passed
prompt sent to the model
Implement in Python 3: `def is_valid_ipv4(s):` Returns True if s is a dotted-quad IPv4 address: four decimal octets 0-255, no leading zeros unless the octet is exactly '0'. Otherwise False.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — ValueError: invalid literal for int() with base 10: '1.2'
def is_valid_ipv4(s):
    if s.count('.') != 3:
        return False
    for i in range(4):
        octet = s[i * 3 : i * 3 + 3]
        if not octet[0] == '0' and int(octet) > 255:
            return False
        if octet[0] == '0' and int(octet) != 0:
            return False
    return True
trial 1 (temp 0.7) — fail — ValueError: invalid literal for int() with base 10: '1.2'
def is_valid_ipv4(s):
    if s.count('.') != 3:
        return False
    for i in range(4):
        octet = s[i * 3 : i * 3 + 3]
        if not octet[0] == '0' and int(octet) > 255:
            return False
        if octet[0] == '0' and int(octet) != 0:
            return False
    return True
trial 2 (temp 0.7) — fail — ValueError: invalid literal for int() with base 10: '1.2'
def is_valid_ipv4(s):
    if s.count('.') != 3:
        return False
    for i in range(4):
        octet = s[i * 3 : i * 3 + 3]
        if not octet[0] == '0' and int(octet) > 255:
            return False
        if octet[0] == '0' and int(octet) != 0:
            return False
    return True

Python (hard) — 3/9 (2.2 min / 1.6k tok)

topological_sort — 0/3 passed
prompt sent to the model
Implement in Python 3: `def topological_sort(deps):` deps maps each node (str) to a list of nodes it depends on. Returns a list of all nodes in an order where every node comes after its dependencies. Returns None if the graph has a cycle (including self-dependencies). Nodes mentioned only as dependencies are included.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError
def topological_sort(deps):
    # Create a set of all nodes
    all_nodes = set(deps.keys())
    # Create a set to keep track of visited nodes
    visited = set()
    # Create a list to store the topological order
    order = []

    def visit(node):
        if node in visited:
            return
        visited.add(node)
        for neighbour in deps.get(node, []):
            visit(neighbour)
        order.append(node)

    # Visit all nodes
    for node in all_nodes:
        visit(node)

    # If all nodes were visited, return the order
    if len(order) == len(all_nodes):
        return order
    else:
        return None
trial 1 (temp 0.7) — fail — AssertionError
def topological_sort(deps):
    # Create a set of all nodes
    all_nodes = set(deps.keys())
    # Create a set to keep track of visited nodes
    visited = set()
    # Create a list to store the topological order
    order = []

    def visit(node):
        if node in visited:
            return
        visited.add(node)
        for neighbour in deps.get(node, []):
            visit(neighbour)
        order.append(node)

    # Visit all nodes
    for node in all_nodes:
        visit(node)

    # If all nodes were visited, return the order
    if len(order) == len(all_nodes):
        return order
    else:
        return None
trial 2 (temp 0.7) — fail — AssertionError
def topological_sort(deps):
    # Create a set of all nodes
    all_nodes = set(deps.keys())
    # Create a set to keep track of visited nodes
    visited = set()
    # Create a list to store the topological order
    order = []

    def visit(node):
        if node in visited:
            return
        visited.add(node)
        for neighbour in deps.get(node, []):
            visit(neighbour)
        order.append(node)

    # Visit all nodes
    for node in all_nodes:
        visit(node)

    # If all nodes were visited, return the order
    if len(order) == len(all_nodes):
        return order
    else:
        return None
json_diff — 0/3 passed
prompt sent to the model
Implement in Python 3: `def json_diff(a, b):` Compares two JSON-like structures (dicts, lists, scalars). Returns a sorted list of dotted key paths where they differ: changed scalar/list values, keys present in only one side. Dicts are recursed into; lists and scalars are compared as whole values. Empty list means identical.
Reply with only a Python code block. No tests, no explanation.
trial 0 (temp 0) — fail — AssertionError
def json_diff(a, b):
    def diff_path(path, a, b):
        if isinstance(a, dict) and isinstance(b, dict):
            return [
                f"{path}.{k}"
                for k in set(a.keys()) ^ set(b.keys())
                for subpath in diff_path(f"{path}.{k}", a[k], b.get(k, {}))
            ]
        elif isinstance(a, list) and isinstance(b, list):
            return [
                f"{path}[{i}]"
                for i, (av, bv) in enumerate(zip(a, b))
                for subpath in diff_path(f"{path}[i]", av, bv)
            ]
        else:
            return [path] if a != b else []

    return diff_path("", a, b)
trial 1 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def diff_path(path, a, b):
        if isinstance(a, dict) and isinstance(b, dict):
            return [
                f"{path}.{k}"
                for k in set(a.keys()) ^ set(b.keys())
                for subpath in diff_path(f"{path}.{k}", a[k], b.get(k, {}))
            ]
        elif isinstance(a, list) and isinstance(b, list):
            return [
                f"{path}[{i}]"
                for i, (av, bv) in enumerate(zip(a, b))
                for subpath in diff_path(f"{path}[i]", av, bv)
            ]
        else:
            return [path] if a != b else []

    return diff_path("", a, b)
trial 2 (temp 0.7) — fail — AssertionError
def json_diff(a, b):
    def diff_path(path, a, b):
        if isinstance(a, dict) and isinstance(b, dict):
            return [
                f"{path}.{k}"
                for k in set(a.keys()) ^ set(b.keys())
                for subpath in diff_path(f"{path}.{k}", a[k], b.get(k, {}))
            ]
        elif isinstance(a, list) and isinstance(b, list):
            return [
                f"{path}[{i}]"
                for i, (av, bv) in enumerate(zip(a, b))
                for subpath in diff_path(f"{path}[i]", av, bv)
            ]
        else:
            return [path] if a != b else []

    return diff_path("", a, b)