Benchmarks — method and results

Status: M9a baseline (2026-10-25); memory-copy corrected in M9b (its blocks are permuted, suite README). Master prompt §19 (what to compare and measure) and §22 rule 3 (no performance claim without a reproducible benchmark). The suite is benchmarks/suite/ (its README defines every workload and build); benchmarks/lz4/ and benchmarks/fbvalid/ are the C6 component benchmarks, with their own READMEs.

1. What is compared

Each of the sixteen workloads (fifteen from M9a, and nbody from M11 G7d) is written once per language with the same algorithm and data structures (the suite README's table), and prints one checksum, which run.py compares with the expected value before it records a time. A program that computes something else is a failed run, not a fast one.

Configuration	What it is	Checks at run time
nova	novac build, its C compiled at -O2 (M5 placement off, the default)	bounds, overflow, null unwrap, allocation failure
c	cc -O2 -std=c11	none
cpp	c++ -O2 -std=c++20, the standard library's containers and unique_ptr	none
rust	rustc -C opt-level=2 -C codegen-units=1 (a release build)	bounds
rust-checked	the same with -C overflow-checks=on	bounds, overflow: what Nova checks
zig	zig build-exe -O ReleaseFast -lc, std.heap.c_allocator	none
zig-safe	-O ReleaseSafe	bounds, overflow, null unwrap: what Nova checks
goose	goose build -O2 (commit 94cc64d)	bounds, always, with its own elimination; signed overflow wraps in a release build (Goose docs/reference.md, Overflow and Bounds checks)

nova, c, cpp and goose go through the same C compiler, named by --cc; Rust and Zig use their own LLVM. That is part of what is measured: a language that compiles to C inherits that C compiler's strengths and weaknesses (C6b's fbvalid ran at 1.76× C's time under gcc 12 and equal to C under clang).

Goose has no heap. Three workloads that free objects one at a time have no Goose version, and in three more Goose's nodes share one pool per list or tree (suite README). Those rows compare memory models, not allocators.

2. What is measured

Metric	How	Notes
Wall time	measure.c: posix_spawn to wait4, CLOCK_MONOTONIC; one discarded run, then the median of --runs runs (default 5); the minimum and maximum are kept in the JSON	process start-up is included, under 2 ms here; every workload runs for about 0.4 s or more
Peak resident memory	ru_maxrss from wait4 in measure.c	measured from a small launcher, because Linux carries the forking process's peak across exec (a Python parent showed 13 MiB for every program)
Allocations	allocount.c, preloaded (DYLD_INSERT_LIBRARIES, LD_PRELOAD), in a separate run: calls to malloc, calloc, realloc, posix_memalign, aligned_alloc; bytes requested; peak live heap in usable bytes (malloc_size, malloc_usable_size)	includes what the C library and the language runtime allocate before main (about 60 on macOS, 1–10 on Linux); Goose's data stacks are mmap'd and not counted
Compile time	wall time of the build command, with each toolchain's global caches warm (one discarded build first); Zig's per-program cache emptied before each build	Nova's includes its C compile; Rust's and Zig's include linking their standard libraries
Binary size	the executable's size on disk, not stripped	Rust and Zig link their standard library statically

3. Machines

Name	CPU	OS	C compilers	Note
M1	Apple M1, 8 cores (4 performance)	macOS 27 (Darwin 27.0.0)	Apple clang 21.0.0	development machine, shared with other work: used for checksums, not published times
Ryzen	AMD Ryzen 5 PRO 3600, 6 cores, 12 threads	Debian 12, Linux 6.1	clang 18.1.8, gcc 12.2.0	shared server: runs under nice, load recorded with each run
Windows VM	Apple M1 (Parallels VM on the development machine), 4 virtual processors	Windows 11 Pro (ARM64)	MSVC 19.51 (ARM64)	MSVC only (§4.1); shares the M1 with other work, so its times are compared with its own C only

Toolchains: rustc 1.99.1; Zig 0.17.0 (official tarballs, sha256 checked); Goose 94cc64d built from source.

4. Reproducing

dotnet build src/Nova.Cli -c Release
python3 benchmarks/suite/run.py --cc clang --runs 5 --json out/benchmarks/suite/clang.json
python3 benchmarks/suite/run.py --cc gcc --runs 5 --json out/benchmarks/suite/gcc.json

--zig, --goose, --rustc and --novac name the toolchains (a missing one is skipped and reported); --langs and --workloads select rows. --nova-flag=--no-bce (repeatable) passes an option to novac, to measure an optimisation with its switch off (optimisations.md §1); benchmarks/lz4/run.py and benchmarks/fbvalid/run.py take it too.

4.1 MSVC (M9f)

run.py needs POSIX (measure.c launches with posix_spawn and reads wait4's peak memory; the allocation counter is an LD_PRELOAD/DYLD interposer), so MSVC has its own runner, benchmarks/suite/run_msvc.py, for Windows:

call "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvarsall.bat" arm64
dotnet build src\Nova.Cli -c Release
python benchmarks\suite\run_msvc.py --runs 5 --json out\benchmarks\msvc.json

• Nova: novac build --emit c, then cl /nologo /std:c11 /O2 /link /STACK:8388608 (the stack of c-backend.md §1.1). C: cl /nologo /std:c11 /O2 on the same main.c as elsewhere.
• The same method otherwise: one discarded run, then the median of --runs wall times (Python's perf_counter, around subprocess.run), and every run's output must equal the workload's checksum.
• Not measured: peak memory, allocations and compile time. Only Nova and C are compared; the other languages' toolchains are not on the Windows machine.
• Under MSVC, rt.h uses no compiler builtins (c-backend.md §1.1): checked arithmetic is the portable code. The MSVC row measures that too.

5. Results

See docs/milestone-9.md (results), which records each run with its date, machine, load and the commit measured. The acceptance run at the end of M9 is there too (9f): the whole suite in every language, each optimisation's share (benchmarks/suite/attribute.py), and the MSVC row (§4.1).