Benchmarks

Keeping it fair

Every algorithm is written by hand, identically, in every language: the same data generation, operation order and data layout. The programs use only standard-library I/O, collections and allocation, with no third-party packages, SIMD intrinsics or threads.

Inputs come from the same SplitMix64 generator, and floating-point results are printed as raw IEEE-754 bits, so every language must agree bit for bit; C++ is built with -ffp-contract=off for that reason. All builds target baseline x86-64, and runtimes run with their defaults, with no GC or JIT tuning for .NET, Go or the JVM.

Each language allocates the way it normally does. BinaryTrees uses new/delete in C++, Box in Rust, the garbage collector in Go, C# and Java, and Allocator::Pool in Rux. WordCount uses each standard library's hash map.

The program benchmark

Each app is built with the language's usual tool: rux, cargo, clang++, go build, dotnet publish, javac and jar, or native-image. After one untimed warm-up build come timed clean release builds. Go starts from a cache holding only the precompiled standard library, so the app itself is always compiled from scratch.

Each program then gets unmeasured warm-up runs and measured runs, with the languages taking turns so that drifts in the machine's state hit all of them alike. Execution time includes process start-up, which for the JIT builds means starting the runtime and compiling.

Every run must exit with code 0 and print exactly the expected output; otherwise the cell is marked FAIL and left out. The page shows medians, and the chart tooltips add the minimum, the standard deviation and the run count.

The compile benchmark

The projects are generated from one specification, so every language compiles the same modules, types and functions and prints the same result. Each module is one of ten kinds — a struct with methods, an enum state machine, dynamic dispatch, a hash map, sorting, a ring buffer, a CRC table, text formatting, small matrices, union-find — and calls a few of the modules before it. C++ builds with CMake and Ninja, Rust with Cargo, C# with dotnet publish, Java with javac and then native-image or jar and jlink.

A clean build removes the build output first: Ninja's object files, Cargo's target folder, the Go build cache except a precompiled standard library, and the MSBuild and Java output folders. Rux compiles its standard library from source as part of every build, while the other languages link a precompiled one, so most of Rux's Baseline time is its standard library.

The same clean builds are then repeated with the runner and every process the build starts pinned to one logical CPU; Cargo and Ninja are also told to run one job. The parallel speed-up is the build on one CPU divided by the build on all of them. A rebuild changes one line in the last module, which only Main calls, and builds again without cleaning: Ninja recompiles one file and links, Go one package, while release Cargo, C# and javac rebuild the whole project.

Reading the numbers

C# and Java appear twice. AOT is one native executable (NativeAOT, GraalVM Native Image). JIT ships its own runtime: C# JIT is published self-contained and trimmed, and Java JIT runs its jar from a minimal jlink runtime. Their executable size is only the program file (the IL assembly or the jar), which cannot run alone, so it is shown in italics and not ranked; their deployable size includes the runtime.

Peak memory and compile memory are the working set and the memory committed by all build processes on Windows, and the maximum resident set of the largest process on Linux, so they are not comparable across operating systems; times are not comparable across machines. Ratios are a language's value divided by Rux's, and the program benchmark summarises them by geometric mean so that a 2× win and a 2× loss cancel out.

The apps, the generator, the runner and full instructions to reproduce these results on your own machine are in rux-lang/Benchmarks.