GradieX docs
GitHub
Menu

v0.0.3

On this page

Quick reference

Version 0.0.3

Performance

v0.0.3 is 1.2–3.2× faster than v0.0.2 per sample, and 1.7–5.5× with mini-batching. Most of the margin on wide layers comes from one compiler flag.

Measured, not inherited

Every figure on this page comes from 0.0.3/bench/run_bench.sh on the current source. The previous numbers in the README were v0.0.2-vs-v0.0.1 and had been carried forward unmeasured; they are gone.

Method

Both engines are driven through one shared harness, bench/bench.cpp, selected by -DBENCH_V2. The timing loop, the input data and the architectures are therefore identical across versions by construction rather than by inspection.
MachineApple M5, 10 cores, macOS; Apple clang 21.0.0
ThreadsOne. #pragma omp simd is instruction-level parallelism; core/ contains no parallel regions.
UnitMicroseconds per sample — one forward pass, one backward pass, one weight update.
EstimatorMinimum of nine trials. Interference only ever adds time, so the fastest run is the least contaminated estimate.
Data256 samples from a fixed LCG, identical in both builds.

At batch size 1 both engines do the same work per sample. v0.0.3 additionally zeroes its gradient buffers on every update — a real cost of the batching architecture, counted here rather than excused. The median and the minimum agree to within 0.06× everywhere, so the choice of estimator moves no conclusion; a clean re-run drifted 1.0 % at the median.

Speedup Over v0.0.2

Grouped bar chart of v0.0.3 speedup over v0.0.2 across four architectures, for three configurations: no OpenMP, with -fopenmp-simd, and with batching. Speedups range from 1.16x to 5.45x and grow with network size. Grouped bar chart of v0.0.3 speedup over v0.0.2 across four architectures, for three configurations: no OpenMP, with -fopenmp-simd, and with batching. Speedups range from 1.16x to 5.45x and grow with network size.
The margin grows with the network. On the smallest architecture the work is dominated by per-layer overhead that neither version avoids; on the widest, single precision and vectorisation both pay.

-O2, no OpenMP — single precision and the structural changes alone:

ArchitectureParamsv0.0.2v0.0.3Speedup
2→8×2→11050.1310.1111.16×
20→32×2→31,8271.0400.6481.63×
64→128×3→1042,63437.43917.4072.15×
784→128×2→10118,28285.40764.3571.33×

-O2 -fopenmp-simd, which is what makes the directives apply:

Architecturev0.0.2v0.0.3Speedupv0.0.3 @ batch 32Speedup
2→8×2→10.1390.1031.35×0.0781.78×
20→32×2→31.0620.5641.88×0.4132.57×
64→128×3→1037.51712.1323.09×7.7634.83×
784→128×2→1085.31126.5893.21×15.6485.45×

-O3 made no measurable difference over -O2 at any size. It is in the matrix and in bench/results.csv, but there is nothing to report from it.

Cost Per Sample

Dot plot of microseconds per sample on a logarithmic axis, one row per architecture, comparing v0.0.2, v0.0.3 and v0.0.3 at batch 32. Costs span from 0.08 to 85 microseconds. Dot plot of microseconds per sample on a logarithmic axis, one row per architecture, comparing v0.0.2, v0.0.3 and v0.0.3 at batch 32. Costs span from 0.08 to 85 microseconds.
A dot plot, not bars. The values span three orders of magnitude, and a bar on a logarithmic axis no longer has length proportional to its value — the encoding would lie. Each row’s connector shows the distance the two releases cover.

The OpenMP Flag Is Most Of It

Line chart showing the speedup from -fopenmp-simd rising from 1.08x at a widest layer of 8 neurons to 2.42x at 784 neurons, on a logarithmic x axis. Line chart showing the speedup from -fopenmp-simd rising from 1.08x at a widest layer of 8 neurons to 2.42x at 784 neurons, on a logarithmic x axis.
The gain tracks layer width. The wider the dot product, the more there is to vectorise. On the narrowest network it is nothing at all; on the widest it is 2.42×, which is most of v0.0.3’s margin there. v0.0.2 is unaffected by the flag — it has no pragmas.
Apple clang can do this

Apple clang rejects -fopenmp, which is the flag usually named, and an earlier revision of these docs concluded that v0.0.3’s vectorisation was therefore inert on macOS. It is not. Apple clang accepts -fopenmp-simd, which enables exactly the SIMD directives and needs no runtime library — and core/ uses nothing else.

clang++ -std=c++20 -O2 -fopenmp-simd main.cpp -o main   # Apple clang
g++-14  -std=c++20 -O3 -fopenmp      main.cpp -o main   # GCC, full OpenMP

Turning it on found two bugs

Unrecognised #pragma omp directives are discarded in total silence — -Wall -Wextra says nothing about them. Two defects had been sitting behind that silence:

  1. A malformed reduction. softmaxCrossEntropyDerivative carried reduction(+:vectorScratchPad[:size]) over a loop that overwrites the array rather than accumulating into it. An array-section reduction gives each lane a private copy initialised to the identity and sums them at the end — the wrong answer. Building the test suite with the flag failed the fused-softmax test immediately.
  2. huber computing in double. 0.5 rather than 0.5f promoted the whole expression in an engine that is float everywhere else, which is also why clang refused to vectorise the loop.

Both are fixed, and tests/run_tests.sh --simd builds with the flag so neither can return unnoticed. With it on, core/ compiles warning-free.

Reproducing

cd 0.0.3
./bench/run_bench.sh            # full matrix, CSV on stdout
./bench/run_bench.sh > out.csv

The runner sweeps -O2 and -O3 against OpenMP on and off, building both engines under each combination. Raw output for the run on this page is checked in at 0.0.3/bench/results.csv. The charts are regenerated from it by docs/make_bench_figures.py.

What Is Not Measured

  • v0.0.1. The old README compared v0.0.2 against v0.0.1 and claimed ~2.4×. That column was dropped rather than carried forward: v0.0.1 is a single translation unit with a different API, and building a fair harness for it was not part of this work.
  • Accuracy per unit time. These are throughput figures. Single precision changes the arithmetic, so a faster update is not automatically a faster path to a given loss — nothing here measures that.
  • Multi-threading. Everything is single-threaded, and there is no parallel for anywhere in core/.
  • Memory. Peak resident size and allocation counts are not tracked. v0.0.2’s zero-allocation assertion was not ported; the flat layout it guarded is unchanged, but nothing currently asserts it.