Version 0.0.1

GradieX — C++ Library Reference

The entire library is one C++17 translation unit, main.cpp, with no dependencies beyond the standard library. Three classes, two enums, and a training loop.

#Building and Testing

C++17 is required. There is no build system, no library to link, and nothing to configure:

g++ -std=c++17 -O2 -Wall -Wextra -pedantic main.cpp -o gradiex
./gradiex

g++ -std=c++17 -O2 -Wall -Wextra -pedantic tests/test_gradiex.cpp -o test_gradiex
./test_gradiex        # 112 passed, 0 failed; exits non-zero on any failure

CI is a GitHub Actions workflow with three jobs. test builds and runs both binaries under GCC and clang on Ubuntu and clang on macOS, with -pedantic -Werror. sanitizers rebuilds the suite with -fsanitize=address,undefined and UBSAN_OPTIONS=halt_on_error=1. release-binary builds gradiex-linux-x86_64 and uploads it as an artifact.

#Overview

Three classes, each owning the next, plus two option structs:

NeuralNetwork   hidden_layers (vector<Layer>) + one output_layer,
                a Loss, and the RNG. Owns the training loop and persistence.
  Layer         vector<Neuron> sharing one activation. Owns the layer-wide
                steps: softmax, and the dropout mask.
    Neuron      one weight vector, one bias, the cached input and
                pre-activation, and this batch's accumulated gradients.

Activation and loss are enum classes, so the hot path switches on an integer rather than comparing strings per neuron per pass. String spellings remain available at the API boundary, where they are parsed exactly once — the changelog records the enum switch as measuring about 17% faster on a 64–128×3–10 network.

One training step:

zero_gradients()                       once per batch

for each sample in the batch:
    forward_pass(x, training, dropout)
        each hidden Layer::forward   ->  z = Wx + b,  a = f(z),  dropout mask
        output Layer::forward        ->  p          (no dropout on the output)
    calculate_loss(p, y)
    accumulate_gradients(y)
        output layer: fused p - y if softmax+CCE, else backward(dL/dp)
        hidden layers, last to first: backward(dL/da)

apply_gradients(learning_rate, 1/batch_count, l2)

No weight moves until apply_gradients(). That is what makes the mini-batch update exactly the mean of the per-sample updates, and it is what guarantees the backward sweep reads pre-update weights — see Invariants.

#Activation and Loss Enums

enum class Activation { Sigmoid, Tanh, ReLU, Linear, LeakyReLU, Softmax };
enum class Loss { MeanSquaredError, BinaryCrossEntropy, CategoricalCrossEntropy };

Activation parse_activation(const string&), string activation_name(Activation)

Round-trip between the enum and its spelling. Accepted names are "sigmoid", "tanh", "ReLU", "linear", "leaky_ReLU" and "softmax". Anything else throws std::invalid_argument. Note the capitalisation: "ReLU" and "leaky_ReLU", not "relu".

Loss parse_loss(const string&), string loss_name(Loss)

The same, for "mean_squared_error", "binary_cross_entropy" and "categorical_cross_entropy".

Both round-trips are asserted by test group 17, along with the claim that enum and string dispatch produce identical values, identical derivatives and identical networks.

#class Neuron

One neuron: a weight vector, a bias, the caches the backward pass needs, and gradient accumulators. All members are public.

#Data Members

#Member Functions

Neuron()

Empty, with every scalar zeroed. Used by load(), which fills the weights from the file.

Neuron(int input_size, mt19937& rng)

He initialization — N(0, sqrt(2 / max(1, input_size))) — drawn from the network's RNG, so the whole network is reproducible from one seed. Bias starts at zero.

static double activate(double, Activation)

Elementwise activation. Sigmoid branches on the sign of its argument, evaluating 1/(1+e^-x) for x >= 0 and e^x/(1+e^x) otherwise, so neither exponential can overflow.

Asking for Softmax throws std::invalid_argument: softmax is layer-wide and cannot be computed from one pre-activation.

static double activation_derivative(double pre_activation, double output, Activation)

Takes both z and a, because sigmoid and tanh are cheaper from the output while ReLU and leaky ReLU need the pre-activation.

Asking for Softmax throws. Its Jacobian is a dense n × n matrix, which a scalar-in / scalar-out signature cannot express; the supported path is the fused one, see Loss Functions.

static double activate(double, const string&), static double activation_derivative(double, double, const string&)

String-accepting wrappers, kept so older call sites keep working. They parse on every call, so they do not belong on a hot path.

void compute_pre_activation(const vector<double>& input)

Caches the input and computes z = w . input + b. Throws if input.size() != weights.size() — shape mismatches are caught here rather than read past the end.

double forward(const vector<double>& input, Activation)

compute_pre_activation followed by activate. Not used on the softmax path, which needs every z before any a.

void zero_gradients()

Clears both accumulators. Called once per batch.

void apply_gradients(double learning_rate, double scale, double l2)

w -= learning_rate * (weight_gradient * scale + l2 * w)
b -= learning_rate * bias_gradient * scale

scale is 1/batch_count, which averages the accumulated batch. L2 is applied to weights and never to the bias, which is the standard choice — decaying a bias just shifts the function without reducing model complexity.

#class Layer

A vector of neurons sharing one activation. Owns the two things that cannot be done per neuron: softmax, and the dropout mask.

#Data Members

#Member Functions

Layer(int neuron_count, int input_size, Activation, mt19937& rng)

Constructs the neurons, each initialised from rng. A string-accepting overload delegates through parse_activation.

vector<double> forward(const vector<double>& input, bool training, double dropout_rate, mt19937* rng)

Computes every pre-activation first, then applies the activation.

For Softmax it subtracts max(z) before exponentiating, so large logits cannot overflow. For everything else it applies Neuron::activate elementwise.

When training is set and dropout_rate > 0, it then applies inverted dropout: survivors are scaled by 1/(1 - p), so inference needs no rescaling and predict() is always deterministic. A rate of 1.0 or more throws. The mask multiplies output_vector only — Neuron::output keeps the unmasked value.

A one-argument overload, forward(input), is inference: no training, no dropout.

vector<double> backward_local(const vector<double>& local_gradients)

Takes gradients already with respect to each neuron's pre-activation. Accumulates dL/dw += local * input_cache and dL/db += local, and returns dL/d(input) for the layer behind it.

This entry point exists so the fused softmax + CCE path can inject p - y directly, skipping the activation derivative that softmax does not have.

vector<double> backward(const vector<double>& gradient_wrt_output)

The ordinary path: applies the dropout mask to the upstream gradient (a dropped unit passes none), multiplies by f'(z), and delegates to backward_local.

Throws on a softmax layer — those must go through the fused path.

vector<double> backward(const vector<double>&, double learning_rate)

Legacy immediate-update form: zero, accumulate this one sample, apply at once. Equivalent to a batch of one with no L2.

void zero_gradients(), void apply_gradients(double learning_rate, double scale, double l2)

Fan the corresponding Neuron call out over the layer.

#TrainOptions and TrainHistory

#class NeuralNetwork

#Data Members

#Member Functions

NeuralNetwork(int input_size, int hidden_layer_count, int layer_width, int output_layer_width, Activation hidden_activation, Activation output_activation, Loss, int64_t seed = -1)

Builds the stack. A negative seed draws one from random_device; the value used is readable as net.seed. A string-accepting overload takes the three names instead.

Four configurations are rejected up front with std::invalid_argument, because none has a correct path through the code:

• non-positive dimensions
• softmax as a hidden activation
• softmax output without categorical cross entropy
• categorical cross entropy without a softmax output

Every hidden layer shares layer_width. Per-layer widths are not reachable from the constructor.

NeuralNetwork()

An empty shell, used by load(). Not useful directly.

int input_size() const, int output_size() const

Read back from the first layer's fan-in and the output layer's neuron count, so they stay correct for a loaded model.

vector<double> forward_pass(const vector<double>& input, bool training = false, double dropout = 0.0)

Chains the layers. Dropout is passed to the hidden layers only — the output layer is always called with training = false.

double calculate_loss(const vector<double>& predictions, const vector<double>& targets) const

The configured loss. MSE and BCE average over the n outputs; CCE sums over the n classes, the standard definition for a one-hot target. Both cross entropies clamp with epsilon = 1e-9, so a confident wrong answer costs a large finite number rather than infinity.

vector<double> loss_gradient(const vector<double>&, const vector<double>&) const

dL/dp. The CCE branch returns the raw -y/p; the supported softmax path never reaches it, because accumulate_gradients uses the fused form instead.

void accumulate_gradients(const vector<double>& targets)

Adds one sample's gradients to the accumulators without touching any weight. On softmax + CCE it computes p - y and calls output_layer.backward_local; otherwise it takes loss_gradient through output_layer.backward. Then it sweeps the hidden layers last to first.

void zero_gradients(), void apply_gradients(double learning_rate, double scale, double l2)

Fan out over every layer.

void backward_pass(const vector<double>& targets, double learning_rate)

Zero, accumulate one sample, apply. The single-sample update, unchanged in behaviour from the original engine.

double evaluate(const vector<vector<double>>& X, const vector<vector<double>>& Y)

Mean loss over a dataset, with dropout off. Returns 0 on empty input.

double accuracy(const vector<vector<double>>& X, const vector<vector<double>>& Y)

Fraction where argmax(prediction) == argmax(target). That is a multi-class measure; for a single sigmoid output it compares the argmax of a one-element vector against itself and returns 1.0 regardless. Use a threshold on predict() for binary tasks.

TrainHistory train(const vector<vector<double>>& X, const vector<vector<double>>& Y, const TrainOptions&)

The full training loop — see Training.

void train(const vector<vector<double>>& X, const vector<vector<double>>& Y, int epochs, double learning_rate, int log_interval = 200)

The original five-argument signature, preserved. It fills a TrainOptions with batch_size = 1, shuffle = false and no regularization, so it is bit-identical to a manual fixed-order, one-sample-at-a-time loop. A test asserts this — see Invariants, item 6.

vector<double> predict(const vector<double>& input)

A forward pass with dropout off. Deterministic, because dropout is inverted.

void save(const string& path) const, static NeuralNetwork load(const string& path)

See Model Persistence.

#Activation Functions

All are elementwise except softmax.

Both sigmoid and softmax are stable out to ±10⁵; test group 7 asserts it.

On the softmax derivative. Every other derivative above is scalar-in / scalar-out. The softmax Jacobian is a dense n × n matrix, dp_i/dz_j = p_i(d_ij - p_j), which that signature cannot express. Rather than return something wrong, Neuron::activation_derivative throws, and the supported path fuses softmax with categorical cross entropy so the Jacobian cancels — see below.

#Loss Functions

MSE and BCE average over the n outputs; CCE sums over the n classes. Both cross entropies clamp p into [1e-9, 1 - 1e-9].

#The fused softmax + CCE path

Substituting the softmax Jacobian into dL/dp = -y/p, the p_i terms cancel and the whole chain collapses to

dL/dz = p - y

which is why Layer::backward_local exists: it accepts gradients already with respect to z, so accumulate_gradients can inject p - y directly. This avoids both the dense Jacobian and the division by p_i, which would be catastrophic for a confidently wrong prediction. The constructor rejects every configuration that would take a different route.

#Training

train(X, Y, options) validates, splits, then loops.

#The validation split

The held-out slice is taken after a shuffle, so the split is not an artefact of how the caller happened to order the data. A non-zero validation_split that rounds to zero samples is bumped to one; a split that would leave no training data throws.

#The batch loop

for each epoch:
    if shuffle: reshuffle the training indices
    for each batch:
        zero_gradients()
        for each sample:  forward (with dropout), loss, accumulate
        apply_gradients(learning_rate, 1 / batch_count, l2)
    record train loss; if a split exists, evaluate and record validation loss

The final partial batch is handled: batch_end is clamped to the end of the index list and batch_count is the actual number of samples, so an uneven split averages each batch over its own size.

Because gradients accumulate and are then scaled by 1/batch_count, the batch update equals the mean of the per-sample gradients exactly. Test group 10 asserts this.

#Regularization

L2 is added at the update, gradient + lambda * w, and never touches the bias.

Dropout is inverted and applies to hidden layers during training only. Survivors are scaled by 1/(1-p), so inference needs no rescaling. A dropped unit passes no gradient, because Layer::backward multiplies the upstream gradient by the same mask.

#Rejected arguments

train() throws std::invalid_argument on an empty or mismatched dataset, non-positive epochs or learning_rate, non-positive batch_size, negative l2, dropout outside [0, 1), validation_split outside [0, 1), and a split that leaves no training data. Test group 5 covers these.

#Model Persistence

Plain text, versioned, written at 17 significant digits — enough to round-trip an IEEE-754 double exactly, so a reloaded network predicts bit-for-bit identically.

GRADIEX-MODEL 1
loss categorical_cross_entropy
seed 20260821
layers 3
layer 0 24 2 tanh            <- index, neurons, fan-in, activation
<bias> <w0> <w1> ...         <- one line per neuron
...

load() rejects a file that is not a GradieX model, a version it does not recognise, layers out of order, and truncated or malformed weights — each with a distinct std::runtime_error. Test group 13 covers the round-trip and the rejections.

Two properties worth knowing. The seed is stored, so a loaded model carries the provenance of the run that produced it. And load() reads each layer's shape independently, so a hand-written model file with varying hidden widths loads and trains correctly — the uniform width is a constructor limitation, not an engine one.

Verified here: the demo's spiral classifier saves, reloads, and predicts identically across all 360 points, maximum delta 0.0e+00.

#Complete Example

#include "main.cpp"     // renaming its main, see Building

int main() {
    // three-class problem, 2 features
    vector<vector<double>> X = /* ... */;
    vector<vector<double>> Y = /* one-hot ... */;

    NeuralNetwork net(2, 2, 24, 3,
                      "tanh", "softmax", "categorical_cross_entropy",
                      20260821);          // seed; omit or pass -1 for random

    TrainOptions options;
    options.epochs           = 400;
    options.learning_rate    = 0.05;
    options.batch_size       = 16;
    options.shuffle          = true;
    options.l2               = 1e-5;
    options.validation_split = 0.25;
    options.log_interval     = 50;

    TrainHistory history = net.train(X, Y, options);

    cout << "best val loss " << history.best_validation_loss
         << " at epoch "     << history.best_epoch     << "\n";
    cout << "accuracy "      << net.accuracy(X, Y)     << "\n";

    net.save("model.gx");
    NeuralNetwork reloaded = NeuralNetwork::load("model.gx");
    // reloaded.predict(x) == net.predict(x), bit for bit
    return 0;
}

This is the second demo in main.cpp, which reaches 100% accuracy on 360 interleaved spiral points.

#Test Suite

g++ -std=c++17 -O2 -Wall -Wextra -pedantic tests/test_gradiex.cpp -o test_gradiex
./test_gradiex

112 checks across 17 groups. All pass. The suite exits non-zero on any failure and CI runs it on three compiler/OS combinations plus a sanitized build.

#Invariants

From the source header. Changing any of these silently breaks correctness.

1. Gradients accumulate; no weight moves until apply_gradients(). This is what makes mini-batching exact, and what guarantees that step 3 of the backward pass, dL/da_j = sum_i dL/dz_i * w_ij, reads w_ij before it is updated. Updating a layer's weights before propagating through it corrupts every layer behind it.

2. input_cache must hold the input from this forward pass. Weight gradients are meaningless against a stale cache.

3. Neuron::output is the activation BEFORE the dropout mask; Layer::output_vector is the value AFTER it. The sigmoid and tanh derivatives are written in terms of the unmasked output, so masking Neuron::output would corrupt them. With dropout off the two are equal, which is exactly why this is easy to break without noticing.

4. Softmax is layer-wide and fused with CCE. Neuron::activate and Neuron::activation_derivative throw rather than return a wrong answer.

5. Activation and Loss are enums on the hot path. Strings are parsed once at the API boundary; do not push a string comparison back into Layer::forward or the per-neuron loops.

6. The five-argument train() is bit-identical to a manual fixed-order, one-sample-at-a-time loop. A test asserts it.

#Known Limitations

Not bugs — the suite passes and the demos are correct — but boundaries worth knowing before building on this release.

1. One width for every hidden layer. Fixed by the constructor. load() has no such restriction, so a hand-written model file is the only way to get varying widths.

2. One output layer, one activation per layer. No multi-head output, no per-neuron activation.

3. No early stopping. TrainHistory records best_epoch and best_validation_loss; nothing acts on them.

4. No per-layer dropout rate. One rate applies to every hidden layer.

5. accuracy() is argmax-based. For a single sigmoid output it compares a one-element vector's argmax against itself and always returns 1.0. Threshold predict() yourself for binary tasks.

6. The library and the demo share a translation unit. Including main.cpp means renaming its main. v0.0.2 fixes this by splitting the engine into headers.

7. Allocation inside the training loop. Layer::forward, backward and backward_local each build and return a vector<double> by value, and every neuron owns two vectors of its own. Correct, and the reason v0.0.2 exists.

8. Mean Absolute Error is absent. Listed in an earlier header revision, never in the Loss enum.

#Not Supported

Optimizers other than plain SGD (no momentum, Adam, or learning-rate schedule), per-layer widths or dropout rates from the constructor, early stopping, convolution, recurrence, attention, embeddings, skip connections, batch or layer normalization, sparse input, streaming datasets, GPU, threading, and SIMD. The inner loops are scalar double arithmetic.

Python bindings arrive in v0.0.2; single-precision arithmetic and SIMD pragmas in v0.0.3.