Version 0.0.2

GradieX — C++ Library Reference

The engine lives entirely in headers under core/. There is no library to link and no build step of its own: include a header and compile.

#Building

C++20 is required. core/initializer.h includes <numbers>, so C++17 will not compile.

clang++ -std=c++20 -O2 main.cpp -o main
./main

No other flags are needed. The #pragma GCC unroll directives in core/activations.h are accepted by both GCC and Apple clang; if your compiler warns about them, add -Wno-unknown-pragmas.

#Overview

A network is a chain of HiddenLayer objects followed by an OutputLayer. Each layer stores its weights as one flat std::vector<double> rather than a vector of vectors, so a layer's weights are contiguous in memory.

Activation, initializer and loss behaviour are supplied as plain function pointers, not as virtual methods or template parameters. Each layer holds five of them: an activation, its derivative, an initializer, a loss, and the loss derivative.

The order of operations for one training step is:

forwardPass(x)
    for each hidden layer:  calculateOutputs()
    for the output layer:   calculateOutputs()

backwardPass(y)
    output layer:  computeErrorSignal(y), computeGradients()
    hidden layers, last to first:
                   computeErrorSignal(next width, next deltas, next weights)
                   computeGradients()
    all layers:    updateWeightsAndBiases(learningRate)

Note that no weights are updated until every error signal has been computed. This matters: a hidden layer's error signal reads the next layer's weights, so updating a layer before propagating through it would corrupt every layer behind it.

#Namespaces and Headers

Including core/network.h pulls in everything.

#Function Pointer Types

core::functions::activations::ActivationFunction

double* __restrict__ (*)(const double* __restrict__ input,
                        double* __restrict__ vectorScratchPad,
                        size_t size)

Applies an activation elementwise over size values, writing into vectorScratchPad and returning it. The scratch pad is allocated once by the layer and reused, so the hot path performs no allocation. The __restrict__ qualifiers promise the compiler that input and output do not alias.

core::functions::activations::ActivationFunctionDerivative

Same signature. Takes the pre-activation values (z), not the activated outputs.

core::functions::initializers::initFunction

double* (*)(double* __restrict__ vectorScratchPad,
            size_t size, size_t size2)

Fills size * size2 values. For xavier and he the two arguments are the fan-in and fan-out respectively.

core::functions::loss::LossFunction

double (*)(const double* __restrict__ actual,
           const double* __restrict__ predicted,
           size_t size)

Returns a single scalar. Note the argument order: target first, prediction second.

core::functions::loss::LossFunctionDerivative

double* (*)(const double* __restrict__ actual,
            const double* __restrict__ predicted,
            double* __restrict__ vectorScratchPad,
            size_t size)

Writes the gradient with respect to predicted into the scratch pad and returns it.

#class core::Layer

Defined in core/layer.h. The base class for both layer kinds. All members are public.

#Constructors

Layer() noexcept = default

Leaves the object empty. Warning: the five function pointer members are indeterminate after this constructor, not null. See Known Bugs, item 6.

Layer(int width, int inputSize, ActivationFunction, ActivationFunctionDerivative, initFunction, LossFunction, LossFunctionDerivative)

Sizes every internal buffer. Does not initialize the weights; call initializeWeightsAndBiases() for that.

#Data Members

The scratch pads are only ever written up to layerWidth elements. The declared size of inputSize * layerWidth is a known defect; see Known Bugs, item 7.

#Member Functions

void initializeWeightsAndBiases()

Calls the stored initializer over the whole weight array with (inputSize, layerWidth) as fan-in and fan-out, then sets every bias to zero.

void calculateOutputs(const double* __restrict__ input)

Copies input into layerInputs, computes z[n] = sum_i w[n][i] * input[i] + b[n] into neuronOutputsUnactivated, then applies the activation into neuronOutputsActivated.

input must point to at least inputSize doubles. This is not checked.

void computeGradients()

Requires neuronDeltas to be current. Writes dL/dw[n][i] = delta[n] * layerInputs[i] and dL/db[n] = delta[n].

Gradients are assigned, not accumulated, so this class cannot express mini-batching as written.

void updateWeightsAndBiases(double learningRate)

Applies w -= learningRate * dL/dw and the same for biases. Plain SGD; no momentum, no weight decay.

void print() const

Writes the layer's geometry, pre-activations, activations, deltas, weights and biases to std::cout. For debugging.

#class core::HiddenLayer

class HiddenLayer : public Layer, inheriting the constructors via using Layer::Layer.

void computeErrorSignal(int nextLayerWidth, const double* nextLayerDeltas, const double* nextLayerWeights)

Computes this layer's error signal from the next layer's:

delta[n] = ( sum over m of nextDeltas[m] * nextWeights[m][n] ) * f'(z[n])

The three arguments describe the layer after this one. Because the next layer's fan-in equals this layer's width, nextLayerWeights is indexed as nextWeights[m * this->layerWidth + n] — note that the stride is this->layerWidth, not this->inputSize.

#class core::OutputLayer

class OutputLayer : public Layer, inheriting the constructors.

void computeErrorSignal(const double* __restrict__ target)

Computes

delta[n] = dL/da[n] * f'(z[n])

by calling the loss derivative into scratchPad1 and the activation derivative into scratchPad2, then multiplying elementwise.

target must point to at least layerWidth doubles. This is not checked.

Warning

this elementwise product cannot express softmax, whose Jacobian is a dense matrix. See Known Bugs, item 2.

#class core::Network

Defined in core/network.h. All members are public.

#Data Members

#Member Functions

Network()

Zero hidden layers, one output slot, learning rate 0.01. The layer vectors are left empty, so the slots must not be used.

Network(int numHiddenLayers, int numOutputLayers, double learningRate)

Resizes both layer vectors to the given counts. Every slot is a null pointer until filled.

void addHiddenLayer(int layerIndex, int width, int inputSize, ActivationFunction, ActivationFunctionDerivative, initFunction, LossFunction, LossFunctionDerivative)

Allocates a HiddenLayer with new and stores it at layerIndex.

Warning

layerIndex is not bounds-checked. An out-of-range index writes past the end of the vector.

void addOutputLayer(int layerIndex, ...)

As above, for the output layer. Same lack of bounds checking.

void initializeAllWeightsAndBiases()

Calls initializeWeightsAndBiases() on every slot.

Warning

dereferences every slot without a null check, so this crashes if any slot was never filled.

void forwardPass(const double* inputs)

Runs each hidden layer in order, then each output layer, feeding each layer's activations to the next. Read the result from outputLayers[0]->neuronOutputsActivated.

void backwardPass(std::vector<double>& targetOutputs)

Computes error signals and gradients for the output layer, then for the hidden layers from last to first, then applies the weight update to every layer. The argument is a non-const reference but is not modified.

void runEpoch(std::vector<double>& inputs, std::vector<double>& targetOutputs)

One forwardPass followed by one backwardPass. Despite the name this processes a single sample, not an epoch.

double computeLoss(const std::vector<double>& targetOutputs) const

Returns the sum of squared differences between the output activations and the targets.

Warning

this ignores the layer's configured loss function and always computes squared error. Training on binaryCrossEntropy and reading the loss from here reports a different quantity from the one being minimised. See Known Bugs, item 1.

void train(std::vector<std::vector<double>>& inputs, std::vector<std::vector<double>>& targetOutputs, int numEpochs)

Per-sample SGD. For each epoch, visits every sample in fixed order, calls runEpoch, accumulates computeLoss, and prints the mean to std::cout. Throws std::invalid_argument if the two containers differ in length; returns immediately if inputs is empty.

There is no way to suppress the per-epoch printing, no shuffling, and no return value. The Python binding reimplements this loop; see Network.train.

void printAllLayers()

Calls print() on every layer.

~Network()

deletes every layer in both vectors.

#Activation Functions

Namespace core::functions::activations. All are elementwise except softmax. Each derivative takes the pre-activation z.

softmax subtracts the row maximum before exponentiating, so it does not overflow. sigmoid does not branch on the sign of its argument and evaluates exp(-x) directly.

#Weight Initializers

Namespace core::functions::initializers.

double* zeros(double* pad, size_t size, size_t size2)

Sets size * size2 values to zero.

double* xavier(double* pad, size_t fanIn, size_t fanOut)

Normal distribution with standard deviation sqrt(2 / (fanIn + fanOut)). Suited to sigmoid and tanh.

double* he(double* pad, size_t fanIn, size_t fanOut)

Normal distribution with standard deviation sqrt(2 / fanIn). Suited to relu and its relatives.

std::mt19937& get_generator()

Returns the shared generator, a function-local static seeded once from std::random_device.

Warning

there is no way to set the seed, so no run is reproducible. See Known Bugs, item 8.

#Loss Functions

Namespace core::functions::loss. Remember the argument order: (actual, predicted, size).

crossEntropy is intended to pair with a softmax output. Because no softmax derivative exists, that pairing cannot currently be trained, and crossEntropyDerivative divides by the prediction — which approaches zero for a confidently wrong answer.

#Weight Storage Layout

neuronWeights is one flat array in row-major order. The weight connecting input i to neuron n is at:

neuronWeights[n * inputSize + i]

so one neuron's weights are contiguous. neuronWeightGradients uses the same layout.

This matters when reading another layer's weights. In HiddenLayer::computeErrorSignal the stride over the next layer's matrix is that layer's fan-in, which equals this layer's width:

nextLayerWeights[m * this->layerWidth + n]

Using inputSize there instead of layerWidth is a natural mistake and produces a network that trains plausibly and is silently wrong.

#Complete Example

A two-input, one-hidden-layer network trained on a linearly separable set.

#include <iostream>
#include <vector>
#include "core/network.h"

int main()
{
    namespace act  = core::functions::activations;
    namespace ini  = core::functions::initializers;
    namespace loss = core::functions::loss;

std::vector<std::vector<double>> X = {
        {0.1, 0.2}, {0.8, 0.9}, {0.2, 0.1}, {0.9, 0.7}
    };
    std::vector<std::vector<double>> Y = {
        {0.0}, {1.0}, {0.0}, {1.0}
    };

// 1 hidden layer, 1 output layer, learning rate 0.05
    core::Network net(1, 1, 0.05);

net.addHiddenLayer(0, /* width */ 4, /* inputSize */ 2,
                       act::tanh, act::tanhDerivative,
                       ini::xavier,
                       loss::mse, loss::mseDerivative);

net.addOutputLayer(0, /* width */ 1, /* inputSize */ 4,
                       act::sigmoid, act::sigmoidDerivative,
                       ini::xavier,
                       loss::mse, loss::mseDerivative);

net.initializeAllWeightsAndBiases();
    net.train(X, Y, 2000);

net.forwardPass(X[1].data());
    std::cout << "prediction: "
              << net.outputLayers[0]->neuronOutputsActivated[0]
              << "\n";
    return 0;
}

Two things to note. The output layer's inputSize must equal the hidden layer's width; nothing checks this. And the activation and its derivative are passed as two separate arguments, so an inconsistent pair such as act::sigmoid with act::reluDerivative compiles without complaint.

#Test Suite

./tests/run_tests.sh             # plain build
./tests/run_tests.sh --asan      # with AddressSanitizer and UBSan
./tests/run_tests.sh --asan Grad  # only tests whose name contains "Grad"

The suite covers core/layer.h and core/network.h separately and exits non-zero on failure.

Several tests fail by design. They are executable specifications for the bugs listed below, each reporting through FAIL_WITH_NOTE with the defect and the fix. A green run is the goal, not the current state.

#Known Bugs and Limitations

1. computeLoss ignores the configured loss. It always sums squared error, so the reported loss is not the quantity being minimised unless you configured mse.

2. Softmax cannot be trained. softmax has a forward implementation but no derivative. Its Jacobian is a dense matrix, which the scalar-in / scalar-out derivative signature cannot express. The correct fix is to fuse it with categorical cross entropy, where the chain collapses to dL/dz = p - y, and to add an entry point that accepts gradients already with respect to z.

3. Only the first output layer back-propagates. backwardPass reads outputLayers[0] when propagating into the hidden stack, so with numOutputLayers > 1 the other heads' gradients are silently discarded.

4. addHiddenLayer and addOutputLayer do not bounds-check layerIndex. An out-of-range index writes past the end of the layer vector.

5. initializeAllWeightsAndBiases dereferences unfilled slots. Constructing a Network with n hidden slots and filling fewer than n crashes.

6. The default Layer constructor leaves function pointers indeterminate. Layer() noexcept = default gives no member initialisers, so activationFunction and the rest hold garbage rather than null. Calling calculateOutputs() on such an object jumps to an arbitrary address. The fix is = nullptr initialisers on all five.

7. The scratch pads are the wrong size. Both are declared inputSize * width but only layerWidth elements are ever used. That overallocates by a factor of inputSize — and is simultaneously too short when inputSize is zero. They should be width.

8. No seed control. get_generator() seeds once from std::random_device and cannot be reseeded, so weight initialisation is never reproducible.

9. Hidden layers carry an unused loss function. Every Layer stores a loss and its derivative, but only the output layer uses them. The constructor still requires both.

10. Activations and their derivatives are independent arguments. Nothing checks that they match, so a mismatched pair compiles and trains incorrectly. The Python binding avoids this by selecting matched pairs by name.

11. Network is unsafe to copy. It owns raw pointers and has a destructor but no copy constructor or assignment operator.

12. Gradients are assigned, not accumulated. computeGradients overwrites, and backwardPass applies the update immediately, so mini-batching cannot be expressed without changing both.

#Not Supported

Optimizers other than plain SGD (no momentum, no Adam, no learning rate schedule), mini-batching, L2 weight decay, dropout, train/validation split, early stopping, per-epoch loss history, model save and load, a trainable softmax output, convolution, recurrence, attention, embeddings, skip connections, batch or layer normalization, sparse input, streaming datasets, GPU, threading, and SIMD. The inner loops are scalar double arithmetic.

See TODO.md in the repository root for measured performance work in priority order.