A progressive learning repository for understanding neural networks from first principles in Go. Each example builds upon the previous, introducing new concepts with educational comments throughout the code.
This repository includes:
- 13 progressive examples from basic predictions to convolutional networks
- Shared neural network package (
pkg/nn/) with reusable components - Multiple activation functions: ReLU, Sigmoid, Tanh, Softmax
- Advanced optimizers: SGD, SGD with Momentum, Adam, RMSprop
- Training utilities: Mini-batch processing, train/validation split, early stopping
- Model persistence: Save/load trained models
- Go 1.16+
- gonum for matrix operations
go mod download├── pkg/nn/ # Shared neural network package
│ ├── activations.go # ReLU, Sigmoid, Tanh, Softmax
│ ├── batch.go # Mini-batch iterator, train/val split
│ ├── convolution.go # 2D convolution, pooling, ConvLayer
│ ├── dropout.go # Dropout regularization
│ ├── initialization.go # Xavier, He weight initialization
│ ├── metrics.go # Training history, visualization
│ ├── network.go # Layer and Network abstractions
│ ├── optimizers.go # SGD, Momentum, Adam, RMSprop
│ └── persistence.go # Save/load models
├── 1-Basic/ # Single neuron prediction
├── 2-MultiInput/ # Multiple inputs, dot product
├── ...
└── 13-Convolution/ # Convolutional neural network
Single neuron prediction with one weight. Demonstrates the fundamental operation: output = input × weight.
go run ./1-Basic/Multiple inputs with weighted sum (dot product). Shows how networks combine multiple features.
go run ./2-MultiInput/Single input producing multiple outputs. Each output has its own weight.
go run ./3-MultiOutput/Full matrix multiplication: multiple inputs → multiple outputs. Foundation for hidden layers.
go run ./4-MultiInputOutput/Simple learning by trial and error. Tests weight adjustments up/down to find optimal values.
go run ./5-HoldColdLearning/Gradient-based learning using derivatives. Calculates delta = (prediction - goal) × input to determine weight updates.
go run ./6-Derivative/Full backpropagation through a 2-layer network with ReLU activation. Demonstrates the chain rule for gradient flow.
Key concepts:
- Forward pass: input → hidden → output
- Backward pass: propagate errors back through layers
- Weight updates using gradients
go run ./7-BackPropagation/Dropout regularization to prevent overfitting. Randomly deactivates neurons during training.
Features:
- Binary dropout mask (0s and 1s)
- Inverted dropout scaling
- Mask applied during backpropagation
go run ./8-DropOut/Comprehensive example showcasing all advanced features:
Features demonstrated:
- Mini-batch training (batch size 16)
- Train/validation split (80/20)
- Adam optimizer
- He/Xavier weight initialization
- Early stopping
- ASCII training curve visualization
Output includes:
- Per-epoch loss and accuracy
- Training summary
- ASCII plot of learning curves
- Final test on XOR problem
go run ./9-BatchGradientDescent/Neural network with sigmoid activation function.
Sigmoid properties:
- Output range: (0, 1)
- Derivative: σ(x) × (1 - σ(x))
- Good for binary classification output
- Can cause vanishing gradients in deep networks
go run ./10-SigmoidActiviation/Neural network with hyperbolic tangent activation.
Tanh properties:
- Output range: (-1, 1)
- Zero-centered (helps learning)
- Derivative: 1 - tanh²(x)
- Stronger gradients than sigmoid
go run ./11-TanhActiviation/Multi-class classification with softmax and cross-entropy loss.
Features:
- Softmax converts raw scores to probabilities
- Cross-entropy loss for classification
- One-hot encoded labels
- Outputs class probabilities that sum to 1
go run ./12-SoftmaxActiviation/Convolutional neural network for image pattern recognition.
Architecture:
- 4×4 input images
- 2×2 convolutional kernels (2 filters)
- Flatten layer
- Fully connected output with sigmoid
Features:
- 2D convolution operation
- Learnable kernels with backpropagation
- Displays learned kernel weights
go run ./13-Convolution/The pkg/nn package provides reusable components that eliminate code duplication:
import "github.com/robcost/go-neural-network/pkg/nn"
// Apply activation to data
output := nn.ReLU(input) // max(0, x)
output := nn.Sigmoid(input) // 1/(1+e^-x)
output := nn.Tanh(input) // (e^x - e^-x)/(e^x + e^-x)
output := nn.Softmax(input) // e^xi / Σe^xj
// Get derivatives
deriv := nn.ReLUDeriv(output)
deriv := nn.SigmoidDeriv(output)
deriv := nn.TanhDeriv(output)// He initialization (best for ReLU)
weights := nn.InitializeWeights(inputSize, outputSize, nn.InitHe)
// Xavier initialization (best for sigmoid/tanh)
weights := nn.InitializeWeights(inputSize, outputSize, nn.InitXavier)// Simple SGD
optimizer := nn.NewSGD(0.01)
// SGD with momentum
optimizer := nn.NewSGDMomentum(0.01, 0.9)
// Adam (recommended default)
optimizer := nn.NewAdam(0.001)
// Update weights
newWeight := optimizer.Update("param_id", currentWeight, gradient)// Create batch iterator
batchIter := nn.NewBatchIterator(inputs, targets, batchSize)
for epoch := 0; epoch < numEpochs; epoch++ {
batchIter.Shuffle() // Shuffle at start of each epoch
for batchIter.HasNext() {
batchInputs, batchTargets := batchIter.Next()
// Train on batch...
}
}trainX, trainY, valX, valY := nn.TrainValidationSplit(
inputs, targets,
0.2, // 20% validation
true, // shuffle
)history := nn.NewTrainingHistory()
// Record each epoch
history.AddEpoch(trainLoss, valLoss, trainAcc, valAcc)
// Check for overfitting
if history.IsOverfitting(patience) {
break // Early stopping
}
// Display results
history.PrintSummary()
history.PlotASCII(50, 10) // ASCII visualization// Save model
err := network.Save("model.gob")
// Load model
network, err := nn.Load("model.gob")He Initialization (for ReLU):
- ReLU zeros out negative values, effectively halving the variance
- He compensates by using √(2/fan_in) instead of √(1/fan_in)
Xavier Initialization (for sigmoid/tanh):
- Keeps variance consistent across layers
- Uses √(2/(fan_in + fan_out))
SGD: Simple but can be slow, gets stuck in local minima
Momentum: Accelerates in consistent gradient directions, dampens oscillations
Adam: Adapts learning rate per-parameter, generally works well out-of-the-box
- Training loss decreasing, validation loss increasing = overfitting
- Both decreasing together = model is learning generalizable patterns
- Use early stopping to halt training when validation loss stops improving
The following examples are planned to bridge the gap from CNNs to transformer-based neural networks:
- 14 - Embeddings: Lookup tables for token representation, converting discrete tokens to dense vectors
- 15 - Simple RNN: Recurrent neural network with hidden state for sequence processing
- 16 - LSTM: Long Short-Term Memory with gates (forget, input, output) and cell state for long-term dependencies
- 17 - Attention: Scaled dot-product attention with Query, Key, Value matrices
- 18 - Multi-Head Self-Attention: Parallel attention heads attending to same sequence
- 19 - Positional Encoding: Sinusoidal encoding to inject sequence position information
- 20 - Layer Normalization: Normalize across features (different from batch normalization)
- 21 - Transformer Block: Complete block with attention + FFN + residual connections
- 22 - Full Transformer: Encoder-decoder or decoder-only architecture
| Component | Key Formula | Purpose |
|---|---|---|
| Embedding | lookup[token_id] |
Convert tokens to vectors |
| RNN | h_t = tanh(W_hh·h_{t-1} + W_xh·x_t) |
Sequential processing |
| LSTM | Gates + cell state | Long-term memory |
| Attention | softmax(QK^T/√d_k)V |
Direct token-to-token connections |
| Positional Encoding | sin/cos(pos/10000^(2i/d)) |
Inject position information |
# Build and test all examples
for dir in */; do
if [[ -f "$dir/main.go" ]]; then
echo "Building $dir..."
go build -o /tmp/test "./$dir" || echo "Failed: $dir"
fi
doneThis is a learning repository. Feel free to:
- Add more examples
- Improve documentation
- Fix bugs
- Add new features to
pkg/nn
Apache License 2.0 - See LICENSE for details.
Inspired by "Grokking Deep Learning" by Andrew Trask and various neural network tutorials. Built as a learning exercise to understand ML fundamentals in Go.