ML from Scratch — CS401 Implementations
A collection of classic supervised-learning algorithms implemented from scratch (no ML frameworks) as a record of the CS401 Machine Learning + Neural Networks course. Every model is trained on a tiny toy dataset so the math is fully transparent: single-feature linear/logistic regression and two-feature (Hours studied, Hours slept) → Passed/Failed datasets for the classifiers.
The flagship model — a trained linear regression with exported weights — is
loadable as a HuggingFace model (model.safetensors + custom transformers
code). The rest are self-contained educational scripts.
Repository layout
| Path | Model / Algorithm | File(s) |
|---|---|---|
src/linear_regression.py |
Linear Regression (gradient descent) | training + MAE/RMSE eval |
src/normal_equation.py |
Linear Regression (closed form) | (XᵀX)⁻¹ Xᵀy |
src/LinearRegression.java |
Linear Regression (Java) | same algorithm, JVM |
notebooks/Foundation.ipynb |
Course Jupyter exercises | LR foundations |
logistic_regression/ |
Logistic Regression | sigmoid + log loss + GD |
linear_discriminant_analysis/ |
Linear Discriminant Analysis | closed-form S⁻¹(μ₁−μ₀) |
k_nearest_neighbors/ |
k-Nearest Neighbors | k=3, leave-one-out eval |
decision_tree/ |
Decision Tree | CART, Gini impurity |
random_forest/ |
Random Forest | 5 bagged trees, √p feature subsets |
support_vector_machines/ |
SVM (linear soft-margin) | hinge-loss subgradient |
perceptron/ |
Perceptron | update-on-mistake + logic gates |
config.json · model.safetensors |
Flagship HF model (LR weights) | loadable via transformers |
modeling_linear_regression.py · configuration_linear_regression.py |
custom model code | AutoModel.from_pretrained |
Flagship HuggingFace model — the trained linear regression
model.safetensors holds theta0 = 0.02888, theta1 = 2.99972 (float32),
produced by 2000 gradient-descent iterations on the 10-example
area→price dataset.
from transformers import AutoModel
model = AutoModel.from_pretrained(
"Eric-Yyvuw/ml-from-scratch", trust_remote_code=True
)
model.predict([150.0]) # tensor([450.0]) — 150 m² → ~450 €k
No-dependency numpy reader:
import json
from safetensors.torch import load_file
config = json.load(open("config.json"))
w = load_file("model.safetensors")
price = lambda x: float(w["theta0"]) + float(w["theta1"]) * x
print(price(150)) # ~450.0
Verified results
| Algorithm | Dataset | Result |
|---|---|---|
| Linear Regression | area (m²) → price (€k), n=10 | MAE 0.0072 · RMSE 0.0084 (h(x)=0.03+3.00x) |
| Logistic Regression | hours studied → passed, n=10 | 10/10 train accuracy; 3.2 h → P(pass)=0.442 |
| Linear Discriminant Analysis | (study, sleep) → passed | 10/10 train accuracy |
| k-Nearest Neighbors (k=3) | (study, sleep) → passed | 9/10 leave-one-out (boundary point misclassified) |
| Decision Tree (CART) | (study, sleep) → passed | 10/10; single split hours studied ≤ 3.50 |
| Random Forest | (study, sleep) → passed | 10/10 (5 trees, majority vote) |
| SVM (linear soft-margin) | (study, sleep) → passed | 10/10 train accuracy |
| Perceptron | (study, sleep) → passed | 10/10 (converges epoch 1) |
| Perceptron gates | AND / OR / XOR | 4/4, 4/4, 2/4 (XOR not linearly separable) |
The shared classifier dataset is deliberately linearly separable, so most models hit 100% training accuracy — the point is the algorithm, not the benchmark.
Run
Pure Python, standard library only (no numpy, no sklearn):
python3 src/linear_regression.py
python3 logistic_regression/logistic_regression.py
python3 k_nearest_neighbors/knn.py
python3 decision_tree/decision_tree.py
python3 random_forest/random_forest.py
python3 support_vector_machines/svm.py
python3 perceptron/perceptron.py
python3 linear_discriminant_analysis/linear_discriminant_analysis.py
Learning objectives
Understand how each algorithm works internally — hypothesis, cost function, gradient/closed-form parameter update, decision boundary, and evaluation — rather than treating models as black boxes. The notebooks and blog-style notes document the derivation for each one.
Limitations
- Educational only; toy datasets (10–14 examples), no train/test split for most classifiers (training accuracy reported) except kNN's leave-one-out.
- Gradient-descent convergence depends on learning rate, initialisation and feature scaling.
- Linear definitions cannot capture non-linear relationships (see the perceptron XOR result for the classic example).
- Predictions are unreliable outside the training-data distribution.
Acknowledgements
CS401 Machine Learning and Neural Networks, Dr. Cathy Ennis (crediting Dr. Ronan Reilly). Implementations and notes are original work done as part of the course.
- Downloads last month
- 2