ML from Scratch — CS401 Implementations

A collection of classic supervised-learning algorithms implemented from scratch (no ML frameworks) as a record of the CS401 Machine Learning + Neural Networks course. Every model is trained on a tiny toy dataset so the math is fully transparent: single-feature linear/logistic regression and two-feature (Hours studied, Hours slept) → Passed/Failed datasets for the classifiers.

The flagship model — a trained linear regression with exported weights — is loadable as a HuggingFace model (model.safetensors + custom transformers code). The rest are self-contained educational scripts.

Repository layout

Path Model / Algorithm File(s)
src/linear_regression.py Linear Regression (gradient descent) training + MAE/RMSE eval
src/normal_equation.py Linear Regression (closed form) (XᵀX)⁻¹ Xᵀy
src/LinearRegression.java Linear Regression (Java) same algorithm, JVM
notebooks/Foundation.ipynb Course Jupyter exercises LR foundations
logistic_regression/ Logistic Regression sigmoid + log loss + GD
linear_discriminant_analysis/ Linear Discriminant Analysis closed-form S⁻¹(μ₁−μ₀)
k_nearest_neighbors/ k-Nearest Neighbors k=3, leave-one-out eval
decision_tree/ Decision Tree CART, Gini impurity
random_forest/ Random Forest 5 bagged trees, √p feature subsets
support_vector_machines/ SVM (linear soft-margin) hinge-loss subgradient
perceptron/ Perceptron update-on-mistake + logic gates
config.json · model.safetensors Flagship HF model (LR weights) loadable via transformers
modeling_linear_regression.py · configuration_linear_regression.py custom model code AutoModel.from_pretrained

Flagship HuggingFace model — the trained linear regression

model.safetensors holds theta0 = 0.02888, theta1 = 2.99972 (float32), produced by 2000 gradient-descent iterations on the 10-example area→price dataset.

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Eric-Yyvuw/ml-from-scratch", trust_remote_code=True
)
model.predict([150.0])   # tensor([450.0]) — 150 m² → ~450 €k

No-dependency numpy reader:

import json
from safetensors.torch import load_file

config = json.load(open("config.json"))
w = load_file("model.safetensors")
price = lambda x: float(w["theta0"]) + float(w["theta1"]) * x
print(price(150))  # ~450.0

Verified results

Algorithm Dataset Result
Linear Regression area (m²) → price (€k), n=10 MAE 0.0072 · RMSE 0.0084 (h(x)=0.03+3.00x)
Logistic Regression hours studied → passed, n=10 10/10 train accuracy; 3.2 h → P(pass)=0.442
Linear Discriminant Analysis (study, sleep) → passed 10/10 train accuracy
k-Nearest Neighbors (k=3) (study, sleep) → passed 9/10 leave-one-out (boundary point misclassified)
Decision Tree (CART) (study, sleep) → passed 10/10; single split hours studied ≤ 3.50
Random Forest (study, sleep) → passed 10/10 (5 trees, majority vote)
SVM (linear soft-margin) (study, sleep) → passed 10/10 train accuracy
Perceptron (study, sleep) → passed 10/10 (converges epoch 1)
Perceptron gates AND / OR / XOR 4/4, 4/4, 2/4 (XOR not linearly separable)

The shared classifier dataset is deliberately linearly separable, so most models hit 100% training accuracy — the point is the algorithm, not the benchmark.

Run

Pure Python, standard library only (no numpy, no sklearn):

python3 src/linear_regression.py
python3 logistic_regression/logistic_regression.py
python3 k_nearest_neighbors/knn.py
python3 decision_tree/decision_tree.py
python3 random_forest/random_forest.py
python3 support_vector_machines/svm.py
python3 perceptron/perceptron.py
python3 linear_discriminant_analysis/linear_discriminant_analysis.py

Learning objectives

Understand how each algorithm works internally — hypothesis, cost function, gradient/closed-form parameter update, decision boundary, and evaluation — rather than treating models as black boxes. The notebooks and blog-style notes document the derivation for each one.

Limitations

  • Educational only; toy datasets (10–14 examples), no train/test split for most classifiers (training accuracy reported) except kNN's leave-one-out.
  • Gradient-descent convergence depends on learning rate, initialisation and feature scaling.
  • Linear definitions cannot capture non-linear relationships (see the perceptron XOR result for the classic example).
  • Predictions are unreliable outside the training-data distribution.

Acknowledgements

CS401 Machine Learning and Neural Networks, Dr. Cathy Ennis (crediting Dr. Ronan Reilly). Implementations and notes are original work done as part of the course.

Downloads last month
2
Safetensors
Model size
2 params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support