The essential algorithms and when to use them

Linear regression

Predicts a number as a weighted sum of features: ŷ = w₀ + w₁x₁ + … + wₙxₙ, fitted by minimising mean squared error. Fast and interpretable (each weight shows a feature's effect, holding others constant). Assumes a roughly linear relationship; sensitive to outliers and correlated features (multicollinearity).

Regularisation: Ridge (L2) shrinks weights smoothly; Lasso (L1) can shrink some to exactly zero (feature selection); Elastic Net mixes both.

Logistic regression

Classification despite the name: applies the sigmoid function to a linear score to get a probability between 0 and 1, then thresholds it (0.5 by default, tune for your costs). A strong, interpretable baseline for binary classification.

Decision trees

Split data with if/else questions that best separate the target (using Gini impurity or entropy for classification, variance for regression). Easy to explain, handle non-linear relationships and mixed feature types, no scaling needed. A single deep tree overfits badly.

Ensembles: random forests and gradient boosting

• Random forest (bagging): many trees, each trained on a bootstrap sample with random feature subsets; average their votes. Reduces variance; robust; few hyperparameters.
• Gradient boosting (XGBoost, LightGBM, CatBoost): trees built sequentially, each fixing the previous ones' errors. Usually the most accurate on tabular data; needs tuning (learning rate, depth, number of trees) and care against overfitting.

For structured/tabular business data, gradient boosting is the default strong model; deep learning dominates images, audio and text.

k-nearest neighbours, SVM and Naive Bayes

• kNN: predicts from the k most similar examples. Simple; slow at prediction time; needs feature scaling.
• SVM: finds the widest-margin boundary between classes; kernels handle non-linear boundaries. Good on small/medium data.
• Naive Bayes: probabilistic, assumes features independent; surprisingly good and fast for text classification (spam).

Unsupervised

• k-means: assigns points to k clusters by nearest centroid; choose k with the elbow method or silhouette score; needs scaling; assumes round clusters.
• Hierarchical clustering and DBSCAN (finds arbitrary shapes and marks noise).
• PCA: projects data onto the directions of greatest variance to reduce dimensions, for visualisation or speed.

A scikit-learn pipeline

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.ensemble import RandomForestClassifier

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

pre = ColumnTransformer([
    ("num", StandardScaler(), ["cgpa", "projects", "internships"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["branch", "city"]),
])
model = Pipeline([("pre", pre), ("clf", RandomForestClassifier(n_estimators=300, random_state=42))])

print(cross_val_score(model, X_train, y_train, cv=5, scoring="f1").mean())
model.fit(X_train, y_train)

The pipeline fits preprocessing inside each fold, which prevents leakage.

Check yourself

• Why is logistic regression a classification algorithm?
• Bagging vs boosting: what does each reduce?
• Which algorithms need feature scaling, and why?