Top 10 machine learning and AI interview questions

Prediction error comes from three sources: bias (error from overly simple assumptions), variance (error from sensitivity to the specific training data) and irreducible noise.

• A high-bias model underfits: a straight line through curved data; poor on training and test data.
• A high-variance model overfits: a very deep tree that memorises noise; excellent on training data, poor on test data.

Increasing complexity lowers bias and raises variance. The goal is the complexity that minimises validation error. Tools: regularisation, cross-validation, more data (reduces variance), ensembles (bagging reduces variance; boosting reduces bias), and better features (reduce bias).

Overfitting is when a model learns the training data's noise rather than the general pattern, so training performance is much better than validation performance.

Prevention:

• More (or augmented) training data.
• Simpler models or fewer features.
• Regularisation: L1/L2 penalties, dropout in neural networks, limiting tree depth or minimum samples per leaf.
• Early stopping when validation loss stops improving.
• Cross-validation for honest estimates and tuning.
• Ensembles like random forests.
• Removing leaky features.

Detect it with learning curves: a large, persistent gap between training and validation scores.

• Precision = TP / (TP + FP): of everything the model flagged positive, the fraction that truly is.
• Recall = TP / (TP + FN): of all actual positives, the fraction the model caught.

Prioritise recall when missing a positive is costly: disease screening, fraud detection, safety defects. Prioritise precision when a false alarm is costly: a spam filter hiding a genuine job offer, auto-rejecting candidates, blocking a customer's card.

They trade off through the decision threshold. F1 (their harmonic mean) summarises both; the PR curve shows the full trade-off. Accuracy is misleading on imbalanced data.

Both add a penalty on weight size to the loss function to reduce overfitting.

• L1 (Lasso): penalty = λ Σ|w|. Pushes some weights to exactly zero, giving sparse models and automatic feature selection. Can be unstable with highly correlated features (picks one arbitrarily).
• L2 (Ridge): penalty = λ Σw². Shrinks all weights smoothly towards zero but rarely to zero. Handles correlated features well and is more stable.
• Elastic Net combines both.

λ (the regularisation strength) is a hyperparameter tuned with cross-validation: too high underfits, too low overfits. In neural networks, L2 is called weight decay.

Random forest (bagging): train many decision trees, each on a bootstrap sample of the rows and a random subset of features at each split; average their predictions (or take a majority vote). The randomness decorrelates the trees, so averaging reduces variance. Trees are trained independently (parallelisable); it's robust and needs little tuning.

Gradient boosting (XGBoost, LightGBM, CatBoost): trains shallow trees sequentially, each fitting the residual errors (gradients of the loss) of the ensemble so far, scaled by a learning rate. It reduces bias and usually achieves higher accuracy on tabular data, but is more sensitive to hyperparameters and can overfit without early stopping.

Both give feature importances; use permutation importance or SHAP values for more trustworthy explanations.

Gradient descent minimises a loss function by repeatedly moving parameters a small step in the direction opposite to the gradient: w ← w − η∇L(w), where η is the learning rate.

• Batch: uses the whole dataset per step. Stable but slow and memory-heavy.
• Stochastic (SGD): one example per step. Fast and noisy; the noise can help escape poor local minima.
• Mini-batch: small batches (32–512). The standard in deep learning: efficient on GPUs with reasonably stable updates.

Learning rate: too high diverges or oscillates; too low is slow or gets stuck. Optimisers such as Momentum, RMSProp and Adam adapt step sizes; learning-rate schedules (warm-up, decay) improve training. Feature scaling makes gradient descent converge faster.

1. Use the right metrics: precision, recall, F1, PR-AUC, not accuracy.
2. Stratified splits so every fold has both classes.
3. Resampling on training data only: undersample the majority, oversample the minority, or SMOTE (synthetic minority examples).
4. Class weights: penalise mistakes on the minority class more (class_weight="balanced", scale_pos_weight in XGBoost).
5. Threshold tuning: pick the threshold that matches business costs.
6. Algorithms that cope well (tree ensembles) or anomaly detection when positives are extremely rare.
7. More minority data if it can be collected.

Always explain the business cost of each error type; that drives every choice above.

• MAE (mean absolute error): average size of errors in the target's units; easy to explain; robust to outliers.
• RMSE: square root of mean squared error; penalises large errors more heavily, so use it when big misses are especially bad.
• R²: proportion of variance explained relative to predicting the mean; can be misleading with few points or non-linear patterns.
• MAPE: average percentage error; intuitive for business, but undefined near zero and asymmetric.

Also: compare against a naive baseline (the mean, or last value for time series), plot residuals (patterns mean the model misses something), check performance across segments, and use time-based validation for forecasts.

A Transformer is a neural network architecture built on self-attention: for each token, it computes how much attention to pay to every other token in the sequence, using query, key and value vectors. Multiple attention heads capture different relationships; positional encodings give word order; feed-forward layers, residual connections and layer normalisation complete each block.

Why it replaced RNNs/LSTMs:

• Parallel training: all tokens are processed at once, while RNNs process sequentially, so Transformers scale to huge datasets on GPUs.
• Long-range dependencies: any two tokens connect directly instead of through many steps that weaken the signal.
• Transfer learning: pre-trained models (BERT for understanding, GPT-style decoders for generation) can be adapted to many tasks.

The trade-off: attention cost grows quadratically with sequence length, which has driven many efficiency techniques.

Retrieval-augmented generation answers questions using your own data:

1. Split documents into chunks and convert them to embeddings stored in a vector index.
2. Embed the user's question and retrieve the most similar chunks (often combined with keyword search and re-ranking).
3. Put those chunks in the prompt and instruct the LLM to answer only from them, with citations.

Use RAG when answers depend on specific, changing or private knowledge (policies, product docs, resumes): updating is as simple as re-indexing; answers can cite sources; hallucination drops.

Use fine-tuning to change behaviour, format, tone or a specialised skill, not to teach facts that change.

Mention evaluation (retrieval recall, answer faithfulness), chunk size trade-offs, access control on retrieved documents, and prompt-injection risks from retrieved content.