,

How Regularization Prevents Overfitting in ML (Simple Guide)

·

· Updated

Generative design lattice bracket evolving into an optimised shape, CAD generative design cover

Quick answer: regularization prevents overfitting by adding a penalty on large coefficients to the model’s loss, controlled by alpha. Ridge (L2) shrinks all coefficients smoothly and suits many useful features. Lasso (L1) can push coefficients to exactly zero, so it also selects features. Elastic Net mixes both and handles groups of correlated features. Always scale features first, and choose alpha by cross-validation.

In the world of machine learning, a little humility goes a long way, especially for your models. That’s where regularization steps in. It is a technique that prevents overfitting, which happens when a model gets a little too enthusiastic about fitting the training data, capturing noise, quirks and coincidences instead of real patterns.

Regularization adds a penalty term to the model’s loss function. The penalty discourages complexity by shrinking large coefficients, which produces simpler models that generalise better to new data. In short, it’s like telling your model: “Don’t try so hard; focus on the big picture.”

Key takeaways

  • Ridge (L2) shrinks every coefficient but rarely to zero. It is a good default when many features each matter a little or are correlated.
  • Lasso (L1) can push coefficients to exactly zero, so it also selects features.
  • Elastic Net mixes both penalties using l1_ratio.
  • Alpha sets the penalty strength. Tune it with cross-validation, and always standardise features first, because penalties depend on coefficient size.

Before we meet the three techniques, let’s cover the three settings that decide how well they work: alpha, the learning rate and feature scaling. Whether you’re building a neural network or a simple linear model, these concepts are your compass.

Three settings that shape a regularized model

1. Alpha (regularization strength)

Applies to: Ridge, Lasso and ElasticNet. alpha controls how much penalty is added to the loss:

  • Ridge: Loss = RSS + alpha * (sum of squared coefficients)
  • Lasso: Loss = RSS / (2n) + alpha * (sum of absolute coefficients)
  • ElasticNet: a weighted mix of both penalties, also with the 1/(2n) scaling on the error term

Note the 1/(2n) factor: scikit-learn scales the error term differently for Lasso and ElasticNet than for Ridge, so the same alpha value is not comparable between them. Tune alpha separately for each.

Alpha valueEffect
0No penalty: the same as ordinary least squares (in practice, use LinearRegression instead)
SmallSlight regularization, keeps flexibility
LargeStrong regularization, forces coefficients to shrink
Regularization strength chart: model curves for low, medium and high alpha, showing how a higher alpha smooths the fit and reduces overfitting

2. Learning rate (gradient-based optimisation)

Applies to: models trained with gradient descent, such as neural networks and SGDRegressor. The learning rate (η) sets the step size of each update.

Learning rateEffect
Too smallSlow convergence, long training time
Just rightFast convergence to minimum loss
Too largeOvershooting, oscillation or divergence

LinearRegression, Ridge, Lasso and ElasticNet in scikit-learn have no learning rate. If you want to tune one, use SGDRegressor.

3. Normalization and standardization

TermMeaning
Min-max normalizationRescales each feature to a fixed range, usually [0, 1] (MinMaxScaler)
StandardizationRescales each feature to mean 0 and standard deviation 1, the z-score (StandardScaler)
  • Features on different scales (height in cm vs income in dollars) distort the penalty, because regularization penalties depend on coefficient size.
  • Gradient-based optimisers also converge better on standardised data.
  • Fit the scaler on the training set only, then apply it to the test set, ideally inside a scikit-learn Pipeline, so no test information leaks into training.

Ridge, Lasso and Elastic Net in scikit-learn

Now the regularization trifecta. Think of Ridge as the gentle nudger, Lasso as the hardliner that loves zeros, and Elastic Net as the diplomatic hybrid keeping everyone civil. All three examples below use the same synthetic dataset so you can compare their coefficients.

Ridge regression (L2 regularization)

  • Adds alpha * sum(coefficients²) to the loss.
  • Larger alpha shrinks all coefficients (rarely to exactly zero), which smooths the solution and reduces overfitting.
  • Solver: Ridge has a closed-form solution, and scikit-learn solves it with methods such as Cholesky, SVD, LSQR or SAG (solver="auto" picks one). There is no learning rate to tune.
  • Standardise first, because L2 penalises all coefficients equally.
from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler

# Step 1: Generate synthetic regression data
X, y = make_regression(n_samples=100, n_features=20, noise=10, random_state=42)

# Step 2: Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Step 3: Standardize the features (fit on training data only)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Step 4: Fit Ridge Regression model
model = Ridge(alpha=1.0)  # alpha is the regularization strength
model.fit(X_train_scaled, y_train)

# Step 5: Predict and evaluate
y_pred = model.predict(X_test_scaled)
mse = mean_squared_error(y_test, y_pred)

print("Mean Squared Error:", mse)
print("Ridge Coefficients:", model.coef_)

Lasso regression (L1 regularization)

  • Adds alpha * sum(abs(coefficients)) to the loss.
  • Encourages sparse models: some coefficients become exactly zero, a built-in feature-selection effect first described by Tibshirani (1996).
  • Solver: coordinate descent, so again no learning rate. For a gradient-based version, use SGDRegressor(loss="squared_error", penalty="l1").
  • Scaling is very important: unequal scales decide which features get zeroed.
from sklearn.linear_model import Lasso
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler

X, y = make_regression(n_samples=100, n_features=20, noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = Lasso(alpha=0.1)
model.fit(X_train_scaled, y_train)

y_pred = model.predict(X_test_scaled)
print("Mean Squared Error:", mean_squared_error(y_test, y_pred))
print("Lasso Coefficients:", model.coef_)  # look for exact zeros

Elastic Net (L1 + L2 regularization)

  • alpha controls the overall strength, and l1_ratio controls the mix: l1_ratio = 1.0 is pure Lasso, and values between 0 and 1 blend both.
  • For a pure L2 penalty, use Ridge rather than l1_ratio = 0. scikit-learn warns that very small l1_ratio values (0.01 or less) are not reliable in ElasticNet.
  • Elastic Net shines when you have correlated features (the Ridge part helps) and still want feature selection (the Lasso part helps).
  • It also uses coordinate descent; for learning-rate tuning use SGDRegressor(loss="squared_error", penalty="elasticnet").
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error

X, y = make_regression(n_samples=100, n_features=20, noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = ElasticNet(alpha=0.1, l1_ratio=0.5)  # 0.5 = equal L1 and L2 mix
model.fit(X_train_scaled, y_train)

y_pred = model.predict(X_test_scaled)
print("Mean Squared Error:", mean_squared_error(y_test, y_pred))
print("ElasticNet Coefficients:", model.coef_)

Which one should you use?

SituationStart withWhy
Many features, most of them usefulRidgeShrinks everything a little and is stable with correlated inputs
You suspect only a few features matterLassoZeros out the rest and gives a simpler, more readable model
Groups of correlated features, and you want selectionElastic NetLasso alone tends to pick one feature from a group at random
Not sureTry all three with cross-validationRidgeCV, LassoCV and ElasticNetCV tune alpha for you

For the full tuning workflow (cross-validation, grid search and choosing a final model), see our linear regression hyperparameter tuning guide.

Frequently asked questions

What is the difference between Ridge and Lasso?

Ridge uses an L2 penalty (squared coefficients) and shrinks coefficients towards zero without removing them. Lasso uses an L1 penalty (absolute values) and can set some coefficients exactly to zero, which removes those features.

Why must I scale features before regularization?

The penalty acts on coefficient size, and coefficient size depends on each feature’s units. Without scaling, features measured in large units are penalised differently from those in small units, for no physical reason.

Can I compare the same alpha between Ridge and Lasso?

No. scikit-learn scales the error term differently for Lasso and Elastic Net than for Ridge, so tune alpha separately for each model with cross-validation.

Conclusion

Regularization is your model’s built-in self-control, reining in overenthusiastic fits. Whether it’s Ridge gently shrinking coefficients, Lasso wielding the zero-sword against irrelevant features, or Elastic Net balancing the two, these techniques help your model learn the signal, not the noise. Tune alpha like a thermostat for complexity, scale your data on the training set only, and you have a recipe for robust predictions. In machine learning, simpler often wins.

Sources

Want a step-by-step plan for learning machine learning?

The 90-Day Machine Learning Roadmap takes you from Python basics to a deployed ML portfolio in 13 weeks, with a plan for every day and a progress tracker. Going for your first ML or data science job? The ML Career Bundle adds an interview kit, Python practice and resume templates.

Prefer video? Watch the tutorials on YouTube · Browse all toolkits


About the author. Anish Rane is an Application Engineer working on diesel engines and generator sets, with a mechanical engineering background, an MSc in Machine Learning & AI and a Data Science & AI specialization from IIT Bombay. Anish writes about where engineering meets machine learning: engines, gears and the Python models that help explain them. All guides by Anish · More about Anish

This guide is for learning. Check calculations against current standards and have safety-critical designs reviewed by a qualified engineer.

Found this useful? Share it:

3 responses to “How Regularization Prevents Overfitting in ML (Simple Guide)”

Leave a Reply

Keep reading