Quick answer: regularization prevents overfitting by adding a penalty on large coefficients to the model’s loss, controlled by alpha. Ridge (L2) shrinks all coefficients smoothly and suits many useful features. Lasso (L1) can push coefficients to exactly zero, so it also selects features. Elastic Net mixes both and handles groups of correlated features. Always scale features first, and choose alpha by cross-validation.
In the world of machine learning, a little humility goes a long way, especially for your models. That’s where regularization steps in. It is a technique that prevents overfitting, which happens when a model gets a little too enthusiastic about fitting the training data, capturing noise, quirks and coincidences instead of real patterns.
Regularization adds a penalty term to the model’s loss function. The penalty discourages complexity by shrinking large coefficients, which produces simpler models that generalise better to new data. In short, it’s like telling your model: “Don’t try so hard; focus on the big picture.”
Key takeaways
- Ridge (L2) shrinks every coefficient but rarely to zero. It is a good default when many features each matter a little or are correlated.
- Lasso (L1) can push coefficients to exactly zero, so it also selects features.
- Elastic Net mixes both penalties using
l1_ratio. - Alpha sets the penalty strength. Tune it with cross-validation, and always standardise features first, because penalties depend on coefficient size.
Before we meet the three techniques, let’s cover the three settings that decide how well they work: alpha, the learning rate and feature scaling. Whether you’re building a neural network or a simple linear model, these concepts are your compass.
Three settings that shape a regularized model
1. Alpha (regularization strength)
Applies to: Ridge, Lasso and ElasticNet. alpha controls how much penalty is added to the loss:
- Ridge:
Loss = RSS + alpha * (sum of squared coefficients) - Lasso:
Loss = RSS / (2n) + alpha * (sum of absolute coefficients) - ElasticNet: a weighted mix of both penalties, also with the 1/(2n) scaling on the error term
Note the 1/(2n) factor: scikit-learn scales the error term differently for Lasso and ElasticNet than for Ridge, so the same alpha value is not comparable between them. Tune alpha separately for each.
| Alpha value | Effect |
|---|---|
0 | No penalty: the same as ordinary least squares (in practice, use LinearRegression instead) |
| Small | Slight regularization, keeps flexibility |
| Large | Strong regularization, forces coefficients to shrink |

2. Learning rate (gradient-based optimisation)
Applies to: models trained with gradient descent, such as neural networks and SGDRegressor. The learning rate (η) sets the step size of each update.
| Learning rate | Effect |
|---|---|
| Too small | Slow convergence, long training time |
| Just right | Fast convergence to minimum loss |
| Too large | Overshooting, oscillation or divergence |
LinearRegression, Ridge, Lasso and ElasticNet in scikit-learn have no learning rate. If you want to tune one, use SGDRegressor.
3. Normalization and standardization
| Term | Meaning |
|---|---|
| Min-max normalization | Rescales each feature to a fixed range, usually [0, 1] (MinMaxScaler) |
| Standardization | Rescales each feature to mean 0 and standard deviation 1, the z-score (StandardScaler) |
- Features on different scales (height in cm vs income in dollars) distort the penalty, because regularization penalties depend on coefficient size.
- Gradient-based optimisers also converge better on standardised data.
- Fit the scaler on the training set only, then apply it to the test set, ideally inside a scikit-learn
Pipeline, so no test information leaks into training.
Ridge, Lasso and Elastic Net in scikit-learn
Now the regularization trifecta. Think of Ridge as the gentle nudger, Lasso as the hardliner that loves zeros, and Elastic Net as the diplomatic hybrid keeping everyone civil. All three examples below use the same synthetic dataset so you can compare their coefficients.
Ridge regression (L2 regularization)
- Adds
alpha * sum(coefficients²)to the loss. - Larger alpha shrinks all coefficients (rarely to exactly zero), which smooths the solution and reduces overfitting.
- Solver: Ridge has a closed-form solution, and scikit-learn solves it with methods such as Cholesky, SVD, LSQR or SAG (
solver="auto"picks one). There is no learning rate to tune. - Standardise first, because L2 penalises all coefficients equally.
from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler
# Step 1: Generate synthetic regression data
X, y = make_regression(n_samples=100, n_features=20, noise=10, random_state=42)
# Step 2: Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Step 3: Standardize the features (fit on training data only)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Step 4: Fit Ridge Regression model
model = Ridge(alpha=1.0) # alpha is the regularization strength
model.fit(X_train_scaled, y_train)
# Step 5: Predict and evaluate
y_pred = model.predict(X_test_scaled)
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)
print("Ridge Coefficients:", model.coef_)
Lasso regression (L1 regularization)
- Adds
alpha * sum(abs(coefficients))to the loss. - Encourages sparse models: some coefficients become exactly zero, a built-in feature-selection effect first described by Tibshirani (1996).
- Solver: coordinate descent, so again no learning rate. For a gradient-based version, use
SGDRegressor(loss="squared_error", penalty="l1"). - Scaling is very important: unequal scales decide which features get zeroed.
from sklearn.linear_model import Lasso
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler
X, y = make_regression(n_samples=100, n_features=20, noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
model = Lasso(alpha=0.1)
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("Mean Squared Error:", mean_squared_error(y_test, y_pred))
print("Lasso Coefficients:", model.coef_) # look for exact zeros
Elastic Net (L1 + L2 regularization)
alphacontrols the overall strength, andl1_ratiocontrols the mix:l1_ratio = 1.0is pure Lasso, and values between 0 and 1 blend both.- For a pure L2 penalty, use
Ridgerather thanl1_ratio = 0. scikit-learn warns that very smalll1_ratiovalues (0.01 or less) are not reliable in ElasticNet. - Elastic Net shines when you have correlated features (the Ridge part helps) and still want feature selection (the Lasso part helps).
- It also uses coordinate descent; for learning-rate tuning use
SGDRegressor(loss="squared_error", penalty="elasticnet").
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
X, y = make_regression(n_samples=100, n_features=20, noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
model = ElasticNet(alpha=0.1, l1_ratio=0.5) # 0.5 = equal L1 and L2 mix
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("Mean Squared Error:", mean_squared_error(y_test, y_pred))
print("ElasticNet Coefficients:", model.coef_)
Which one should you use?
| Situation | Start with | Why |
|---|---|---|
| Many features, most of them useful | Ridge | Shrinks everything a little and is stable with correlated inputs |
| You suspect only a few features matter | Lasso | Zeros out the rest and gives a simpler, more readable model |
| Groups of correlated features, and you want selection | Elastic Net | Lasso alone tends to pick one feature from a group at random |
| Not sure | Try all three with cross-validation | RidgeCV, LassoCV and ElasticNetCV tune alpha for you |
For the full tuning workflow (cross-validation, grid search and choosing a final model), see our linear regression hyperparameter tuning guide.
Frequently asked questions
What is the difference between Ridge and Lasso?
Ridge uses an L2 penalty (squared coefficients) and shrinks coefficients towards zero without removing them. Lasso uses an L1 penalty (absolute values) and can set some coefficients exactly to zero, which removes those features.
Why must I scale features before regularization?
The penalty acts on coefficient size, and coefficient size depends on each feature’s units. Without scaling, features measured in large units are penalised differently from those in small units, for no physical reason.
Can I compare the same alpha between Ridge and Lasso?
No. scikit-learn scales the error term differently for Lasso and Elastic Net than for Ridge, so tune alpha separately for each model with cross-validation.
Conclusion
Regularization is your model’s built-in self-control, reining in overenthusiastic fits. Whether it’s Ridge gently shrinking coefficients, Lasso wielding the zero-sword against irrelevant features, or Elastic Net balancing the two, these techniques help your model learn the signal, not the noise. Tune alpha like a thermostat for complexity, scale your data on the training set only, and you have a recipe for robust predictions. In machine learning, simpler often wins.
Sources
- scikit-learn. 1.1 Linear Models (Ridge, Lasso and Elastic Net objectives).
- scikit-learn. Ridge, Lasso and ElasticNet API documentation.
- scikit-learn. Common pitfalls and recommended practices (fit preprocessing on training data only).
- Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267-288.
Want a step-by-step plan for learning machine learning?
The 90-Day Machine Learning Roadmap takes you from Python basics to a deployed ML portfolio in 13 weeks, with a plan for every day and a progress tracker. Going for your first ML or data science job? The ML Career Bundle adds an interview kit, Python practice and resume templates.
Prefer video? Watch the tutorials on YouTube · Browse all toolkits




3 responses to “How Regularization Prevents Overfitting in ML (Simple Guide)”
[…] How Regularization Prevents Overfitting in ML […]
[…] How Regularization Prevents Overfitting in ML […]
[…] performance improvements, especially in real-world regression problems, your best bet lies in using regularized linear models like Ridge Regression, Lasso, or the ever-diplomatic Elastic Net. These models don’t just fit […]