,

How Logistic Regression Works: Beginner to Pro Guide

·

· Updated

Layered data panels with a neural network and rising trend chart, machine learning hub cover

Logistic regression is a supervised machine learning model for classification: it predicts the probability that an example belongs to a class, such as spam or not spam, diabetic or not diabetic. Despite its name, it is a classifier. It is called “regression” because, inside, it computes a linear combination of the inputs, just like linear regression, and then turns that number into a probability.

Key takeaways

  • Logistic regression passes a linear score through the sigmoid function to get a probability between 0 and 1.
  • A threshold turns the probability into a class. The default is 0.5, but you can move it when false positives and false negatives have different costs.
  • Coefficients act on the log-odds, not directly on the probability.
  • The model is fitted by maximum likelihood estimation, solved iteratively by an optimiser such as L-BFGS.

Types of logistic regression

  • Binomial (binary) logistic regression: two outcomes, such as yes/no. It uses the sigmoid function to produce a probability between 0 and 1.
  • Multinomial logistic regression: three or more unordered classes, such as “bike”, “sedan” or “SUV”. It uses the softmax function to turn scores into probabilities that add up to 1.
  • Ordinal logistic regression: three or more ordered classes, such as “Low”, “Medium” and “High”. Instead of softmax, it models one underlying score with a set of cut-off thresholds (a cumulative logit), which respects the order of the classes.

Assumptions to check first

  1. Independent observations: each data point should stand on its own. Think of coworkers at a virtual meeting: no side chats or shared secrets.
  2. The right outcome type: binary for binomial logistic regression. With more than two classes (cat/dog/hamster), use the multinomial version with softmax.
  3. Linearity of the log-odds: each predictor should have a roughly straight-line relationship with the log-odds of the outcome.
  4. Watch out for extreme outliers: a few unusual points can pull the fitted curve noticeably.
  5. Enough data: small datasets give unstable coefficients and shaky predictions, especially with many predictors.

Why a hard boundary is not enough

If linear regression is the friendly neighbourhood line of best fit, logistic regression is its more decisive cousin who deals in zeros and ones. It answers yes/no questions like “Is this email spam?” or “Will this customer churn?” Instead of fitting a line through values, it fits probabilities bounded between 0 and 1. You could say it’s regression with commitment issues: it never promises more than 1.

Consider a medical example. Suppose a person has a blood sugar reading of 195 mg/dL, and you do not know whether they have diabetes. Would you classify them as diabetic or not?

Step-function decision boundary at a blood sugar level of 200, with red and blue markers
A hard decision boundary at a blood sugar level of 200 mg/dL, shown as a step function.

A simple rule says anyone at or above 200 mg/dL is diabetic. (200 mg/dL is a real diagnostic cut-off used by the American Diabetes Association for some glucose tests, though diagnosis depends on the test type and is made by a clinician.) But 195 mg/dL is very close to that line. This person could easily be a non-diabetic with slightly high blood sugar, and the data also includes people at 220 mg/dL who are not diabetic. A hard boundary can’t express that uncertainty. A probability can.

The sigmoid function and the threshold

The sigmoid function maps any real number to a value between 0 and 1, tracing a smooth S-shaped curve. Logistic regression uses it to turn a linear score into a probability.

To make a yes/no decision, we compare that probability with a threshold. The usual default is 0.5: above it, predict class 1; below it, class 0. In medicine or fraud detection you might lower the threshold to catch more true cases, at the cost of more false alarms.

Sigmoid curve of diabetes probability against blood sugar level, centered at 200
A sigmoid curve models the probability of diabetes smoothly, with a gradual transition from 0 to 1 instead of a sharp step.

The logistic regression equation

For one input, the probability of the outcome is:

P(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}

  • P(x) : probability of the outcome (for example, having diabetes) given the input x (for example, blood sugar level)
  • \beta_0 : the intercept
  • \beta_1 : the coefficient for x
  • e : the base of the natural logarithm

Rearranged, the same model says the log-odds are linear: \ln\frac{P}{1-P} = \beta_0 + \beta_1 x . So \beta_1 is the change in log-odds for a one-unit increase in x . It is not the change in probability, which depends on where you are on the S-curve.

Fitting the model: maximum likelihood

How do we find the best \beta_0 and \beta_1 ? Suppose we have 10 people, where people 5, 7, 8, 9 and 10 are diabetic. A good curve gives high probabilities to the diabetics and low probabilities to everyone else, so we want to make this product as large as possible:

(1 - P_1)(1 - P_2)(1 - P_3)(1 - P_4)(1 - P_6) \cdot P_5 P_7 P_8 P_9 P_{10}

Sigmoid curve showing diabetes probability by blood sugar level
A fitted sigmoid curve showing diabetes probability by blood sugar level.

In general form, this is the likelihood function:

\left[ \prod_{i \in \text{non-diabetics}} (1 - P_i) \right] \cdot \left[ \prod_{i \in \text{diabetics}} P_i \right]

Changing \beta_0 and \beta_1 reshapes the sigmoid curve, and at one combination the likelihood is maximised. Finding it is called maximum likelihood estimation (MLE). There is no closed-form answer, so libraries solve it iteratively: scikit-learn’s LogisticRegression uses the L-BFGS optimiser by default and also adds a small L2 penalty unless you turn it off.

Try it in Python

import numpy as np
from sklearn.linear_model import LogisticRegression

# Toy data: blood sugar (mg/dL) and diagnosis (1 = diabetic). Illustration only.
X = np.array([[120], [135], [150], [160], [175], [190], [205], [215], [230], [250]])
y = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])

model = LogisticRegression().fit(X, y)

print("Probability of diabetes at 195 mg/dL:", model.predict_proba([[195]])[0, 1].round(2))
print("Predicted class (0.5 threshold):", model.predict([[195]])[0])
print("Coefficient (log-odds per mg/dL):", model.coef_[0, 0].round(3))

On this toy data the model gives the 195 mg/dL person a probability of about 0.76, so at the default 0.5 threshold it predicts “diabetic”, but with honest uncertainty attached. This is a teaching example, not medical advice.

Frequently asked questions

Is logistic regression used for regression or classification?

Classification. Despite the name, it predicts the probability that an example belongs to a class, then applies a threshold to make a yes/no decision.

Should the threshold always be 0.5?

No. 0.5 is only the default. When missing a positive case is costly, as in medical screening or fraud detection, a lower threshold catches more positives at the cost of more false alarms.

How is logistic regression trained?

By maximum likelihood estimation: the coefficients are chosen so the predicted probabilities are as high as possible for the examples that really are positive and as low as possible for those that are not.

Sources

Want a step-by-step plan for learning machine learning?

The 90-Day Machine Learning Roadmap takes you from Python basics to a deployed ML portfolio in 13 weeks, with a plan for every day and a progress tracker. Going for your first ML or data science job? The ML Career Bundle adds an interview kit, Python practice and resume templates.

Prefer video? Watch the tutorials on YouTube · Browse all toolkits


About the author. Anish Rane is an Application Engineer working on diesel engines and generator sets, with a mechanical engineering background, an MSc in Machine Learning & AI and a Data Science & AI specialization from IIT Bombay. Anish writes about where engineering meets machine learning: engines, gears and the Python models that help explain them. All guides by Anish · More about Anish

This guide is for learning. Check calculations against current standards and have safety-critical designs reviewed by a qualified engineer.

Found this useful? Share it:

9 responses to “How Logistic Regression Works: Beginner to Pro Guide”

Leave a Reply

Keep reading