Chapter 3 — Linear Regression

Correlation, CAPM, multiple regression, variable selection

Prof. Xuhu Wan

Chapter 3 · Introduction to Business Analytics

Linear Regression

Correlation, CAPM (simple regression), multiple regression, train/test, variable selection.

Prof. Xuhu Wan

ISOM, HKUST Business School · Wan Academy · 2026 Edition

Correlation — r vs r²

r ≈ 0.74 between NVDA and SPY daily returns.

Do not say “74 % of NVDA’s movement is explained by SPY.” That’s wrong.

Do say “r² = 0.55, so 55 % of NVDA’s variance is linearly explained by SPY.” The remaining 45 % is idiosyncratic.

The relationship between r and r² is the most-confused fact in introductory regression.

CAPM — α and β

The Capital Asset Pricing Model:

\[r_{\text{stock}} - r_f = \alpha + \beta\,(r_m - r_f) + \varepsilon\]

  • β (beta) — sensitivity to the market
    • β > 1 = aggressive (amplifies)
    • β < 1 = defensive (dampens)
  • α (Jensen’s alpha) — abnormal return after adjusting for market risk
    • α > 0 = outperforms benchmark
    • α ≈ 0 = efficient-market prediction
  • r_f — risk-free rate

We subtract r_f from both sides because CAPM models excess returns.

Fit CAPM with statsmodels

Important

sm.add_constant(X) is required — without it, statsmodels fits a model with no intercept. This is the single most common bug for analysts moving from R or Stata.

Reading the Output

The model.summary() table:

coef std err t P>|t| [0.025 0.975]
const (α) 0.0043 0.001 4.32 0.000 0.0024 0.0063
Mkt_excess (β) 2.221 0.103 21.65 0.000 2.020 2.422
R² = 0.542
  • β = 2.22 → NVDA moves ≈ 2.2 % per 1 % market move
  • 95 % CI for β = [2.02, 2.42] doesn’t contain 1 → significantly aggressive
  • p-values near zero → both α and β statistically nonzero
  • R² = 0.54 → market explains 54 % of NVDA’s daily variance

Variable Selection — AIC vs BIC

\[\text{AIC} = -2\ln L + 2k \qquad \text{BIC} = -2\ln L + k\ln n\]

Note

AIC penalty +2k is small → keeps more variables, optimised for forecasting.

BIC penalty +k ln n grows with sample size → keeps fewer variables, optimised for identifying the true model.

No criterion is simultaneously efficient and consistent — a fundamental statistical impossibility. Use AIC if you care about prediction; BIC if you care about which features are real.

Multiple Regression: Predicting Final-Exam Scores

Five candidate predictors of a final-exam score on a 100-point scale — three clearly useful, one with a small effect, one in genuine doubt.

  • +1 study hour → ≈ +1.8 pts · +1.0 GPA → ≈ +12 pts (controlling for the rest).
  • Extracurriculars come in negative and small — modest opportunity cost, not catastrophic.
  • Same one-line API as simple regression; just pass a list of predictor columns.

Worked Example: NBA Player Stats Predict Wins

Daryl Morey ran the Houston Rockets on regression-driven scouting — the analytics that fuelled the league’s three-point revolution.

  • Linear-probability model is a teaching tool; logistic regression is the right tool for binary outcomes.

Working with an AI Copilot

When you paste a model.summary() table into an AI and ask “is this good?”, it will almost always reply “p < 0.05 → significant” without checking whether the regression assumptions actually hold or whether your features leak future information.

  1. Always ask the AI to check LINE assumptions — Linearity, Independence, Normality of residuals, Equal variance (homoskedasticity).
  2. For “is X significant?”, also ask “what would change this conclusion?” — sample size, outliers, omitted variables.
  3. The AI cannot detect a data leak — that requires human judgment about how the data was constructed and what was knowable at prediction time.

Mistakes Library: LTCM (1998)

Warning

Long-Term Capital Management, founded 1994, hired Robert Merton and Myron Scholes — fresh Nobel laureates (1997) — as principals. The fund held $4.8B in equity supporting $129B in fixed-income arbitrage positions, all priced off models that assumed credit spreads were normally distributed.

In August 1998 Russia defaulted. Spreads moved 8+ standard deviations — a 1-in-10²⁰ event under their Gaussian model, i.e. effectively impossible. LTCM lost $4.6B in four months; the Federal Reserve coordinated a $3.6B bailout from 14 banks to prevent systemic collapse.

Lesson for regression: every regression coefficient, p-value, and confidence interval is conditional on the distributional assumptions you fed in. If the tails are fatter than you assumed, your “significant” result is a story, not a fact.

Decision Memo — Should the Team Sign This Player?

Regression output is not the deliverable — a defensible recommendation is. Translate the coefficients into a one-page memo.

To: General Manager, Eastern Conference team From: <Your name>, basketball analytics intern Subject: Sign free agent X to a 1-year prove-it deal Date: 2026-05-15

Recommendation: Sign X on a 1-year, performance-incentivised contract.

Evidence: - Linear-probability fit: +1 assist → +0.04 win probability (p < 0.01). - X averaged 6.2 assists/game last year vs. league median 3.8. - Turnover rate is league-average (β = −0.05, p = 0.03).

Caveats: - Linear-probability predictions outside [0,1] are common. - Assists are partially endogenous (better teammates → more assists). - 12-row sample is illustrative only.

Next step: Re-run with logistic regression on the full 82-game season; build a 20-game hold-out test.

Chapter Summary

Concept Tool
Correlation df.corr()
Regression sm.OLS(y, sm.add_constant(X)).fit()
Reading output .summary()
CI for β .conf_int()
Prediction .predict() / .get_prediction()
Variable selection AIC / BIC / Adj R² / Mallow’s Cp

Full treatment of CAPM (simple), the pharmacy multiple-regression case, residual diagnostics, and the student-performance multi-regression in the book — Chapter 3.

Next: Chapter 4 — Clustering.

Prof. Xuhu Wan · HKUST ISOM · Introduction to Business Analytics