Chapter 3 · Section 8 practice
Test-Set Accuracy and Honest Prediction Scope
Work from question 1 to 10. Edit your Python box, click Run Code, review the result, then click Submit attempt. Reference answers unlock after all ten attempts are submitted. This records completion, not correctness. Each box runs independently. Progress is saved in this browser.
1. Recognise overfitting
Warm-up. A simpler model has training RMSE s=0.4 and validation R²=0.7; a flexible model has training RMSE s=0.2 and validation R²=0.5. Print which model validation R² favours and explain why.
Write your answer, then run it.
2. Predict test rows
Warm-up. Print the first three test predictions.
Write your answer, then run it.
3. Read prediction errors
Warm-up. Print the first three test errors, observed minus predicted.
Write your answer, then run it.
4. Calculate test R-squared
Build your skills. Print test R-squared.
Write your answer, then run it.
5. Read prediction scope
Build your skills. Print why this CAPM test is conditional rather than a standalone forecast before market close.
Write your answer, then run it.
6. Compare training and test fit
Build your skills. Print training and test adjusted R-squared, then training and test RMSE in percentage points, using the course n-k-1 formula. Keep the training model fixed and round to four decimals.
Write your answer, then run it.
7. Check time separation
Build your skills. Print whether the last training date precedes the first test date.
Write your answer, then run it.
8. Check the fitted sample
Challenge. Print model.nobs and the number of training rows.
Write your answer, then run it.
9. Reconstruct RMSE s
Challenge. Calculate fitted CAPM RMSE s using n-2 and compare it with model.mse_resid.
Write your answer, then run it.
10. Interpret negative test R-squared
Challenge. If test R-squared is -0.2, print its interpretation.
Write your answer, then run it.
Reference answers are locked until all 10 attempts are submitted.
1. Recognise overfitting
simple_validation, flexible_validation = 0.7, 0.5
print(simple_validation > flexible_validation)
print(
"Validation favours the simple model. The flexible model fits training more closely but predicts held-out records worse."
)Expected output
True
Validation favours the simple model. The flexible model fits training more closely but predicts held-out records worse.These constructed values illustrate overfitting; validation is used for choice and a separate final test is still needed.
2. Predict test rows
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
p = model.predict(sm.add_constant(test[["Market"]]))
print(p.head(3).round(5))Expected output
Date
2024-08-08 0.05707
2024-08-09 0.01344
2024-08-12 0.00438
dtype: float64Coefficients remain fitted on training rows.
3. Read prediction errors
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
p = model.predict(sm.add_constant(test[["Market"]]))
print((test["Excess"] - p).head(3).round(5))Expected output
Date
2024-08-08 0.00400
2024-08-09 -0.01574
2024-08-12 0.03618
dtype: float64One observation can differ from its fitted mean.
4. Calculate test R-squared
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
p = model.predict(sm.add_constant(test[["Market"]]))
y = test["Excess"]
print(round(1 - ((y - p) ** 2).sum() / ((y - y.mean()) ** 2).sum(), 4))Expected output
0.4387The denominator uses test outcomes around their own mean for retrospective scoring.
5. Read prediction scope
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
print(
"The test uses realised market returns. A future prediction needs supplied or forecast market inputs."
)Expected output
The test uses realised market returns. A future prediction needs supplied or forecast market inputs.Known coefficients do not make future inputs known.
6. Compare training and test fit
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
import numpy as np
p = model.predict(sm.add_constant(test[["Market"]]))
y = test["Excess"]
n, k = len(test), 1
sse = ((y - p) ** 2).sum()
sst = ((y - y.mean()) ** 2).sum()
adj_test = 1 - (sse / (n - k - 1)) / (sst / (n - 1))
rmse_test = np.sqrt(sse / (n - k - 1))
print(round(model.rsquared_adj, 4), round(adj_test, 4))
print(round(np.sqrt(model.mse_resid) * 100, 4), round(rmse_test * 100, 4))Expected output
0.3295 0.433
2.6693 2.1504Higher adjusted R-squared and lower RMSE indicate better scores. Use the sample size of each set, with k=1, and never refit the test outcomes.
7. Check time separation
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
print(train.index.max() < test.index.min())Expected output
TrueChronological separation prevents future rows entering this fitted model.
8. Check the fitted sample
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
print(int(model.nobs), len(train))Expected output
400 400The model must only use training outcomes.
9. Reconstruct RMSE s
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
import numpy as np
n = len(train)
sse = (model.resid ** 2).sum()
print(round(np.sqrt(sse / (n - 2)), 6))
print(round(np.sqrt(model.mse_resid), 6))Expected output
0.026693
0.026693RMSE and residual standard error are the same s in this course.
10. Interpret negative test R-squared
import pandas as pd
prices = pd.read_csv(
"https://busanalytics-book.pages.dev/data/nvda_spy_daily_2023_2024.csv",
index_col="Date", parse_dates=True).sort_index()
factors = pd.read_csv(
"https://busanalytics-book.pages.dev/data/ff5_daily_2023_2024.csv",
index_col="Date", parse_dates=True) / 100
returns = prices.pct_change(fill_method=None)
data = returns.copy()
# Series columns match the Date index, not the row position.
data["Mkt-RF"] = factors["Mkt-RF"]
data["SMB"] = factors["SMB"]
data["HML"] = factors["HML"]
data["RMW"] = factors["RMW"]
data["CMA"] = factors["CMA"]
data["RF"] = factors["RF"]
# Keep complete dates for all model comparisons.
data = data.dropna()
data["Excess"] = data["NVDA"] - data["RF"]
data["Market"] = data["SPY"] - data["RF"]
cut = int(len(data) * 0.8)
train = data.iloc[:cut].copy()
test = data.iloc[cut:].copy()
import statsmodels.api as sm
X = sm.add_constant(train[["Market"]])
model = sm.OLS(train["Excess"], X).fit()
print(
"The predictions have larger squared error than the retrospective test-mean reference. The test mean was not an available forecast."
)Expected output
The predictions have larger squared error than the retrospective test-mean reference. The test mean was not an available forecast.A training OLS fit with an intercept does not guarantee nonnegative test R-squared.