Chapter 3 · Section 9 practice
Extension: Pharmacy Profit and Model Comparison
Work from question 1 to 10. Edit your Python box, click Run Code, review the result, then click Submit attempt. Reference answers unlock after all ten attempts are submitted. This records completion, not correctness. Each box runs independently. Progress is saved in this browser.
1. Check the allocation
Warm-up. Print the numbers of training, validation and test cities, and whether all three index sets are disjoint.
Write your answer, then run it.
2. Count candidate subsets
Warm-up. Use combinations to count all nonempty subsets of the six inputs.
Write your answer, then run it.
3. Compare validation scores
Warm-up. Print full-model and selected-model validation adjusted R-squared and course RMSE s in USD.
Write your answer, then run it.
4. Inspect the training p-values
Build your skills. Print the name and p-value of the full model’s largest slope p-value.
Write your answer, then run it.
5. Read the explicit selected list
Build your skills. Print the selected predictor list and its training residual degrees of freedom.
Write your answer, then run it.
6. Rank every candidate
Build your skills. Enumerate all 63 subsets, score validation RMSE s, and print the best predictor list.
Write your answer, then run it.
7. Predict the proposed store
Build your skills. Print the selected training model’s predicted Profit for the supplied Kansas City inputs.
Write your answer, then run it.
8. Calculate test RMSE s
Challenge. Freeze the selected training model and calculate course RMSE s on the 22 test cities.
Write your answer, then run it.
9. Calculate test adjusted R-squared
Challenge. Calculate adjusted R-squared for the selected model’s held-out predictions.
Write your answer, then run it.
10. Compare an input while holding others fixed
Challenge. Print the fitted Profit difference for a one-percentage-point higher age-65+ share under the selected model.
Write your answer, then run it.
Reference answers are locked until all 10 attempts are submitted.
1. Check the allocation
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
print(len(train), len(validation), len(test))
print(
not (
set(train.index) & set(validation.index)
or set(train.index) & set(test.index)
or set(validation.index) & set(test.index)
)
)Expected output
66 22 22
TrueThe fixed cross-sectional allocation is 66/22/22; the proposed row is outside all three sets.
2. Count candidate subsets
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
from itertools import combinations
print(sum(len(list(combinations(features, k))) for k in range(1, 7)))Expected output
63Six possible inputs give 2**6−1=63 nonempty candidates, each with an intercept.
3. Compare validation scores
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
def score(fitted, rows, columns):
prediction = fitted.predict(sm.add_constant(rows[columns]))
errors = rows["Profit"] - prediction
n, k = len(rows), len(columns)
sse = (errors ** 2).sum()
sst = ((rows["Profit"] - rows["Profit"].mean()) ** 2).sum()
adjusted = 1 - (sse / (n - k - 1)) / (sst / (n - 1))
rmse_s = (sse / (n - k - 1)) ** 0.5
return adjusted, rmse_s
print(score(full, validation, features))
print(score(reduced, validation, short_features))Expected output
(0.30583892745886343, 17092.43347999681)
(0.45197458688865944, 15187.074372108971)Use the same validation cities and the correct k for each model.
4. Inspect the training p-values
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
slopes = full.pvalues.loc[features]
print(slopes.idxmax(), slopes.max())Expected output
Income 0.9936295235625769The intercept is excluded when identifying a slope. A large p-value is not proof of no relationship.
5. Read the explicit selected list
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
print(short_features)
print(int(reduced.df_resid))Expected output
['Disposable Income', 'Birth Rate (per 1,000)', '% 65 or Older']
62Three predictors plus an intercept leave 66−3−1=62 training residual degrees of freedom.
6. Rank every candidate
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
def score(fitted, rows, columns):
prediction = fitted.predict(sm.add_constant(rows[columns]))
errors = rows["Profit"] - prediction
n, k = len(rows), len(columns)
sse = (errors ** 2).sum()
sst = ((rows["Profit"] - rows["Profit"].mean()) ** 2).sum()
adjusted = 1 - (sse / (n - k - 1)) / (sst / (n - 1))
rmse_s = (sse / (n - k - 1)) ** 0.5
return adjusted, rmse_s
from itertools import combinations
rows = []
for k in range(1, len(features) + 1):
for subset in combinations(features, k):
columns = list(subset)
fitted = sm.OLS(train["Profit"], sm.add_constant(train[columns])).fit()
adjusted, rmse_s = score(fitted, validation, columns)
rows.append({"columns": columns, "k": k,
"Validation_adj_R2": adjusted,
"Validation_RMSE_s_USD": rmse_s})
results = pd.DataFrame(rows)
best = results.sort_values("Validation_RMSE_s_USD").iloc[0]
print(best["columns"])Expected output
['Disposable Income', 'Birth Rate (per 1,000)', '% 65 or Older']The validation ranking selects Disposable Income, Birth Rate and the age-65+ share without using final-test profits.
7. Predict the proposed store
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
new = pd.DataFrame(
{
"const": [1.0],
"Disposable Income": [22642],
"Birth Rate (per 1,000)": [14.4],
"% 65 or Older": [11.4],
}
)
print(reduced.predict(new).iloc[0])Expected output
184278.61459528247The supplied demographics allow a conditional prediction even though Profit is missing for this proposed store.
8. Calculate test RMSE s
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
prediction = reduced.predict(sm.add_constant(test[short_features]))
errors = test["Profit"] - prediction
n, k = len(test), len(short_features)
print(((errors ** 2).sum() / (n - k - 1)) ** 0.5)Expected output
15869.873570210166Use test outcomes only for this final evaluation, not to refit coefficients.
9. Calculate test adjusted R-squared
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
prediction = reduced.predict(sm.add_constant(test[short_features]))
sse = ((test["Profit"] - prediction) ** 2).sum()
sst = ((test["Profit"] - test["Profit"].mean()) ** 2).sum()
n, k = len(test), len(short_features)
print(1 - (sse / (n - k - 1)) / (sst / (n - 1)))Expected output
0.8095173911787292The score uses the mean and size of the same 22-city test sample and three predictors.
10. Compare an input while holding others fixed
import pandas as pd
import statsmodels.api as sm
raw = pd.read_csv("https://busanalytics-book.pages.dev/data/profit.csv")
features = [
"Income", "Disposable Income", "Birth Rate (per 1,000)",
"Soc Security (per 1,000)", "CV Death (per 100,000)", "% 65 or Older"
]
# The proposed Kansas City location has no observed Profit.
observed = raw.dropna(subset=["Profit"]).copy()
# These are different cities, not a time series.
shuffled = observed.sample(frac=1, random_state=42)
n = len(shuffled)
train = shuffled.iloc[:int(n * 0.6)].copy()
validation = shuffled.iloc[int(n * 0.6):int(n * 0.8)].copy()
test = shuffled.iloc[int(n * 0.8):].copy()
full = sm.OLS(train["Profit"], sm.add_constant(train[features])).fit()
short_features = ["Disposable Income", "Birth Rate (per 1,000)", "% 65 or Older"]
reduced = sm.OLS(train["Profit"], sm.add_constant(train[short_features])).fit()
print(reduced.params["% 65 or Older"])Expected output
5983.114243125648A slope applies to a one-unit increase in its own input unit. Here one unit is one percentage point.