Every exercise in 45_debugging_drill.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.
Drill D2.1. A mean of means
The task
Why it matters: the headline of a survey report is a figure for everyone the survey stands for. A figure that gives a small province the same say as a large one describes no one.
The report has a table of each province's weighted rice yield, made by province_yields(rice), below; that table is right. Under it, weighted_yield(rice) should give the headline: the weighted average yield of all the rice plots of the seven provinces, each plot's yield counted as many times as its weight says.
Input → output: the rice table → the weighted average of yield_t_ha, weighted by weight, in tonnes per hectare, to two decimal places.
| call | should give |
|---|---|
weighted_yield(rice) |
4.5 (the report's 4.50 t/ha) |
Tools: weighted means (module 4, section 3 (30_summarize_and_combine.py)).
Predict. The function averages the province table. Nueva Ecija has the highest yield and by far the largest weights. Will its answer be above or below the right one?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
4.25 |
The mean of the seven province yields: each province counts once. | the weighted mean over the plots |
4.35 |
The plain mean of the plots' yields: the weights are left out. | multiply by weight, divide by its sum |
| A number far above 5 | The weighted sum was divided by the number of plots, not by the sum of the weights. | rice["weight"].sum() |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: the function averaged the seven province yields, a mean of means, which gives every province the same say. A weighted mean over all the plots gives each plot the say its weight gives it, so Nueva Ecija, with the largest weights and the highest yields, pulls the figure up: 4.50, not 4.25.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def weighted_yield(rice: pd.DataFrame) -> float: """The weighted average rice yield of all the plots, in tonnes per hectare, to two decimal places.""" # Drill D2.1: the task is in the text above this cell. Find the mistake and fix it. return round((rice["yield_t_ha"] * rice["weight"]).sum() / rice["weight"].sum(), 2)Why each part is there
- The weighted mean is the sum of each yield times its weight, divided by the sum of the weights, over every plot at once:
(rice["yield_t_ha"] * rice["weight"]).sum() / rice["weight"].sum(). - Each province's figure in the table is right; only averaging them is wrong. An average of averages equals the overall average only when every group weighs the same, which the seven provinces don't: Nueva Ecija's rice plots carry weights adding up to 317,945, Ilocos Norte's to 43,227.
np.average(rice["yield_t_ha"], weights=rice["weight"])gives the same figure.
Drill D2.2. A join that multiplies rows
The task
Why it matters: a weighted total multiplies every plot by its household's weight. A plot counted twice adds its harvest twice, and nobody reading the total can tell.
rice_harvest_t(plots, households) should give the estimated rice harvest of all the farm households the survey stands for: each rice plot's production_kg times its household's weight, added up, in tonnes. The weights come from households with a join on hh_id.
Input → output: the plots and households tables → the weighted total rice harvest, in whole tonnes.
| call | should give |
|---|---|
rice_harvest_t(plots, households) |
3752516 |
Tools: joins and validate= (module 4, section 5 (30_summarize_and_combine.py)); duplicates (module 5, section 5 (40_cleaning_and_time.py)).
Predict. There are 1,034 rice plots. How many rows will the join give?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
| A total a little too big | Households in the table twice: their plots are joined, and counted, twice. | drop_duplicates() before the join |
| MergeError: Merge keys are not unique in right dataset; not a many-to-one merge | validate= found the households in twice, as it should. Now remove them. |
drop_duplicates() on the weights |
| KeyError: 'weight' | Only hh_id was kept from households, or the weights were renamed on the way. |
households[["hh_id", "weight"]] |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: 6 households are in households twice, so their hh_id matched two rows, and the join gave each of their rice plots twice: 1,037 rows from 1,034 plots. Each copy added its harvest times its weight again, 15,325 t too many.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def rice_harvest_t(plots: pd.DataFrame, households: pd.DataFrame) -> int: """The estimated rice harvest of all the farm households the survey stands for, in tonnes.""" rice_plots = plots[plots["crop"] == "Rice"] # Drill D2.2: the task is in the text above this cell. Find the mistake and fix it. joined = rice_plots.merge(households[["hh_id", "weight"]].drop_duplicates(), on="hh_id", how="left", validate="many_to_one") return round((joined["production_kg"] * joined["weight"]).sum() / 1000)Why each part is there
.drop_duplicates()on the weights keeps one row per household, so each plot finds exactly one weight.validate="many_to_one"writes down what the join must be: many plots to one household. Had it been there, the first run would have stopped with aMergeErrorthat names the households in twice, instead of quietly multiplying rows.- Comparing the number of rows before and after a join, as the trace did, finds this every time: a left join on a key that is unique on the right side keeps exactly as many rows as it started with.
Drill D2.3. Plots that drop out of the totals
The task
Why it matters: a table of totals by group should add up to the grand total. When it doesn't, a reader either loses trust in the table or, worse, doesn't notice.
The enumerators left crop blank for 14 plots. harvest_by_crop(plots) should give the harvest of the survey's plots by crop, in tonnes, with the plots whose crop was not recorded in a row of their own, its crop missing (NaN), so the rows add up to every plot's harvest.
Input → output: the plots table → one value per crop, in tonnes, plus one for the missing crop.
| crop | harvest (t) |
|---|---|
| Banana | 2,627.40 |
| Corn | 2,044.36 |
| Rice | 3,840.65 |
| Vegetables | 1,046.39 |
| NaN | 64.79 |
Tools: grouping and missing keys (module 4, section 1 (30_summarize_and_combine.py)).
Predict. How many rows will the function give?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
Four rows, no NaN |
groupby dropped the plots with no crop. |
groupby's dropna= |
A row named not recorded |
The blanks were filled with a label. The task keeps the missing key as NaN. |
dropna=False instead of fillna |
| Values in kilograms | The division by 1,000 was lost. | / 1000 |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: groupby leaves out the rows whose key is missing, unless told otherwise. The 14 plots with no crop recorded were in no group, so their 64.79 t vanished from the table, and its rows added up to 9,558.8 t instead of 9,623.59 t.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def harvest_by_crop(plots: pd.DataFrame) -> pd.Series: """The plots' harvest by crop, in tonnes, with the plots whose crop was not recorded in a row of their own.""" # Drill D2.3: the task is in the text above this cell. Find the mistake and fix it. return plots.groupby("crop", dropna=False)["production_kg"].sum() / 1000Why each part is there
dropna=Falsekeeps the missing key as a group of its own, shown asNaN, so every plot is in exactly one row.- Adding up the rows and comparing with the column's own total, as the trace did, is the quickest check on any table of totals.
- Filling the blanks first with a label,
plots["crop"].fillna("not recorded"), also keeps them, and reads better in a published table; the task here asks for the missing key itself.
Drill D2.4. Dates read month-first
The task
Why it matters: the survey's fieldwork ran from October to December 2025. A table that shows interviews in March describes fieldwork that never happened.
The file writes each interview date day first: 19/12/2025 is 19 December 2025, as the codebook says. interviews_by_month(dates) should turn the texts into dates and count the interviews in each month.
Input → output: the interview dates as text → the number of interviews in each month, months in order.
| call | should give |
|---|---|
interviews_by_month(interview_dates).to_dict() |
{Period('2025-10', 'M'): 381, Period('2025-11', 'M'): 384, Period('2025-12', 'M'): 425} |
Tools: parsing day-first dates, and periods (module 5, section 3 (40_cleaning_and_time.py)).
Predict. 09/11/2025 and 19/12/2025 are two of the dates. Which month will the function put each one in?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
| Months from January to December | The dates are read month first. | format= |
Still wrong with dayfirst=True |
A format= is given too, and the format wins. |
the order of %d and %m |
| ValueError: time data "…" doesn't match format | errors="coerce" was removed while the format is still wrong. The error names a date the format cannot read. |
format= |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: the format read the dates month first, as in the United States, but the file writes them day first. A date whose first number is above 12, such as 19/12/2025, has no 19th month, so it failed, and errors="coerce" turned it into a missing date (NaT) without a word; a date such as 09/11/2025 was read as 11 September. Only 446 of the 1,190 interviews were counted, spread over the whole year.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def interviews_by_month(dates: pd.Series) -> pd.Series: """How many interviews in each month, from the dates as the file writes them.""" # Drill D2.4: the task is in the text above this cell. Find the mistake and fix it. parsed = pd.to_datetime(dates, format="%d/%m/%Y", errors="coerce") return parsed.dt.to_period("M").value_counts().sort_index()Why each part is there
format="%d/%m/%Y"reads the day first, then the month, then the four-digit year, as the codebook says the file writes them.errors="coerce"hid the failures. It is safe only with a check after it: counting the missing dates (parsed.isna().sum()) would have shown 744 at once.- Months that the survey never worked in were the clue: a result that contradicts what you know about the data, here October to December, means the code is reading it wrong.
Drill D2.5. Square meters mixed with hectares
The task
Why it matters: a yield in tonnes per hectare needs every area in hectares. A few areas in another unit move the average without making it look strange.
mean_yield(plots, crop) should give the average yield of one crop's plots, unweighted: each plot's harvest in tonnes divided by its area in hectares, then the mean of those yields, to two decimal places. In plots, 151 areas are in square meters (area_unit is "sqm"); 10,000 square meters make a hectare.
Input → output: the plots table and a crop → the mean yield of that crop's plots, in tonnes per hectare.
| call | should give |
|---|---|
mean_yield(plots, "Rice") |
4.35 |
Tools: one unit for a measure (module 5, section 2 (40_cleaning_and_time.py)).
Predict. Run Try it. Does its answer look like a rice yield?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
4.1 for rice |
Areas in square meters are used as hectares. | area_unit, and divide those areas by 10,000 |
| Thousands of tonnes per hectare | Every area was divided by 10,000, hectares too. | divide only where area_unit is "sqm" |
0.0 or a tiny yield |
Areas multiplied by 10,000 instead of divided. | 10,000 square meters make 1 hectare |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: area holds each plot's area in its own unit, and 58 rice plots are in square meters. Divided by 1,940 instead of 0.194, their yields came out ten thousand times too small, close to 0, and pulled the mean down from 4.35 to 4.10: a figure that still looks like a rice yield.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def mean_yield(plots: pd.DataFrame, crop: str) -> float: """The average yield of one crop's plots, in tonnes per hectare, to two decimal places.""" crop_plots = plots[plots["crop"] == crop] # Drill D2.5: the task is in the text above this cell. Find the mistake and fix it. area_ha = crop_plots["area"].where(crop_plots["area_unit"] == "ha", crop_plots["area"] / 10_000) return round((crop_plots["production_kg"] / 1000 / area_ha).mean(), 2)Why each part is there
.where(crop_plots["area_unit"] == "ha", crop_plots["area"] / 10_000)keeps the areas already in hectares and divides the others by 10,000, so every area is in one unit before any yield is worked out.- The name
area_hapromised hectares, and nothing made it true. Naming a column after its unit helps only when the code checks the unit, here witharea_unit. - Nothing raised an error and the answer looked plausible. Only a figure known another way, the report's 4.35, or a check with a plot in square meters could show it: this is the kind of mistake checks are for.