Skip to content
Learn Python with BNHR

Capstone

On this page
  1. Overview
  2. K1.1 · Weighted yields by province, for one crop
  3. K1.2 · The months whose file never came
  4. K1.3 · The weighted totals, from the raw export

Every exercise in 70_capstone.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.

Exercise K1.1. Weighted yields by province, for one crop

The task

Why it matters: this is the number the extension office has never had: each province's yield for a crop, standing for all its farm plots, with the count of plots behind it so a figure from five plots is not read like one from two hundred.

Input → output: the plots, the households and a crop → one row per province where the crop is grown, the province as the index, with plots (how many of the crop's plots) and yield_t_ha (the weighted mean of the plots' yields, in tonnes per hectare, 2 decimals). Lowest yield first, so the provinces to look at come first; ties by province name.

A plot's yield is production_kg ÷ area_ha ÷ 1000. The weighted mean multiplies each plot's yield by its household's weight, adds those up, and divides by the sum of the weights (module 4, section 3 (30_summarize_and_combine.py)). On the farm survey, all rice plots together give 4.50 t/ha this way.

province plots yield_t_ha
Camarines Sur 161 …
… … …

If the check is red: common causes

what you see the usual cause look at
Aurora's rice is 4.5 in the small survey Unweighted: the mean of 3.0, 5.0 and 5.5. multiply by weight before adding up
Aurora's rice is 4.71 or 5.13 in the small survey Total harvest ÷ total area (unweighted or weighted), not the mean of the plots' yields. the definition above
One plot's yield is tiny area instead of area_ha. Step 2
Yields in the thousands Kilograms per hectare, not tonnes. / 1000
The right numbers in another order Not sorted lowest first, or ties not broken by province. sort_values(["yield_t_ha", "province"])
A province with no plots of the crop appears The households were the left side of the join. start from the crop's plots

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise K1.2. The months whose file never came

The task

Why it matters: a missing month is invisible in a combined table: there are simply no rows for it, and a line chart draws straight across. The analyst's report must say which months are missing, every time, without anyone looking.

Input → output: a list of file names (as [p.name for p in folder.iterdir()] gives them) → the months between the first and the last price file that have no file, as texts "YYYY-MM", in order. A price file is named exactly farmgate_YYYY-MM.csv; any other name in the folder (a README.md, a copy named farmgate_2024-01 (1).csv, a farmgate_2023-10_draft.csv) is not a price file and is ignored. No price files at all: an empty list.

file names missing months
farmgate_2024-11.csv, farmgate_2025-02.csv, farmgate_2024-12.csv ["2025-01"]
farmgate_2023-01.csv, farmgate_2023-04.csv ["2023-02", "2023-03"]

Tools: re.fullmatch(r"farmgate_(\d{4}-\d{2})\.csv", name) picks out the month (Python Essentials, text); pd.period_range(first, last, freq="M") lists every month between two (module 5, section 4 (40_cleaning_and_time.py)), and .strftime("%Y-%m") writes each as text.

If the check is red: common causes

what you see the usual cause look at
Wrong across the new year The months were not sorted, or were compared as numbers. sorted(...) of the texts, then period_range
Months before the first file are listed A draft or another file was taken for a price file. re.fullmatch, not re.search
ValueError or IndexError with no price files found[0] of an empty list. return [] first
Periods instead of texts period_range gives periods. .strftime("%Y-%m")
Every month is listed The comparison is between a period and a text. compare texts with texts

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise K1.3. The weighted totals, from the raw export

The task

Why it matters: this is the function the officer will defend on Friday. It must take the export exactly as it comes and give the totals the published table rounds, so that the only difference left is rounding.

Input → output: the raw households and plots (course.households(clean=False), course.plots(clean=False): production already numbers, every trap still there) → one row per province, the province as the index, with whole numbers:

column what it is
farm_households the sum of the households' weights
rice_area_ha the weighted total area of rice plots, in hectares
rice_production_t the weighted total rice harvest, in tonnes

Every province of the households has a row; a province with no rice plots shows 0 for rice (a weighted total of nothing is 0, not unknown).

Tools: drop_duplicates() (module 5, section 5 (40_cleaning_and_time.py)), one unit for area (module 5, section 2 (40_cleaning_and_time.py)), weighted totals (module 4, section 3 (30_summarize_and_combine.py)), merge(..., validate="many_to_one") (module 4, section 5 (30_summarize_and_combine.py)).

If the check is red: common causes

what you see the usual cause look at
Aurora has 700 farm households in the small export The duplicate household was not dropped. households.drop_duplicates()
MergeError: Merge keys are not unique in right dataset The same: a household still listed twice. Step 1 before the join
Rice area in the thousands Square meters added as hectares. Step 2
Aurora's rice area is 800 The duplicate plot was counted twice. plots.drop_duplicates()
Benguet is missing, or shows a missing value The rice totals have no row for a province with no rice. start the table from the households; fillna(0)
IntCastingNaNError astype(int) on a missing value. fillna(0) before astype(int)

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Something new

Identify results
LayerExplore a debugging drill
Debugging drill D1Debugging drill: five broken functions
AboutThe debugging drill of Python for Data Management, after module 3: five broken functions, each with one everyday mistake, to read, find, fix and explain in twenty minutes.
CoursePython for Data Management, free