Every exercise in 25_debugging_drill.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.
Drill D1.1. Codes that lose their leading zero
The task
Why it matters: the PSGC code is how the survey joins to every other table about places. A code that loses a digit matches nothing, without an error.
The code list names each province by a 10-digit code: the first five digits of its towns' codes, then five zeros. Ilocos Norte is 0102800000. with_province_code(path) should read the households file, keep psgc as the 10-digit text the codebook describes, and add each household's province_code in the same form.
Input → output: the path of households.csv → the households table with psgc as text and a province_code column.
| call | should give |
|---|---|
with_province_code(HOUSEHOLDS_CSV)["province_code"].head(3).tolist() |
['0102800000', '0102800000', '0102800000'] |
with_province_code(HOUSEHOLDS_CSV)["province_code"].isin(code_list["province_code"]).sum() |
1196: every household's province is in the code list |
Tools: reading with the right types (module 3, section 2 (20_dataframes.py)).
Predict. The first household's code in the file is 0102820000. What province code will the function give it?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
psgc has 9 digits for most households |
The column is still read as numbers. | read_csv's dtype= for psgc |
province_code is right, but psgc still has 9 digits |
The zeros were put back in the province code only. The task keeps psgc itself as 10-digit text. |
where psgc is read |
| AttributeError: Can only use .str accessor with string values | .astype(str) was removed while psgc is still a number. |
read psgc as text from the start |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: read_csv saw a column of digits and read it as whole numbers, and a number has no leading zero, so 0102820000 became 102820000. The province code was then cut from the wrong digits: 1028200000 instead of 0102800000.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def with_province_code(path) -> pd.DataFrame: """The households file, with psgc as 10-digit text and each household's province code beside it.""" # Drill D1.1: the task is in the text above this cell. Find the mistake and fix it. table = pd.read_csv(path, dtype={"psgc": str}) table["province_code"] = table["psgc"].astype(str).str[:5] + "00000" return tableWhy each part is there
dtype={"psgc": str}tellsread_csvto keep the column as text, exactly as the file writes it, so all ten digits stay.- The
.astype(str)on the next line was someone's answer to an earlier error:.strworks only on text, and pandas refused it on numbers. Turning the numbers into text came too late, after the zero was gone, so it turned an error into a wrong answer. - Only the codes that start with 0 were hurt: Bukidnon's (
10…) and Cotabato's (12…) came out right, which is why the check also tries a small file with a code of each kind.
Drill D1.2. `and` for `&`, and missing brackets
The task
Why it matters: almost every analysis filters rows on two conditions at once. The way pandas combines them is not the way Python's if does.
irrigated_rice(plots) should give the plots that grow rice and are irrigated: the rows whose crop is "Rice" and whose irrigated is "Yes".
Input → output: the plots table → the rows of irrigated rice plots, every column kept.
| call | should give |
|---|---|
len(irrigated_rice(plots)) |
651 |
Tools: filtering on two conditions (module 3, section 3 (20_dataframes.py)).
Predict. Will the function run? If it stops, what will the last line of the error say?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
| ValueError: The truth value of a Series is ambiguous | and (or or) between two columns of True and False. |
& and | |
| TypeError: operation 'rand_' not supported for dtype 'str' | & without brackets: Python combined "Rice" with a column before comparing. |
a pair of brackets around each comparison |
| Too many rows, rice or irrigated | | (or) where the task asks for both. |
& |
| No rows at all | A value spelled unlike the file: "rice", "yes". |
the values in plots["crop"] and plots["irrigated"] |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: and asks Python for one True or False about each side, and a whole column of them has no single answer, so pandas stopped with the truth value of a Series is ambiguous. Combining two columns of True and False row by row is &, and each comparison needs its own brackets.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def irrigated_rice(plots: pd.DataFrame) -> pd.DataFrame: """The plots that grow rice and are irrigated.""" # Drill D1.2: the task is in the text above this cell. Find the mistake and fix it. return plots[(plots["crop"] == "Rice") & (plots["irrigated"] == "Yes")]Why each part is there
&combines the two conditions row by row: a row is kept when it is rice and irrigated.- The brackets are needed because
&comes before==in Python's order of operations. Without them,plots["crop"] == "Rice" & plots["irrigated"] == "Yes"is read as"Rice" & plots["irrigated"]first, which pandas refuses with aTypeErrorabout'rand_'. - The same goes for
|(or) and~(not), which replaceorandnotin a pandas filter, as module 2 showed for NumPy masks.
Drill D1.3. Comparing with NaN
The task
Why it matters: before anyone reports fertilizer use, they need to know how many plots have no fertilizer recorded. A count of zero says the column is complete; if that is false, every average built on it is too.
In plots.csv, a plot without fertilizer recorded has its fertilizer_kg left blank, which pandas reads as NaN, "not a number". count_no_fertilizer(plots) should count those plots.
Input → output: the plots table → how many plots have no fertilizer_kg, as a whole number.
| call | should give |
|---|---|
count_no_fertilizer(plots) |
195 |
Tools: NaN (module 2, section 4 (10_arrays.py)).
Predict. The function compares each value with np.nan. What count will it give?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
0 |
A comparison with np.nan, with == or is: never true row by row. |
.isna() |
| One more than expected in the small table | The zero was counted too: fillna(0) == 0, or == 0. A plot given 0 kg has fertilizer recorded. |
count missing values only |
| The number of plots with fertilizer | .notna() where the task asks for the missing ones. |
.isna() |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: NaN is not equal to anything, not even to NaN, so == np.nan was False on every row and the count was 0, without an error. Finding missing values takes a test made for it, .isna().
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def count_no_fertilizer(plots: pd.DataFrame) -> int: """How many plots have no fertilizer recorded.""" # Drill D1.3: the task is in the text above this cell. Find the mistake and fix it. return int(plots["fertilizer_kg"].isna().sum())Why each part is there
.isna()gives True where a value is missing (NaN, and pandas' own<NA>), and False elsewhere;.sum()counts the Trues.int(...)hands back a plain Python whole number, as the task asks, instead of a NumPy one.- A count of exactly 0 from a column you know has blanks is the sign of a comparison that can never be true.
np.nan == np.nanis False, as module 2 showed: that is how NaN is defined, in NumPy and everywhere else.
Drill D1.4. An assignment that doesn't stick
The task
Why it matters: a sum of areas means something only if every area is in the same unit. This table will then say every plot is in hectares; if the numbers were not converted, it says something false about 151 plots.
to_hectares(plots) should give a copy of the plots with every area in hectares: an area in square meters divided by 10,000, and area_unit set to "ha" on every row. The table it is given stays as it was.
Input → output: the plots table → a copy, every area in hectares, every area_unit "ha".
| call | should give |
|---|---|
to_hectares(plots).loc[1, "area"] |
0.194: HH-0001's plot 2, 1,940 square meters |
to_hectares(plots)["area_unit"].unique().tolist() |
['ha'] |
to_hectares(plots)["area"].max() |
4.008 |
Tools: changing values under copy-on-write (module 3, section 4 (20_dataframes.py)).
Predict. HH-0001's second plot is 1,940 square meters. After the function, what will its area and its area_unit be?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
Areas of thousands of "ha" |
The new values went into a copy: two pairs of brackets in a row, table["area"][sqm] = .... |
table.loc[rows, column] = ... |
| Every area divided by 10,000, hectares too | The division is on the whole column, not the square-meter rows. | sqm on both sides |
Still thousands of "ha", with .loc in the line |
.loc picks the rows, then a second pair of brackets the column: table.loc[sqm]["area"] = ... is still a chained assignment. |
the rows and the column inside one .loc[...] |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: table["area"][sqm] = ... is a chained assignment: it sets values on table["area"], a column taken out of the table, and under copy-on-write that column behaves as a copy. The new values went into the copy, and table kept its square meters, while the next line labeled them all hectares.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def to_hectares(plots: pd.DataFrame) -> pd.DataFrame: """A copy of the plots with every area in hectares.""" table = plots.copy() sqm = table["area_unit"] == "sqm" # Drill D1.4: the task is in the text above this cell. Find the mistake and fix it. table.loc[sqm, "area"] = table.loc[sqm, "area"] / 10_000 table["area_unit"] = "ha" return tableWhy each part is there
table.loc[sqm, "area"] = ...names the rows and the column in one step ontableitself, so the values land in the table.- pandas said so: it gave a warning,
ChainedAssignmentError, whose first sentence is A value is being set on a copy of a DataFrame or Series through chained assignment. A warning does not stop the code, so the function went on and gave back a table that lies. table.assign(area=table["area"].where(~sqm, table["area"] / 10_000))is another way, making a new column instead of changing values in place; it gives the same table.
Drill D1.5. Numbers read as text
The task
Why it matters: sorting, summing and averaging only work on numbers. A column of text sorts and adds as text, without an error.
plots.csv writes large harvests with a thousands separator: "1,640". largest_harvests(path, n) should read the plots file and give the n largest harvests, in kilograms, largest first, as a list of whole numbers.
Input → output: the path of plots.csv and a count → a list of the n largest values of production_kg.
| call | should give |
|---|---|
largest_harvests(PLOTS_CSV, 3) |
[84190, 70780, 59090] |
Tools: reading with the right types (module 3, section 2 (20_dataframes.py)).
Predict. The file's first plot harvested "1,640" kg. Will the first harvest the function gives be bigger or smaller than that?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
['990', '990', '990'] |
The column is text, sorted letter by letter. | read_csv's thousands= |
| ValueError: invalid literal for int() | .astype(int) on texts that still have their commas. |
remove the commas first, or read them right |
| The smallest harvests | ascending=False was lost. |
sort_values |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: "1,640" is not a number to read_csv, because of the comma, so the whole column was read as text. Text sorts letter by letter, so "990" came before "84,190" (9 is after 8), and the "largest" harvests were 990 kg, as text.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def largest_harvests(path, n: int) -> list: """The n largest harvests in the plots file, in kilograms, largest first.""" # Drill D1.5: the task is in the text above this cell. Find the mistake and fix it. table = pd.read_csv(path, thousands=",") return table["production_kg"].sort_values(ascending=False).head(n).tolist()Why each part is there
thousands=","tellsread_csvthat a comma inside a number separates the thousands, so"1,640"is read as the whole number1640and the column isint64.- The column's type showed it at once:
str, not a number. Asking for a column'sdtypeis the quickest check after reading any file. - Removing the commas after reading,
table["production_kg"].str.replace(",", "").astype(int), also gives the right numbers; tellingread_csvis shorter, and it does it for every column.
Drill D1.6. Rows that no longer line up
The task
Why it matters: a report shows labels, not codes. A label put on the wrong household is worse than no label: nothing about it looks wrong.
head_sex is a code: 1 male, 2 female, 9 not stated. with_sex_label(households) should give the households it is given with a new column, head_sex_label, holding "male", "female" or "not stated", each on its own household's row. It is used on any part of the table: all the households, one province's, or the households in another order.
Input → output: a households table, or part of one → the same rows, with head_sex_label beside head_sex.
| call | should give |
|---|---|
with_sex_label(households)["head_sex_label"].value_counts().to_dict() |
{'male': 927, 'female': 245, 'not stated': 24} |
with_sex_label(households[households["province"] == "Isabela"])["head_sex_label"].isna().sum() |
0: every household labeled |
Tools: the index and alignment (module 3, section 1 (20_dataframes.py)).
Predict. Run Try it below, on all the households. Does each label match its code?
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
NaN labels on one province's rows |
The new Series is numbered 0, 1, 2, …, and those numbers are not the rows' index labels. | pd.Series(..., index=...), or .map on the column |
| Labels on the wrong households after sorting | The same: the values line up by position, but pandas lines them up by index label. | the index of the Series you assign |
.reset_index(drop=True) makes the labels right but the check is still red |
The table's own index was changed. The task keeps the rows as given. | give the labels the table's index instead |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
What was wrong: a new pd.Series gets the index 0, 1, 2, … whatever rows it came from, and pandas puts a column into a table by index label, not by position. On the whole table the labels happen to agree; on one province's rows (labels 170 to 339) or on sorted rows, each label went to the row whose index matched, or nowhere, which left NaN.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def with_sex_label(households: pd.DataFrame) -> pd.DataFrame: """The households, each with its head's sex as a label beside the code.""" # Drill D1.6: the task is in the text above this cell. Find the mistake and fix it. labels = pd.Series(["male" if code == 1 else "female" if code == 2 else "not stated" for code in households["head_sex"]], index=households.index) return households.assign(head_sex_label=labels)Why each part is there
index=households.indexgives the new Series the table's own index labels, so each label goes back to the household it was made from, in any part of the table and in any order.households["head_sex"].map({1: "male", 2: "female", 9: "not stated"})does the same in one step:.mapkeeps the index of the column it starts from.- Nothing raised an error, and on the whole table the answer was right. Only a check on a part of the table, or on the rows in another order, could find it: alignment is pandas' great help, until a Series carries an index from somewhere else.