Skip to content
Learn Python with BNHR

Debugging drill: a table that lies

On this page
  1. Overview
  2. D1.1 · Codes that lose their leading zero
  3. D1.2 · `and` for `&`, and missing brackets
  4. D1.3 · Comparing with NaN
  5. D1.4 · An assignment that doesn't stick
  6. D1.5 · Numbers read as text
  7. D1.6 · Rows that no longer line up

Every exercise in 25_debugging_drill.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.

Drill D1.1. Codes that lose their leading zero

The task

Why it matters: the PSGC code is how the survey joins to every other table about places. A code that loses a digit matches nothing, without an error.

The code list names each province by a 10-digit code: the first five digits of its towns' codes, then five zeros. Ilocos Norte is 0102800000. with_province_code(path) should read the households file, keep psgc as the 10-digit text the codebook describes, and add each household's province_code in the same form.

Input → output: the path of households.csv → the households table with psgc as text and a province_code column.

call should give
with_province_code(HOUSEHOLDS_CSV)["province_code"].head(3).tolist() ['0102800000', '0102800000', '0102800000']
with_province_code(HOUSEHOLDS_CSV)["province_code"].isin(code_list["province_code"]).sum() 1196: every household's province is in the code list

Tools: reading with the right types (module 3, section 2 (20_dataframes.py)).

Predict. The first household's code in the file is 0102820000. What province code will the function give it?

If the check is red: common causes

what you see the usual cause look at
psgc has 9 digits for most households The column is still read as numbers. read_csv's dtype= for psgc
province_code is right, but psgc still has 9 digits The zeros were put back in the province code only. The task keeps psgc itself as 10-digit text. where psgc is read
AttributeError: Can only use .str accessor with string values .astype(str) was removed while psgc is still a number. read psgc as text from the start

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Drill D1.2. `and` for `&`, and missing brackets

The task

Why it matters: almost every analysis filters rows on two conditions at once. The way pandas combines them is not the way Python's if does.

irrigated_rice(plots) should give the plots that grow rice and are irrigated: the rows whose crop is "Rice" and whose irrigated is "Yes".

Input → output: the plots table → the rows of irrigated rice plots, every column kept.

call should give
len(irrigated_rice(plots)) 651

Tools: filtering on two conditions (module 3, section 3 (20_dataframes.py)).

Predict. Will the function run? If it stops, what will the last line of the error say?

If the check is red: common causes

what you see the usual cause look at
ValueError: The truth value of a Series is ambiguous and (or or) between two columns of True and False. & and |
TypeError: operation 'rand_' not supported for dtype 'str' & without brackets: Python combined "Rice" with a column before comparing. a pair of brackets around each comparison
Too many rows, rice or irrigated | (or) where the task asks for both. &
No rows at all A value spelled unlike the file: "rice", "yes". the values in plots["crop"] and plots["irrigated"]

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Drill D1.3. Comparing with NaN

The task

Why it matters: before anyone reports fertilizer use, they need to know how many plots have no fertilizer recorded. A count of zero says the column is complete; if that is false, every average built on it is too.

In plots.csv, a plot without fertilizer recorded has its fertilizer_kg left blank, which pandas reads as NaN, "not a number". count_no_fertilizer(plots) should count those plots.

Input → output: the plots table → how many plots have no fertilizer_kg, as a whole number.

call should give
count_no_fertilizer(plots) 195

Tools: NaN (module 2, section 4 (10_arrays.py)).

Predict. The function compares each value with np.nan. What count will it give?

If the check is red: common causes

what you see the usual cause look at
0 A comparison with np.nan, with == or is: never true row by row. .isna()
One more than expected in the small table The zero was counted too: fillna(0) == 0, or == 0. A plot given 0 kg has fertilizer recorded. count missing values only
The number of plots with fertilizer .notna() where the task asks for the missing ones. .isna()

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Drill D1.4. An assignment that doesn't stick

The task

Why it matters: a sum of areas means something only if every area is in the same unit. This table will then say every plot is in hectares; if the numbers were not converted, it says something false about 151 plots.

to_hectares(plots) should give a copy of the plots with every area in hectares: an area in square meters divided by 10,000, and area_unit set to "ha" on every row. The table it is given stays as it was.

Input → output: the plots table → a copy, every area in hectares, every area_unit "ha".

call should give
to_hectares(plots).loc[1, "area"] 0.194: HH-0001's plot 2, 1,940 square meters
to_hectares(plots)["area_unit"].unique().tolist() ['ha']
to_hectares(plots)["area"].max() 4.008

Tools: changing values under copy-on-write (module 3, section 4 (20_dataframes.py)).

Predict. HH-0001's second plot is 1,940 square meters. After the function, what will its area and its area_unit be?

If the check is red: common causes

what you see the usual cause look at
Areas of thousands of "ha" The new values went into a copy: two pairs of brackets in a row, table["area"][sqm] = .... table.loc[rows, column] = ...
Every area divided by 10,000, hectares too The division is on the whole column, not the square-meter rows. sqm on both sides
Still thousands of "ha", with .loc in the line .loc picks the rows, then a second pair of brackets the column: table.loc[sqm]["area"] = ... is still a chained assignment. the rows and the column inside one .loc[...]

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Drill D1.5. Numbers read as text

The task

Why it matters: sorting, summing and averaging only work on numbers. A column of text sorts and adds as text, without an error.

plots.csv writes large harvests with a thousands separator: "1,640". largest_harvests(path, n) should read the plots file and give the n largest harvests, in kilograms, largest first, as a list of whole numbers.

Input → output: the path of plots.csv and a count → a list of the n largest values of production_kg.

call should give
largest_harvests(PLOTS_CSV, 3) [84190, 70780, 59090]

Tools: reading with the right types (module 3, section 2 (20_dataframes.py)).

Predict. The file's first plot harvested "1,640" kg. Will the first harvest the function gives be bigger or smaller than that?

If the check is red: common causes

what you see the usual cause look at
['990', '990', '990'] The column is text, sorted letter by letter. read_csv's thousands=
ValueError: invalid literal for int() .astype(int) on texts that still have their commas. remove the commas first, or read them right
The smallest harvests ascending=False was lost. sort_values

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Drill D1.6. Rows that no longer line up

The task

Why it matters: a report shows labels, not codes. A label put on the wrong household is worse than no label: nothing about it looks wrong.

head_sex is a code: 1 male, 2 female, 9 not stated. with_sex_label(households) should give the households it is given with a new column, head_sex_label, holding "male", "female" or "not stated", each on its own household's row. It is used on any part of the table: all the households, one province's, or the households in another order.

Input → output: a households table, or part of one → the same rows, with head_sex_label beside head_sex.

call should give
with_sex_label(households)["head_sex_label"].value_counts().to_dict() {'male': 927, 'female': 245, 'not stated': 24}
with_sex_label(households[households["province"] == "Isabela"])["head_sex_label"].isna().sum() 0: every household labeled

Tools: the index and alignment (module 3, section 1 (20_dataframes.py)).

Predict. Run Try it below, on all the households. Does each label match its code?

If the check is red: common causes

what you see the usual cause look at
NaN labels on one province's rows The new Series is numbered 0, 1, 2, …, and those numbers are not the rows' index labels. pd.Series(..., index=...), or .map on the column
Labels on the wrong households after sorting The same: the values line up by position, but pandas lines them up by index label. the index of the Series you assign
.reset_index(drop=True) makes the labels right but the check is still red The table's own index was changed. The task keeps the rows as given. give the labels the table's index instead

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Something new

Identify results
LayerExplore a module
ModuleDataFrames
AboutModule 3 of Arrays & DataFrames: Series and the index, reading files with the right types, selecting and sorting, and new columns under pandas 3's copy-on-write. Its sections, its notebooks to read on the web, and the companion's solutions and explanations.
CourseArrays & DataFrames, free