Debugging drill: a report that goes wrong¶

The barangay office asked for five figures from its files, and someone wrote a short script to give them. The script stops with an error, and once that is fixed, it gives four wrong figures out of five, without a word. This drill is that script, one function per figure, each with a mistake from modules 2 to 6: the kind that turns up when real files meet real code.

Work through it as you did Debugging drill D1: predict what a function gives, run it, trace the line that makes the difference, fix it and watch its check turn green, and say what the mistake was, in one sentence. Each check starts red.

The office worked the figures out by hand, so you know what the script should say:

figure the office's
Farmland 11.8 ha
Largest household H022, 10 members
Average household Malanday 4.5, Tumana 4.1, Concepcion Uno 5.1
Harvest reported 45.55 t
People in farming households 106
In [1]:
import csv

import course
from course import PATHS, check_cases, try_it

The script¶

read_rows reads a CSV file into rows, each a dictionary from column name to text (module 6, section 3). report gives the five figures, each from a function of its own, below. Neither of these two has a mistake.

In [2]:
def read_rows(path):
    """Every row of a CSV file, as a dictionary from column name to text."""
    with open(path, encoding="utf-8", newline="") as f:
        return list(csv.DictReader(f))


def report(path):
    """The office's five figures, from the households' file and the harvests' file beside it."""
    rows = read_rows(path)
    largest = largest_household(rows)
    averages = ", ".join(f"{name} {average_members(in_barangay(rows, name))}" for name in ["Malanday", "Tumana", "Concepcion Uno"])
    return [
        f"Farmland: {farmland_ha(rows)} ha",
        f"Largest household: {largest['id']}, {largest['members']} members",
        f"Average household: {averages}",
        f"Harvest reported: {total_harvest(path.parent / 'harvests.txt')} t",
        f"People in farming households: {farming_members(rows)}",
    ]


rows = read_rows(PATHS.data / "households.csv")
len(rows), rows[0]
Out[2]:
(30,
 {'id': 'H001',
  'barangay': 'Malanday',
  'members': '5',
  'head_age': '47',
  'farms': 'yes',
  'crop': 'Palay',
  'area_sqm': '5000',
  'harvest_kg': '2500'})

Why every open in the course says encoding="utf-8". A text file is stored as bytes, and its encoding says which bytes stand for which letter. Without encoding=, Python uses the computer's own: on macOS and Linux that is UTF-8, the encoding the course's files are saved in, but on Windows, before Python 3.15, it is usually another, and every letter outside plain English comes out wrong. This is what a barangay's name saved in UTF-8 looks like, read the Windows way:

In [3]:
"Parañaque".encode("utf-8").decode("cp1252")
Out[3]:
'Parañaque'

No error: just a wrong name, which then matches nothing. encoding="utf-8" reads the same letters on every computer. This mistake has no check below, because on a Mac or on Linux it does not show at all, which is exactly why it is worth writing every time.

D2.1 · Farmland¶

farmland_ha(rows) should add up every household's farm area, given in square metres in the column area_sqm, and give it in hectares (10,000 square metres), to two decimal places.

call should return
farmland_ha(rows) 11.8

Predict. Read the function, then the row shown above: will it run?

In [4]:
def farmland_ha(rows):
    """The farm area of every household, in hectares, to two decimal places."""
    total = 0
    for row in rows:
        total = total + int(row["area_m2"])
    return round(total / 10000, 2)

Run. The next cell runs it on the file's rows, as the report does, and stops.

In [5]:
farmland_ha(rows)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[5], line 1
----> 1 farmland_ha(rows)

Cell In[4], line 5, in farmland_ha(rows)
      1 def farmland_ha(rows):
      2     """The farm area of every household, in hectares, to two decimal places."""
      3     total = 0
      4     for row in rows:
----> 5         total = total + int(row["area_m2"])
      6     return round(total / 10000, 2)

KeyError: 'area_m2'

Trace. Read the error from the bottom up. Its last line, KeyError: 'area_m2', says a dictionary was asked for a key it does not have, and names the key. The line above it, in farmland_ha, is where. Each row's keys are the file's column names, so ask a row what they are:

In [6]:
list(rows[0])
Out[6]:
['id',
 'barangay',
 'members',
 'head_age',
 'farms',
 'crop',
 'area_sqm',
 'harvest_kg']
In [7]:
check_cases("Drill D2.1", farmland_ha, [("households.csv", (rows,)), ("its first three rows", (rows[:3],))], what="lists of rows",
            hint="Compare the key the error names with the file's column names.")
Out[7]:

Drill D2.1: not right yet. 0 of 2 lists of rows are right. The first one that is not: households.csv. Your code gave "KeyError: 'area_m2'". Compare the key the error names with the file's column names.

Still stuck after ten minutes? The solution and why it works are on the companion ↗.

If the check is red: common causes for Drill D2.1
  • KeyError with another name: copy the column's name from the file's first line, area_sqm, letter by letter.
  • TypeError: unsupported operand type(s) for +: 'int' and 'str': the value is text, as every value from a CSV file is. Keep int(...) around it.

D2.2 · The largest household¶

largest_household(rows) should give the row of the household with the most members.

call should return
largest_household(rows) H022's row: 10 members

Predict. Which household will Try it give?

In [8]:
def largest_household(rows):
    """The row of the household with the most members."""
    return max(rows, key=lambda row: row["members"])
In [9]:
try_it(largest_household, rows)
Out[9]:

largest_household([{'id': 'H001', 'barangay': 'Malanday', 'members': '5', 'he…) gave back {'id': 'H005', 'barangay': 'Concepcion Uno', 'members': '9', 'head_age': '52', 'farms': 'yes', 'crop': '', 'area_sqm': '6000', 'harvest_kg': '0'} (type dict).

Trace. It gives H005, with 9 members, and H022 has 10. max compared the members, so look at what it compared: what type is a row's members, and how do two texts compare?

In [10]:
type(rows[0]["members"]), "9" > "10", 9 > 10
Out[10]:
(str, True, False)
In [11]:
check_cases(
    "Drill D2.2", largest_household,
    [("households.csv", (rows,)), ("households of 10, 9 and 2 members", ([{"id": "A", "members": "10"}, {"id": "B", "members": "9"}, {"id": "C", "members": "2"}],)),
     ("households of 3 and 12 members", ([{"id": "D", "members": "3"}, {"id": "E", "members": "12"}],))],
    what="lists of rows", hint="What type are the values `max` compares?",
)
Out[11]:

Drill D2.2: not right yet. 0 of 3 lists of rows are right. The first one that is not: households.csv. Your code gave {'id': 'H005', 'barangay': 'Concepcion Uno', 'members': '9', 'head_age': '52', 'farms': 'yes', 'crop': '', 'area_sqm': …. What type are the values max compares?

Still stuck after ten minutes? The solution and why it works are on the companion ↗.

If the check is red: common causes for Drill D2.2
  • H005: the members are still compared as text. Turn each into a number in the key: int(row["members"]).
  • A number instead of a row: max(int(row["members"]) for row in rows) gives the highest number, 10, not the household. Keep the rows, and give max a key.

D2.3 · The average household, by barangay¶

The report gives each barangay's average household. in_barangay(rows, name) picks one barangay's rows, and has no mistake. average_members(group) should give the average number of members of the households in group, to one decimal place.

call should return
average_members(in_barangay(rows, "Malanday")) 4.5
average_members(in_barangay(rows, "Tumana")) 4.1

Predict. Malanday has 11 households with 50 members between them. What will Try it give?

In [12]:
def in_barangay(rows, name):
    """The rows of one barangay."""
    return [row for row in rows if row["barangay"] == name]


def average_members(group):
    """The average number of members of the households in group, to one decimal place."""
    total = 0
    for row in group:
        total = total + int(row["members"])
    return round(total / len(rows), 1)
In [13]:
try_it(average_members, in_barangay(rows, "Malanday"))
Out[13]:

average_members([{'id': 'H001', 'barangay': 'Malanday', 'members': '5', 'he…) gave back 1.7 (type float).

Trace. 1.7 is far too low for a barangay of households of four and five. Now try the function on the whole file, and compare the numbers of rows: it is right for the whole file and wrong for any part of it. So it must use something besides group. Read it for a name it was not given (module 5, section 2).

In [14]:
average_members(rows), len(rows), len(in_barangay(rows, "Malanday"))
Out[14]:
(4.6, 30, 11)
In [15]:
check_cases(
    "Drill D2.3", average_members,
    [(f'in_barangay(rows, "{name}")', (in_barangay(rows, name),)) for name in ["Malanday", "Tumana", "Concepcion Uno"]] + [("the whole file", (rows,))],
    what="groups of rows", hint="Right for the whole file? Then which name in the function is not `group`?",
)
Out[15]:

Drill D2.3: not right yet. 1 of 4 groups of rows are right. The first one that is not: in_barangay(rows, "Malanday"). Your code gave 1.7. Right for the whole file? Then which name in the function is not group?

Still stuck after ten minutes? The solution and why it works are on the companion ↗.

If the check is red: common causes for Drill D2.3
  • Right for the whole file, too low for a barangay: the function still divides by the number of all the rows, a global, instead of the households it was given, len(group).
  • ZeroDivisionError: the group is empty, because a barangay's name is spelt differently from the file's. Copy it from the file.

D2.4 · The harvest¶

harvests.txt has one harvest per line, in tonnes, and Missing where a farm did not report, as in Exercise 6.5. total_harvest(path) should add up the numbers, skip the Missing lines, and round the total to two decimal places.

call should return
total_harvest(PATHS.data / "harvests.txt") 45.55

Predict. Read the function: what will Try it give?

In [16]:
def total_harvest(path):
    """The harvests in the file added up, in tonnes, skipping the lines that are not numbers."""
    total = 0
    with open(path, encoding="utf-8") as f:
        for line in f:
            try:
                total = total + flaot(line)
            except:
                continue
    return round(total, 2)
In [17]:
try_it(total_harvest, PATHS.data / "harvests.txt")
Out[17]:

total_harvest('data/harvests.txt') gave back 0 (type int).

Trace. 0, and no error: every line was skipped, the numbers too. An except: with no name catches every error (module 6, section 4), so whatever went wrong on each line was skipped with it. Make it catch only the error it is there for, except ValueError:, run the function's cell and Try it again, and read what it was hiding.

In [18]:
check_cases("Drill D2.4", total_harvest, [("harvests.txt", (PATHS.data / "harvests.txt",))], what="files",
            hint="What does the `except:` catch, besides the `Missing` lines?")
Out[18]:

Drill D2.4: not right yet. 0 of 1 files are right. The first one that is not: harvests.txt. Your code gave 0. What does the except: catch, besides the Missing lines?

Still stuck after ten minutes? The solution and why it works are on the companion ↗.

If the check is red: common causes for Drill D2.4
  • 0 still: the bare except: is still hiding the error. Name the one to skip, except ValueError:, and read what shows up.
  • NameError: name 'flaot' is not defined: that is the hidden error, a misspelt name. It is float.
  • ValueError: could not convert string to float: 'Missing\n': the try is gone. Keep it, with except ValueError:.

D2.5 · People in farming households¶

farming_members(rows) should add up the members of the households that farm: those whose farms column says yes.

call should return
farming_members(rows) 106

Predict. Run Try it. Does its answer look like a number of people in farming households?

In [19]:
def farming_members(rows):
    """The number of people in the households that farm."""
    total = 0
    for row in rows:
        if row["farms"]:
            total = total + int(row["members"])
    return total
In [20]:
try_it(farming_members, rows)
Out[20]:

farming_members([{'id': 'H001', 'barangay': 'Malanday', 'members': '5', 'he…) gave back 137 (type int).

Trace. 137 is the number of people in all 30 households: every household was counted. if row["farms"]: asks whether the value counts as true. What is the value for H003, which does not farm, and does it count as true?

In [21]:
rows[2]["farms"], bool(rows[2]["farms"]), bool("")
Out[21]:
('no', True, False)
In [22]:
check_cases(
    "Drill D2.5", farming_members,
    [("households.csv", (rows,)), ("one household that does not farm", ([{"id": "H003", "members": "2", "farms": "no"}],)),
     ("one that farms and one that does not", ([{"id": "H001", "members": "5", "farms": "yes"}, {"id": "H004", "members": "1", "farms": "no"}],))],
    what="lists of rows", hint="What does `if` make of the text `\"no\"`?",
)
Out[22]:

Drill D2.5: not right yet. 0 of 3 lists of rows are right. The first one that is not: households.csv. Your code gave 137. What does if make of the text "no"?

Still stuck after ten minutes? The solution and why it works are on the companion ↗.

If the check is red: common causes for Drill D2.5
  • 137, everyone: any text but "" counts as true. Compare it: row["farms"] == "yes".
  • 0: the comparison is with a value the file does not have, such as "Yes" or True. The file says yes, in lower case.

The whole report¶

Run the script, and compare its figures with the office's at the top.

In [23]:
for line in report(PATHS.data / "households.csv"):
    print(line)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[23], line 1
----> 1 for line in report(PATHS.data / "households.csv"):
      2     print(line)

Cell In[2], line 13, in report(path)
      9     rows = read_rows(path)
     10     largest = largest_household(rows)
     11     averages = ", ".join(f"{name} {average_members(in_barangay(rows, name))}" for name in ["Malanday", "Tumana", "Concepcion Uno"])
     12     return [
---> 13         f"Farmland: {farmland_ha(rows)} ha",
     14         f"Largest household: {largest['id']}, {largest['members']} members",
     15         f"Average household: {averages}",
     16         f"Harvest reported: {total_harvest(path.parent / 'harvests.txt')} t",

Cell In[4], line 5, in farmland_ha(rows)
      1 def farmland_ha(rows):
      2     """The farm area of every household, in hectares, to two decimal places."""
      3     total = 0
      4     for row in rows:
----> 5         total = total + int(row["area_m2"])
      6     return round(total / 10000, 2)

KeyError: 'area_m2'

What the five mistakes have in common¶

One stopped with an error, which said where to look. The other four ran, and gave a wrong figure without a word: the wrong household, averages far too low, no harvest at all, and everyone counted as a farmer. Two of them came from the same place: a file gives text, and text does not behave like the numbers and yes-or-no answers it stands for. The third was a function reaching outside itself for a name it was not given, and the fourth an except that caught far more than it was there for. None of them would have been found without a figure known another way, here the office's, which is what a check is.

Next: Meet marimo, in 60_meet_marimo.py, opened with uv run marimo edit 60_meet_marimo.py. Or, if you want harder exercises first, the stretch exercises, in 81_stretch.ipynb.