Text, files and errors¶

Until now the barangay's records were typed into the notebook. Real records arrive as files, written by other people and other programs: names with stray spaces, numbers with a word where a number should be, a line that does not fit. This module reads them, cleans the text, and handles what does not fit, without the notebook falling over.

section what you do
1 · Working with text clean, split, join and test text with its methods
2 · Patterns check and find text by its shape, with regular expressions
3 · Reading and writing files text files line by line, and CSV files as records
4 · Errors and exceptions handle the line that does not fit, and stop with a clear message when you must

The files are in the course folder's data/ folder; course.PATHS.data is where it is.

In [1]:
import course
from course import PATHS, check_cases, try_it

PATHS.data, sorted(p.name for p in PATHS.data.iterdir())
Out[1]:
(PosixPath('~/essentials/data'),
 ['brgy_population.txt', 'harvests.txt', 'households.csv'])
In [2]:
course.quizzes_at("6.0")

1 · Working with text¶

Text (a str) has methods, called with a dot, which give back a new text: text never changes in place.

| method | gives | example | |:--|:--|:--| | .strip() | the text without spaces at both ends | " Tumana ".strip() is "Tumana" | | .lower(), .upper(), .title() | the text in another case | "tumana".title() is "Tumana" | | .replace(old, new) | every old replaced | "Brgy. 143".replace("Brgy.", "Barangay") | | .split(sep) | a list of the parts between sep | "NCR|Tumana|880".split("|") | | .split() | the words, however many spaces between them | "Barangay 143".split() | | sep.join(parts) | the parts joined by sep | " ".join(["Barangay", "143"]) | | .startswith(x), .endswith(x) | True or False | "# a comment".startswith("#") | | .isdigit() | whether every character is a digit | "0128".isdigit() |

In [3]:
raw = "   concepcion    UNO  "
raw.strip(), raw.split(), " ".join(raw.split()).title()
Out[3]:
('concepcion    UNO', ['concepcion', 'UNO'], 'Concepcion Uno')

Formatting with f-strings you know from module 2. Two more things worth knowing: len(text) counts characters, and text[0], text[-3:] take characters and slices, as with a list.

Exercise 6.1 · Clean a barangay's name¶

Names typed by hand come with stray spaces and mixed case. Finish clean_name(text) so that it returns the name with no spaces at the ends, a single space between words, and each word capitalised.

call returns
clean_name(" concepcion UNO ") "Concepcion Uno"
clean_name("malanday") "Malanday"
In [4]:
def clean_name(text):
    # Step 1: split the text into its words (.split() with no input ignores any run of spaces)
    # Step 2: join the words with a single space, and capitalise each word (.title())
    raise NotImplementedError("Exercise 6.1")  # delete this line when your code is ready
In [5]:
try_it(clean_name, "   concepcion    UNO  ")
Out[5]:

clean_name(' concepcion UNO '): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.

In [6]:
check_cases("Exercise 6.1", clean_name, [(repr(t), (t,)) for t in ["   concepcion    UNO  ", "malanday", "TUMANA", " barangay 143 ", ""]],
            what="names", hint="`.split()`, then `\" \".join(...)`, then `.title()`.")
Out[6]:

Exercise 6.1: not attempted yet. .split(), then " ".join(...), then .title().

If the check is red: common causes for Exercise 6.1
  • Several spaces between words: .strip() only removes spaces at the ends. Split the words and join them with one space.
  • "Concepcion UNO": .capitalize() changes only the first letter of the whole text. Use .title().

2 · Patterns¶

Some checks are about a text's shape: a mobile number is 09 and nine more digits. A regular expression (regex) describes such a shape, and the re module tests text against it.

in a pattern means
\d one digit
\d{9} exactly nine digits
[A-Z] one capital letter
? the thing before it, or nothing
\| either side (09\|\+639)
( … ) a group, to use with \| or to take out

re.fullmatch(pattern, text) checks the whole text against the pattern, and gives a match or None. re.findall(pattern, text) gives every part of the text that fits. Write patterns as raw strings, r"...", so the backslashes stay as they are.

In [7]:
import re

bool(re.fullmatch(r"09\d{9}", "09171234567")), bool(re.fullmatch(r"09\d{9}", "0917-123-4567")), re.findall(r"\d{4}", "rounds 2023 and 2024")
Out[7]:
(True, False, ['2023', '2024'])

Exercise 6.2 · A valid mobile number¶

A Philippine mobile number is written 09 and nine digits, or +639 and nine digits. Finish is_mobile(text) so that it returns True when the whole text is a mobile number in one of those forms, and False otherwise.

call returns
is_mobile("09171234567") True
is_mobile("+639171234567") True
is_mobile("0917-123-4567") False
In [8]:
def is_mobile(text):
    # Step 1: write a pattern: (09 or +639), then nine digits; the + needs a backslash
    # Step 2: return whether re.fullmatch finds the pattern in the whole text (bool(...))
    raise NotImplementedError("Exercise 6.2")  # delete this line when your code is ready
In [9]:
try_it(is_mobile, "09171234567")
Out[9]:

is_mobile('09171234567'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.

In [10]:
check_cases("Exercise 6.2", is_mobile,
            [(repr(t), (t,)) for t in ["09171234567", "+639171234567", "0917-123-4567", "0917123456", "091712345678", "639171234567", "call 09171234567", "+63 917 123 4567"]],
            what="numbers", hint="`re.fullmatch(r\"(09|\\+639)\\d{9}\", text)`.")
Out[10]:

Exercise 6.2: not attempted yet. re.fullmatch(r"(09|\+639)\d{9}", text).

If the check is red: common causes for Exercise 6.2
  • True for a number inside other text: re.search or re.match look at only part of the text. Use re.fullmatch.
  • False for +639…: the + needs a backslash, \+.
  • A match object instead of True: return bool(...) of it.

3 · Reading and writing files¶

open(path) opens a file; with open(path, encoding="utf-8") as f: opens it for the lines indented under it, and closes it after, even when something goes wrong. Looping over the file gives its lines, each with the \n at its end (which .strip() removes). Always give encoding="utf-8": without it, Windows may read the file in another encoding, and names with ñ come out wrong.

In [11]:
with open(PATHS.data / "brgy_population.txt", encoding="utf-8") as f:
    for line in f:
        print(repr(line))
'# Population by barangay, as sent by the city office (2020)\n'
'# region | barangay | population\n'
'\n'
'NCR|   Malanday | 1401\n'
'NCR|Tumana| 880\n'
'   NCR| Concepcion Uno| 1029\n'
'NCR | Barangay 143 |2210\n'
'NCR|Barangay 144 | 1655\n'
'\n'
'NCR| Barangay 145|  975\n'
'# End of report\n'

The file is messy in ways files often are: comment lines starting with #, blank lines, and spaces around the | separators. Each line that matters is region, barangay and population.

Writing uses open(path, "w", encoding="utf-8") and f.write(text). "w" replaces whatever the file held; "a" adds to its end. write does not add a line break: put \n at the end of each line.

In [12]:
report = PATHS.root / "report.txt"
with open(report, "w", encoding="utf-8") as f:
    f.write("Malanday: 1,401 people\n")
    f.write("Tumana: 880 people\n")
report.read_text(encoding="utf-8")
Out[12]:
'Malanday: 1,401 people\nTumana: 880 people\n'

PATHS.data / "brgy_population.txt" joins folder and file names into a path (with pathlib), which works on Windows and on a Mac alike. A path can read a whole file at once with .read_text(encoding="utf-8").

Exercise 6.3 · Population by barangay¶

Finish read_population(path) so that it reads a report like brgy_population.txt and returns a dictionary of each barangay's population, as a whole number. Skip blank lines and lines starting with #.

call returns
read_population(PATHS.data / "brgy_population.txt") {"Malanday": 1401, "Tumana": 880, …}
In [13]:
def read_population(path):
    # Step 1: start an empty dictionary, and open the file with encoding="utf-8"
    # Step 2: for each line, strip it; skip it if it is empty or starts with "#"
    # Step 3: split it on "|", strip each part, and store int(population) under the barangay
    raise NotImplementedError("Exercise 6.3")  # delete this line when your code is ready
In [14]:
try_it(read_population, PATHS.data / "brgy_population.txt")
Out[14]:

read_population('data/brgy_population.txt'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.

In [15]:
check_cases("Exercise 6.3", read_population, [("brgy_population.txt", (PATHS.data / "brgy_population.txt",))], what="files",
            hint="Strip each line, skip blanks and comments, strip each part.")
Out[15]:

Exercise 6.3: not attempted yet. Strip each line, skip blanks and comments, strip each part.

If the check is red: common causes for Exercise 6.3
  • ValueError: not enough values to unpack: a blank or comment line was split. Skip lines that are empty or start with # before splitting.
  • Keys with spaces, such as " Malanday ": strip each part after splitting.
  • Populations as text: turn them into numbers with int(...).

CSV files. A CSV file is a table as text: a header line with the column names, then one line per row, values separated by commas. The csv module reads it as records: csv.DictReader gives each row as a dictionary keyed by the column names, exactly the shape of the barangay's records in modules 3 to 5. Every value comes back as text, though, so numbers need int(...) and yes/no answers need turning into True or False.

In [16]:
import csv

with open(PATHS.data / "households.csv", encoding="utf-8", newline="") as f:
    rows = list(csv.DictReader(f))
len(rows), rows[0]
Out[16]:
(30,
 {'id': 'H001',
  'barangay': 'Malanday',
  'members': '5',
  'head_age': '47',
  'farms': 'yes',
  'crop': 'Palay',
  'area_sqm': '5000',
  'harvest_kg': '2500'})

Exercise 6.4 · The households, as records¶

Finish read_households(path) so that it reads households.csv and returns a list of records like the ones of modules 3 to 5: members, head_age, area_sqm and harvest_kg as whole numbers, farms as True for yes and False otherwise, and crop as None when it is empty.

call returns (the first record)
read_households(PATHS.data / "households.csv")[0] {"id": "H001", "barangay": "Malanday", "members": 5, "head_age": 47, "farms": True, "crop": "Palay", "area_sqm": 5000, "harvest_kg": 2500}
In [17]:
def read_households(path):
    # Step 1: open the file (encoding="utf-8", newline=""), and read it with csv.DictReader
    # Step 2: for each row, build a record with the right types (int(...), == "yes", or None for an empty crop)
    # Step 3: return the list of records
    raise NotImplementedError("Exercise 6.4")  # delete this line when your code is ready
In [18]:
try_it(read_households, PATHS.data / "households.csv")
Out[18]:

read_households('data/households.csv'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.

In [19]:
check_cases("Exercise 6.4", read_households, [("households.csv", (PATHS.data / "households.csv",))], what="files",
            hint="Every value comes as text: give each column its type.")
Out[19]:

Exercise 6.4: not attempted yet. Every value comes as text: give each column its type.

If the check is red: common causes for Exercise 6.4
  • Numbers as text ("5"): give each number column int(...).
  • farms as "yes": compare it, row["farms"] == "yes".
  • crop as "": an empty crop should be None; row["crop"] or None does it.

4 · Errors and exceptions¶

When a value does not fit, Python raises an exception: int("Missing") raises a ValueError. Module 2 read what the message says. Here, the program decides what to do about it.

Handling an exception. try: runs a block; if it raises a given exception, the except block runs instead, and the program goes on.

In [20]:
values = []
for text in ["10.5", "Missing", "5.0"]:
    try:
        values.append(float(text))
    except ValueError:
        print(f"skipped {text!r}")
values
skipped 'Missing'
Out[20]:
[10.5, 5.0]

Catch only the exception you expect, by name (except ValueError:), never everything (except:): a bare except also hides your own mistakes, a misspelt name or a missing file, and the program carries on with wrong results.

Raising one. When a function gets a value it cannot work with, the right answer is often to stop, with a message that says what was wrong: raise ValueError("…"). A clear stop at the bad value is better than a wrong result three steps later.

In [21]:
def members_of(text):
    members = int(text)
    if members < 1:
        raise ValueError(f"a household has at least one member, not {members}")
    return members


members_of("4")
Out[21]:
4

Exercise 6.5 · The total harvest, despite the gaps¶

harvests.txt has one harvest in tonnes per line, and Missing where a farm did not report. Finish total_harvest(path) so that it returns the total of the numbers, rounded to two decimal places, skipping the lines that are not numbers.

call returns
total_harvest(PATHS.data / "harvests.txt") 45.55
In [22]:
def total_harvest(path):
    # Step 1: start a total at 0, and open the file
    # Step 2: for each line, try to add float(line) to the total; except ValueError, skip it
    # Step 3: return the total rounded to 2 decimal places
    raise NotImplementedError("Exercise 6.5")  # delete this line when your code is ready
In [23]:
try_it(total_harvest, PATHS.data / "harvests.txt")
Out[23]:

total_harvest('data/harvests.txt'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.

In [24]:
check_cases("Exercise 6.5", total_harvest, [("harvests.txt", (PATHS.data / "harvests.txt",))], what="files", hint="`try: … except ValueError:` around `float(line)`.")
Out[24]:

Exercise 6.5: not attempted yet. try: … except ValueError: around float(line).

If the check is red: common causes for Exercise 6.5
  • ValueError: could not convert string to float: 'Missing': the conversion is not inside try.
  • Many decimals: round the total to 2.

Exercise 6.6 · Stop at a bad value¶

Finish checked_age(text) so that it returns the age in text as a whole number, and raises a ValueError with the message age must be between 0 and 120, not X (X the number read) when it is outside that range. Text that is not a number raises int's own ValueError, which is fine.

call returns
checked_age("47") 47
checked_age("150") raises ValueError: age must be between 0 and 120, not 150
In [25]:
def checked_age(text):
    # Step 1: turn the text into a whole number
    # Step 2: if it is below 0 or above 120, raise ValueError(f"age must be between 0 and 120, not {age}")
    # Step 3: return the age
    raise NotImplementedError("Exercise 6.6")  # delete this line when your code is ready
In [26]:
try_it(checked_age, "150")
Out[26]:

checked_age('150'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.

In [27]:
check_cases("Exercise 6.6", checked_age, [(repr(t), (t,)) for t in ["47", "0", "120", "150", "-3", "abc"]], what="ages",
            hint="Raise `ValueError` with the exact message for an age out of range.", raises=True)
Out[27]:

Exercise 6.6: not attempted yet. Raise ValueError with the exact message for an age out of range.

If the check is red: common causes for Exercise 6.6
  • None for "150": the function returns instead of raising. Use raise ValueError(...).
  • A different message: the check compares the message too: age must be between 0 and 120, not 150.
  • Wrong for "0" or "120": both are allowed; the test is 0 <= age <= 120.
In [28]:
course.quizzes_at("6.4")

Next: Debugging drill D2, in 55_debugging_drill.ipynb, then Meet marimo, in 60_meet_marimo.py, opened with uv run marimo edit 60_meet_marimo.py.