Text, files and errors¶
Until now the barangay's records were typed into the notebook. Real records arrive as files, written by other people and other programs: names with stray spaces, numbers with a word where a number should be, a line that does not fit. This module reads them, cleans the text, and handles what does not fit, without the notebook falling over.
| section | what you do |
|---|---|
| 1 · Working with text | clean, split, join and test text with its methods |
| 2 · Patterns | check and find text by its shape, with regular expressions |
| 3 · Reading and writing files | text files line by line, and CSV files as records |
| 4 · Errors and exceptions | handle the line that does not fit, and stop with a clear message when you must |
The files are in the course folder's data/ folder; course.PATHS.data is where it is.
import course
from course import PATHS, check_cases, try_it
PATHS.data, sorted(p.name for p in PATHS.data.iterdir())
(PosixPath('~/essentials/data'),
['brgy_population.txt', 'harvests.txt', 'households.csv'])
course.quizzes_at("6.0")
1 · Working with text¶
Text (a str) has methods, called with a dot, which give back a new text: text never changes in place.
| method | gives | example |
|:--|:--|:--|
| .strip() | the text without spaces at both ends | " Tumana ".strip() is "Tumana" |
| .lower(), .upper(), .title() | the text in another case | "tumana".title() is "Tumana" |
| .replace(old, new) | every old replaced | "Brgy. 143".replace("Brgy.", "Barangay") |
| .split(sep) | a list of the parts between sep | "NCR|Tumana|880".split("|") |
| .split() | the words, however many spaces between them | "Barangay 143".split() |
| sep.join(parts) | the parts joined by sep | " ".join(["Barangay", "143"]) |
| .startswith(x), .endswith(x) | True or False | "# a comment".startswith("#") |
| .isdigit() | whether every character is a digit | "0128".isdigit() |
raw = " concepcion UNO "
raw.strip(), raw.split(), " ".join(raw.split()).title()
('concepcion UNO', ['concepcion', 'UNO'], 'Concepcion Uno')
Formatting with f-strings you know from module 2. Two more things worth knowing: len(text) counts characters, and text[0], text[-3:] take characters and slices, as with a list.
Exercise 6.1 · Clean a barangay's name¶
Names typed by hand come with stray spaces and mixed case. Finish clean_name(text) so that it returns the name with no spaces at the ends, a single space between words, and each word capitalised.
| call | returns |
|---|---|
clean_name(" concepcion UNO ") |
"Concepcion Uno" |
clean_name("malanday") |
"Malanday" |
def clean_name(text):
# Step 1: split the text into its words (.split() with no input ignores any run of spaces)
# Step 2: join the words with a single space, and capitalise each word (.title())
raise NotImplementedError("Exercise 6.1") # delete this line when your code is ready
try_it(clean_name, " concepcion UNO ")
clean_name(' concepcion UNO '): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.
check_cases("Exercise 6.1", clean_name, [(repr(t), (t,)) for t in [" concepcion UNO ", "malanday", "TUMANA", " barangay 143 ", ""]],
what="names", hint="`.split()`, then `\" \".join(...)`, then `.title()`.")
Exercise 6.1: not attempted yet. .split(), then " ".join(...), then .title().
If the check is red: common causes for Exercise 6.1
- Several spaces between words:
.strip()only removes spaces at the ends. Split the words and join them with one space. "Concepcion UNO":.capitalize()changes only the first letter of the whole text. Use.title().
2 · Patterns¶
Some checks are about a text's shape: a mobile number is 09 and nine more digits. A regular expression (regex) describes such a shape, and the re module tests text against it.
| in a pattern | means |
|---|---|
\d |
one digit |
\d{9} |
exactly nine digits |
[A-Z] |
one capital letter |
? |
the thing before it, or nothing |
\| |
either side (09\|\+639) |
( … ) |
a group, to use with \| or to take out |
re.fullmatch(pattern, text) checks the whole text against the pattern, and gives a match or None. re.findall(pattern, text) gives every part of the text that fits. Write patterns as raw strings, r"...", so the backslashes stay as they are.
import re
bool(re.fullmatch(r"09\d{9}", "09171234567")), bool(re.fullmatch(r"09\d{9}", "0917-123-4567")), re.findall(r"\d{4}", "rounds 2023 and 2024")
(True, False, ['2023', '2024'])
Exercise 6.2 · A valid mobile number¶
A Philippine mobile number is written 09 and nine digits, or +639 and nine digits. Finish is_mobile(text) so that it returns True when the whole text is a mobile number in one of those forms, and False otherwise.
| call | returns |
|---|---|
is_mobile("09171234567") |
True |
is_mobile("+639171234567") |
True |
is_mobile("0917-123-4567") |
False |
def is_mobile(text):
# Step 1: write a pattern: (09 or +639), then nine digits; the + needs a backslash
# Step 2: return whether re.fullmatch finds the pattern in the whole text (bool(...))
raise NotImplementedError("Exercise 6.2") # delete this line when your code is ready
try_it(is_mobile, "09171234567")
is_mobile('09171234567'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.
check_cases("Exercise 6.2", is_mobile,
[(repr(t), (t,)) for t in ["09171234567", "+639171234567", "0917-123-4567", "0917123456", "091712345678", "639171234567", "call 09171234567", "+63 917 123 4567"]],
what="numbers", hint="`re.fullmatch(r\"(09|\\+639)\\d{9}\", text)`.")
Exercise 6.2: not attempted yet. re.fullmatch(r"(09|\+639)\d{9}", text).
If the check is red: common causes for Exercise 6.2
Truefor a number inside other text:re.searchorre.matchlook at only part of the text. Usere.fullmatch.Falsefor+639…: the+needs a backslash,\+.- A match object instead of
True: returnbool(...)of it.
3 · Reading and writing files¶
open(path) opens a file; with open(path, encoding="utf-8") as f: opens it for the lines indented under it, and closes it after, even when something goes wrong. Looping over the file gives its lines, each with the \n at its end (which .strip() removes). Always give encoding="utf-8": without it, Windows may read the file in another encoding, and names with ñ come out wrong.
with open(PATHS.data / "brgy_population.txt", encoding="utf-8") as f:
for line in f:
print(repr(line))
'# Population by barangay, as sent by the city office (2020)\n' '# region | barangay | population\n' '\n' 'NCR| Malanday | 1401\n' 'NCR|Tumana| 880\n' ' NCR| Concepcion Uno| 1029\n' 'NCR | Barangay 143 |2210\n' 'NCR|Barangay 144 | 1655\n' '\n' 'NCR| Barangay 145| 975\n' '# End of report\n'
The file is messy in ways files often are: comment lines starting with #, blank lines, and spaces around the | separators. Each line that matters is region, barangay and population.
Writing uses open(path, "w", encoding="utf-8") and f.write(text). "w" replaces whatever the file held; "a" adds to its end. write does not add a line break: put \n at the end of each line.
report = PATHS.root / "report.txt"
with open(report, "w", encoding="utf-8") as f:
f.write("Malanday: 1,401 people\n")
f.write("Tumana: 880 people\n")
report.read_text(encoding="utf-8")
'Malanday: 1,401 people\nTumana: 880 people\n'
PATHS.data / "brgy_population.txt" joins folder and file names into a path (with pathlib), which works on Windows and on a Mac alike. A path can read a whole file at once with .read_text(encoding="utf-8").
Exercise 6.3 · Population by barangay¶
Finish read_population(path) so that it reads a report like brgy_population.txt and returns a dictionary of each barangay's population, as a whole number. Skip blank lines and lines starting with #.
| call | returns |
|---|---|
read_population(PATHS.data / "brgy_population.txt") |
{"Malanday": 1401, "Tumana": 880, …} |
def read_population(path):
# Step 1: start an empty dictionary, and open the file with encoding="utf-8"
# Step 2: for each line, strip it; skip it if it is empty or starts with "#"
# Step 3: split it on "|", strip each part, and store int(population) under the barangay
raise NotImplementedError("Exercise 6.3") # delete this line when your code is ready
try_it(read_population, PATHS.data / "brgy_population.txt")
read_population('data/brgy_population.txt'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.
check_cases("Exercise 6.3", read_population, [("brgy_population.txt", (PATHS.data / "brgy_population.txt",))], what="files",
hint="Strip each line, skip blanks and comments, strip each part.")
Exercise 6.3: not attempted yet. Strip each line, skip blanks and comments, strip each part.
If the check is red: common causes for Exercise 6.3
ValueError: not enough values to unpack: a blank or comment line was split. Skip lines that are empty or start with#before splitting.- Keys with spaces, such as
" Malanday ": strip each part after splitting. - Populations as text: turn them into numbers with
int(...).
CSV files. A CSV file is a table as text: a header line with the column names, then one line per row, values separated by commas. The csv module reads it as records: csv.DictReader gives each row as a dictionary keyed by the column names, exactly the shape of the barangay's records in modules 3 to 5. Every value comes back as text, though, so numbers need int(...) and yes/no answers need turning into True or False.
import csv
with open(PATHS.data / "households.csv", encoding="utf-8", newline="") as f:
rows = list(csv.DictReader(f))
len(rows), rows[0]
(30,
{'id': 'H001',
'barangay': 'Malanday',
'members': '5',
'head_age': '47',
'farms': 'yes',
'crop': 'Palay',
'area_sqm': '5000',
'harvest_kg': '2500'})
Exercise 6.4 · The households, as records¶
Finish read_households(path) so that it reads households.csv and returns a list of records like the ones of modules 3 to 5: members, head_age, area_sqm and harvest_kg as whole numbers, farms as True for yes and False otherwise, and crop as None when it is empty.
| call | returns (the first record) |
|---|---|
read_households(PATHS.data / "households.csv")[0] |
{"id": "H001", "barangay": "Malanday", "members": 5, "head_age": 47, "farms": True, "crop": "Palay", "area_sqm": 5000, "harvest_kg": 2500} |
def read_households(path):
# Step 1: open the file (encoding="utf-8", newline=""), and read it with csv.DictReader
# Step 2: for each row, build a record with the right types (int(...), == "yes", or None for an empty crop)
# Step 3: return the list of records
raise NotImplementedError("Exercise 6.4") # delete this line when your code is ready
try_it(read_households, PATHS.data / "households.csv")
read_households('data/households.csv'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.
check_cases("Exercise 6.4", read_households, [("households.csv", (PATHS.data / "households.csv",))], what="files",
hint="Every value comes as text: give each column its type.")
Exercise 6.4: not attempted yet. Every value comes as text: give each column its type.
If the check is red: common causes for Exercise 6.4
- Numbers as text (
"5"): give each number columnint(...). farmsas"yes": compare it,row["farms"] == "yes".cropas"": an empty crop should beNone;row["crop"] or Nonedoes it.
4 · Errors and exceptions¶
When a value does not fit, Python raises an exception: int("Missing") raises a ValueError. Module 2 read what the message says. Here, the program decides what to do about it.
Handling an exception. try: runs a block; if it raises a given exception, the except block runs instead, and the program goes on.
values = []
for text in ["10.5", "Missing", "5.0"]:
try:
values.append(float(text))
except ValueError:
print(f"skipped {text!r}")
values
skipped 'Missing'
[10.5, 5.0]
Catch only the exception you expect, by name (except ValueError:), never everything (except:): a bare except also hides your own mistakes, a misspelt name or a missing file, and the program carries on with wrong results.
Raising one. When a function gets a value it cannot work with, the right answer is often to stop, with a message that says what was wrong: raise ValueError("…"). A clear stop at the bad value is better than a wrong result three steps later.
def members_of(text):
members = int(text)
if members < 1:
raise ValueError(f"a household has at least one member, not {members}")
return members
members_of("4")
4
Exercise 6.5 · The total harvest, despite the gaps¶
harvests.txt has one harvest in tonnes per line, and Missing where a farm did not report. Finish total_harvest(path) so that it returns the total of the numbers, rounded to two decimal places, skipping the lines that are not numbers.
| call | returns |
|---|---|
total_harvest(PATHS.data / "harvests.txt") |
45.55 |
def total_harvest(path):
# Step 1: start a total at 0, and open the file
# Step 2: for each line, try to add float(line) to the total; except ValueError, skip it
# Step 3: return the total rounded to 2 decimal places
raise NotImplementedError("Exercise 6.5") # delete this line when your code is ready
try_it(total_harvest, PATHS.data / "harvests.txt")
total_harvest('data/harvests.txt'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.
check_cases("Exercise 6.5", total_harvest, [("harvests.txt", (PATHS.data / "harvests.txt",))], what="files", hint="`try: … except ValueError:` around `float(line)`.")
Exercise 6.5: not attempted yet. try: … except ValueError: around float(line).
If the check is red: common causes for Exercise 6.5
ValueError: could not convert string to float: 'Missing': the conversion is not insidetry.- Many decimals: round the total to 2.
Exercise 6.6 · Stop at a bad value¶
Finish checked_age(text) so that it returns the age in text as a whole number, and raises a ValueError with the message age must be between 0 and 120, not X (X the number read) when it is outside that range. Text that is not a number raises int's own ValueError, which is fine.
| call | returns |
|---|---|
checked_age("47") |
47 |
checked_age("150") |
raises ValueError: age must be between 0 and 120, not 150 |
def checked_age(text):
# Step 1: turn the text into a whole number
# Step 2: if it is below 0 or above 120, raise ValueError(f"age must be between 0 and 120, not {age}")
# Step 3: return the age
raise NotImplementedError("Exercise 6.6") # delete this line when your code is ready
try_it(checked_age, "150")
checked_age('150'): not written yet. Write your code, delete the raise NotImplementedError line, and run the function's cell.
check_cases("Exercise 6.6", checked_age, [(repr(t), (t,)) for t in ["47", "0", "120", "150", "-3", "abc"]], what="ages",
hint="Raise `ValueError` with the exact message for an age out of range.", raises=True)
Exercise 6.6: not attempted yet. Raise ValueError with the exact message for an age out of range.
If the check is red: common causes for Exercise 6.6
Nonefor"150": the function returns instead of raising. Useraise ValueError(...).- A different message: the check compares the message too:
age must be between 0 and 120, not 150. - Wrong for
"0"or"120": both are allowed; the test is0 <= age <= 120.
course.quizzes_at("6.4")
Next: Debugging drill D2, in 55_debugging_drill.ipynb, then Meet marimo, in 60_meet_marimo.py, opened with uv run marimo edit 60_meet_marimo.py.