Every exercise in 81_stretch_day1.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.
Stretch S1.1. A strict code
The task
code_to_text pads whatever it is given. A strict version should refuse what cannot be a code: raise a ValueError with a clear message when the value (after removing spaces) is not made only of digits, is longer than width, or is a decimal that is not whole (12.5). Missing values still give None.
| call | gives |
|---|---|
strict_code(100305000.0, 10) |
"0100305000" |
strict_code("12A", 4) |
ValueError |
strict_code("01234567890", 10) |
ValueError (11 digits) |
strict_code(12.5, 4) |
ValueError |
If the check is red
An empty text is not made of digits either.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Exercise 1.1's steps, with a refusal wherever a value cannot be a code. Quietly padding a bad value would send it on into every join; a ValueError stops it where it entered, with a message that says why.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def strict_code(value, width: int) -> str | None: """A code as text of `width` characters, or a ValueError for anything that cannot be a code.""" if value is None or pd.isna(value): return None if isinstance(value, float): if not value.is_integer(): raise ValueError(f"{value!r} is not a whole number") value = int(value) text = str(value).strip() if not text.isdigit(): raise ValueError(f"{value!r} is not made only of digits") if len(text) > width: raise ValueError(f"{value!r} has more than {width} digits") return text.zfill(width)Why each part is there
- Missing stays
None, as in Exercise 1.1: a missing code is a gap to report, not an error in the file. - A decimal must be whole before it becomes a whole number:
value.is_integer()refuses12.5instead of lettingint(12.5)quietly cut it to12. .isdigit()refuses anything that is not only digits, after.strip():'12A', and also the empty text, because''.isdigit()isFalse. That is what the check's hint points at.len(text) > widthrefuses a code that is too long, such as eleven digits for a 10-character code..zfillwould hand it back unchanged and the error would surface far away, as a join that finds nothing.
Stretch S1.2. Find the header row yourself
The task
Exercise 3.1 used header=3 because you looked at the workbook. A pipeline that reads next quarter's edition should not depend on someone looking. Write find_header_row(path, first_heading): read the sheet with header=None, and return the position (from 0) of the first row whose first cell is first_heading. Raise a ValueError if there is none.
| call | gives |
|---|---|
find_header_row(PATHS.reference / "psgc_2023Q4.xlsx", "Code") |
3 |
If the check is red
Position from 0, and a ValueError when no row matches.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Read the sheet with no header at all, so every row is data, then look down the first column for the heading. The row where it sits is the header row.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def find_header_row(path: Path, first_heading: str) -> int: """The position of the row whose first cell is `first_heading`.""" sheet = pd.read_excel(path, header=None, dtype=str) matches = sheet.index[sheet.iloc[:, 0].str.strip().eq(first_heading).fillna(False)] if len(matches) == 0: raise ValueError(f"no row starts with {first_heading!r} in {path.name}") return int(matches[0])Why each part is there
pd.read_excel(path, header=None, dtype=str)keeps every row, numbered from 0, so a row's number is its position, which is whatheader=takes.sheet.iloc[:, 0]is the first column;.str.strip().eq(first_heading)isTruewhere its cell, without spaces around it, is the heading. Empty cells give no match.sheet.index[...]lists the matching row numbers. With none, the function raises aValueErrornaming the heading and the file: a new edition with a renamed heading should stop the pipeline, not be read from the wrong row.int(matches[0])returns the first match as a plain whole number. The check runs it on two editions of the code list, so the answer comes from the file, never from someone having looked at it.
Stretch S1.3. What a careless join costs
The task
The careless join in section 3 copied the people of one city. How much does that change the weighted total? Write weighted_drift(round_df, code_list): the weighted total (sum of weight) after round_df.merge(code_list, left_on="area_code", right_on="code", how="left"), minus the weighted total before, rounded to a whole number. Use the code list with its duplicate (from your load_code_list).
The point of the exercise is the size of the number, compared with the region's population.
If the check is red
Only the people of the duplicated code are copied: their weights are counted twice.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Join without removing the duplicate, and compare the weighted totals before and after: the difference is exactly what the careless join added.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def weighted_drift(round_df: pd.DataFrame, code_list: pd.DataFrame) -> int: """How many weighted persons a careless join adds.""" joined = round_df.merge(code_list, left_on="area_code", right_on="code", how="left") return round(joined["weight"].sum() - round_df["weight"].sum())Why each part is there
round_df.merge(code_list, left_on="area_code", right_on="code", how="left")is the careless join: nodrop_duplicates, novalidate. Each person whose code is listed twice comes out twice.joined["weight"].sum() - round_df["weight"].sum()is the drift. Only the copied people add to it, each with their full weight, so it equals the weighted population of the duplicated city.round(...)gives a whole number of people. Set it beside the region's population: a few rows copied in the sample can be thousands of people in the estimate, which is why Exercise 3.2'svalidate="many_to_one"stops instead.
Stretch S1.4. An API reader that cannot loop forever
The task
A broken API can name, as its next page, a page you have already read. Your Exercise 4.3 would then go round for ever. Write read_pages(folder): like read_api, but keep a set of the pages already read, and raise a ValueError naming the page when next points back to one. Return the records as a list of (region, year, value_text), in reading order.
If the check is red
Two records when the pages end; a ValueError when they loop.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Exercise 4.3's loop with a memory: before reading a page, check that it has not been read already.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def read_pages(folder: Path) -> list[tuple[str, int, str]]: """Every record of the API's pages, refusing pages that loop back.""" records, seen, page_name = [], set(), "page-1.json" while page_name is not None: if page_name in seen: raise ValueError(f"the pages loop back to {page_name}") seen.add(page_name) page = json.loads((folder / page_name).read_text(encoding="utf-8")) records += [(r["key"][0], int(r["key"][1]), r["values"][0]) for r in page["data"]] page_name = page["next"] return recordsWhy each part is there
seenis a set of page names;page_name in seenis a quick yes or no however many pages there are. A page met twice means the pages go round in a circle, and the function raises aValueErrornaming that page.seen.add(page_name)comes before the page is read, so the check covers every page, including the first.- This version reads each page with
json.loads, so a page is a dictionary:page["data"]andpage["next"], notpage.data. It skips the shape checkPxPagegives in Exercise 4.3; here the point is the loop. - Each record becomes a
(region, year, value_text)tuple, withint(...)on the year, in the order the pages give them, which is what the check compares.