Skip to content
Learn Python with BNHR

Share: the release

On this page
  1. Overview
  2. 2.1 · Age groups
  3. 2.2 · People by province, sex and age group
  4. 3.1 · Suppress small cells
  5. 4.1 · A codebook from the contract
  6. 5.1 · Open every file you wrote

Every exercise in 31_share_round.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.

Exercise 2.1. Age groups

The task

Why it matters: a release groups ages into bands: single years would make too many small cells.

Input → output: an age (a number, or missing) → its group's label, or None when the age is missing.

age group
0 to 14 "0-14"
15 to 24 "15-24"
25 to 44 "25-44"
45 to 64 "45-64"
65 and over "65+"

AGE_GROUPS (in the setup cell) lists them as (lowest, highest, label): loop over it and return the label whose range holds the age. Check for a missing age first (pd.isna(age)).

If the check is red: common causes

what you see the usual cause look at
14 or 15 in the wrong band < instead of <= at one end. both ends included
An error for a missing age NaN reached the comparison. check pd.isna(age) first

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 2.2. People by province, sex and age group

The task

Why it matters: this is the table the release is about. Built by code from the stored round, it is the same table every quarter.

Input → output: a round's stored rows → one row per province, sex and age group, with respondents (how many people answered: rows) and weighted_persons (the sum of their weights, rounded to a whole number).

Prepare three columns, then group:

  • province: the first five characters of area_code, then "00000"; where area_code is empty (an unknown area), "unknown";
  • sex: sex_label, with missing values as "Not stated";
  • age_group: age through your age_group (.map(age_group)), with missing values as "Not stated";
  • group: .groupby(["province", "sex", "age_group"]).agg(respondents=("hhid", "size"), weighted_persons=("weight", "sum")), then .reset_index(), and round weighted_persons with .round().

If the check is red: common causes

what you see the usual cause look at
Fewer than six respondents in total Rows with a missing province, sex or age were dropped: groupby leaves out missing keys. fill them with "unknown" / "Not stated" first
Provinces like 01001 The "00000" was not added. area_code.str[:5] + "00000"
Decimals in weighted_persons Not rounded. .round()

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 3.1. Suppress small cells

The task

Why it matters: this is the step that makes the release safe to publish. It runs on every release, by code, so no small cell slips through by hand.

Input → output: the release table and a threshold → the table ready to publish: a column suppressed (True where respondents is below the threshold), weighted_persons empty where suppressed, and no respondents column (the number of respondents in a cell is not published).

Tools: a condition (primer §2 (90_primers.py)), .where(condition) (keeps values where the condition is True, empty elsewhere), and .drop(columns=...).

If the check is red: common causes

what you see the usual cause look at
respondents is still there It was not dropped. .drop(columns="respondents")
Small cells show a 0 instead of nothing A zero is a number: it says "no one", which is false. .where(~small) leaves them empty
The cell with exactly 30 is hidden <= instead of <. below the threshold only

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 4.1. A codebook from the contract

The task

Input → output: a pandera schema → one row per column, with:

column of the codebook from the pandera column example
column its name sex
description column.description Sex, from the questionnaire
type str(column.dtype), put in words with TYPE_NAMES text
may be missing column.nullable, as "yes" or "no" no
rules each check's check.error, joined with "; "; empty text when there are none isin(['Female', 'Male', 'Not stated'])

Loop over schema.columns.items(): each item is a column's name and its pandera column. TYPE_NAMES.get(text, text) gives the words for a type, or the type itself when it is not in the list.

If the check is red: common causes

what you see the usual cause look at
rules shows objects like <Check in_range…> The checks were joined, not their text. check.error
rules shows in_range without its limits check.name is the rule's name only. check.error has the values
type shows float64 The type was not put in words. TYPE_NAMES.get(...)
may be missing is True or False The yes/no was not written as text. "yes" if column.nullable else "no"

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 5.1. Open every file you wrote

The task

Why it matters: an export can "succeed" and still write the wrong thing: a CSV whose empty cells became zeros, a database missing rows. The only proof is to read each file back.

Input → output: a release folder → one row per data file (release_table.parquet, release_table.csv, release.sqlite), with file, rows, and suppressed hidden: True when, in that file, every row marked suppressed has an empty weighted_persons.

Read each with its own reader: pd.read_parquet(path), pd.read_csv(path), and for SQLite pd.read_sql_query("SELECT * FROM release_table", con) with con = sqlite3.connect(path) (close it after: with closing(sqlite3.connect(path)) as con:). SQLite stores True as 1, so make the column yes/no with .astype(bool) before using it as a condition.

If the check is red: common causes

what you see the usual cause look at
The CSV with zeros passes Only the row count is checked. the suppressed rows' weighted_persons must be empty
KeyError: 'suppressed' in the SQLite table The query did not select every column. SELECT * FROM release_table
SQLite's suppressed rows are not found 1 was compared with True without converting. .astype(bool)

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Something new

Identify results
LayerExplore a stretch exercises
Stretch exercises S3Stretch: charts
AboutStretch exercises of Arrays & DataFrames, after module 7: chart exercises on the survey and the prices, in the library of your choice, each with its check and its solution on the companion.
CourseArrays & DataFrames, free