Skip to content
Learn Python with BNHR

Stretch · A new edition and Share

On this page
  1. Overview
  2. S3.1 · Complementary suppression
  3. S3.2 · Parts that add up to the whole
  4. S3.3 · The crosswalk the other way, by shares
  5. S3.4 · A canned query with two parameters
  6. S3.5 · Matching names to codes

Every exercise in 83_stretch_day3.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.

Stretch S3.1. Complementary suppression

The task

suppress hides small cells, but a province's total is published too: when only one of its cells is hidden, the total minus the visible cells gives it back (the explainer in 31_share_round.py, section 3, shows it). Write suppress_complementary(table, threshold):

  • hide every cell with fewer respondents than the threshold, and mark it "small cell" in a column suppressed;
  • in each province where exactly one cell is hidden that way, also hide the province's other cell with the fewest respondents, marked "complementary" (the first one, if two tie);
  • suppressed is empty text for a published cell; weighted_persons is empty for both kinds of hidden cell; respondents is dropped.

The input has the columns province, age_group, respondents and weighted_persons, like explain.SUPPRESSION_TABLE. A province where every cell is small has nothing left to hide.

If the check is red

At 10, one cell of 0600300000 is small, so a second is hidden; at 30, two are small, which is safe already.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Stretch S3.2. Parts that add up to the whole

The task

Round each part of a distribution on its own, and the parts may add up to 999 or 1,001 per thousand: a reader who adds them finds the release "wrong". The largest remainder method fixes it. Write per_thousand(values): for a Series of non-negative numbers, each value's share per 1,000, as whole numbers that add up to exactly 1,000:

  1. the exact shares: values / values.sum() * 1000;
  2. round each down (numpy.floor, or // 1);
  3. hand out what is missing to 1,000, one each, to the values with the largest remainders (exact minus rounded down); on a tie, the earlier value first;
  4. keep the Series' index, and give back whole numbers (.astype(int)).

Tip: remainders.sort_values(ascending=False, kind="stable") sorts largest first and keeps ties in their order.

If the check is red

Three equal parts give 334, 333 and 333: the first one gets the spare unit.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Stretch S3.3. The crosswalk the other way, by shares

The task

In 30_new_edition.py you mapped Round 3 back to the old geography, because new → old never copies anyone. Sometimes the question goes the other way: Round 1's totals in the new geography. A municipality that split has two successors, and the old data cannot say who lives in which; the usual answer is to share out its total by each successor's share in a round that has both (Round 3).

Write allocate_to_2024q2(totals, crosswalk, shares):

  • totals: area_code_2023q4 and weighted_persons, one row per old code;
  • crosswalk: code_2024q2 and code_2023q4, as from your load_crosswalk;
  • shares: code_2024q2 and share, only for successors of a split: the shares of one old code's successors add up to 1;
  • result: code_2024q2 and weighted_persons, one row per new code, sorted by code. A code with no successor in the crosswalk keeps its own code; a successor with no share gets the whole total (share 1).

Join the totals to the crosswalk (old → new: validate="one_to_many", since one old code may have two successors), then to the shares (validate="many_to_one"), multiply, and add up by new code. The grand total must not change: check it yourself.

If the check is red

The split's 2,000 becomes 800 and 1,200; 0600303000 keeps its own code; the grand total stays 3,800.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Stretch S3.4. A canned query with two parameters

The task

Write region_sex_sql(): the release's cells for one region and one sex, with the columns province, age_group, weighted_persons and suppressed, sorted by province then age group. Two parameters, :region and :sex: Datasette shows a box for each.

Then add it to your portal: the button below adds it to datasette.json as a second canned query, beside region_summary, and adds suppressed to the table's facets. Restart Datasette to see it.

If the check is red

Region 01, Female: three cells, sorted by province then age group.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Stretch S3.5. Matching names to codes

The task

Another agency sends a list of local governments by name: reference/lgu_names_other_agency.csv, with the province's name and a count of facilities. To join it to anything, each name needs its code. Names are written differently (Penablanca, Sta. Cruz, TALAVERA), one has a typo, two are not municipalities at all, and the same municipality name exists in several provinces: the province is part of the match.

Write match_names(lgus, places):

  • lgus: lgu_name and province, as in the file;
  • places: code, name and province for every city and municipality (from places() below);
  • result: the rows of lgus with three more columns: code, matched_name (the code list's name), and match: "exact" when the two names are the same once normalised, "close" when difflib.get_close_matches(name, candidates, n=1, cutoff=0.8) finds one among the same province's normalised names, or "none" (and empty code and name).

Normalise a name: lower case; accents removed (unicodedata.normalize("NFKD", text), then drop the characters for which unicodedata.combining(c) is true); sta. written santa, a leading s. written san; spaces collapsed.

A close match is a guess: a report should show it for a person to confirm, never join on it silently.

If the check is red

The two San Jose rows get two different codes: one per province. Cabiaa is close to Cabiao.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the check's hint above covers the usual slip. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Something new

Identify results
LayerExplore a stretch exercises
Stretch exercises S3Stretch: A new edition and Share
AboutStretch exercises of Python for Data Management, after module 6: more on a new edition and on sharing the release, with less help, each with its check and its solution on the companion.
CoursePython for Data Management, free