Every exercise in 50_polars.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.
Exercise 6.1. Read the plots, the polars way
The task
Why it matters: in module 3 (20_dataframes.py) you read plots.csv with the right types and in module 5 you cleaned it. Every polars analysis of the plots starts with the same steps, so they belong in one function.
Input → output: the plots file (a path, or the file's contents as bytes: pl.read_csv takes either) → a polars DataFrame with the file's eight columns, then area_ha:
production_kgas whole numbers (Int64), its thousands separators removed;- each row that is an exact repeat of another kept only once;
area_ha:areawhenarea_unitis"ha",area / 10_000when it is"sqm".
| hh_id | plot_no | crop | area | area_unit | … | production_kg | fertilizer_kg | area_ha |
|---|---|---|---|---|---|---|---|---|
| HH-0001 | 1 | Rice | 0.439 | ha | … | 1640 | 100.7 | 0.439 |
| HH-0001 | 2 | Rice | 1940.0 | sqm | … | 700 | null | 0.194 |
A blank crop stays null: polars reads an empty field as missing, and the plot is still a plot.
Tools: pl.read_csv(source, schema_overrides={"production_kg": pl.String}) so the column is text whatever its first rows hold; .str.replace_all(",", "").cast(pl.Int64); .unique(maintain_order=True); and for the unit, polars' version of pandas' where:
pl.when(pl.col("area_unit") == "sqm").then(pl.col("area") / 10_000).otherwise(pl.col("area"))
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
InvalidOperationError: conversion from str to i64 failed |
The cast comes before the commas are removed. | .str.replace_all(",", "") first, then .cast(pl.Int64) |
| InvalidOperationError: expected String type, got: i64 (second file) | polars guessed numbers, so there is no text to clean. | schema_overrides={"production_kg": pl.String} |
| the first file has 6 rows, not 5 | The repeated row is still there. | .unique(maintain_order=True) |
area_ha is huge for the sqm plots |
Multiplied by 10,000 instead of dividing, or the condition is the wrong way round. | .then(pl.col("area") / 10_000) where the unit is "sqm" |
production_kg is Float64 |
Cast to a decimal type. | .cast(pl.Int64) |
| fewer rows than expected | Rows with a blank crop or fertilizer were dropped. | no drop_nulls(): a missing value is not a reason to lose the plot |
| ColumnNotFoundError | A column name misspelt. | the names in the message's list of valid columns |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
The pandas reading of module 3 and the cleaning of module 5, as polars steps chained on one table: say the type on the way in, fix the text, drop the repeats, add the unit-free area.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def read_plots(source) -> pl.DataFrame: """The plots file in polars: production as whole numbers, repeated rows once, and area_ha.""" # Exercise 6.1: the task and examples are in the text above this cell. table = pl.read_csv(source, schema_overrides={"production_kg": pl.String}) return ( table.with_columns(pl.col("production_kg").str.replace_all(",", "").cast(pl.Int64)) .unique(maintain_order=True) .with_columns(area_ha=pl.when(pl.col("area_unit") == "sqm").then(pl.col("area") / 10_000).otherwise(pl.col("area"))) )Why each part is there
schema_overrides={"production_kg": pl.String}makes the column text whatever polars sees in its first 100 rows. Without it, a file whose first rows have no comma is read as numbers, and.str.replace_allthen fails with InvalidOperationError: expected String type, got: i64. That is the second case of the check..str.replace_all(",", "")removes every comma, then.cast(pl.Int64)makes whole numbers. Casting first fails: conversion fromstrtoi64failed for every value with a comma..unique(maintain_order=True)keeps one of each set of identical rows, in the file's order. Withoutmaintain_orderthe rows come back in any order, which is still correct but harder to compare with the file.pl.when(...).then(...).otherwise(...)is a condition, row by row, like pandas'where: square meters are divided by 10,000 (a hectare is 100 m × 100 m), hectares are kept. Multiplying instead makes a 1,940 m² plot 19 million hectares.
Exercise 6.2. Irrigated rice, and its yield
The task
Why it matters: filtering and adding a column are the two steps of almost every analysis. In module 3 (20_dataframes.py) you wrote them in pandas; here they are filter and with_columns.
Input → output: the clean plots, in polars (with area_ha) → only the plots whose crop is "Rice" and whose irrigated is "Yes", with five columns, in this order: hh_id, plot_no, area_ha, production_kg, and a new yield_t_ha, the harvest in tonnes per hectare.
| hh_id | plot_no | area_ha | production_kg | yield_t_ha |
|---|---|---|---|---|
| HH-0001 | 2 | 0.194 | 700 | 3.608… |
A yield in tonnes per hectare is production_kg / area_ha / 1000: kilograms per hectare, then 1,000 kilograms to the tonne. Use area_ha, never area, which is still in square meters for some plots.
Tools: filter with two conditions (separated by a comma, or joined with & and brackets), with_columns(yield_t_ha=...), select(...).
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
| TypeError: the truth value of an Expr is ambiguous | and between the conditions, or & without brackets around each comparison. |
filter(cond1, cond2), or (cond1) & (cond2) |
| rain-fed rice or corn in the result | Only one of the two conditions is applied. | both conditions in filter |
| yields 1,000 times too big | Kilograms per hectare, not tonnes. | / 1000 at the end |
| one yield thousands of times too small | area used instead of area_ha. |
pl.col("area_ha") |
| the columns in another order, or one too many | select is missing, or lists other columns. |
select("hh_id", "plot_no", "area_ha", "production_kg", "yield_t_ha") |
| ColumnNotFoundError: unable to find column "yield_t_ha" | select comes before with_columns, so the column does not exist yet. |
the order of the steps |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Three contexts, one after the other: filter keeps the rows, with_columns adds the yield, select keeps the columns asked for, in their order.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def irrigated_rice(plots: pl.DataFrame) -> pl.DataFrame: """The irrigated rice plots, with their yield in tonnes per hectare.""" # Exercise 6.2: the task and examples are in the text above this cell. return ( plots.filter(pl.col("crop") == "Rice", pl.col("irrigated") == "Yes") .with_columns(yield_t_ha=pl.col("production_kg") / pl.col("area_ha") / 1000) .select("hh_id", "plot_no", "area_ha", "production_kg", "yield_t_ha") )Why each part is there
filter(pl.col("crop") == "Rice", pl.col("irrigated") == "Yes"): conditions given tofilterseparated by commas must all hold, like&. A plot with no crop isnull, and a comparison withnullis not true, so it is left out.pl.col("production_kg") / pl.col("area_ha") / 1000is kilograms per hectare, then tonnes.area_hamatters: the check's small table has a plot of 2,500 m², whose yield fromareawould be 2,500 times too small.select(...)names the five columns in the order asked: a table's columns are part of its answer.
Exercise 6.3. The weighted rice yield by province
The task
Why it matters: this is the headline number of the survey, from module 4 (30_summarize_and_combine.py): how much rice a hectare gives in each province, as the province's farms would show it, not only the sample's. Each household stands for weight farm households, so a plot counts as many times as its household's weight. Unweighted, the rice yield of the whole survey is 4.35 t/ha; weighted, 4.50.
Input → output: the clean plots and the clean households, both in polars → one row per province, sorted by province, with four columns:
| province | plots | yield_t_ha | weighted_t_ha |
|---|---|---|---|
| the province | how many rice plots | the mean of the plots' yields | the same mean, weighted by each plot's household weight |
The steps: keep the rice plots; join each plot to its household's province and weight (a household has many plots, one plot one household: validate="m:1"); add each plot's yield, as in Exercise 6.2; then group by province.
A worked example: two rice plots in one province, yields 3.0 and 6.0 t/ha, from households of weight 300 and 100. Unweighted, (3.0 + 6.0) / 2 = 4.5. Weighted, (3.0 × 300 + 6.0 × 100) / (300 + 100) = 3.75.
Tools: join(..., on="hh_id", how="left", validate="m:1"); group_by("province").agg(plots=pl.len(), yield_t_ha=..., weighted_t_ha=...); the weighted mean as one expression, (pl.col("x") * pl.col("weight")).sum() / pl.col("weight").sum(); sort("province").
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
weighted_t_ha equals yield_t_ha |
The weight is not in the formula, or both columns use the same expression. | (pl.col("yield_t_ha") * pl.col("weight")).sum() / pl.col("weight").sum() |
weighted_t_ha hundreds of times too big |
The mean of yield times weight, not divided by the weights. | divide by pl.col("weight").sum() |
| Iloilo's mean is 3.94, not 4.6 | Total harvest over total area, not the mean of the plots' yields. | the yield of each plot first, then .mean() |
| more plots than expected, or Corn in the numbers | The rice filter is missing. | filter(pl.col("crop") == "Rice") |
| ColumnNotFoundError: unable to find column "province" | The join is missing, or comes after group_by. |
join before grouping |
| ComputeError: join keys did not fulfill m:1 validation | The households table lists a household twice. | the clean households, or unique() on them first |
a province row that is null |
A plot whose household is not in the households table. | how="left" keeps it: check the keys |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Bring each plot its household's province and weight with a join, compute each plot's yield, then let one agg give three numbers per province: a count, a plain mean and a weighted mean.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def weighted_yield_by_province(plots: pl.DataFrame, households: pl.DataFrame) -> pl.DataFrame: """Rice plots, mean yield and weighted mean yield (t/ha), one row per province.""" # Exercise 6.3: the task and examples are in the text above this cell. return ( plots.filter(pl.col("crop") == "Rice") .join(households.select("hh_id", "province", "weight"), on="hh_id", how="left", validate="m:1") .with_columns(yield_t_ha=pl.col("production_kg") / pl.col("area_ha") / 1000) .group_by("province") .agg( plots=pl.len(), yield_t_ha=pl.col("yield_t_ha").mean(), weighted_t_ha=(pl.col("yield_t_ha") * pl.col("weight")).sum() / pl.col("weight").sum(), ) .sort("province") )Why each part is there
filter(pl.col("crop") == "Rice")first, so the join and the group work only on the rows the answer is about.join(households.select("hh_id", "province", "weight"), on="hh_id", how="left", validate="m:1"): many plots to one household.validatestops with a ComputeError if a household appeared twice, which would copy its plots and count them twice. Selecting the three columns keeps the joined table small.- Each plot's yield is computed before grouping, because the mean is a mean of the plots' yields, not the province's total harvest over its total area (that gives a third number, which the check's small table tells apart).
(pl.col("yield_t_ha") * pl.col("weight")).sum() / pl.col("weight").sum()is the weighted mean, worked out inside each group. Taking(yield × weight).mean()instead gives a number in "tonnes times households", hundreds of times too big.- In
agg,yield_t_ha=pl.col("yield_t_ha").mean()reuses the name: the result column replaces the plots' column of the same name.sort("province")gives a fixed order, whichgroup_bydoes not.
Exercise 6.4. A lazy query
The task
Why it matters: a query written lazily runs as it is on a file a thousand times bigger, because polars reads only what it needs. Returning the query, not its result, also lets whoever calls your function add steps before anything runs.
Input → output: the raw plots file (a path or bytes; pl.scan_csv takes either) → a LazyFrame, not run, that gives one row for each value of irrigated, among the rice plots of the file with its repeated rows dropped:
| irrigated | plots | production_kg |
|---|---|---|
| No | how many rice plots | their total harvest, in kg |
| Yes | … | … |
Tools: pl.scan_csv(source, schema_overrides=...), then the steps of Exercise 6.1 that this needs (production as whole numbers, unique()), filter, group_by("irrigated").agg(plots=pl.len(), production_kg=...), sort("irrigated"). Do not end with .collect(): the check collects it, after making sure it is still a query that starts by scanning the file.
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
| not a LazyFrame but a DataFrame | The query ends with .collect(), or starts with pl.read_csv. |
pl.scan_csv(...), and no .collect() |
starts by scanning the file is False |
pl.read_csv(...).lazy(): lazy, but only after reading everything. |
pl.scan_csv(source, ...) |
| one plot too many in the first file | The repeated row is counted. | .unique() |
| InvalidOperationError when the check runs it | The production column is text when summed, or numbers when cleaned. | schema_overrides, then .str.replace_all and .cast, as in Exercise 6.1 |
| Corn in the numbers | The rice filter is missing. | filter(pl.col("crop") == "Rice") |
A lazy query's errors appear only when it runs, at .collect(): the line the error points to is the collect, and the mistake is in one of the steps before it.
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
The eager steps you know, started from scan_csv instead of read_csv, and returned before .collect(): a description of the work, which polars optimizes and runs only when someone collects it.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def rice_by_irrigation(source) -> pl.LazyFrame: """A lazy query: rice plots and their total harvest, irrigated and not, from the raw plots file.""" # Exercise 6.4: the task and examples are in the text above this cell. return ( pl.scan_csv(source, schema_overrides={"production_kg": pl.String}) .with_columns(pl.col("production_kg").str.replace_all(",", "").cast(pl.Int64)) .unique() .filter(pl.col("crop") == "Rice") .group_by("irrigated") .agg(plots=pl.len(), production_kg=pl.col("production_kg").sum()) .sort("irrigated") )Why each part is there
pl.scan_csv(source, schema_overrides={"production_kg": pl.String})reads nothing yet; it starts a plan whose first step is a scan of the file.pl.read_csv(...).lazy()would give a LazyFrame too, but only after reading the whole file, which is why the check looks for the scan in the plan.- The text is cleaned and cast as in Exercise 6.1, and
.unique()drops the repeated rows, so the four plots entered twice in the survey are counted once. filter(pl.col("crop") == "Rice")is written after the cleaning, where it reads well. polars pushes it down into the scan anyway: the plan shows it asSELECTIONunderCsv SCAN.agg(plots=pl.len(), production_kg=pl.col("production_kg").sum())gives the count and the total;sort("irrigated")a fixed order once it runs.- No
.collect(): the function returns the query. Whoever calls it decides when it runs, and can add steps first.
Exercise 6.5. Back to pandas, with the types it had
The task
Why it matters: the household table went through polars for a fast step, and must now go back to a pandas function from module 4 (30_summarize_and_combine.py) that looks households up by hh_id in the index, and treats a not-stated age as missing, not as a decimal.
Input → output: households in polars (with hh_id, and head_sex and head_age as whole numbers that may be null) → a pandas DataFrame, indexed by hh_id, with head_sex and head_age as pandas' nullable whole numbers ("Int64"). Every other column as .to_pandas() gives it.
| hh_id (index) | province | … | head_sex | head_age |
|---|---|---|---|---|
| HH-0001 | Ilocos Norte | … | 1 | 58 |
| HH-0012 | … | … | <NA> |
47 |
Tools: .to_pandas(), .set_index("hh_id"), .astype({"head_sex": "Int64", "head_age": "Int64"}).
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
head_age is float64 |
.to_pandas() alone: a whole-number column with gaps comes back as decimals. |
.astype({"head_sex": "Int64", "head_age": "Int64"}) |
| only the second table fails | astype only where there are gaps; with none, the type is int64. |
"Int64" in every case |
int64, not Int64 |
A lowercase "int64": the ordinary type, which cannot hold <NA>. |
"Int64", with a capital I |
int64[pyarrow] |
use_pyarrow_extension_array=True: Arrow types, not pandas' own. |
plain .to_pandas() |
the index is 0, 1, 2… and hh_id is a column |
The index is not set. | .set_index("hh_id") |
| AttributeError: 'DataFrame' object has no attribute 'set_index' | set_index called on the polars table, before .to_pandas(). |
the order of the steps |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Cross with .to_pandas(), then restore by hand the two pandas ideas polars does not carry: the index, and whole numbers that can be missing.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def households_for_pandas(households: pl.DataFrame) -> pd.DataFrame: """The households as a pandas table indexed by hh_id, with the not-stated codes still missing whole numbers.""" # Exercise 6.5: the task and examples are in the text above this cell. return households.to_pandas().set_index("hh_id").astype({"head_sex": "Int64", "head_age": "Int64"})Why each part is there
.to_pandas()crosses through pyarrow. It gives a fresh index 0, 1, 2…, and turnshead_sexandhead_ageintofloat64wherever they have anull, because the ordinary pandas whole number cannot hold a gap..set_index("hh_id")moves the key into the index, where the pandas code from module 4 looks for it. polars kept it as a column all along..astype({"head_sex": "Int64", "head_age": "Int64"})brings back pandas' nullable whole numbers:<NA>for not stated, and58, not58.0. The check compares every column's type, sofloat64fails even though the numbers are equal.- The check's second table has no
nullat all. There.to_pandas()givesint64, a whole number but not the nullable one, soastypeis needed in every case, not only when there are gaps.
Exercise 6.6. The prices, into polars
The task
Why it matters: course.prices() gives every farmgate price in one pandas table, with month as a monthly period, as module 5 (40_cleaning_and_time.py) left it. polars has no period type, so the month must become something polars understands before it crosses: a date, the first day of the month.
Input → output: the prices in pandas (month a monthly Period, then province, commodity, price) → the same rows in polars, with month as Date and the other three columns as they were.
| month | province | commodity | price |
|---|---|---|---|
| 2022-01-01 | Ilocos Norte | Palay | 18.2 |
Tools: prices["month"].dt.to_timestamp() (in pandas: the first moment of each month), prices.assign(...), pl.from_pandas(...), then .with_columns(pl.col("month").cast(pl.Date)) (in polars: the date, without a time of day).
If the check is red: common causes
| what you see | the usual cause | look at |
|---|---|---|
month shows numbers such as 624, its type Extension(...) |
The period crossed as it is. | .dt.to_timestamp() before pl.from_pandas |
month is Datetime(time_unit='us', ...) |
The time of day is still there. | .cast(pl.Date) |
month is String |
astype(str): text, not a date. Sorting and date ranges then work on letters. |
.dt.to_timestamp() |
| AttributeError: 'DataFrame' object has no attribute 'assign' | assign called after pl.from_pandas: polars says with_columns. |
convert the month in pandas, before crossing |
| AttributeError: Can only use .dt accessor with datetimelike values | .dt.to_timestamp() on a column that is not a period. |
prices["month"], from course.prices() |
Still stuck after ten minutes? Ask your table, then raise your hand.
The solution, and why it works
Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.
The idea
Turn the one column polars cannot read into one it can, while the table is still in pandas, cross, then trim the time of day polars does not need.
The function, solved
Copy it with the button on the code's frame, and paste it over all the code in the exercise's cell, so the indentation stays right.
def prices_for_polars(prices: pd.DataFrame) -> pl.DataFrame: """The prices in polars, with each month as the date of its first day.""" # Exercise 6.6: the task and examples are in the text above this cell. return pl.from_pandas(prices.assign(month=prices["month"].dt.to_timestamp())).with_columns(pl.col("month").cast(pl.Date))Why each part is there
prices.assign(month=prices["month"].dt.to_timestamp())replaces each period with the first moment of its month, a pandasdatetime64, and returns a new table: the caller'spricesis not changed.pl.from_pandas(...)now readsmonthas polars'Datetime. Crossing the period itself gives an "extension" column holding months since January 1970 (624 for January 2022), which the check's type comparison tells apart..with_columns(pl.col("month").cast(pl.Date))keeps the date and drops the time of day, which is always midnight here and means nothing for a month.DatetimeandDateshow the same day but are different types, and the check compares types.