Skip to content
Learn Python with BNHR

polars

On this page
  1. Overview
  2. 6.1 · Read the plots, the polars way
  3. 6.2 · Irrigated rice, and its yield
  4. 6.3 · The weighted rice yield by province
  5. 6.4 · A lazy query
  6. 6.5 · Back to pandas, with the types it had
  7. 6.6 · The prices, into polars

Every exercise in 50_polars.py, with the solution and why it works. Each answer stays closed until you swear you tried it first. Opening one changes nothing in your notebook; your check still has to go green from your own code. Read the whole notebook, with its lessons.

Exercise 6.1. Read the plots, the polars way

The task

Why it matters: in module 3 (20_dataframes.py) you read plots.csv with the right types and in module 5 you cleaned it. Every polars analysis of the plots starts with the same steps, so they belong in one function.

Input → output: the plots file (a path, or the file's contents as bytes: pl.read_csv takes either) → a polars DataFrame with the file's eight columns, then area_ha:

  • production_kg as whole numbers (Int64), its thousands separators removed;
  • each row that is an exact repeat of another kept only once;
  • area_ha: area when area_unit is "ha", area / 10_000 when it is "sqm".
hh_id plot_no crop area area_unit … production_kg fertilizer_kg area_ha
HH-0001 1 Rice 0.439 ha … 1640 100.7 0.439
HH-0001 2 Rice 1940.0 sqm … 700 null 0.194

A blank crop stays null: polars reads an empty field as missing, and the plot is still a plot.

Tools: pl.read_csv(source, schema_overrides={"production_kg": pl.String}) so the column is text whatever its first rows hold; .str.replace_all(",", "").cast(pl.Int64); .unique(maintain_order=True); and for the unit, polars' version of pandas' where:

pl.when(pl.col("area_unit") == "sqm").then(pl.col("area") / 10_000).otherwise(pl.col("area"))

If the check is red: common causes

what you see the usual cause look at
InvalidOperationError: conversion from str to i64 failed The cast comes before the commas are removed. .str.replace_all(",", "") first, then .cast(pl.Int64)
InvalidOperationError: expected String type, got: i64 (second file) polars guessed numbers, so there is no text to clean. schema_overrides={"production_kg": pl.String}
the first file has 6 rows, not 5 The repeated row is still there. .unique(maintain_order=True)
area_ha is huge for the sqm plots Multiplied by 10,000 instead of dividing, or the condition is the wrong way round. .then(pl.col("area") / 10_000) where the unit is "sqm"
production_kg is Float64 Cast to a decimal type. .cast(pl.Int64)
fewer rows than expected Rows with a blank crop or fertilizer were dropped. no drop_nulls(): a missing value is not a reason to lose the plot
ColumnNotFoundError A column name misspelt. the names in the message's list of valid columns

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 6.2. Irrigated rice, and its yield

The task

Why it matters: filtering and adding a column are the two steps of almost every analysis. In module 3 (20_dataframes.py) you wrote them in pandas; here they are filter and with_columns.

Input → output: the clean plots, in polars (with area_ha) → only the plots whose crop is "Rice" and whose irrigated is "Yes", with five columns, in this order: hh_id, plot_no, area_ha, production_kg, and a new yield_t_ha, the harvest in tonnes per hectare.

hh_id plot_no area_ha production_kg yield_t_ha
HH-0001 2 0.194 700 3.608…

A yield in tonnes per hectare is production_kg / area_ha / 1000: kilograms per hectare, then 1,000 kilograms to the tonne. Use area_ha, never area, which is still in square meters for some plots.

Tools: filter with two conditions (separated by a comma, or joined with & and brackets), with_columns(yield_t_ha=...), select(...).

If the check is red: common causes

what you see the usual cause look at
TypeError: the truth value of an Expr is ambiguous and between the conditions, or & without brackets around each comparison. filter(cond1, cond2), or (cond1) & (cond2)
rain-fed rice or corn in the result Only one of the two conditions is applied. both conditions in filter
yields 1,000 times too big Kilograms per hectare, not tonnes. / 1000 at the end
one yield thousands of times too small area used instead of area_ha. pl.col("area_ha")
the columns in another order, or one too many select is missing, or lists other columns. select("hh_id", "plot_no", "area_ha", "production_kg", "yield_t_ha")
ColumnNotFoundError: unable to find column "yield_t_ha" select comes before with_columns, so the column does not exist yet. the order of the steps

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 6.3. The weighted rice yield by province

The task

Why it matters: this is the headline number of the survey, from module 4 (30_summarize_and_combine.py): how much rice a hectare gives in each province, as the province's farms would show it, not only the sample's. Each household stands for weight farm households, so a plot counts as many times as its household's weight. Unweighted, the rice yield of the whole survey is 4.35 t/ha; weighted, 4.50.

Input → output: the clean plots and the clean households, both in polars → one row per province, sorted by province, with four columns:

province plots yield_t_ha weighted_t_ha
the province how many rice plots the mean of the plots' yields the same mean, weighted by each plot's household weight

The steps: keep the rice plots; join each plot to its household's province and weight (a household has many plots, one plot one household: validate="m:1"); add each plot's yield, as in Exercise 6.2; then group by province.

A worked example: two rice plots in one province, yields 3.0 and 6.0 t/ha, from households of weight 300 and 100. Unweighted, (3.0 + 6.0) / 2 = 4.5. Weighted, (3.0 × 300 + 6.0 × 100) / (300 + 100) = 3.75.

Tools: join(..., on="hh_id", how="left", validate="m:1"); group_by("province").agg(plots=pl.len(), yield_t_ha=..., weighted_t_ha=...); the weighted mean as one expression, (pl.col("x") * pl.col("weight")).sum() / pl.col("weight").sum(); sort("province").

If the check is red: common causes

what you see the usual cause look at
weighted_t_ha equals yield_t_ha The weight is not in the formula, or both columns use the same expression. (pl.col("yield_t_ha") * pl.col("weight")).sum() / pl.col("weight").sum()
weighted_t_ha hundreds of times too big The mean of yield times weight, not divided by the weights. divide by pl.col("weight").sum()
Iloilo's mean is 3.94, not 4.6 Total harvest over total area, not the mean of the plots' yields. the yield of each plot first, then .mean()
more plots than expected, or Corn in the numbers The rice filter is missing. filter(pl.col("crop") == "Rice")
ColumnNotFoundError: unable to find column "province" The join is missing, or comes after group_by. join before grouping
ComputeError: join keys did not fulfill m:1 validation The households table lists a household twice. the clean households, or unique() on them first
a province row that is null A plot whose household is not in the households table. how="left" keeps it: check the keys

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 6.4. A lazy query

The task

Why it matters: a query written lazily runs as it is on a file a thousand times bigger, because polars reads only what it needs. Returning the query, not its result, also lets whoever calls your function add steps before anything runs.

Input → output: the raw plots file (a path or bytes; pl.scan_csv takes either) → a LazyFrame, not run, that gives one row for each value of irrigated, among the rice plots of the file with its repeated rows dropped:

irrigated plots production_kg
No how many rice plots their total harvest, in kg
Yes … …

Tools: pl.scan_csv(source, schema_overrides=...), then the steps of Exercise 6.1 that this needs (production as whole numbers, unique()), filter, group_by("irrigated").agg(plots=pl.len(), production_kg=...), sort("irrigated"). Do not end with .collect(): the check collects it, after making sure it is still a query that starts by scanning the file.

If the check is red: common causes

what you see the usual cause look at
not a LazyFrame but a DataFrame The query ends with .collect(), or starts with pl.read_csv. pl.scan_csv(...), and no .collect()
starts by scanning the file is False pl.read_csv(...).lazy(): lazy, but only after reading everything. pl.scan_csv(source, ...)
one plot too many in the first file The repeated row is counted. .unique()
InvalidOperationError when the check runs it The production column is text when summed, or numbers when cleaned. schema_overrides, then .str.replace_all and .cast, as in Exercise 6.1
Corn in the numbers The rice filter is missing. filter(pl.col("crop") == "Rice")

A lazy query's errors appear only when it runs, at .collect(): the line the error points to is the collect, and the mistake is in one of the steps before it.

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 6.5. Back to pandas, with the types it had

The task

Why it matters: the household table went through polars for a fast step, and must now go back to a pandas function from module 4 (30_summarize_and_combine.py) that looks households up by hh_id in the index, and treats a not-stated age as missing, not as a decimal.

Input → output: households in polars (with hh_id, and head_sex and head_age as whole numbers that may be null) → a pandas DataFrame, indexed by hh_id, with head_sex and head_age as pandas' nullable whole numbers ("Int64"). Every other column as .to_pandas() gives it.

hh_id (index) province … head_sex head_age
HH-0001 Ilocos Norte … 1 58
HH-0012 … … <NA> 47

Tools: .to_pandas(), .set_index("hh_id"), .astype({"head_sex": "Int64", "head_age": "Int64"}).

If the check is red: common causes

what you see the usual cause look at
head_age is float64 .to_pandas() alone: a whole-number column with gaps comes back as decimals. .astype({"head_sex": "Int64", "head_age": "Int64"})
only the second table fails astype only where there are gaps; with none, the type is int64. "Int64" in every case
int64, not Int64 A lowercase "int64": the ordinary type, which cannot hold <NA>. "Int64", with a capital I
int64[pyarrow] use_pyarrow_extension_array=True: Arrow types, not pandas' own. plain .to_pandas()
the index is 0, 1, 2… and hh_id is a column The index is not set. .set_index("hh_id")
AttributeError: 'DataFrame' object has no attribute 'set_index' set_index called on the polars table, before .to_pandas(). the order of the steps

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Exercise 6.6. The prices, into polars

The task

Why it matters: course.prices() gives every farmgate price in one pandas table, with month as a monthly period, as module 5 (40_cleaning_and_time.py) left it. polars has no period type, so the month must become something polars understands before it crosses: a date, the first day of the month.

Input → output: the prices in pandas (month a monthly Period, then province, commodity, price) → the same rows in polars, with month as Date and the other three columns as they were.

month province commodity price
2022-01-01 Ilocos Norte Palay 18.2

Tools: prices["month"].dt.to_timestamp() (in pandas: the first moment of each month), prices.assign(...), pl.from_pandas(...), then .with_columns(pl.col("month").cast(pl.Date)) (in polars: the date, without a time of day).

If the check is red: common causes

what you see the usual cause look at
month shows numbers such as 624, its type Extension(...) The period crossed as it is. .dt.to_timestamp() before pl.from_pandas
month is Datetime(time_unit='us', ...) The time of day is still there. .cast(pl.Date)
month is String astype(str): text, not a date. Sorting and date ranges then work on letters. .dt.to_timestamp()
AttributeError: 'DataFrame' object has no attribute 'assign' assign called after pl.from_pandas: polars says with_columns. convert the month in pandas, before crossing
AttributeError: Can only use .dt accessor with datetimelike values .dt.to_timestamp() on a column that is not a period. prices["month"], from course.prices()

Still stuck after ten minutes? Ask your table, then raise your hand.

The solution, and why it works

Try it yourself first: the check names the input that failed, and the common causes above cover most red checks. Open this when you have been stuck for ten minutes, or when your check is green and you want to compare.

Something new

Identify results
LayerExplore a stretch exercises
Stretch exercises S3Stretch: charts
AboutStretch exercises of Arrays & DataFrames, after module 7: chart exercises on the survey and the prices, in the library of your choice, each with its check and its solution on the companion.
CourseArrays & DataFrames, free