Working with large CSVs efficiently in Jupyter

Notes for anyone hitting "the kernel died" when reading data extracts into pandas in a workspace with a fixed memory limit. By default the Jupyter app has 10GB of memory.

The most common Jupyter crashes are out-of-memory errors where the container dies and becomes unavailable (or says it suddenly can't authenticate). A library like Pandas reads the whole CSV/TSV/Text files into memory, even if you don't need to work with all of the columns (or rows) of data in it.

Start by reading only the columns you need. Most tables have a lot of columns the analysis never touches, and dropping them on read is usually enough by itself. This applies to every file, not only the obvious big ones. A table doesn't have to be the largest to be mostly dead weight.

# reads every column
flowsheet = pd.read_csv("flowsheet_rows.csv")

# reads only what's used
flowsheet = pd.read_csv(
    "flowsheet_rows.csv",
    usecols=["project_id", "start_datetime"]
)

To find the columns a table needs, look at what the code does with it: the columns it selects, filters on, groups by, or plots. Only those go in usecols.

On very wide or very long tables you may also hit a low_memory mixed-types warning, or an IndexError from pandas' chunked reader. Don't use low_memory=False to get rid of it. That re-reads the whole file to work out the types and puts the memory straight back. Set the column types explicitly instead, and parse the dates afterwards:

lab = pd.read_csv(
    path / "lab_components.csv",
    usecols=[
        "project_id",
        "start_datetime",
        "component_name",
        "NumericValue",
        "Unit"
    ],
    dtype={
        "project_id": "str",
        "start_datetime": "str",
        "component_name": "str",
        "NumericValue": "float64",
        "Unit": "str"
    },
)

lab["start_datetime"] = pd.to_datetime(
    lab["start_datetime"],
    errors="coerce"
)

If you're comparing two extracts, you often don't need both loaded in full. The comparison (row counts, patient counts, date ranges, per-patient spans) is a set of summaries you can get from a few columns of each. If you do need to switch between them, restart the kernel in between so the previous one isn't still sitting in memory.

Checking before you load

memcheck.py estimates how much memory a set of CSVs will need without loading them fully, so it won't crash the kernel while checking. Point it at a directory or a list of files, and set the limit to whatever the session has:

from memcheck import estimate_many

estimate_many("../data/raw/my_extract", limit_gb=10)

# with the columns you'd actually keep, to see the footprint drop
estimate_many(
    "../data/raw/my_extract",
    usecols=["project_id", "start_datetime"],
    limit_gb=10
)

It prints a per-file breakdown and says whether it's likely to fit.

When trimming columns isn't enough

Even with every file trimmed, there's a limit. You might be able load one large dataset selectively and work with it, but when an operation like a concat or group-by builds a temporary copy on top of what's already loaded, there may not be enough memory for the way Pandas works. Trimming columns gets the code running, but it doesn't remove the limitations though.

Moving to Polars

Polars reads data lazily, straight from your CSV/TSV/Text. It only pulls the columns and rows a query needs, and can stream through data too big to hold all at once. Do the loading and the heavy reducing in Polars, then convert the result back to pandas so your existing matplotlib / seaborn / reporting code still works. You only convert back what you need to plot or report.

import polars as pl

result = (
    pl.scan_csv("lab_components.csv")   # lazy, nothing read yet
      .filter(pl.col("component_name") == "Creatinine")
      .select(["project_id", "start_datetime", "NumericValue"])
      .collect()          # runs now, streaming
      .to_pandas()        # small result, back to pandas
)

The filter and column selection happen inside the scan, so the full table is never held in memory. For example, you get the Creatinine rows, three columns, and the result.

For a sense of scale: an EMR data dump for a cohort study, around twenty tables with a couple of very large ones (tens of millions of rows). With columns trimmed, the comparison ran in about 126 seconds in pandas, and only finished with other notebooks closed to free up memory. The same work in Polars ran in about 25 seconds without clearing anything. That's about 5x faster here. It varies by workload, but the memory side is the consistent difference: pandas only just fits, Polars has room to spare because it never holds the whole dataset at once.

Installing (if not already available):

pip install polars

If import polars fails with an "illegal instruction" error on the machine's CPU, install polars-lts-cpu instead. Same API. (pyarrow, needed for .to_pandas(), is usually already there.)

pandas to Polars: common translations

Same operations, different syntax. These cover the usual data-prep idioms. The plotting doesn't change because you convert back to pandas first.

Task	pandas	Polars
Read a CSV	`pd.read_csv(path)`	`pl.read_csv(path)` / `pl.scan_csv(path)` (lazy)
Read only some columns	`pd.read_csv(path, usecols=[...])`	`pl.scan_csv(path).select([...])`
Select columns	`df[["a", "b"]]`	`df.select(["a", "b"])`
Filter rows	`df[df["x"] == 1]`	`df.filter(pl.col("x") == 1)`
Filter, multiple conditions	`df[(df.x == 1) & (df.y > 0)]`	`df.filter((pl.col("x") == 1) & (pl.col("y") > 0))`
Parse a date column	`pd.to_datetime(df["d"], errors="coerce")`	`pl.col("d").str.to_datetime(strict=False)`
Drop nulls in a column	`df.dropna(subset=["d"])`	`df.drop_nulls("d")`
New/derived column	`df["c"] = df["a"] - df["b"]`	`df.with_columns((pl.col("a") - pl.col("b")).alias("c"))`
Group + aggregate	`df.groupby("id")["x"].agg(["min", "max"])`	`df.group_by("id").agg(pl.col("x").min().alias("min"), pl.col("x").max().alias("max"))`
Unique values	`df["id"].unique()`	`df.select("id").unique()`
Count unique	`df["id"].nunique()`	`df.select(pl.col("id").n_unique())`
Days between dates	`(df["b"] - df["a"]).dt.days`	`(pl.col("b") - pl.col("a")).dt.total_days()`
Rename a column	`df.rename(columns={"a": "b"})`	inline via `.alias("b")` in select/agg
Sort	`df.sort_values("x", ascending=False)`	`df.sort("x", descending=True)`
Convert to pandas		`df.to_pandas()`

Two differences to know about. Polars has no row index, so set_index, reset_index and .loc[label] have no equivalent; you use .filter() and .select() instead. And scan_csv is lazy: it builds a plan and nothing runs until .collect(), which is what lets it read only the columns and rows you need. Put filters and column selections before .collect().

A side-by-side example

pandas:

lab_creat = lab[lab["component_name"] == "Creatinine"][
    ["project_id", "start_datetime", "NumericValue", "Unit"]
].copy()

lab_creat["date"] = pd.to_datetime(
    lab_creat["start_datetime"],
    errors="coerce"
)

per_patient = lab_creat.groupby(
    "project_id"
)["NumericValue"].agg(["min", "max"])

Polars, with the filter and selection pushed into the scan:

lab_creat = (
    pl.scan_csv("lab_components.csv")
      .filter(pl.col("component_name") == "Creatinine")
      .select(["project_id", "start_datetime", "NumericValue", "Unit"])
      .with_columns(
          pl.col("start_datetime")
            .str.to_datetime(strict=False)
            .alias("date")
      )
)

per_patient = (
    lab_creat.group_by("project_id")
      .agg(
          pl.col("NumericValue").min().alias("min"),
          pl.col("NumericValue").max().alias("max")
      )
      .collect()
      .to_pandas()
)

This version never holds the whole table. You get the Creatinine rows, four columns, and the per-patient summary.

Updated on September 03, 2026