Estimating CSV Memory Requirements
Overview
Large CSV extracts can easily exceed the memory available to a Jupyter workspace when loaded into pandas. This commonly results in notebook crashes, kernel restarts, or out-of-memory errors.
The CSV Memory Estimator provides a way to estimate the memory footprint of one or more CSV files without loading the entire dataset into memory. This allows you to assess whether a workload is likely to fit within the available memory before running your analysis.
The utility works by:
- Counting the total number of rows in the source file.
- Loading a sample of rows.
- Measuring the actual memory consumed by that sample.
- Calculating memory-per-row.
- Scaling the result to estimate the size of the full DataFrame.
This approach provides a practical estimate of how much memory pandas is likely to require while avoiding the risk of crashing the notebook during the assessment itself.
Installing the Utility
Create a file called:
memcheck.pyand paste the following code into it.
Complete Utility Code
from __future__ import annotations
"""
Pre-flight memory estimator for reading CSVs into pandas.
Estimates the resident RAM needed to load one or many CSV files as pandas
DataFrames, WITHOUT loading them fully.
Method:
- Read a sample of rows
- Measure actual pandas memory usage
- Calculate bytes-per-row
- Count total rows
- Estimate final DataFrame size
Useful for determining whether a dataset is likely to fit within a
Jupyter workspace before loading it.
"""
from pathlib import Path
import pandas as pd
def _count_data_rows(path: str | Path) -> int:
"""Count data rows excluding the header."""
with open(path, "rb") as f:
total = sum(1 for _ in f)
return max(total - 1, 0)
def estimate_csv_memory(
path: str | Path,
usecols: list[str] | None = None,
sample_rows: int = 50_000,
) -> dict:
path = Path(path)
header_cols = pd.read_csv(path, nrows=0).columns.tolist()
if usecols is not None:
cols = [c for c in usecols if c in header_cols]
missing = [c for c in usecols if c not in header_cols]
else:
cols = None
missing = []
total_rows = _count_data_rows(path)
sample = pd.read_csv(
path,
usecols=cols,
nrows=sample_rows,
)
n_sample = len(sample)
if n_sample == 0:
est_bytes = 0.0
bytes_per_row = 0.0
else:
sample_bytes = sample.memory_usage(deep=True).sum()
bytes_per_row = sample_bytes / n_sample
est_bytes = bytes_per_row * total_rows
return {
"file": path.name,
"rows": total_rows,
"cols_read": len(cols) if cols is not None else len(header_cols),
"cols_total": len(header_cols),
"sampled_rows": n_sample,
"bytes_per_row": bytes_per_row,
"est_bytes": est_bytes,
"est_gb": est_bytes / 1024**3,
"missing_cols": missing,
"sample_was_whole_file": total_rows <= n_sample,
}
def estimate_many(
files,
usecols_map: dict | None = None,
usecols: list[str] | None = None,
limit_gb: float = 10.0,
overhead_factor: float = 1.3,
sample_rows: int = 50_000,
) -> pd.DataFrame:
if isinstance(files, (str, Path)) and Path(files).is_dir():
paths = sorted(Path(files).glob("*.csv"))
else:
paths = [Path(p) for p in files]
rows = []
for p in paths:
cols = usecols
if usecols_map:
cols = (
usecols_map.get(p.name)
or usecols_map.get(p.stem)
or usecols
)
rows.append(
estimate_csv_memory(
p,
usecols=cols,
sample_rows=sample_rows,
)
)
df = pd.DataFrame(rows)
if df.empty:
print("No CSV files found.")
return df
total_gb = df["est_gb"].sum()
peak_gb = total_gb * overhead_factor
show = df[
["file", "rows", "cols_read", "cols_total", "est_gb"]
].copy()
show["est_gb"] = show["est_gb"].round(3)
with pd.option_context("display.max_rows", None):
print(show.to_string(index=False))
print("-" * 60)
print(f"Sum of final frames: {total_gb:6.2f} GB")
print(f"Estimated peak (x{overhead_factor}): {peak_gb:6.2f} GB")
print(f"Session limit: {limit_gb:6.2f} GB")
any_missing = [
(r["file"], r["missing_cols"])
for _, r in df.iterrows()
if r["missing_cols"]
]
if any_missing:
print("\nWARNING: requested columns not found:")
for f, m in any_missing:
print(f" {f}: {m}")
print()
if peak_gb <= limit_gb:
print(
f"VERDICT: likely FITS "
f"({peak_gb:.1f} GB peak <= {limit_gb} GB)"
)
elif total_gb <= limit_gb:
print(
f"VERDICT: RISKY "
f"({total_gb:.1f} GB fits but "
f"{peak_gb:.1f} GB peak exceeds limit)"
)
else:
print(
f"VERDICT: will NOT fit "
f"({total_gb:.1f} GB exceeds "
f"{limit_gb} GB limit)"
)
return dfExample: Estimate an Entire Extract
To estimate all CSV files within an extract directory:
from memcheck import estimate_many
estimate_many(
"../data/raw/my_extract",
limit_gb=10
)Example output:
file rows cols_read cols_total est_gb
patients.csv 5000000 12 12 0.89
encounters.csv 7500000 24 24 2.10
lab_components.csv 34000000 18 18 7.34
------------------------------------------------------------
Sum of final frames: 10.33 GB
Estimated peak (x1.3): 13.43 GB
Session limit: 10.00 GB
VERDICT: will NOT fitExample: Estimate After Column Reduction
You can estimate the impact of using usecols before changing your code:
estimate_many(
"../data/raw/my_extract",
usecols=[
"project_id",
"start_datetime"
],
limit_gb=10
)This is often the quickest way to identify large memory savings.
Example: Different Columns for Different Files
Many workflows use different columns from different datasets.
estimate_many(
"../data/raw/my_extract",
usecols_map={
"patients": [
"project_id"
],
"encounters": [
"project_id",
"admission_date"
],
"lab_components": [
"project_id",
"component_name",
"NumericValue"
]
},
limit_gb=10
)Understanding the Verdict
Likely FITS
VERDICT: likely FITSExpected peak memory usage is below the configured workspace limit.
RISKY
VERDICT: RISKYThe final DataFrames may fit, but temporary copies created during joins, concatenations or aggregations are likely to push memory usage over the limit.
Will NOT Fit
VERDICT: will NOT fitThe estimated memory required exceeds the available memory before any processing takes place.
In this situation you should:
- Reduce columns using
usecols - Process data in chunks
- Use Polars or DuckDB
- Increase workspace memory allocation
Best Practice
Before loading any large extract:
- Run
estimate_many(). - Review the predicted peak memory usage.
- Reduce columns where possible.
- Re-run the estimate.
- Only proceed when the workload comfortably fits within available memory.
A few seconds spent estimating memory can prevent the considerably longer process of recovering from a crashed notebook session.
Summary
The CSV Memory Estimator provides a safe way to predict how much memory a pandas workload is likely to consume before any large files are fully loaded. By sampling rows and extrapolating memory usage, it helps users avoid kernel crashes, identify oversized workloads, and make informed decisions about column selection and processing strategies.
Using this utility as part of your normal workflow can significantly reduce memory-related issues when working with large extracts in Jupyter workspaces.