Processing Very Large Datasets with Parquet - Without Copying It

How to analyse multi-terabyte time-series and sensor datasets held in object storage, directly from your analysis environment — across Python, R, MATLAB, and interactive visualisation — without downloading the data and without loading it into memory.

The problem

Research datasets from wearables, sensors, and other digital-health technologies get large fast — hundreds of millions to billions of rows, terabytes on disk. The instinct is to copy the data into each workspace before analysing it. At scale that's slow, expensive (storage duplication and egress), and often simply impractical.

Parquet plus object storage offers a better model: leave the data where it is, and read only the parts each query needs. This article shows the pattern and gives working examples in four tools, all reading the same dataset in object storage.

The example dataset throughout is a synthetic ALS (motor neurone disease) wearable study — accelerometry and derived metrics, hundreds of millions of observations, a few GB of Parquet — but the pattern applies to any large columnar time-series dataset.

Why Parquet makes this possible

Parquet is a columnar file format with two properties that make selective reading possible:

  1. Footer metadata. Every Parquet file carries a footer describing its schema, row counts, and per-column statistics (including the min/max of each column in each row group). You can characterise an entire dataset — its shape, schema, and size — by reading only footers, without touching any of the actual data.
  2. Row groups + columns are independently addressable. Data is stored column-by-column, in blocks called row groups. A reader can fetch only the columns a query references (column projection) and skip entire row groups whose statistics show they can't match a filter (row-group pruning). Combined with partitioning (organising files into folders by, say, date or subject), a well-laid-out Parquet dataset lets a query read a tiny fraction of the total bytes.

The result: a bounded question ("this subject, this time window, these two columns") touches a tiny slice of a huge dataset — and the raw data never leaves object storage in bulk.

Important distinction. Object storage (S3-compatible) serves bytes; it does not run your computation. The efficiency here is reduced data movement — fetching only the columns and row groups you need — not server-side compute. Your analysis runs in your workspace; you just move far less data into it.

Prerequisites

  • A Parquet dataset in S3-compatible object storage (AWS S3, Azure Blob via an S3 layer, MinIO, etc.).
  • Network access: your workspace must be allowed to reach the object-store endpoint. If your environment restricts outbound traffic, allow-list the specific bucket URL in the workspace firewall — you don't need broad egress, just the one bucket.
  • If behind an HTTP proxy: some client libraries (notably PyArrow's C++ S3 layer) don't read the HTTP(S)_PROXY environment variables automatically — you pass the proxy explicitly (shown below).
  • Libraries per tool: pyarrow + s3fs (Python), duckdb/duckplyr (R), a recent MATLAB (parquetDatastore), and a plotting library for the visualisation.

The pattern, in five steps

  1. Connect to object storage by reference.
  2. Inspect the dataset from footer metadata (schema, row counts — no data read).
  3. Estimate/plan which columns and row groups a query needs.
  4. Analyse by streaming batches with column projection and row-group pruning — process hundreds of millions of rows without holding them in memory.
  5. Curate derived datasets and write them back to storage — without copying the source.

Python (PyArrow)

A single self-contained script showing the whole pattern — open by reference, inspect from metadata, stream a filtered/projected scan, write a derived summary back:

#!/usr/bin/env python3
"""Stream and process large Parquet from S3 — no download, no full in-memory load."""
import os

# In a non-AWS container the AWS SDK otherwise stalls ~15s per call trying to
# reach the EC2 metadata endpoint. Disable it before importing pyarrow.
os.environ.setdefault("AWS_EC2_METADATA_DISABLED", "true")

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.dataset as ds
import pyarrow.fs as pafs
import pyarrow.parquet as pq
from urllib.parse import urlparse
import s3fs

BUCKET = "your-bucket"
SOURCE = f"s3://{BUCKET}/wearable-data"
DERIVED = f"s3://{BUCKET}/wearable-data/derived/summary"
REGION = os.getenv("AWS_DEFAULT_REGION", "eu-west-2")

def filesystem() -> pafs.S3FileSystem:
    # Anonymous read shown; pass access_key/secret_key for authenticated buckets.
    # Proxy is passed explicitly — PyArrow's C++ S3 SDK does not read HTTP(S)_PROXY.
    kwargs = {"region": REGION, "anonymous": True}
    proxy = os.getenv("https_proxy") or os.getenv("http_proxy")
    if proxy:
        pp = urlparse(proxy if "://" in proxy else f"http://{proxy}")
        kwargs["proxy_options"] = {"scheme": pp.scheme or "http",
                                   "host": pp.hostname, "port": pp.port}
    return pafs.S3FileSystem(**kwargs)

def path(uri: str) -> str:
    u = urlparse(uri)
    return u.netloc + u.path

# 1 & 2: open by reference; inspect from footer metadata (no row bodies read)
fs = filesystem()
dataset = ds.dataset(path(SOURCE), filesystem=fs, format="parquet", partitioning="hive")
print(f"Dataset: {dataset.count_rows():,} rows")
print("Schema:", dataset.schema.names)

# 3 & 4: stream a projected, pruned scan — memory-bounded over the whole dataset
scanner = dataset.scanner(
    columns=["patient_id", "metric_type", "value"],    # column projection
    filter=(pc.field("metric_type") == "gait_speed"),  # predicate -> row-group pruning
    use_threads=True,
)
running_sum, running_count = 0.0, 0
for batch in scanner.to_batches():        # one batch in memory at a time
    values = batch.column("value")
    running_sum += pc.sum(values).as_py() or 0.0
    running_count += len(values)
mean_value = running_sum / running_count if running_count else float("nan")
print(f"Scanned {running_count:,} rows; mean = {mean_value:.3f}")

# 5: write a curated derived dataset back to storage (no full-source copy).
# Writing via s3fs uses a single PutObject for small files, avoiding the
# always-multipart behaviour of PyArrow's C++ writer (matters for object
# stores that only permit single-part / anonymous writes).
summary = pa.table({
    "metric_type": pa.array(["gait_speed"]),
    "rows_analysed": pa.array([running_count], type=pa.int64()),
    "mean_value": pa.array([mean_value], type=pa.float64()),
})
s3 = s3fs.S3FileSystem(anon=True)
with s3.open(f"{path(DERIVED)}/summary-0.parquet", "wb") as f:
    pq.write_table(summary, f)
print(f"Wrote derived summary to {DERIVED}")

The key lines are the scanner(...) call — columns= does column projection, filter= drives row-group pruning — and the for batch in scanner.to_batches() loop, which streams the data through rather than materialising it. The dataset can be far larger than memory.

R (DuckDB via duckplyr)

DuckDB reads Parquet directly from S3 over its httpfs extension. Queries are lazy — the read only happens on collect(), and DuckDB pushes down column projection and row-group pruning automatically:

library(duckplyr)
library(dplyr)

# Configure DuckDB's S3 access (proxy + httpfs for object storage)
duckplyr::db_exec(sprintf("SET http_proxy='%s'", Sys.getenv("https_proxy")))
duckplyr::db_exec("INSTALL httpfs")
duckplyr::db_exec("LOAD httpfs")

# Open the partitioned dataset by reference (lazy — no data read yet)
data <- read_parquet_duckdb(
  "s3://your-bucket/wearable-data/study_id=*/year=*/month=*/part-*.parquet",
  prudence = "stingy",                       # be conservative about what to fetch
  options = list(hive_partitioning = TRUE)
)

# Windowed summary — reads only the columns it needs, only matching row groups
summary <- data |>
  filter(month == "01") |>
  summarise(
    observations = n(),
    mean_value   = mean(value, na.rm = TRUE),
    minimum      = min(value, na.rm = TRUE),
    maximum      = max(value, na.rm = TRUE),
    .by = metric_type
  ) |>
  collect()                                  # triggers the pruned read

print(summary)

A useful thing to observe: a query that touches one column (e.g. a count by quality flag) is faster and moves far less data than one that also reads a heavy numeric column — even over the same number of rows. Transfer scales with the columns you touch, not the row count. DuckDB reads only the needed columns from object storage and computes locally, returning a small result.

MATLAB

MATLAB's parquetDatastore treats the remote Parquet as a streaming datastore — a reference that reads in chunks, not a full load:

files = [ ...
    "https://your-bucket.s3.eu-west-2.amazonaws.com/wearable-data/study_id=STUDY-001/year=2026/month=01/part-00000.parquet"
    "https://your-bucket.s3.eu-west-2.amazonaws.com/wearable-data/study_id=STUDY-001/year=2026/month=02/part-00001.parquet"
    ];

ds = parquetDatastore(files);
preview(ds);                                   % peek at schema + first rows
disp(ds);
fprintf("Dataset contains %d Parquet files\n", numel(ds.Files));
fprintf("Variables: %d\n", numel(ds.VariableNames));
disp(ds.VariableNames')

The datastore streams through the data in chunks rather than reading it all into memory, so the same large dataset is usable from MATLAB without a bulk copy — useful when part of your research community works in MATLAB rather than Python or R.

Interactive visualisation

You can visualise a large slice interactively without materialising every point in the browser. The pattern: a pruned read pulls only the needed columns and time window from object storage; a level-of-detail (LOD) charting engine then draws only what the screen resolution needs, so millions of points remain interactive.

import pyarrow.compute as pc
import pyarrow.dataset as ds
import numpy as np
import xy   # a level-of-detail charting library

# Pruned read: only the columns and time window we plot leave storage
dataset = ds.dataset("your-bucket/wearable-data", filesystem=fs,
                     format="parquet", partitioning="hive")
table = dataset.scanner(
    columns=["observed_at", "value", "patient_id"],
    filter=(pc.field("metric_type") == "gait_speed"),
).to_table()

# Marshal to the charting engine's fast path (numeric, single-chunk)
x = pc.cast(pc.cast(table["observed_at"], pa.int64()), pa.float64()).combine_chunks()
y = pc.cast(table["value"], pa.float64()).combine_chunks()

chart = xy.scatter_chart(
    xy.scatter(x, y, density=True),
    xy.x_axis(type_="time", format="%Y-%m-%d", label="observed_at"),
    xy.y_axis(label="gait_speed"),
)
chart.to_html("preview.html")     # or render inline in a notebook

Note the two distinct efficiencies: the read from object storage is pruned (only needed columns/window), and the render is level-of-detail (only what the screen needs). Neither streams pixels from storage; together they let you explore a large dataset that was never downloaded in full.

Choosing what leaves storage: full scan vs selective query

The size of the win depends on how the data is laid out and how bounded the query is:

  • A whole-dataset scan that only needs some columns still benefits from column projection — you skip the columns you don't reference. Modest but real (e.g. reading 6 of 11 columns).
  • A bounded query (a time window, a subject) additionally benefits from row-group and partition pruning — the reader skips entire row groups whose statistics can't match. This can be dramatic: a narrow time window against a large dataset may touch a tiny fraction of row groups.

The practical implication: lay your Parquet out for how you query it. Partition by the fields you filter on (e.g. subject and time granularity), and sort/cluster within files by timestamp, so row-group statistics are tight and pruning is effective. A naive, unpartitioned, unsorted dump of Parquet files gets almost none of these benefits and forces full scans — the format is only half the story; the layout is the rest.

Materialise derived tiers to avoid recomputation

High-frequency raw data (e.g. 30 Hz accelerometry) is usually too granular to query directly for analytical questions. Rather than recompute windowed aggregates from raw every time, compute them once and store them as derived Parquet tiers:

  • Raw — the full-resolution samples.
  • Rollup — windowed aggregates (per epoch, per hour) computed from raw.
  • Derived — analytical features/summaries (per subject, per day).

Queries then read the smallest tier that answers them, and you never re-scan billions of raw rows for a daily trend. The transform/aggregate pattern in the Python example (writing a curated derived dataset back to storage) is the building block for this — an "ingest/curate once, read many" model that avoids duplicating the raw corpus per project.

Summary

  • Parquet's footer metadata + columnar layout let you inspect, cost, and query a huge dataset while reading only the bytes you need.
  • Column projection and row-group/partition pruning minimise data movement — provided the data is laid out for how you query it.
  • The pattern works identically across Python, R, MATLAB, and interactive visualisation — the same data in object storage, read selectively by each tool, never copied in bulk, never fully in memory.
  • Materialise derived tiers so you query rollups, not raw, for analytical questions.
  • The compute runs in your workspace; the efficiency is in moving less data — keep compute close to the data and location matters little.

You can read more about this in our blog.

Updated on August 05, 2026
Back to top