Analysing Large Scale Parquet datasets from remote storage

How to analyse multi-terabyte time-series and sensor datasets stored in object storage directly from your analysis environment, across Python, R, MATLAB, and interactive visualisation tools, without downloading the dataset in full or loading it entirely into memory.

The challenge

Datasets generated by wearables, sensors, and other digital health technologies can grow rapidly. Studies frequently produce hundreds of millions or billions of records, resulting in datasets that occupy gigabytes or terabytes of storage.

A common approach is to copy data into individual workspaces before analysis begins. As datasets increase in size, this becomes more difficult to manage. Transfer times increase, storage capacity is consumed by duplicate copies, and moving data between environments can become a significant operational overhead.

When data is stored as Parquet files in object storage, analysis can be performed directly against the source dataset. Rather than retrieving every file, analysis tools can access only the sections of data required for a particular query.

The examples in this article use a large wearable-device dataset containing accelerometry measurements and derived metrics, represented by hundreds of millions of observations stored as Parquet files. The same techniques apply to many other large columnar datasets.

Why Parquet is well suited to large datasets

Parquet includes several features that help large datasets remain practical to analyse.

Metadata stored in file footers

Each Parquet file contains metadata describing its schema, row count, and statistics for individual columns within each row group. These statistics typically include minimum and maximum values.

This information can be read independently of the underlying records. As a result, tools can inspect dataset structure, schemas, and record counts without scanning the data itself.

Independent access to columns and row groups

Parquet stores data in a column-oriented format and organises records into row groups.

This allows readers to:

  • Read only the columns referenced by a query.
  • Exclude row groups that cannot contain matching records based on stored statistics.
  • Combine these optimisations with dataset partitioning to further reduce data access.

For example, a query limited to a specific participant and time period may access only a small subset of files, row groups, and columns even when the overall dataset spans many terabytes.

Object storage and computation

Object storage and computation serve different roles.

Object storage services provide access to the underlying data. Query execution continues to occur within the analysis environment itself. The efficiency comes from reducing the amount of data that must be transferred from storage rather than performing computation within the storage platform.

Prerequisites

The examples in this guide assume:

  • A Parquet dataset stored in S3-compatible object storage such as AWS S3, MinIO, or Azure Blob Storage through an S3-compatible interface.
  • Network connectivity from the analysis environment to the object storage endpoint.
  • Any required firewall rules configured to allow access to the target bucket.
  • Explicit proxy configuration where required by client libraries.
  • Appropriate libraries available for each environment:
    • Python: pyarrow and s3fs
    • R: DuckDB and duckplyr
    • MATLAB with support for parquetDatastore
    • A visualisation library suitable for interactive plotting

Workflow

The same general approach is used regardless of language or tool:

  1. Connect to the dataset in object storage.
  2. Inspect dataset metadata.
  3. Identify the columns and records required for the analysis.
  4. Process data in batches using projection and pruning techniques.
  5. Write derived outputs back to storage when required.

This approach allows datasets that are substantially larger than available memory to be analysed efficiently.

Python (PyArrow)

The following example opens a dataset directly from object storage, inspects its metadata, streams a filtered query, and writes a derived summary back to storage.

#!/usr/bin/env python3
"""Stream and process large Parquet from S3 — no download, no full in-memory load."""

import os

# In a non-AWS container the AWS SDK otherwise stalls ~15s per call trying to
# reach the EC2 metadata endpoint. Disable it before importing pyarrow.
os.environ.setdefault("AWS_EC2_METADATA_DISABLED", "true")

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.dataset as ds
import pyarrow.fs as pafs
import pyarrow.parquet as pq
from urllib.parse import urlparse
import s3fs

BUCKET = "your-bucket"
SOURCE = f"s3://{BUCKET}/wearable-data"
DERIVED = f"s3://{BUCKET}/wearable-data/derived/summary"
REGION = os.getenv("AWS_DEFAULT_REGION", "eu-west-2")


def filesystem() -> pafs.S3FileSystem:
    # Anonymous read shown; pass access_key/secret_key for authenticated buckets.
    # Proxy is passed explicitly — PyArrow's C++ S3 SDK does not read HTTP(S)_PROXY.
    kwargs = {"region": REGION, "anonymous": True}
    proxy = os.getenv("https_proxy") or os.getenv("http_proxy")
    if proxy:
        pp = urlparse(proxy if "://" in proxy else f"http://{proxy}")
        kwargs["proxy_options"] = {
            "scheme": pp.scheme or "http",
            "host": pp.hostname,
            "port": pp.port
        }
    return pafs.S3FileSystem(**kwargs)


def path(uri: str) -> str:
    u = urlparse(uri)
    return u.netloc + u.path


# 1 & 2: open by reference; inspect from footer metadata (no row bodies read)
fs = filesystem()
dataset = ds.dataset(
    path(SOURCE),
    filesystem=fs,
    format="parquet",
    partitioning="hive"
)

print(f"Dataset: {dataset.count_rows():,} rows")
print("Schema:", dataset.schema.names)

# 3 & 4: stream a projected, pruned scan
scanner = dataset.scanner(
    columns=["patient_id", "metric_type", "value"],
    filter=(pc.field("metric_type") == "gait_speed"),
    use_threads=True,
)

running_sum, running_count = 0.0, 0
for batch in scanner.to_batches():
    values = batch.column("value")
    running_sum += pc.sum(values).as_py() or 0.0
    running_count += len(values)

mean_value = (
    running_sum / running_count
    if running_count
    else float("nan")
)
print(f"Scanned {running_count:,} rows; mean = {mean_value:.3f}")

# 5: write a curated derived dataset back to storage
summary = pa.table({
    "metric_type": pa.array(["gait_speed"]),
    "rows_analysed": pa.array([running_count], type=pa.int64()),
    "mean_value": pa.array([mean_value], type=pa.float64()),
})

s3 = s3fs.S3FileSystem(anon=True)
with s3.open(f"{path(DERIVED)}/summary-0.parquet", "wb") as f:
    pq.write_table(summary, f)

print(f"Wrote derived summary to {DERIVED}")

The scanner() configuration defines the columns required for the query and the filter used to limit the data read. The to_batches() iterator processes the dataset incrementally, allowing datasets much larger than available memory to be analysed.

R (DuckDB via duckplyr)

DuckDB can query Parquet files stored in object storage through its httpfs extension. Queries remain lazy until results are collected.

library(duckplyr)
library(dplyr)

# Configure DuckDB's S3 access (proxy + httpfs for object storage)
duckplyr::db_exec(sprintf("SET http_proxy='%s'", Sys.getenv("https_proxy")))
duckplyr::db_exec("INSTALL httpfs")
duckplyr::db_exec("LOAD httpfs")

# Open the partitioned dataset by reference (lazy — no data read yet)
data <- read_parquet_duckdb(
  "s3://your-bucket/wearable-data/study_id=*/year=*/month=*/part-*.parquet",
  prudence = "stingy",
  options = list(hive_partitioning = TRUE)
)

# Windowed summary
summary <- data |>
  filter(month == "01") |>
  summarise(
    observations = n(),
    mean_value = mean(value, na.rm = TRUE),
    minimum = min(value, na.rm = TRUE),
    maximum = max(value, na.rm = TRUE),
    .by = metric_type
  ) |>
  collect()

print(summary)

Query performance is often influenced by the number and type of columns being read. A query that accesses a small categorical column generally transfers less data than one that also requires large numerical columns, even when both operate on the same number of rows. DuckDB reads only the columns referenced by the query and performs computation locally before returning a result.

MATLAB

MATLAB's parquetDatastore provides access to Parquet files through a datastore interface that reads data incrementally rather than loading complete datasets into memory.

files = [
    "https://your-bucket.s3.eu-west-2.amazonaws.com/wearable-data/study_id=STUDY-001/year=2026/month=01/part-00000.parquet"
    "https://your-bucket.s3.eu-west-2.amazonaws.com/wearable-data/study_id=STUDY-001/year=2026/month=02/part-00001.parquet"
];

ds = parquetDatastore(files);

preview(ds);   % peek at schema + first rows
disp(ds);

fprintf("Dataset contains %d Parquet files\n", numel(ds.Files));
fprintf("Variables: %d\n", numel(ds.VariableNames));
disp(ds.VariableNames')

The datastore reads data in manageable chunks, allowing datasets larger than available memory to be processed directly from their source location. This provides a workflow for MATLAB users that is comparable to the Python and R examples.

Interactive visualisation

Large datasets can also be explored interactively without rendering every point individually. A common pattern is to combine selective data retrieval with a level-of-detail rendering engine.

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.dataset as ds
import xy  # a level-of-detail charting library

dataset = ds.dataset(
    "your-bucket/wearable-data",
    filesystem=fs,
    format="parquet",
    partitioning="hive"
)

table = dataset.scanner(
    columns=["observed_at", "value", "patient_id"],
    filter=(pc.field("metric_type") == "gait_speed"),
).to_table()

x = pc.cast(
    pc.cast(table["observed_at"], pa.int64()),
    pa.float64()
).combine_chunks()
y = pc.cast(table["value"], pa.float64()).combine_chunks()

chart = xy.scatter_chart(
    xy.scatter(x, y, density=True),
    xy.x_axis(type_="time", format="%Y-%m-%d", label="observed_at"),
    xy.y_axis(label="gait_speed"),
)

chart.to_html("preview.html")

Two separate optimisations are involved. The data retrieval step reads only the required columns and records from storage, and the visualisation layer reduces rendering work to the level of detail needed for the current display. Together, these techniques support interactive exploration of datasets that are far larger than the information ultimately shown on screen.

Choosing what data is read

The amount of data transferred from storage depends on both the query and the structure of the dataset.

Queries that require only a subset of columns benefit from column projection because unnecessary columns are never read. Even when every row must be examined, reducing the number of columns can significantly reduce data transfer.

Additional savings are available when queries include filters. Partition pruning and row-group pruning enable readers to exclude sections of the dataset that cannot contain matching records. Queries restricted to specific participants, dates, or time ranges may therefore access only a small fraction of the available data.

Dataset layout plays an important role in achieving these benefits. Partitioning should reflect commonly used filtering dimensions such as study, participant, or time period. Within files, data should be organised in a way that supports effective row-group statistics. Datasets stored as unpartitioned Parquet files often require substantially more scanning despite using the same file format.

Creating derived data tiers

High-frequency sensor data is often more detailed than required for routine analytical work. Recomputing aggregates from raw measurements for every query introduces unnecessary processing overhead.

A common approach is to maintain multiple levels of derived data:

  • Raw – original measurements at full resolution.
  • Rollup – aggregated measures calculated across fixed windows such as epochs or hours.
  • Derived – analytical features and summary measures generated from the underlying data.

Queries can then operate against the most appropriate tier for the task. This reduces repeated processing of raw data and helps provide consistent analytical inputs across projects.

The pattern shown in the Python example, where a derived dataset is written back to object storage, provides a simple example of this workflow.

Summary

Parquet stores metadata and column statistics that allow analysis tools to examine dataset structure before reading the underlying records.

Column projection, row-group pruning, and partition pruning reduce the amount of data transferred from object storage by limiting reads to the data required for a particular query.

The same approach can be used across Python, R, MATLAB, and visualisation tools, allowing multiple analysis environments to work directly against shared datasets without maintaining separate copies.

For large-scale analytical platforms, storing rollups and derived datasets alongside raw data can further reduce processing costs and improve query performance. Query execution continues to run within the analysis environment, while object storage acts as the shared source of data.

Updated on September 04, 2026
Back to top