Efficiently Working with Large CSV Datasets in R Web Applications

As R web applications become more widely used for exploratory analysis, operational dashboards, and data products, performance issues can emerge when working with large CSV files. Applications that perform well with a few thousand rows can become slow, memory-intensive, or unstable when datasets grow into the millions of records.

This article discusses common challenges associated with loading large CSV files into R web applications and presents several approaches for improving performance. The examples are aimed at data scientists who are building analytical applications rather than software engineers designing large-scale data platforms.

The challenge

A common pattern in many R web applications is to load one or more CSV files into memory when the application starts.

library(shiny)
library(readr)

server <- function(input, output, session) {

  data <- read_csv("large_dataset.csv")

  output$summary <- renderTable({
    data |>
      dplyr::group_by(category) |>
      dplyr::summarise(
        observations = n(),
        mean_value = mean(value, na.rm = TRUE)
      )
  })
}

This works well for smaller datasets, but several issues appear as data volumes increase:

  • The entire CSV file must be parsed before analysis can begin.
  • All columns are loaded into memory, whether they are needed or not.
  • Every application worker may hold its own copy of the dataset.
  • Application startup times increase.
  • Memory consumption grows quickly.
  • Interactive filters may repeatedly scan the entire dataset.

These issues are not specific to any particular framework. They arise because CSV is a simple storage format with limited support for efficient querying.

Why CSV becomes a bottleneck

CSV remains popular because it is portable, easy to understand, and supported by virtually every analytical tool. However, it has several characteristics that limit scalability.

A CSV file contains rows of text. To work with the data, R must:

  1. Read the entire file.
  2. Parse every row.
  3. Infer or apply data types.
  4. Store the resulting data frame in memory.

If a dataset contains 10 million rows, every import requires processing all 10 million rows regardless of whether the analysis only needs two columns or a small subset of records.

Consider a dataset containing:

  • 1 million records: approximately 100 MB
  • 10 million records: approximately 1 GB
  • 100 million records: approximately 10 GB

An R web application running on a shared server may struggle when multiple users each need access to datasets of this size.

Option 1: Convert CSV to Parquet

For analytical workloads, converting CSV files to Parquet is often the simplest improvement.

Parquet stores data in a columnar format. Unlike CSV, query engines can:

  • Read only required columns.
  • Skip large portions of data.
  • Access metadata without reading the entire file.
  • Reduce storage requirements through compression.

One-time conversion

The CSV only needs to be converted once.

library(arrow)

dataset <- read.csv("wearable_data.csv")

write_parquet(
  dataset,
  "wearable_data.parquet"
)

For larger collections of CSV files:

library(arrow)

tbl <- open_dataset(
  "raw_csv_directory",
  format = "csv"
)

write_dataset(
  tbl,
  path = "parquet_dataset",
  format = "parquet"
)

After conversion, analyses can read directly from Parquet rather than repeatedly processing CSV files.

Reading Parquet efficiently in an R web application

Instead of loading an entire dataset into memory, Parquet can be queried using Arrow's dataset interface.

library(shiny)
library(arrow)
library(dplyr)

dataset <- open_dataset(
  "parquet_dataset",
  format = "parquet"
)

ui <- fluidPage(
  selectInput(
    "metric",
    "Metric",
    choices = c("steps", "activity", "sleep")
  ),

  tableOutput("summary")
)

server <- function(input, output, session) {

  output$summary <- renderTable({

    dataset |>
      filter(metric_type == input$metric) |>
      summarise(
        observations = n(),
        mean_value = mean(value, na.rm = TRUE)
      ) |>
      collect()

  })

}

shinyApp(ui, server)

Notice that the dataset is not fully materialised. The filtering and aggregation operations are pushed to Arrow and only the final result is returned to R.

This approach can significantly reduce both memory consumption and data transfer.

Option 2: Query the data using DuckDB

DuckDB has become a popular solution for analytical workloads because it can query large files efficiently without requiring a separate database server.

For many R web applications, DuckDB provides an excellent balance between simplicity and performance.

Opening a CSV with DuckDB

library(DBI)
library(duckdb)

con <- dbConnect(duckdb())

dbExecute(
  con,
  "
  CREATE VIEW wearable_data AS
  SELECT *
  FROM read_csv_auto('wearable_data.csv')
  "
)

DuckDB does not need to import the CSV into a traditional database table. It can query the file directly.

Using DuckDB within an R web application

library(shiny)
library(DBI)
library(duckdb)

con <- dbConnect(duckdb())

dbExecute(
  con,
  "
  CREATE VIEW wearable_data AS
  SELECT *
  FROM read_csv_auto('wearable_data.csv')
  "
)

ui <- fluidPage(

  dateInput(
    "start_date",
    "Start Date"
  ),

  tableOutput("results")

)

server <- function(input, output, session) {

  output$results <- renderTable({

    sql <- sprintf(
      "
      SELECT
        metric_type,
        COUNT(*) AS observations,
        AVG(value) AS mean_value
      FROM wearable_data
      WHERE observed_at >= '%s'
      GROUP BY metric_type
      ",
      input$start_date
    )

    dbGetQuery(con, sql)

  })

}

shinyApp(ui, server)

DuckDB performs the filtering and aggregation before returning results to the application, reducing the amount of data that must be processed in memory.

Combining DuckDB and Parquet

DuckDB is particularly effective when querying Parquet datasets.

Suppose a dataset has already been converted from CSV to Parquet:

wearable-data/
├── year=2024/
├── year=2025/
└── year=2026/

DuckDB can query the data directly:

library(DBI)
library(duckdb)

con <- dbConnect(duckdb())

results <- dbGetQuery(
  con,
  "
  SELECT
    metric_type,
    AVG(value) AS mean_value
  FROM read_parquet('wearable-data/**/*.parquet')
  WHERE year = 2026
  GROUP BY metric_type
  "
)

print(results)

Only the relevant files and columns are accessed.

For analytical datasets containing tens or hundreds of millions of records, this can result in substantial reductions in processing time and memory usage.

Option 3: Pre-compute analytical datasets

Many analytical applications repeatedly execute the same calculations.

For example:

data |>
  group_by(patient_id) |>
  summarise(
    average_steps = mean(step_count)
  )

If the source dataset contains hundreds of millions of records, repeatedly recomputing these summaries can become expensive.

A common approach is to create derived datasets ahead of time.

Raw data

accelerometry.parquet

Daily summaries

daily_activity.parquet

Participant summaries

participant_summary.parquet

The application can then use the most appropriate level of aggregation.

Instead of reading 500 million observations, a dashboard may only need a few thousand daily summaries.

Option 4: Cache expensive operations

Some analyses are repeated frequently across users.

For example:

reactive({
  dataset |>
    filter(study == input$study) |>
    summarise(
      observations = n()
    ) |>
    collect()
})

If users repeatedly request the same result, caching can reduce computation.

library(cachem)

cache <- cache_mem(max_size = 500 * 1024^2)

Combined with Parquet or DuckDB, caching can significantly improve responsiveness for commonly requested views.

A practical migration path

For many teams, a complete redesign is unnecessary.

Stage 1

Current approach:

CSV -> read.csv() -> R web application

Suitable for small datasets.

Stage 2

Convert CSV to Parquet:

Parquet -> Arrow -> R web application

Improves storage efficiency and query performance.

Stage 3

Introduce DuckDB:

Parquet -> DuckDB -> R web application

Adds scalable analytical querying.

Stage 4

Create curated analytical datasets:

Raw Parquet
    ↓
Derived Parquet
    ↓
DuckDB
    ↓
R web application

Appropriate for larger production-scale analytical applications.

When should you move away from CSV?

A useful rule of thumb is:

  • Up to 100 MB: CSV is usually sufficient.
  • 100 MB to 1 GB: Consider Parquet.
  • 1 GB to 10 GB: Parquet or DuckDB strongly recommended.
  • 10 GB+: Parquet, DuckDB and derived datasets recommended.

The exact threshold depends on the complexity of the analysis and the resources available to the application.

Summary

CSV files are easy to use but can become a significant bottleneck in R web applications as datasets grow. The entire file must be parsed and loaded before analysis begins, which increases startup time, memory consumption, and processing overhead.

Converting large datasets to Parquet is often the simplest improvement. Analytical queries can access only the columns and records they need, reducing both storage requirements and runtime costs.

DuckDB provides another effective option by allowing SQL queries to run directly against CSV or Parquet files without requiring a separate database server. For many analytical applications, it offers a straightforward path to improved scalability while remaining easy to deploy alongside existing R infrastructure.

For larger deployments, creating derived datasets and caching frequently requested results can further reduce query costs and improve user experience.

The most effective approach is usually to minimise the amount of data that the application needs to read, process, and transfer for each request. By combining Parquet, DuckDB, and appropriate aggregation strategies, R web applications can remain responsive even when underlying datasets grow into the millions or hundreds of millions of records.

Updated on September 04, 2026
Back to top