How to Handle Large CSV File Imports Without Crashing Your App

Where large imports actually break#

The naive import path looks reasonable in a code review. Read the file the user selected, parse it into an array of objects, loop over the array validating each row, then POST the result to your API. It works perfectly on the 200-row sample the developer tested with, and it works on the 5,000-row file the first customer uploads. Then someone exports their entire product catalogue and the tab goes white.

Nothing in that sequence is wrong in isolation. The problem is that every step in it buffers, and the buffers stack. By the time you are validating, you are holding the raw text, the parsed array, and the partially built output at the same time — three copies of the same data, in the memory of a browser tab that is also trying to render your application.

It is also worth being precise about what "large" means, because file size is the wrong metric. Cost tracks the number of row objects you allocate, not megabytes on disk. A 40 MB file with two million narrow rows will hurt considerably more than a 400 MB file with fifty thousand wide ones, because the first allocates forty times as many JavaScript objects. Size your defences against row count.

Stream the parse, and get it off the main thread#

The first change is to stop treating the file as a value and start treating it as a stream. Every serious CSV parser supports incremental parsing, where you receive rows in chunks as they are read rather than as one array at the end. Papa Parse calls this a chunk callback; the effect is that peak memory becomes the size of one chunk instead of the size of the file.

import Papa from 'papaparse'

function parseInChunks(file, { onChunk, onDone, onError }) {
  let rowsSeen = 0

  Papa.parse(file, {
    header: true,
    skipEmptyLines: true,
    worker: true,          // parse off the main thread
    chunkSize: 1024 * 512, // 512 KB of file per chunk

    chunk(results, parser) {
      rowsSeen += results.data.length

      // Hand the chunk downstream, then let it be garbage collected.
      // Never push results.data into an array that outlives this call.
      const keepGoing = onChunk(results.data, results.errors, rowsSeen)

      // Backpressure: pause until the slow stage downstream catches up.
      if (keepGoing instanceof Promise) {
        parser.pause()
        keepGoing.then(() => parser.resume(), onError)
      }
    },

    complete() { onDone(rowsSeen) },
    error: onError,
  })
}

Two details in that snippet carry most of the benefit. worker: true moves parsing to a Web Worker, so the main thread stays free to paint — this is the difference between an app that shows a moving progress bar and an app the operating system offers to force-quit. And the parser.pause() / parser.resume() pair is backpressure: without it, the parser will happily read the whole file while your validation or upload code falls further behind, and you are back to holding everything in memory with extra steps.

Validate in chunks, and cap what you keep#

Validation is where the second buffer usually hides. Even with a streaming parser, code that collects { row, errors } for every row rebuilds the full dataset in memory — and if the file is broadly malformed, the error list can be larger than the data that produced it. A user who uploads a file with the wrong schema entirely will generate one error per row per rule.

The discipline is to keep aggregates rather than records. Count what passed, retain a bounded sample of what failed, and stop collecting detail past a threshold — nobody reads the 4,000th instance of the same error, and the first fifty tell the user exactly what to fix.

const MAX_ERRORS_KEPT = 200

const summary = {
  rows: 0,
  valid: 0,
  failed: 0,
  byRule: new Map(),   // rule -> count, cheap and complete
  samples: [],         // bounded detail for the UI
}

function validateChunk(rows, startIndex) {
  const clean = []

  for (let i = 0; i < rows.length; i++) {
    summary.rows++
    const failures = checkRow(rows[i])   // your rules, returns string[]

    if (failures.length === 0) {
      summary.valid++
      clean.push(rows[i])
      continue
    }

    summary.failed++
    for (const rule of failures) {
      summary.byRule.set(rule, (summary.byRule.get(rule) || 0) + 1)
    }
    if (summary.samples.length < MAX_ERRORS_KEPT) {
      summary.samples.push({ line: startIndex + i + 2, failures })
    }
  }

  return clean   // only clean rows travel further down the pipeline
}

Note what leaves the function: only the clean rows. Failed rows are represented by counters and a sample. That single decision keeps memory proportional to the batch size rather than to the failure rate, which matters because the worst files are the ones where almost everything fails.

• Count by rule, not by row — "date_format: 12,481 rows" is more useful to a user than twelve thousand identical messages.
• Keep the line number, not the row. Users find problems in their spreadsheet by line, and a number costs four bytes instead of an object.
• Fail the import early if the first few thousand rows are almost entirely invalid. It is nearly always the wrong file, and running the other 1.99 million rows helps nobody.
• Run expensive checks — uniqueness, database lookups — over the batch rather than per row, so one query answers a thousand questions.

Deliver in bounded batches, with backpressure#

The final buffer is the request body. Posting two million rows in a single call fails in several ways at once: proxies and load balancers enforce body size limits, gateways time out long before the write completes, and a failure at 98% loses everything that came before it. Batching solves all three, but only if you also bound concurrency.

const BATCH_SIZE = 500
const MAX_IN_FLIGHT = 4

async function deliver(batches, importId) {
  const running = new Set()

  for (let i = 0; i < batches.length; i++) {
    const task = send(batches[i], `${importId}:${i}`)
      .finally(() => running.delete(task))

    running.add(task)

    // Backpressure: never let more than MAX_IN_FLIGHT requests run.
    if (running.size >= MAX_IN_FLIGHT) await Promise.race(running)
  }

  await Promise.all(running)
}

async function send(rows, idempotencyKey, attempt = 0) {
  const res = await fetch('/api/import', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Idempotency-Key': idempotencyKey,
    },
    body: JSON.stringify({ rows }),
  })

  if (res.ok) return

  // Retry only what is retryable, and back off.
  if ((res.status === 429 || res.status >= 500) && attempt < 4) {
    const wait = Math.min(2 ** attempt * 500, 8000)
    await new Promise((r) => setTimeout(r, wait))
    return send(rows, idempotencyKey, attempt + 1)
  }

  throw new Error(`Batch ${idempotencyKey} failed: ${res.status}`)
}

The idempotency key is not optional decoration. Retries are guaranteed at this scale — a mobile connection drops, a deploy cycles a pod, a gateway returns 502 after your database already committed the write. Without a key the retry duplicates rows, and duplicate customer records are a worse outcome than a failed import, because nobody notices them for weeks.

Keep the interface honest while it works#

A technically correct import that appears frozen will be cancelled. Once parsing runs in a worker and delivery is batched, you have real progress information available at every moment — use it, because the perception of a long import is set almost entirely by whether something on screen is moving.

• Drive the progress bar from bytes read or rows processed, and update it per chunk rather than per row — a value that changes sixty times a second is a rendering cost with no informational gain.
• Show counts as they accrue: rows read, rows valid, rows failed. Users tolerate slow far better than they tolerate opaque.
• Virtualise any preview grid. Rendering two million table rows into the DOM will freeze the tab even when the data layer is doing everything right.
• Offer cancel, and make it real — abort the fetches, terminate the worker, and tell the server which batches to roll back or ignore.
• Never block the tab with a modal spinner that hides the numbers. That is the pattern that produces "it hung" reports for imports that were succeeding.

What the receiving end has to get right#

Streaming on the client moves the problem to your server, which now receives many small requests instead of one enormous one. That is a much better shape, but it introduces its own requirements — and they are the ones most often discovered in production rather than in design.

• Honour idempotency keys. Record the key, and return the original result for a repeat rather than writing again. This is what makes client retries safe.
• Insert in bulk. One statement per batch, not one per row. A per-row insert loop is a thousand network round trips wearing a trench coat.
• Return 429 with `Retry-After` when you are saturated. A well-behaved client will slow down; a client with no signal will keep hammering until something breaks.
• Keep transactions short. Wrapping an entire two-million-row import in one transaction holds locks for minutes and blocks everything else touching those tables.
• Decide partial-failure policy before launch. If batch 300 of 400 fails, do the first 299 stay? Both answers are defensible; discovering you never chose is not.

This is also the point where the work stops being about CSV at all. Everything above — batching, idempotency, backpressure, partial failure — is ordinary distributed-systems plumbing, which is precisely why it takes longer than teams estimate. The parsing was never the hard part. See the five components of an import pipeline for how these pieces fit together.

Or don’t build this part#

Every technique here is well understood, and none of it is differentiated work — no customer has ever chosen a product because its chunk boundaries were well chosen. It is simply a few weeks of engineering plus a long tail of edge cases that only surface on real customer files.

A hosted importer absorbs the whole category. CSVbox streams uploads up to 500 MB, validates in chunks, handles up to two million rows in a single import, and delivers in batches to your destination with retries and progress already wired in. The browser stays responsive because nothing ever holds the file, and you get the failure semantics without designing them.

If you are weighing that trade honestly, the hidden cost of building your own importer covers what follows the initial build, and the roundup of embedded importers covers the options.