Node Response Backpressure and Export Aborts

Working case

A permit officer downloads 240,000 inspection rows over a weak mobile connection. The database cursor emits rows much faster than the socket sends them, and a naive loop calls response.write for each row. Resident memory climbs while only a small fraction of the file has left the server. The officer closes the tab at row 81,000, but the query keeps reading to the end. A bounded implementation fetches a batch, writes it, waits when backpressure appears, and closes the cursor when the client connection closes. If the export must survive navigation, it becomes a queued artifact request instead of an in-flight HTTP stream.

Implementation boundary

function waitForDrain(response) {
  return new Promise((resolve, reject) => {
    function cleanup() {
      response.off("drain", onDrain);
      response.off("close", onClose);
    }
    function onDrain() { cleanup(); resolve(); }
    function onClose() { cleanup(); reject(new Error("client_closed")); }
    response.once("drain", onDrain);
    response.once("close", onClose);
  });
}

async function writeExportRows(response, rows) {
  for await (const inspectionRow of rows) {
    if (response.destroyed) throw new Error("client_closed");
    const line = `${inspectionRow.permitId},${inspectionRow.revision}\n`;
    if (!response.write(line)) await waitForDrain(response);
  }
  if (response.destroyed) throw new Error("client_closed");
  response.end();
}

Create the cursor only after authorization and filters are fixed. Emit headers before body bytes, but remember that after the first byte an error cannot be converted into a clean JSON response. Preserve CSV escaping and newline rules as part of the data contract. Prefer pipeline for compatible readable and writable streams because it coordinates backpressure and failure propagation. When manually generating rows, pause on false and resume on drain; attach close, error, and abort cleanup paths. Distinguish a normal finish from a premature close in metrics. If a database page is fetched before the socket drains, cap the page size and avoid prefetching more pages until writing resumes.

Cost and boundaries

A bounded stream keeps application buffering proportional to one database batch plus stream high-water buffers, O(K), rather than O(N) rows for an N-row export. Work remains O(N) in generated bytes and row serialization. Slow clients occupy a connection and database cursor longer, so set an export duration budget and concurrency cap. The stream high-water mark is a threshold, not a hard memory ceiling across every upstream buffer. For repeated or very large downloads, materialize an object-storage artifact once and serve it with its own retention and authorization policy. Measure queued bytes, drain waits, cursor duration, canceled scans, and completed bytes.

Failure trace

Use a client that reads a few bytes and pauses for several seconds; memory should settle within the selected batch budget. Abort after the header, halfway through a quoted CSV field, and immediately after the last byte. A premature close must release the cursor and count as canceled, while a normal finish counts as delivered. Inject a database error after bytes have been sent and ensure the stream ends with an incomplete-download signal in logs; do not append a JSON error to the CSV. Revoke the officer's permission before starting a second request and make it fail before any private row is sent.

Verification

A slow reader keeps queued memory bounded.
Client close releases the database cursor.
A completed response and a canceled response have separate metrics.

Practice drill

Build a CSV export endpoint for permit 447 that emits 83 rows in batches of seven. Use a deliberately slow writable client to force write to return false. Count calls to the row provider and assert it does not fetch the next batch until drain. Abort after row 19 and verify cursor cleanup. Add a background alternative that stores an artifact with owner, expiry, and revision metadata. Compare the live stream and artifact path for a second download, a midstream failure, and permission revocation. Explain which path carries the lower database load for repeat readers.

Decision note

Live streams own short-lived work; durable exports own an artifact and a status record.

Common Mistakes

Treating write(false) as a failed write rather than a pause signal.
Continuing a database scan after the response closes.
Appending an error JSON document to a partial CSV.

Apply and check

Build Project: Native Node Permit Gateway and review Web Development: Native Node Runtime Contracts.