Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchTo handle large files in Node.js without buffering each file in full, connect readable and writable streams with stream/promises.pipeline(). For uploads, stream the request body—or each file stream from a multipart parser—into a temporary file. For downloads, stream a file into the HTTP response. pipeline() propagates errors, manages backpressure, and can be cancelled with an AbortSignal; validation, authorization, file naming, and cleanup are still responsibilities of your application.
What streaming changes
Node’s HTTP API is designed to stream request and response data rather than buffer entire messages. An incoming HTTP request is an IncomingMessage, which is a readable stream; a client-side ClientRequest is writable for sending an upload. On the server, ServerResponse is writable, so a file can flow from disk to the response as it is read.
Streams apply backpressure: when a destination cannot accept data as quickly as a source produces it, the flow is regulated rather than requiring the application to accumulate the whole file in memory. This avoids whole-file buffering, but it does not mean memory use is zero or that performance is guaranteed. Buffers, transforms, concurrent requests, and other application work still use memory.
Node documents a default highWaterMark of 64 * 1024 bytes for fs.createReadStream(). That is a stream buffering threshold, not a promise that a transfer uses exactly 64 KiB of memory or a throughput figure. Tune stream settings only after measuring the actual workload.
Recommended Free Tools
#1 Best Overall
Stream an upload to disk
For an endpoint that accepts a raw binary request body, use req as the readable source and fs.createWriteStream() as the destination. Check authorization and any required metadata before accepting the body. Enforce a byte limit while reading: a Content-Length check can reject an oversized request early, but it is not sufficient on its own because the header can be absent or untrusted.
The following helper illustrates a bounded raw upload. It writes to a unique temporary file, counts bytes as they pass through a transform, and renames the file only after the pipeline completes. The storage directory should be created and protected by the application, outside the public web root. The caller should supply an authorized destination name rather than deriving a filesystem path from user input.
Rank #2
import { createWriteStream } from 'node:fs';
import { mkdir, rename, rm } from 'node:fs/promises';
import { randomUUID } from 'node:crypto';
import { join } from 'node:path';
import { Transform } from 'node:stream';
import { pipeline } from 'node:stream/promises';
async function receiveRawUpload(req, storageDir, finalName, maxBytes, signal) {
await mkdir(storageDir, { recursive: true });
const tempPath = join(storageDir, `${randomUUID()}.upload`);
const finalPath = join(storageDir, finalName);
let received = 0;
const enforceLimit = new Transform({
transform(chunk, encoding, callback) {
received += chunk.length;
if (received > maxBytes) {
callback(new Error('Upload exceeds the permitted size'));
} else {
callback(null, chunk);
}
}
});
try {
await pipeline(
req,
enforceLimit,
createWriteStream(tempPath, { flags: 'wx' }),
{ signal }
);
await rename(tempPath, finalPath);
return { path: finalPath, bytes: received };
} catch (error) {
await rm(tempPath, { force: true });
throw error;
}
}Use a temporary name that cannot collide with another request; flags: 'wx' also prevents accidentally overwriting an existing temporary file. Keep the temporary and final paths on the same filesystem if you rely on rename as the publish step. On failure or cancellation, remove partial output. A limit error or client disconnect may close the request connection, so decide how your server reports errors before response headers have been sent and avoid attempting to reuse a destroyed request.
- Validate the user’s authorization and intended operation before writing.
- Do not trust a supplied filename or path. Generate storage names and keep display names as separately validated metadata.
- Check allowed type and other content rules; a filename extension or client-provided content type alone does not establish file contents.
- Do not make a file available to other users until the write finishes and required validation or scanning succeeds.
Stream multipart uploads one file at a time
multipart/form-data is a framing format for fields and files; it is not itself a file stream abstraction. Use a multipart parser or framework adapter that exposes each uploaded file as a readable stream. Pipe each file stream into its own temporary destination, and apply size limits and validation to each file as well as to the overall request where appropriate.
Rank #3
NestJS’s file-upload documentation demonstrates this pattern with await pipeline(file.stream, createWriteStream(path)). The parser handles multipart boundaries and fields; Node’s stream pipeline handles movement of the file bytes. Avoid first collecting a multipart file into an in-memory buffer if the goal is to support large uploads.
Stream a file to an HTTP download
Resolve a file identifier through an authorization-aware lookup; do not concatenate a user-supplied path onto a storage directory. Once the authorized path is known, inspect the file and set the response status and headers before starting the stream. Set Content-Type to an appropriate media type, include Content-Length when serving the complete file and its size is known, and use Content-Disposition: attachment when the browser should download rather than display it inline.
Rank #4
import { createReadStream } from 'node:fs';
import { stat } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
async function sendFile(res, authorizedPath, contentType, downloadName) {
const info = await stat(authorizedPath);
if (!info.isFile()) {
res.writeHead(404).end();
return;
}
const controller = new AbortController();
res.on('close', () => {
if (!res.writableEnded) controller.abort();
});
res.writeHead(200, {
'Content-Type': contentType,
'Content-Length': info.size,
'Content-Disposition': `attachment; filename="${downloadName}"`
});
try {
await pipeline(createReadStream(authorizedPath), res, {
signal: controller.signal
});
} catch (error) {
if (!res.destroyed) res.destroy(error);
}
}This is a pattern, not a complete route: authorizedPath, contentType, and downloadName must come from trusted application logic. In production, encode or otherwise safely format the download filename for an HTTP header. The file can also change between the metadata lookup and opening the stream, so applications that permit concurrent file replacement should account for that race.
If the client disconnects, aborting the pipeline prevents continuing work on the transfer. Handle filesystem and stream errors according to whether the response has started: before headers, the route may return an error status; after streaming has begun, it generally cannot replace the partial response with a new status.
Add HTTP range requests for resumable or partial downloads
Range support is application behavior built on top of Node’s HTTP and filesystem streams. For a single byte range, parse and validate the requested offsets against the file size, then create a read stream with inclusive start and end values. Return 206 Partial Content with Content-Range: bytes start-end/total, Accept-Ranges: bytes, and Content-Length equal to end - start + 1.
| Request or condition | Response behavior |
|---|---|
| Complete file requested | Return 200; the length, if supplied, is the complete file size. |
| Valid supported byte range | Return 206, the selected bytes, and a matching Content-Range. |
| Range cannot be satisfied by the file | Return 416 Range Not Satisfiable; include Content-Range: bytes */total. |
Decide explicitly how the endpoint treats malformed, multiple, or unsupported ranges. A simple implementation can support a single range and apply a documented policy to other forms; it should not accidentally interpret unvalidated offsets. A range end beyond the file’s last byte must not cause the server to claim bytes it did not send. Range responses need their own correct length and headers rather than reusing the full-file values.
Use pipeline for errors, cleanup, and cancellation
A bare .pipe() connects streams, but it does not provide the same centralized completion and error handling as pipeline(). In an HTTP handler, the promise returned by stream/promises.pipeline() resolves when the stream chain completes and rejects when a stage fails. The promise API accepts an AbortSignal; aborting it destroys the pipeline and rejects with an AbortError.
- On upload failure, remove the incomplete temporary file and do not publish it.
- On download cancellation, stop reading the source when the response is closed.
- Distinguish cancellation, size-limit violations, permission failures, and storage errors in application logging and response policy.
- Do not send a success response until an upload pipeline has completed and the application’s required validation has passed.
Insert compression or other transforms
A transform can sit between a readable source and writable destination without requiring the whole file to be loaded first. Node’s zlib documentation uses promise-based pipeline() to connect createReadStream(input), createGzip(), and createWriteStream(output). The same stream shape can support encryption, hashing, metering, or content inspection when each stage is designed to honor backpressure and is included in the pipeline’s error and cancellation handling.
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';
await pipeline(
createReadStream('input.txt'),
createGzip(),
createWriteStream('input.txt.gz')
);Choose the right upload path
Raw HTTP, multipart parsing, and managed object-storage transfers solve different parts of the problem. Node’s core streams provide data flow and backpressure; a parser, storage SDK, or hosting service adds protocol handling, policy, or durability. Compare the options against the actual requirements rather than assuming that a stream alone provides resumability, scanning, or durable storage.
Quick Recap
| Option | Protocol shape | What to plan for |
|---|---|---|
| Raw Node.js endpoint | One binary request body streamed from IncomingMessage. |
Implement request authorization, byte limits, file validation, cancellation, temporary-file cleanup, and storage policy. |
| Multipart parser or framework adapter | Fields and file parts are parsed; each file should be exposed as a stream for bounded-memory handling. | Configure parser limits and validation, and ensure the adapter does not buffer whole files before exposing them. |
| Managed object-storage transfer | Defined by the chosen provider’s API or SDK. | Verify its size limits, resumability, cancellation behavior, validation and scanning hooks, durability, and operational visibility; these are not supplied by Node’s core stream APIs. |
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
