How to stream to an HTTP response
© https://nodejs.org/en/

How to stream to an HTTP response

Use streams to efficiently send a large file as an HTTP response

By Mario Kandut

Europe’s developer-focused job platform

Let companies apply to you

Developer-focused, salary and tech stack upfront.

Just one profile, no job applications!

A Node.js stream can help process large files, larger than the free memory of your computer, since it processes the data in small chunks.

Streams are a built-in feature in Node.js and represent asynchronous flow of data. This article explains how to stream a large file to an HTTP response in Node.js.

💰 The Pragmatic Programmer: journey to mastery. 💰 One of the best books in software development, sold over 200,000 times.

Streams in Node.js

Send a stream back to the client as a response to an HTTP request

This article is using Node v16.14.0 and ExpressJS 4.17.2.

In previous articles I have covered a lot of basic stream handling and theoretical background. Now, let's look at some implementation. In this tutorial we are going to use streams to efficiently send a file as an HTTP response and download it.

The big advantage of streams is that you can process large files (bigger than the available memory). In general, reading a large file into memory is an inefficient use of resources.

We are going to use ExpressJS with some CSV sample data from a previous article, How to use streams to ETL data.

Overview

  1. Initialize project and install dependencies
  2. Create a readable stream from a file
  3. Error handling and set headers for downloading file

1. Initialize project and install dependencies

We start with creating a folder for the project

mkdir streams-http
cd streams-http

We are going to use npm packages, hence, we have to initialize the project to get a package.json Initialize empty project to install dependencies, add -y flag to agree to everything.

npm init -y

Install ExpressJS.

npm i express

Create a folder for sample data and add the CSV data.

mkdir data
cd data
touch sample-data.csv

Copy all sample data into sample-data.csv and save it. Use copy+paste or fs.writeFile in the REPL or with the -p flag in the terminal.

id,firstName,lastName,email,email2,randomized
100,Jobi,Taam,[email protected],[email protected],Z lsmDLjL
101,Dacia,Elephus,[email protected],[email protected],Za jfPaJof
102,Arlina,Bibi,[email protected],[email protected],zmzlfER
103,Lindie,Torray,[email protected],[email protected],ibVggFEh
104,Modestia,Leonard,[email protected],[email protected]," Tit KCrdh"
105,Karlee,Cornelia,[email protected],[email protected],PkQCUXzq
106,Netty,Travax,[email protected],[email protected],psJKWDBrXm
107,Dede,Romelda,[email protected],[email protected],heUrfT
108,Sissy,Crudden,[email protected],[email protected],cDJxC
109,Sherrie,Sekofski,[email protected],[email protected],dvYHUJ
110,Sarette,Maryanne,[email protected],[email protected],rskGIJNF
111,Selia,Waite,[email protected],[email protected],DOPBe
112,Karly,Tjon,[email protected],[email protected],zzef nCMVL
113,Sherrie,Berriman,[email protected],[email protected],rQqmjw
114,Nadine,Greenwald,[email protected],[email protected],JZsmKafeIf
115,Antonietta,Gino,[email protected],[email protected],IyuCBqwlj
116,June,Dorothy,[email protected],[email protected],vyCTyOjt
117,Belva,Merriott,[email protected],[email protected],MwwiGEjDfR
118,Robinia,Hollingsworth,[email protected],[email protected],wCaIu
119,Dorthy,Pozzy,[email protected],[email protected],fmWOUCIM
120,Barbi,Buffum,[email protected],[email protected],VOZEKSqrZa
121,Priscilla,Hourigan,[email protected],[email protected],XouVGeWwJ
122,Tarra,Hunfredo,[email protected],[email protected],NVzIduxd
123,Madalyn,Westphal,[email protected],[email protected],XIDAOx
124,Ruthe,McAdams,[email protected],[email protected],iwVelLKZH
125,Maryellen,Brotherson,[email protected],[email protected],nfoiVBjjqw
126,Shirlee,Mike,[email protected],[email protected],MnTkBSFDfo
127,Orsola,Giule,[email protected],[email protected],VPrfEYJi
128,Linzy,Bennie,[email protected],[email protected],ZHctp
129,Vanessa,Cohdwell,[email protected],[email protected],RvUcbJihHf
130,Jaclyn,Salvidor,[email protected],[email protected],gbbIxz
131,Mildrid,Pettiford,[email protected],[email protected],snyeV
132,Carol-Jean,Eliathas,[email protected],[email protected],EAAjYHiij
133,Susette,Ogren,[email protected],[email protected]," BhYgr"
134,Farrah,Suanne,[email protected],[email protected],hYZbZIc
135,Cissiee,Idelia,[email protected],[email protected],PNuxbvjx
136,Alleen,Clara,[email protected],[email protected],YkonJWtV
137,Merry,Letsou,[email protected],[email protected],sLfCumcwco
138,Fanny,Clywd,[email protected],[email protected],Go kx
139,Trixi,Pascia,[email protected],[email protected],lipLcqRAHr
140,Sandie,Quinn,[email protected],[email protected],KrGazhI
141,Dania,Wenda,[email protected],[email protected],CXzs kDv
142,Kellen,Vivle,[email protected],[email protected],RrKPYqq
143,Jany,Whittaker,[email protected],[email protected],XAIufn
144,Lusa,Fillbert,[email protected],[email protected],FBFQnPm
145,Farrah,Edee,[email protected],[email protected],TrCwKb
146,Felice,Peonir,[email protected],[email protected],YtVZywf
147,Starla,Juan,[email protected],[email protected],aUTvjVNyw
148,Briney,Elvyn,[email protected],[email protected],tCEvgeUbwF
149,Marcelline,Ricarda,[email protected],[email protected],sDwIlLckbd
150,Mureil,Rubie,[email protected],[email protected],HbcfbKd
151,Nollie,Dudley,[email protected],[email protected],EzjjrNwVUm
152,Yolane,Melony,[email protected],[email protected],wfqSgpgL
153,Brena,Reidar,[email protected],[email protected],iTlvaS
154,Glenda,Sabella,[email protected],[email protected],zzaWxeI
155,Paola,Virgin,[email protected],[email protected],gJO hXTWZl
156,Aryn,Erich,[email protected],[email protected],qUoLwH
157,Tiffie,Borrell,[email protected],[email protected],cIYuVMHwF
158,Anestassia,Daniele,[email protected],[email protected],JsDbQbc
159,Ira,Glovsky,[email protected],[email protected],zKITnYXyhC
160,Sara-Ann,Dannye,[email protected],[email protected],wPClmU
161,Modestia,Zina,[email protected],[email protected],YRwcMqPK
162,Kelly,Poll,[email protected],[email protected],zgklmO
163,Ernesta,Swanhildas,[email protected],[email protected],tWafP
164,Giustina,Erminia,[email protected],[email protected],XgOKKAps
165,Jerry,Kravits,[email protected],[email protected],olzBzS
166,Magdalena,Khorma,[email protected],[email protected],BBKPB
167,Lory,Pacorro,[email protected],[email protected],YmWQB
168,Carilyn,Ethban,[email protected],[email protected],KUXenrJh
169,Tierney,Swigart,[email protected],[email protected],iQCQJ
170,Beverley,Stacy,[email protected],[email protected],NMrS Zpa f
171,Ida,Dex,[email protected],[email protected],hiIgOCxNg
172,Sam,Hieronymus,[email protected],[email protected],dLSkVe
173,Lonnie,Colyer,[email protected],[email protected],ZeDosRy
174,Rori,Ethban,[email protected],[email protected],SXFZQmX
175,Lelah,Niles,[email protected],[email protected],NwxvCXeszl
176,Kathi,Hepsibah,[email protected],[email protected],SOcAOSn
177,Dominga,Cyrie,[email protected],[email protected],IkjDyuqK
178,Pearline,Bakerman,[email protected],[email protected],vHVCkQ
179,Selma,Gillan,[email protected],[email protected],hSZgpBNsw
180,Bernardine,Muriel,[email protected],[email protected],AnSDTDa U
181,Ermengarde,Hollingsworth,[email protected],[email protected],IYQZ Nmv
182,Marguerite,Newell,[email protected],[email protected],kSaD uaHH
183,Albertina,Nisbet,[email protected],[email protected],Y jHyluB
184,Chere,Torray,[email protected],[email protected],loElYdo
185,Vevay,O'Neill,Vevay.O'[email protected],Vevay.O'[email protected],uLZSdatVn
186,Ann-Marie,Gladstone,[email protected],[email protected],fwKlEksI
187,Donnie,Lymann,[email protected],[email protected],deBrqXyyjf
188,Myriam,Posner,[email protected]m,[email protected],gEMZo
189,Dale,Pitt,[email protected],[email protected],OeMdG
190,Cindelyn,Thornburg,[email protected],[email protected],kvhFmKGoMZ
191,Maisey,Hertzfeld,[email protected],[email protected],OajjJ
192,Corina,Heisel,[email protected],[email protected],luoDJeHo
193,Susette,Marcellus,[email protected],[email protected],AXHtR AyV
194,Lanae,Sekofski,[email protected],[email protected],FgToedU
195,Linet,Beebe,[email protected],[email protected],DYGfRP
196,Emilia,Screens,[email protected],[email protected],LXUcleSs
197,Tierney,Avi,[email protected],[email protected],VegzbHH
198,Pollyanna,Thar,[email protected],[email protected],GjYeEGK
199,Darci,Elephus,[email protected],[email protected],DaQNdN

Create an index.js file (in root folder), which will be the main file for our code.

cd .. # if you are in the data folder
touch index.js

2. Create a readable stream from a file

First, we are going to create a basic express server, which listens on port 3000. Open index.js in your IDE and add the following code.

const express = require('express');

const app = express();
const PORT = 3000;

app.listen(PORT, () =>
  console.log(`Server listening on port ${PORT}`),
);

Start the node server with running node index.js in the project root folder. You should see Server listening on port 3000 in your terminal. Terminate the server with CTRL+C.

Let's make another route to download the csv file. Add a GET handler with the route /get-data to index.js.

app.get('/get-data', (req, res, next) => {
  // TBD
});

Now we have a route on which we are going to download the file in the end. We can proceed to create a readable stream to read the file. For creating a stream we have to import the fs module.

const fs = require('fs');

Create a stream to read file sample-data.csv.

app.get('/dl', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
});

The constant fileStream represents the data stream from the file. This stream we directly pipe into the response.

app.get('/dl', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.pipe(res);
});

Start the server again node index.js. And open a web browser with http://localhost:3000/get-data. You should see the csv file.

3. Error handling and set headers for downloading file

Now we are sending a file as a stream, but it should download. Let's make it happen. As always, we have to think of error handling first. What could go wrong? The file could not exist.

To handle this, we have to listen to the open event on the readStream to check, if the file exists and only pipe the stream if it does.

app.get('/get-data', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.on('open', () => {
    fileStream.pipe(res);
  });
});

And if the file doesn't exist, we return the error as response.

app.get('/get-data', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.on('open', () => {
    fileStream.pipe(res);
  });
  fileStream.on('error', err => {
    next(err);
  });
});

In most cases the pipeline method should be used, but pipeline destroys streams when an error occurs, and we would not be able to send a response back. Hence, for this use case manually error handling is acceptable.

At the moment the browser is displaying the file inline, it's loading the file in the browser. To tell the browser to download the file, we have to set:

  • a Content-Type header on the response to specify what file we are sending, and
  • the Content-Disposition header to attachment with a file name.

Express has a method for this attachment("FILENAME"). It sets the HTTP response Content-Disposition header field to “attachment”, and if a filename is given, then it sets the Content-Type based on the extension name via res.type(), and sets the Content-Disposition “filename=” parameter.

When using res.attachment('streamed-sample-data'), the content-type header will be set to text/csv and the content-disposition to the streamed-sample-data.csv.

app.get('/get-data', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.on('open', () => {
    res.attachment('streamed-sample-data.csv');
    fileStream.pipe(res);
  });
  fileStream.on('error', err => {
    next(err);
  });
});

Restart your node server and go to http://localhost:3000/get-data. The file streamed-sample-data.csv should be downloaded.

TL;DR

  • Error handling has to be done always, especially when working with streams.
  • Error handling should be done in most cases with pipeline, only if the stream should not be destroyed, manually error handling is necessary.
  • The express method attachment() is used to add Content-Type and Content-Disposition headers to a response.
  • For the future, the Express framework has a method for sending files via stream sendFile().

Thanks for reading and if you have any questions, use the comment function or send me a message @mariokandut.

If you want to know more about Node, have a look at these Node Tutorials.

References (and Big thanks):

ExpressJS, HeyNode, Node.js - Streams, MDN - Streams, MDN - HTTP

More node articles:

How to dynamically load ESM in CJS

How to convert a CJS module to an ESM

How to create a CJS module

How to stream to an HTTP response

How to handle binary data in Node.js?

How to use streams to ETL data?

How to connect streams with pipeline?

How to handle stream errors?

How to connect streams with pipe?

What Is a Node.js Stream?

Handling Errors in Node (asynchronous)

Handling Errors in Node.js (synchronous)

Introduction to errors in Node.js

Callback to promise-based functions

ETL: Load Data to Destination with Node.js

ETL: Transform Data with Node.js

ETL: Extract Data with Node.js

Event Emitters in Node.js

How to set up SSL locally with Node.js?

How to use async/await in Node.js

What is an API proxy?

How to make an API request in Node.js?

How does the Event Loop work in Node.js

How to wait for multiple Promises?

How to organize Node.js code

Understanding Promises in Node.js

How does the Node.js module system work?

Set up and test a .env file in Node

How to Use Environment Variables in Node

How to clean up node modules?

Restart a Node.js app automatically

How to update a Node dependency - NPM?

What are NPM scripts?

How to uninstall npm packages?

How to install npm packages?

How to create a package.json file?

What Is the Node.js ETL Pipeline?

What is data brokering in Node.js?

How to read and write JSON Files with Node.js?

What is package-lock.json?

How to install Node.js locally with nvm?

How to update Node.js?

How to check unused npm packages?

What is the Node.js fs module?

What is Semantic versioning?

The Basics of Package.json explained

How to patch an NPM dependency

What is NPM audit?

Beginner`s guide to NPM

Getting started with Node.js

Scroll to top ↑