Limitations, benefits and use cases for Node.js streams
By Mario Kandut
Europe’s developer-focused job platform
Let companies apply to you
Developer-focused, salary and tech stack upfront.
Just one profile, no job applications!
Streams are a built-in feature in Node.js and represent asynchronous flow of data. Streams are also a way to handle reading and/or writing files. A Node.js stream can help process large files larger than the free memory of your computer, since it processes the data in small chunks.
This article is based on Node v16.14.0.
💰 The Pragmatic Programmer: journey to mastery. 💰 One of the best books in software development, sold over 200,000 times.
This is the first article of a series about streams in Node.js. It aims to give an overview of different types of streams and what the limitations, benefits and use-cases are.
Streams in Node.js
Streams are an interface for working with streaming data. Think of a Unix pipe
| as a mental model of streams.
Essentially, a stream is a collection of data, which isn't available at once. The streamed data arrives in small chunks.
As a result we handle each data chunk when it arrives asynchronously.
In Node.js streams are used in many built-in modules to handle async data processing, for example,
http module uses streaming interfaces with
Stream data is a buffer by default, unless it is configured to with objects. This means it helps to buffer the data in memory.
Streams let us work with data that is too large to fit into memory. We can work with a chunk of data at a time. For instance, you are working with a 50gb file of analytics data with millions of rows. If you read this file into memory, it will take very long and eventually hit the memory limit of Node.js or of your local machine. Handling this file with a stream, we can process each row from the dataset at a time and don't have to read the file into memory. Hence, streams are memory efficient.
Streams are also useful in other scenarios. For example reading a large file into memory (assuming it fits), it would take some time to be readable. When consuming data from a stream, it's readable the moment a chunk of data arrives. This means streams are time efficient compared to reading data into memory.
Streams can be combined to and with other streams. For instance, the output of one stream can be used as the input for another stream. This allows us to combine streams into a pipeline through which data can flow between the streams. Hence, streams are composable.
There are 5 types of streams in the built-in
stream module of Node.js. docs
Out in the wild there is a high possibility you will encounter
All streams are instances of
EventEmitter. EventEmitters are used to emit and respond to events asynchronously.
Read more about EventEmitters in the article Event Emitters in Node.js.
Events emitted by streams can be used to read and/or write data, manage the stream state, and handle errors.
Though streams are instances of
EventEmitter it is not recommended, to handle streams like events and just listen to the events.
Instead, the recommended way is to use the
pipeline methods, which consume streams and handle the events for you.
Working with stream events is useful, when a more controlled way of how the stream is consumed is needed. For instance, triggering an event when a particular stream ends or begins. Have a look at the official Node.js docs regarding Streams for more information on this.
Readable stream events
data- emitted when the stream outputs a data chunk.
readable- emitted when there is data ready to be read from the stream.
end- emitted when no more data is available.
error- emitted when an error has occurred within the stream, and an error object is passed to the handler. Unhandled stream errors can crash the application.
Writable stream events
drainwill be emitted, when the writable stream's internal buffer has been cleared and is ready to have more data written into it.
finishwill be emitted, when all data has been written.
errorwill be emitted when an error occurred while writing data, and an error object is passed to the handler. Unhandled stream errors can crash the application.
Thanks for reading and if you have any questions, use the comment function or send me a message @mariokandut.
References (and Big thanks):
Never miss an article.