How to handle binary data in Node.js?
© https://nodejs.org/en/

How to handle binary data in Node.js?

Handle binary data in Node.js with the Buffer constructor

By Mario Kandut

Europe’s developer-focused job platform

Let companies apply to you

Developer-focused, salary and tech stack upfront.

Just one profile, no job applications!

This article is based on Node v16.14.0.

Handling binary data in server-side programming is an essential feature, and a must-have for every developer dealing with Node.js. In Node.js binary data is handled with the Buffer constructor. Let's have a look at the anatomy of a Buffer instance.

The Buffer instance in Node.js

💰 The Pragmatic Programmer: journey to mastery. 💰 One of the best books in software development, sold over 200,000 times.

The Buffer constructor is a global, hence, no import is needed to use it.

Type node -p "Buffer" in your terminal and have a look at the output.

[Function: Buffer] {
  poolSize: 8192,
  from: [Function: from],
  of: [Function: of],
  alloc: [Function: alloc],
  allocUnsafe: [Function: allocUnsafe],
  allocUnsafeSlow: [Function: allocUnsafeSlow],
  isBuffer: [Function: isBuffer],
  compare: [Function: compare],
  isEncoding: [Function: isEncoding],
  concat: [Function: concat],
  byteLength: [Function: byteLength],
  [Symbol(kIsEncodingSymbol)]: [Function: isEncoding]
}

The Buffer constructor was introduced into Node.js, when Javascript did not have a native binary type. Javascript evolved and different views of a buffer got added to the language, like an ArrayBuffer or other typed arrays.

For instance, an ArrayBuffer instance can be accessed by a Float64array, where each set of 8 bytes is interpreted as a 64-bit floating-point number. Have a look at the MDN article Javascript Typed Arrays. When these new data structures where added, the Buffer constructor internals were refactored on top of the Uint8array typed array. This means a buffer object is both, an instance of a Buffer and an instance of Uint8array.

Let's open the REPL and double-check this.

# enter REPL
node

## Allocates a new Buffer of size bytes.
const buffer = Buffer.alloc(10)

buffer instanceof Buffer
## returns true

buffer instanceof Uint8Array
## returns true

Important: The method Buffer.prototpye.slice overrides the Uint8Array.prototype.slice method. The Uint8Array method will take a copy of a buffer, the Buffer method will return a buffer instance that references the binary data.

Allocating Buffers

Usually a constructor is called with the new keyword, with the Buffer constructor this is deprecated. The correct and safe way to allocate a buffer of a certain amount of bytes is to use Buffer.allocate, like:

const buffer = Buffer.allocate(10);

The Buffer.alloc function produces a zero-filled buffer by default. Let's use the dynamic evaluation to see the output directly.

node -p "Buffer.alloc(10)"
## the output should be <Buffer 00 00 00 00 00 00 00 00 00 00>

When a buffer is printed to the terminal, the ellipsis in <Buffer ...> are hexadecimal numbers. For example a single byte buffer with a decimal value of 100, is 1100100 in binary and 64 in hexadecimal. Hence, the output would be <Buffer 64>

There is also an unsafe way to allocate buffers.

const buffer = Buffer.allocUnsafe(10);

Any times a buffer is created, it's allocated (or assigned) from unallocated (or unassigned) memory. Unassigned memory is only unlinked, it is never wiped. This implies, that unless the buffer is overwritten, (zero-filled), it can contain fragments of previously deleted data. This poses a security risk. The method allocUnsafe is used only for advanced use cases, like performance optimization. If you have to create a buffer, only use the safe method Buffer.alloc.

Converting Strings to Buffers

The String primitive in JavaScript is a frequently used data structure.

A buffer can also be created from a string by using Buffer.from. The string characters are converted to by values.

const buffer = Buffer.from('Hello World');

Let's dynamically evaluate this.

node -p "Buffer.from('Hello World')"

The output is <Buffer 48 65 6c 6c 6f 20 57 6f 72 6c 64>.

In order to convert a string to a binary representation, an encoding has to be defined. The default encoding for Buffer.from is UTF8. The UTF8 encoding may have up to four bytes per character, so string length will not always match the converted buffer size. Especially, when dealing with emojis.

node -p "'🔥'.length"
## will return 2
node -p "Buffer.from('🔥').length"
## will return 4

When the first argument passed to Buffer.from is a string, a second argument can be passed to set the encoding. Two types of encodings are available in this context: character encodings and binary-to-text encodings. UTF8 is one character encoding, UTF16LE is another one. Different encodings result in different buffer sizes.

Converting Buffers to Strings

To convert a buffer to a string, call the toString method on a Buffer instance. Let's try it out, use the REPL or create a file and run it with node.

const buffer = Buffer.from('hello world');
console.log(buffer); // prints <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>
console.log(buffer.toString()); // prints  'hello world'

The toString method also accepts an encoding argument.

const buffer = Buffer.from('mario');
console.log(buffer); // prints <Buffer 6d 61 72 69 6f>
console.log(buffer.toString('hex')); // prints '6d6172696f'

JSON Serializing and Deserializing Buffers

A very common format for serialization is JSON, especially when working with JavaScript-based applications. When JSON.stringify encounters any object, it attempts to call the toJSON method on that object, if it exists. Buffer instances have a toJSON method, which returns a plain JavaScript object.

node -p "Buffer.from('hello world').toJSON()"

Calling toJSON on the above Buffer instance returns the following JSON.

{
  "type": "Buffer",
  "data": [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]
}

Buffer instances are represented in JSON by an object, that has a type property with a string value of Buffer and a data property with an array of numbers, representing the value of each byte.

TL;DR

  • If you have to create a buffer, only use the safe method Buffer.alloc.
  • The Buffer.alloc function produces a zero-filled buffer by default.
  • The Buffer constructor does not need the new keyword.
  • There is an unsafe way to allocate buffer allocUnsafe(), this poses a security risk, though there are some advanced use cases.

Thanks for reading and if you have any questions, use the comment function or send me a message @mariokandut.

If you want to know more about Node, have a look at these Node Tutorials.

References (and Big thanks):

JSNAD - Using Buffers Node.js - Buffer, MDN - Javascript Typed Arrays

More node articles:

How to dynamically load ESM in CJS

How to convert a CJS module to an ESM

How to create a CJS module

How to stream to an HTTP response

How to handle binary data in Node.js?

How to use streams to ETL data?

How to connect streams with pipeline?

How to handle stream errors?

How to connect streams with pipe?

What Is a Node.js Stream?

Handling Errors in Node (asynchronous)

Handling Errors in Node.js (synchronous)

Introduction to errors in Node.js

Callback to promise-based functions

ETL: Load Data to Destination with Node.js

ETL: Transform Data with Node.js

ETL: Extract Data with Node.js

Event Emitters in Node.js

How to set up SSL locally with Node.js?

How to use async/await in Node.js

What is an API proxy?

How to make an API request in Node.js?

How does the Event Loop work in Node.js

How to wait for multiple Promises?

How to organize Node.js code

Understanding Promises in Node.js

How does the Node.js module system work?

Set up and test a .env file in Node

How to Use Environment Variables in Node

How to clean up node modules?

Restart a Node.js app automatically

How to update a Node dependency - NPM?

What are NPM scripts?

How to uninstall npm packages?

How to install npm packages?

How to create a package.json file?

What Is the Node.js ETL Pipeline?

What is data brokering in Node.js?

How to read and write JSON Files with Node.js?

What is package-lock.json?

How to install Node.js locally with nvm?

How to update Node.js?

How to check unused npm packages?

What is the Node.js fs module?

What is Semantic versioning?

The Basics of Package.json explained

How to patch an NPM dependency

What is NPM audit?

Beginner`s guide to NPM

Getting started with Node.js

Scroll to top ↑