A journey to Asynchronous Programming: NodeJS FS.Promises API

This post is about how Node.Js performs tasks, how and why one would use async methods and why and when one should use sync methods. We will also see how to make use of the new FS. Promises API.

Throughout this post, we will look at the many ways to write code the asynchronous way in Javascript and also look at:

✅ How asynchronous code fits in the event loop
✅ When you should resort to synchronous methods
✅ How to promisify FS methods and the FS.Promises API

Prerequisites
#

To make the best use of this post, one must already be:

✅ Sufficiently experienced in Javascript and NodeJS fundamentals (Variables, Functions, etc.)
✅ Had some level of exposure to the FileSystem module (Optional)

Refer to the resources section if you wish to satisfy the prerequisites and gain the best use of this post

Outcomes
#

Once you're done reading this post you will feel confident about asynchronous programming and will have learned something new but also know:

  • What an event loop and the call stack is and how it fits with async programming

  • About synchronous methods and when to use them

  • About callback hell, the demerits of using callback methods in fs

  • That there are other libraries out there like bluebird that solved this issue, but now there's native support for fs modules with async/await, and their version numbers

  • To use the fs. promise API next time to avoid callback hell

⭐- If you hang tight there's also some bonus content regarding some best practices when using certain FileSystem methods!

The Event Loop and the Call Stack: How does asynchronous flow work
#

Javascript achieves concurrency through what is known as an event loop. This event loop is what is responsible for executing the code you write, processing any event that fires, etc. This event loop is what makes it possible for Javascript to run on a single thread and handle asynchronous tasks, this just means that Javascript does one thing at a time.

This might sound like a limitation but it is definitely something that helps. It allows you to work without worrying about concurrency issues and surprisingly the event loop is non-blocking! Of course, unless you as a developer purposely do something to block it.

The event loop looks more or less like this:

This loop runs as long as your program runs and hence called the event loop.

To better understand asynchronous programming though, one must understand the following concepts:

  • Call Stack: The call stack is where all your code gets pushed into and then executed one by one. Each function pushed into the call stack is called a "frame". These frames stack up on top of each other, and the last one that was pushed gets executed the first.

  • Heap: All objects/non-primitives are allocated in the heap. This is a large amount of memory that mostly does not have any proper structure.

  • Callback Queue: This is where the asynchronous code gets thrown into, and waits till their time for execution comes.

  • Job Queue: A recently introduced type of queue reserved primarily for Promises. All thenables from your Promise code, are pushed into this queue.

Let's take a look at the following code example and see how a typical execution flow looks like:

Initially the synchronous tasks console.log() will be run in the order they were pushed into the call stack. Then the Promise thenables will be pushed into the Job Queue, while the setTimeout 's callback function is pushed into the Callback Queue. However, as the Job Queue is given a higher priority than the Callback Queue, the thenables are executed before the callback functions. What's a promise or a thenable, you ask? That's what we will look at in the next topic!

Promises and async/await: The solution to callback hell#

As you previously saw in the setTimeout, a callback function is one of the ways that Javascript allows you to write asynchronous code. In Javascript, even Functions are objects and because of this a function can take another function as an argument/parameter and can also be returned by functions.

Functions that take another function as an argument is called a Higher-Order Function. A function that is passed as an argument to another function is what is known as a callback. But quite often, having a whole lot of callbacks look like this:

Taken from callbackhell, this shows how extremely complex and difficult it might get to maintain callbacks in a large codebase. Don't panic! That's why we have promises.

A promise is an object that will produce some value in the future. When? We can't say, it depends. However, the value that is produced is one of two things. It is either a resolved value or a reason why it couldn't be resolved, which usually indicates something is wrong.

A promise goes through a lifecycle that can be visualized like the following:


Taken from a great resource on promises, MDN.

But still, this didn't provide the cleanliness we wanted because it was quite easy to have a whole lot of thenables one after the other. This is why the async/await syntax was introduced, which looks like the following:

Looks a whole lot better than what you saw in all the previous code examples!

Synchronous FS methods: What they are and when to use#

Before we jump into the exciting FS.promises API that I previously used, we must talk about the often unnoticed and unnecessarily avoided synchronous FileSystem methods. Remember how I mentioned previously that you can purposely block the event loop? A synchronous FS method does just that. Now you might have heard quite a lot of times about how you should avoid synchronous FS methods like the plague, but trust me because they block the event loop, there are times when you can use them.

Typically, a synchronous method will include the word Sync in it's name, so watch out for them!

A synchronous function should be used over an asynchronous one when:

  • You need a task to complete before any of the tasks that come after are performed.

  • The task that you're doing seems more synchronous than asynchronous in nature.

A typical use case to satisfy both the above use cases can be expressed like this:

DataStore is a means of storing products, and you'll easily notice the use of synchronous methods. The reason for this use is that it is completely acceptable to use a synchronous method like this as the constructor function is run only once per every creation of a new instance of DataStore. Also, it is essential to see if the file is available and create the file before it will be used by any other function.

Asynchronous FS methods: Why they are better and how#

The asynchronous FileSystem methods in NodeJS, commonly use callbacks because, during the time they were made, Promises and async/await hadn't come out nor were they at experimental stages. The key advantage these methods provide over their synchronous siblings is the fact that you do not end up blocking the event loop when you use them. This allows us to write better more performant code.

When code is run asynchronously, the CPU does not wait idly by until a task is completed but moves on to the next set of tasks. For example, let us take a task that takes 200ms to complete. If a synchronous method is used, CPU will be occupied for the entire 200ms but if you use around 190ms of that time is freed up and can now be used by the CPU to perform any other tasks that are available.

A typical code example of asynchronous FileSystem methods are:

As you can see, they are distinguished by the lack of Sync and the apparent usage of callback functions. When secret.txt has been completely read, the callback function will be executed and the secret data stored will be printed on the console.

Warning: Mixing sync and async code#


As humans, we're prone to making silly mistakes and when frustrated or when we experience a lot of stress, we tend to make unwise decisions, one such decision is mixing synchronous code with asynchronous code!

Let's look at the following situation:

Due to the nature of how NodeJS tackles operations, it is very much likely that the secret.txt file is deleted before we actually read it. Thankfully here though, we are catching the error so we will know that the file doesn't exist anymore. It is best to not mix asynchronous code with synchronous code, being consistent is mandatory in a modern codebase.

The history of promisifying FileSystem#

Way back when FS.promises was introduced, developers had to resort to a few troublesome techniques. You might not need them anymore, but in the unlikely event you end up using an old version of NodeJS knowing how to achieve promisification will help greatly.

One method is to use the promisify method from the NodeJS util module:

But as you can see, this allows you to only turn one method into its promisified version at a time, so some developers often used an external module known as bluebird that allowed one to do this:

Some developers still use bluebird as opposed to the natively implemented Promises API, due to performance reasons.

FS.promises API: Native support, One less thing to worry about.#

As of NodeJS version 10.0, you can now use FS.promises a solution to all the problems that you'd face with thenables when you use Promises. You can neatly and directly use the FS.promises API and the clean async/await syntax. You do not have to use any other external dependencies.

To use the FS.promises API you would do something like the following:

It's much cleaner than the code you saw from the callback hell example, and the promises example as well! One must note however that async/await is simply syntax sugar, meaning it uses the Promise API under the hood.

⭐ Take this with you: Use NodeJs File Streams to reduce memory footprint#

File streams are unfortunately one of the most unused or barely known concepts in the FileSystem module. To understand how a FileStream works, you must look at the Streams API in the NodeJS docs.

One very common use case of FileStreams is when you must copy a large file, quite often whether you use an asynchronous method or synchronous method, this leads to a large amount of memory usage and a long time. This can be avoided by using the FileSystem methods fs.createReadStream and fs.createWriteStream.

Conclusion
#

Phew! That was long, wasn't it? But now you must feel pretty confident regarding asynchronous programming, and you can now use the FS.promises API instead of the often used callback methods in the FileSystem module. Over time, we will see more changes in NodeJS, it is after all written in a language that is widely popular. What you should do now is check out the resources section and read some more about this or try out Fullstack Node.Js to further improve your confidence and get a lot of other different tools under your belt!

Resources#

  1. Eloquent Javascript: Asynchronous Programming

  2. NodeJS Docs: Fs.promises

  3. JSConf Talk: What the heck is an event loop anyway?

  4. Fullstack Node.Js

  5. An in-depth yet friendly guide in Promises