NodeJS, an Elixir Library for Calling Node.js Functions

Here at Revelry, we’re big proponents of Elixir. Recently, we started doing some work with the Stellar blockchain network, including this cool thing: We’re building an Elixir Stellar client. (It’s open source – please contribute if you’re interested!)

In this process, we ran into a snag: While there were a lot of Stellar SDKs for various languages, Elixir was not one of them.

So we set about doing the responsible thing: writing our own.

(Well, Bryan Joseph did. We mostly just gave him time to do it and cheered him on.)

Turns out, writing a fresh new blockchain client is no arbitrary task.

We’re still committed to the Elixir Stellar client, but we wanted to get up and running a little faster. So we had some decisions to make.

These were the options in front of us:

1. Commit more time to the Elixir Stellar client, and wait for it to be done, or:
2. Use a different language for the application, or:
3. Mash up Elixir with something else that has a solid Stellar client already.

Option 1: Finishing the Elixir Stellar client first sounds great, and we should absolutely do that. However, we’re talking about blockchain code here. This demands a lot of care, given the security implications. It’s clearly not something that should be rushed. We are also slowed down by the fact that Stellar uses a serialization format called XDR, and as I write this, a fully-functioning XDR library for Elixir is not yet complete.

Option 2: Using a different language. Well, the thing is… We really, really like Elixir. Its performance, reliability, and ergonomics have given our apps a big boost. It would be a shame to abandon it just because it lacks one library that we’re too impatient to finish first. Plus, it’s really good at interfacing with other systems.

We went with option 3. Stellar’s JavaScript SDK is one of the better supported and more mature Stellar clients available, and we already use JavaScript for browser code and React Native apps.

So, we decided to incorporate Node.js.

In the process, we created a generic library that provides an API for calling JavaScript functions from Elixir code.

It’s simply named nodejs, and it’s available as a hex package.

Having this Elixir-to-Node bridge allows us to use a tried-and-tested JavaScript library for our app while we get our Elixir client off the ground.

The end product is a package that lets you write in Elixir:

NodeJS.call({"math", "add"}, [1, 2])

And get back {:ok, 3} from a JavaScript function.

You can use it to do the things that Node.js is better at, like…

• Render a React component server-side.
• Devise a scheme to share some code between the server and a browser.
• Or something else I didn’t think of. (Hit me up in the comments.)

At this time, there are some (mostly innocuous) limitations and rules to follow around paths and serialization, so definitely check out the README when getting started.

At this point, if you’re thinking “Dang, I could use this,” and you don’t care how it works, you can stop reading and jump right over to Hex to get started:But if you do want to know more of the implementation details, read on!
I’ll take you through it.

How NodeJS works

Let’s start with the simplest and most universally familiar topic first.

Serialization and data formats

Elixir and JavaScript need a common language with which to speak to each other. We picked JSON because it’s common, easy, and mostly just works. Then we defined what the payload structures should look like.

For function call requests we went with:

[
  [module_name, optional_module_export_key], # [module_name] by itself calls the default JS export
  [arg1, arg2, ..., argx]
]

For example, the following request:

[["math", "add"], [1, 2]]

Is like this JavaScript:

require('./math').add(1, 2)

It sends a JSON-serialized response back to Elixir. For responses, we need to be able to indicate success vs. failure and to pack up return values or errors, respectively. A success response payload looks like:

[true, return_value_json]

While an error response looks like:

[false, error_message_and_stack_trace_string]

On the Elixir side, we unpack these into the {:ok, result} and {:error, error} tuples that you’re accustomed to.

This is Elixir, so we’re obviously going to talk about processes

If you’ve been working in Elixir for a while, you are probably at least somewhat familiar with OTP processes and the Elixir modules for using them, even if that familiarity stops at “I’ve heard of GenServer before.” I’m not going to do a deep dive here. (The docs do a much better job of competently explaining these concepts than I can.) But what I will explain is which Elixir modules we use and how and why.

The basic flow goes like this:

• Start a Supervisor.
• The Supervisor starts a pool of child processes (workers), each of which is a GenServer.
• Each GenServer opens a Port.

The Port starts node and shuttles data between the Elixir app and the nodeserver. The GenServer manages the lifecycle and request/response loop of our Port. And the Supervisor provides fault-tolerance, restarting any GenServerin the pool that might crash.

If you’re now thinking, “I could use this package as a blueprint to do the same thing for Python, Java, etc.” you are correct! Personally, I would love to be able to use the vast scientific and machine learning ecosystem of Python without giving up Elixir, and there’s no reason we could not apply the same strategy there.

What’s next

The library is presently pretty stable, has very good test coverage, and does the job with strong performance characteristics. I’ve kicked around some ideas like finding a more space-efficient serialization format than JSON, but JSON is doing the job well, so I’m waiting for feedback before I do anything about it. A better future idea might be to provide some macros that define more ergonomically-mapped modules/function calls, like:

NodeJS.module MyApp.Math, "math" do
  js add: 2 // defines a 2-argument function named `add`.
  js subtract: 2
  js multiply: 2
  js divide: 2
end

# `MyApp.Math.add(1, 2)` calls `NodeJS.call({"math", "add"}, [1, 2])` 

The other thing on my mind (that I touched on earlier) is not necessarily changing anything about the library, but using it as the basis for an Elixir wrapper around React that can do server-side rendering.

If you’ve got any other thoughts about how to use or improve on this idea, please let me know! And if you give it a try and find some rough edges, even better would be to head over to our GitHub repo and file an issue.

---

Elixir JavaScript JSON Node Stellar