Domain Module Postmortem

Evan Lucas

Domain Module Postmortem

It's possible for a developer to create a new domain and then simply run domain.enter(). Which then acts as a catch-all for any exception in the future that couldn't be observed by the thrower. Allowing a module author to intercept the exceptions of unrelated code in a different module. Preventing the originator of the code from knowing about its own exceptions.

Here's an example of how one indirectly linked modules can affect another:

// module a.js
const import bb = require: anyrequire('./b');
const import cc = require: anyrequire('./c');

// module b.js
const const d: anyd = require: anyrequire('domain').create();
const d: anyd.on('error', () => {
  /* silence everything */
});
const d: anyd.enter();

// module c.js
const import depdep = require: anyrequire('some-dep');
import depdep.method(); // Uh-oh! This method doesn't actually exist.

Since module b enters the domain but never exits any uncaught exception will be swallowed. Leaving module c in the dark as to why it didn't run the entire script. Leaving a potentially partially populated module.exports. Doing this is not the same as listening for 'uncaughtException'. As the latter is explicitly meant to globally catch errors. The other issue is that domains are processed prior to any 'uncaughtException' handlers, and prevent them from running.

Another issue is that domains route errors automatically if no 'error' handler was set on the event emitter. There is no opt-in mechanism for this, and automatically propagates across the entire asynchronous chain. This may seem useful at first, but once asynchronous calls are two or more modules deep and one of them doesn't include an error handler the creator of the domain will suddenly be catching unexpected exceptions, and the thrower's exception will go unnoticed by the author.

The following is a simple example of how a missing 'error' handler allows the active domain to hijack the error:

const import domaindomain = require: anyrequire('domain');
const import netnet = require: anyrequire('net');
const const d: anyd = import domaindomain.create();
const d: anyd.on('error', err: anyerr => var console: Consoleconsole.Console.error(...data: any[]): void[MDN Reference](https://developer.mozilla.org/docs/Web/API/console/error_static)error(err: anyerr.message));

const d: anyd.run(() =>
  import netnet
    .createServer(c: anyc => {
      c: anyc.end();
      c: anyc.write('bye');
    })
    .listen(8000)
);

Even manually removing the connection via d.remove(c) does not prevent the connection's error from being automatically intercepted.

Failures that plagues both error routing and exception handling are the inconsistencies in how errors are bubbled. The following is an example of how nested domains will and won't bubble the exception based on when they happen:

const import domaindomain = require: anyrequire('domain');
const import netnet = require: anyrequire('net');
const const d: anyd = import domaindomain.create();
const d: anyd.on('error', () => var console: Consoleconsole.Console.error(...data: any[]): void[MDN Reference](https://developer.mozilla.org/docs/Web/API/console/error_static)error('d intercepted an error'));

const d: anyd.run(() => {
  const const server: anyserver = import netnet
    .createServer(c: anyc => {
      const const e: anye = import domaindomain.create(); // No 'error' handler being set.
      const e: anye.run(() => {
        // This will not be caught by d's error handler.
        setImmediate(() => {
          throw new var Error: ErrorConstructor
new (message?: string, options?: ErrorOptions) => Error (+1 overload)Error('thrown from setImmediate');
        });
        // Though this one will bubble to d's error handler.
        throw new var Error: ErrorConstructor
new (message?: string, options?: ErrorOptions) => Error (+1 overload)Error('immediately thrown');
      });
    })
    .listen(8080);
});

It may be expected that nested domains always remain nested, and will always propagate the exception up the domain stack. Or that exceptions will never automatically bubble. Unfortunately both these situations occur, leading to potentially confusing behavior that may even be prone to difficult to debug timing conflicts.

API Gaps

While APIs based on using EventEmitter can use bind() and errback style callbacks can use intercept(), alternative APIs that implicitly bind to the active domain must be executed inside of run(). Meaning if module authors wanted to support domains using a mechanism alternative to those mentioned they must manually implement domain support themselves. Instead of being able to leverage the implicit mechanisms already in place.

Error Propagation

Propagating errors across nested domains is not straight forward, if even possible. Existing documentation shows a simple example of how to close() an http server if there is an error in the request handler. What it does not explain is how to close the server if the request handler creates another domain instance for another async request. Using the following as a simple example of the failing of error propagation:

const module d1
const d1: anyd1 = domain.create();
module d1
const d1: anyd1.foo = true; // custom member to make more visible in console
module d1
const d1: anyd1.on('error', er: anyer => {
  /* handle error */
});

module d1
const d1: anyd1.run(() =>
  function setTimeout(handler: TimerHandler, timeout?: number, ...arguments: any[]): number[MDN Reference](https://developer.mozilla.org/docs/Web/API/Window/setTimeout)setTimeout(() => {
    const const d2: anyd2 = domain.create();
    const d2: anyd2.bar = 43;
    const d2: anyd2.on('error', er: anyer => var console: Consoleconsole.Console.error(...data: any[]): void[MDN Reference](https://developer.mozilla.org/docs/Web/API/console/error_static)error(er: anyer.message, domain._stack));
    const d2: anyd2.run(() => {
      function setTimeout(handler: TimerHandler, timeout?: number, ...arguments: any[]): number[MDN Reference](https://developer.mozilla.org/docs/Web/API/Window/setTimeout)setTimeout(() => {
        function setTimeout(handler: TimerHandler, timeout?: number, ...arguments: any[]): number[MDN Reference](https://developer.mozilla.org/docs/Web/API/Window/setTimeout)setTimeout(() => {
          throw new var Error: ErrorConstructor
new (message?: string, options?: ErrorOptions) => Error (+1 overload)Error('outer');
        });
        throw new var Error: ErrorConstructor
new (message?: string, options?: ErrorOptions) => Error (+1 overload)Error('inner');
      });
    });
  })
);

Even in the case that the domain instances are being used for local storage so access to resources are made available there is still no way to allow the error to continue propagating from d2 back to d1. Quick inspection may tell us that simply throwing from d2's domain 'error' handler would allow d1 to then catch the exception and execute its own error handler. Though that is not the case. Upon inspection of domain._stack you'll see that the stack only contains d2.

This may be considered a failing of the API, but even if it did operate in this way there is still the issue of transmitting the fact that a branch in the asynchronous execution has failed, and that all further operations in that branch must cease. In the example of the http request handler, if we fire off several asynchronous requests and each one then write()'s data back to the client many more errors will arise from attempting to write() to a closed handle. More on this in Resource Cleanup on Exception.

Resource Cleanup on Exception

The following script contains a more complex example of properly cleaning up in a small resource dependency tree in the case that an exception occurs in a given connection or any of its dependencies. Breaking down the script into its basic operations:

'use strict';

const import domaindomain = require: anyrequire('domain');
const import EEEE = require: anyrequire('events');
const import fsfs = require: anyrequire('fs');
const import netnet = require: anyrequire('net');
const import utilutil = require: anyrequire('util');
const const print: anyprint = process._rawDebug;

const const pipeList: any[]pipeList = [];
const const FILENAME: "/tmp/tmp.tmp"FILENAME = '/tmp/tmp.tmp';
const const PIPENAME: "/tmp/node-domain-example-"PIPENAME = '/tmp/node-domain-example-';
const const FILESIZE: 1024FILESIZE = 1024;
var var uid: numberuid = 0;

// Setting up temporary resources
const const buf: anybuf = Buffer(const FILESIZE: 1024FILESIZE);
for (var var i: numberi = 0; var i: numberi < const buf: anybuf.length; var i: numberi++) const buf: anybuf[var i: numberi] = ((var Math: MathAn intrinsic object that provides basic mathematics functionality and constants.Math.Math.random(): numberReturns a pseudorandom number between 0 and 1.random() * 1e3) % 78) + 48; // Basic ASCII
import fsfs.writeFileSync(const FILENAME: "/tmp/tmp.tmp"FILENAME, const buf: anybuf);

function function ConnectionResource(c: any): voidConnectionResource(c: anyc) {
  import EEEE.call(this);
  this.ConnectionResource._connection: any_connection = c: anyc;
  this.ConnectionResource._alive: any_alive = true;
  this.ConnectionResource._domain: any_domain = import domaindomain.create();
  this.ConnectionResource._id: any_id = var Math: MathAn intrinsic object that provides basic mathematics functionality and constants.Math.Math.random(): numberReturns a pseudorandom number between 0 and 1.random().Number.toString(radix?: number): stringReturns a string representation of an object.
@paramradix Specifies a radix for converting numeric values to strings. This value is only used for numbers.toString(32).String.substr(from: number, length?: number): stringGets a substring beginning at the specified location and having the specified length.
@deprecatedA legacy feature for browser compatibility@paramfrom The starting position of the desired substring. The index of the first character in the string is zero.@paramlength The number of characters to include in the returned substring.substr(2).String.substr(from: number, length?: number): stringGets a substring beginning at the specified location and having the specified length.
@deprecatedA legacy feature for browser compatibility@paramfrom The starting position of the desired substring. The index of the first character in the string is zero.@paramlength The number of characters to include in the returned substring.substr(0, 8) + ++var uid: numberuid;

  this.ConnectionResource._domain: any_domain.add(c: anyc);
  this.ConnectionResource._domain: any_domain.on('error', () => {
    this.ConnectionResource._alive: boolean_alive = false;
  });
}
import utilutil.inherits(class ConnectionResource
function ConnectionResource(c: any): voidConnectionResource, import EEEE);

class ConnectionResource
function ConnectionResource(c: any): voidConnectionResource.Function.prototype: anyprototype.ConnectionResource.end(chunk: any): voidend = function function (local function) end(chunk: any): voidend(chunk: anychunk) {
  this.ConnectionResource._alive: boolean_alive = false;
  this.ConnectionResource._connection: any_connection.end(chunk: anychunk);
  this.emit('end');
};

class ConnectionResource
function ConnectionResource(c: any): voidConnectionResource.Function.prototype: anyprototype.ConnectionResource.isAlive(): booleanisAlive = function function (local function) isAlive(): booleanisAlive() {
  return this.ConnectionResource._alive: boolean_alive;
};

class ConnectionResource
function ConnectionResource(c: any): voidConnectionResource.Function.prototype: anyprototype.ConnectionResource.id(): stringid = function function (local function) id(): stringid() {
  return this.ConnectionResource._id: string_id;
};

class ConnectionResource
function ConnectionResource(c: any): voidConnectionResource.Function.prototype: anyprototype.ConnectionResource.write(chunk: any): anywrite = function function (local function) write(chunk: any): anywrite(chunk: anychunk) {
  this.emit('data', chunk: anychunk);
  return this.ConnectionResource._connection: any_connection.write(chunk: anychunk);
};

// Example begin
import netnet
  .createServer(c: anyc => {
    const const cr: ConnectionResourcecr = new constructor ConnectionResource(c: any): ConnectionResourceConnectionResource(c: anyc);

    const const d1: anyd1 = import domaindomain.create();
    import fsfs.open(
      const FILENAME: "/tmp/tmp.tmp"FILENAME,
      'r',
      const d1: anyd1.intercept(fd: anyfd => {
        function streamInParts(fd: any, cr: any, pos: any): voidstreamInParts(fd: anyfd, const cr: ConnectionResourcecr, 0);
      })
    );

    function pipeData(cr: any): voidpipeData(const cr: ConnectionResourcecr);

    c: anyc.on('close', () => const cr: ConnectionResourcecr.ConnectionResource.end(chunk: any): voidend());
  })
  .listen(8080);

function function streamInParts(fd: any, cr: any, pos: any): voidstreamInParts(fd: anyfd, cr: anycr, pos: anypos) {
  const const d2: anyd2 = import domaindomain.create();
  var function (local var) alive: booleanalive = true;
  const d2: anyd2.on('error', er: anyer => {
    const print: anyprint('d2 error:', er: anyer.message);
    cr: anycr.end();
  });
  import fsfs.read(
    fd: anyfd,
    new Buffer(10),
    0,
    10,
    pos: anypos,
    const d2: anyd2.intercept((bRead: anybRead, buf: anybuf) => {
      if (!cr: anycr.isAlive()) {
        return import fsfs.close(fd: anyfd);
      }
      if (cr: anycr._connection.bytesWritten < const FILESIZE: 1024FILESIZE) {
        // Documentation says callback is optional, but doesn't mention that if
        // the write fails an exception will be thrown.
        const const goodtogo: anygoodtogo = cr: anycr.write(buf: anybuf);
        if (const goodtogo: anygoodtogo) {
          function setTimeout(handler: TimerHandler, timeout?: number, ...arguments: any[]): number[MDN Reference](https://developer.mozilla.org/docs/Web/API/Window/setTimeout)setTimeout(() => function streamInParts(fd: any, cr: any, pos: any): voidstreamInParts(fd: anyfd, cr: anycr, pos: anypos + bRead: anybRead), 1000);
        } else {
          cr: anycr._connection.once('drain', () =>
            function streamInParts(fd: any, cr: any, pos: any): voidstreamInParts(fd: anyfd, cr: anycr, pos: anypos + bRead: anybRead)
          );
        }
        return;
      }
      cr: anycr.end(buf: anybuf);
      import fsfs.close(fd: anyfd);
    })
  );
}

function function pipeData(cr: any): voidpipeData(cr: anycr) {
  const const pname: stringpname = const PIPENAME: "/tmp/node-domain-example-"PIPENAME + cr: anycr.id();
  const const ps: anyps = import netnet.createServer();
  const const d3: anyd3 = import domaindomain.create();
  const const connectionList: any[]connectionList = [];
  const d3: anyd3.on('error', er: anyer => {
    const print: anyprint('d3 error:', er: anyer.message);
    cr: anycr.end();
  });
  const d3: anyd3.add(const ps: anyps);
  const ps: anyps.on('connection', conn: anyconn => {
    const connectionList: any[]connectionList.Array<any>.push(...items: any[]): numberAppends new elements to the end of an array, and returns the new length of the array.
@paramitems New elements to add to the array.push(conn: anyconn);
    conn: anyconn.on('data', () => {}); // don't care about incoming data.
    conn: anyconn.on('close', () => {
      const connectionList: any[]connectionList.Array<any>.splice(start: number, deleteCount?: number): any[] (+1 overload)Removes elements from an array and, if necessary, inserts new elements in their place, returning the deleted elements.
@paramstart The zero-based location in the array from which to start removing elements.@paramdeleteCount The number of elements to remove.@returnsAn array containing the elements that were deleted.splice(const connectionList: any[]connectionList.Array<any>.indexOf(searchElement: any, fromIndex?: number): numberReturns the index of the first occurrence of a value in an array, or -1 if it is not present.
@paramsearchElement The value to locate in the array.@paramfromIndex The array index at which to begin the search. If fromIndex is omitted, the search starts at index 0.indexOf(conn: anyconn), 1);
    });
  });
  cr: anycr.on('data', chunk: anychunk => {
    for (var function (local var) i: numberi = 0; function (local var) i: numberi < const connectionList: any[]connectionList.Array<any>.length: numberGets or sets the length of the array. This is a number one higher than the highest index in the array.length; function (local var) i: numberi++) {
      const connectionList: any[]connectionList[function (local var) i: numberi].write(chunk: anychunk);
    }
  });
  cr: anycr.on('end', () => {
    for (var function (local var) i: numberi = 0; function (local var) i: numberi < const connectionList: any[]connectionList.Array<any>.length: numberGets or sets the length of the array. This is a number one higher than the highest index in the array.length; function (local var) i: numberi++) {
      const connectionList: any[]connectionList[function (local var) i: numberi].end();
    }
    const ps: anyps.close();
  });
  const pipeList: any[]pipeList.Array<any>.push(...items: any[]): numberAppends new elements to the end of an array, and returns the new length of the array.
@paramitems New elements to add to the array.push(const pname: stringpname);
  const ps: anyps.listen(const pname: stringpname);
}

process.on('SIGINT', () => process.exit());
process.on('exit', () => {
  try {
    for (var function (local var) i: numberi = 0; function (local var) i: numberi < const pipeList: any[]pipeList.Array<any>.length: numberGets or sets the length of the array. This is a number one higher than the highest index in the array.length; function (local var) i: numberi++) {
      import fsfs.unlinkSync(const pipeList: any[]pipeList[function (local var) i: numberi]);
    }
    import fsfs.unlinkSync(const FILENAME: "/tmp/tmp.tmp"FILENAME);
  } catch (function (local var) e: unknowne) {}
});

When a new connection happens, concurrently:
- Open a file on the file system
- Open Pipe to unique socket
Read a chunk of the file asynchronously
Write chunk to both the TCP connection and any listening sockets
If any of these resources error, notify all other attached resources that they need to clean up and shutdown

As we can see from this example a lot more must be done to properly clean up resources when something fails than what can be done strictly through the domain API. All that domains offer is an exception aggregation mechanism. Even the potentially useful ability to propagate data with the domain is easily countered, in this example, by passing the needed resources as a function argument.

One problem domains perpetuated was the supposed simplicity of being able to continue execution, contrary to what the documentation stated, of the application despite an unexpected exception. This example demonstrates the fallacy behind that idea.

Attempting proper resource cleanup on unexpected exception becomes more complex as the application itself grows in complexity. This example only has 3 basic resources in play, and all of them with a clear dependency path. If an application uses something like shared resources or resource reuse the ability to cleanup, and properly test that cleanup has been done, grows greatly.

In the end, in terms of handling errors, domains aren't much more than a glorified 'uncaughtException' handler. Except with more implicit and unobservable behavior by third-parties.

Resource Propagation

Another use case for domains was to use it to propagate data along asynchronous data paths. One problematic point is the ambiguity of when to expect the correct domain when there are multiple in the stack (which must be assumed if the async stack works with other modules). Also the conflict between being able to depend on a domain for error handling while also having it available to retrieve the necessary data.

The following is a involved example demonstrating the failing using domains to propagate data along asynchronous stacks:

const import domaindomain = require: anyrequire('domain');
const import netnet = require: anyrequire('net');

const const server: anyserver = import netnet
  .createServer(c: anyc => {
    // Use a domain to propagate data across events within the
    // connection so that we don't have to pass arguments
    // everywhere.
    const const d: anyd = import domaindomain.create();
    const d: anyd.data = { connection: anyconnection: c: anyc };
    const d: anyd.add(c: anyc);
    // Mock class that does some useless async data transformation
    // for demonstration purposes.
    const const ds: DataStreamds = new constructor DataStream(cb: any): DataStreamDataStream(function dataTransformed(chunk: any): voiddataTransformed);
    c: anyc.on('data', chunk: anychunk => const ds: DataStreamds.DataStream.data(chunk: any): voiddata(chunk: anychunk));
  })
  .listen(8080, () => var console: Consoleconsole.Console.log(...data: any[]): void[MDN Reference](https://developer.mozilla.org/docs/Web/API/console/log_static)log(`listening on 8080`));

function function dataTransformed(chunk: any): voiddataTransformed(chunk: anychunk) {
  // FAIL! Because the DataStream instance also created a
  // domain we have now lost the active domain we had
  // hoped to use.
  import domaindomain.active.data.connection.write(chunk: anychunk);
}

function function DataStream(cb: any): voidDataStream(cb: anycb) {
  this.DataStream.cb: anycb = cb: anycb;
  // DataStream wants to use domains for data propagation too!
  // Unfortunately this will conflict with any domain that
  // already exists.
  this.DataStream.domain: anydomain = import domaindomain.create();
  this.DataStream.domain: anydomain.data = { inst: thisinst: this };
}

class DataStream
function DataStream(cb: any): voidDataStream.Function.prototype: anyprototype.DataStream.data(chunk: any): voiddata = function function (local function) data(chunk: any): voiddata(chunk: anychunk) {
  // This code is self contained, but pretend it's a complex
  // operation that crosses at least one other module. So
  // passing along "this", etc., is not easy.
  this.DataStream.domain: anydomain.run(function () {
    // Simulate an async operation that does the data transform.
    setImmediate(() => {
      for (var function (local var) i: numberi = 0; function (local var) i: numberi < chunk: anychunk.length; function (local var) i: numberi++)
        chunk: anychunk[function (local var) i: numberi] = ((chunk: anychunk[function (local var) i: numberi] + var Math: MathAn intrinsic object that provides basic mathematics functionality and constants.Math.Math.random(): numberReturns a pseudorandom number between 0 and 1.random() * 100) % 96) + 33;
      // Grab the instance from the active domain and use that
      // to call the user's callback.
      const const self: anyself = import domaindomain.active.data.inst;
      const self: anyself.cb.call(const self: anyself, chunk: anychunk);
    });
  });
};

The above shows that it is difficult to have more than one asynchronous API attempt to use domains to propagate data. This example could possibly be fixed by assigning parent: domain.active in the DataStream constructor. Then restoring it via domain.active = domain.active.data.parent just before the user's callback is called. Also the instantiation of DataStream in the 'connection' callback must be run inside d.run(), instead of simply using d.add(c), otherwise there will be no active domain.

In short, for this to have a prayer of a chance usage would need to strictly adhere to a set of guidelines that would be difficult to enforce or test.

Performance Issues

A significant deterrent from using domains is the overhead. Using node's built-in http benchmark, http_simple.js, without domains it can handle over 22,000 requests/second. Whereas if it's run with NODE_USE_DOMAINS=1 that number drops down to under 17,000 requests/second. In this case there is only a single global domain. If we edit the benchmark so the http request callback creates a new domain instance performance drops further to 15,000 requests/second.

While this probably wouldn't affect a server only serving a few hundred or even a thousand requests per second, the amount of overhead is directly proportional to the number of asynchronous requests made. So if a single connection needs to connect to several other services all of those will contribute to the overall latency of delivering the final product to the client.

Using AsyncWrap and tracking the number of times init/pre/post/destroy are called in the mentioned benchmark we find that the sum of all events called is over 170,000 times per second. This means even adding 1 microsecond overhead per call for any type of setup or tear down will result in a 17% performance loss. Granted, this is for the optimized scenario of the benchmark, but I believe this demonstrates the necessity for a mechanism such as domain to be as cheap to run as possible.

Looking Ahead

The domain module has been soft deprecated since Dec 2014, but has not yet been removed because node offers no alternative functionality at the moment. As of this writing there is ongoing work building out the AsyncWrap API and a proposal for Zones being prepared for the TC39. At such time there is suitable functionality to replace domains it will undergo the full deprecation cycle and eventually be removed from core.

Node.js Foundation Individual Membership Director election opens Friday, January 20 Node.js Foundation Individual Membership Now Open

Domain Module Postmortem

Domain Module Postmortem

Usability Issues

Implicit Behavior

API Gaps

Error Propagation

Resource Cleanup on Exception

Resource Propagation

Performance Issues

Looking Ahead