Mosca MQTT Persistence Performance

While quietly rolling out the public beta of Adafruit IO, we noticed some CPU spikes in our Node.js processes that handle MQTT. We currently use a slightly modified version of Mosca by @mcollina as our MQTT broker. It is a very solid package, but we seemed to be pushing the limits of Mosca’s Redis persistence module.

Debugging

After examining the output of node --perf in chrome://tracing, I seemed to have found the culprit. It looked as if Mosca was waiting on Redis to return the result of HKEYS for retained topics.

v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  QuickSort native array.js:710:26
v8  InnerArraySort native array.js:670:24 {1}
v8  sort native array.js:885:19 {2}
v8  /home/deploy/io/releases/20151202164744/node/node_modules/mosca/lib/persistence/redis.js:220:44
v8  RedisClient.return_reply /home/deploy/io/releases/20151202164744/node/node_modules/redis/index.js:608:47 {2}
v8  /home/deploy/io/releases/20151202164744/node/node_modules/redis/index.js:306:34
v8  doNTCallback0 node.js:416:27
v8  _tickCallback node.js:335:27 {1}

We retain all current values for topics, so they are immediately returned when a user connects to Adafruit IO and subscribes to a topic. This enables the user to return to a previous state on startup without making any calls to the REST API.

Improving Performance with Redis using Lua

In order to improve performance, I decided to do some preprocessing in Lua on the Redis server before returning hash keys to Node.js. MQTT has two standard wildcard characters: + is a wildcard for matching one topic level, and # can be used for matching multiple. Given Lua’s limited string pattern matching abilities, I decided to narrow down the results using Lua and do the final matching in Node.js.

Here’s the Lua script:

local cursor = 0
local response = {}
local match
local hscan
repeat
  hscan = redis.call('HSCAN', KEYS[1], cursor, 'MATCH', ARGV[1])
  cursor = tonumber(hscan[1])
  match = hscan[2]
  if match then
    for idx = 1, #match, 2 do
      response[match[idx]] = match[idx + 1]
    end
  end
until cursor == 0
return cjson.encode(response)

You can call the script from Node.js using Redis EVAL like this:

'use strict';

// this example is es6, so it will only work in node >=4.0.0.
// it assumes you have a redis hash created with at least topic/1/1
// and topic/2/1 as keys.
const redis = require('redis'),
      async = require('async');

const client = redis.createClient(6379, '127.0.0.1');

// syntax highlighting is currently broken for es6 template strings :\
const script = "\
local cursor = 0 \
local response = {} \
local match \
local hscan \
repeat \
  hscan = redis.call('HSCAN', KEYS[1], cursor, 'MATCH', ARGV[1]) \
  cursor = tonumber(hscan[1]) \
  match = hscan[2] \
  if match then \
    for idx = 1, #match, 2 do \
      response[match[idx]] = match[idx + 1] \
    end \
  end \
until cursor == 0 \
return cjson.encode(response)";

client.eval(script, 1, 'test', 'topic/*/999997', function(err, topics){
  if (err) throw err;
  console.log('found', topics);
  client.end();
});

The HSCAN wildcard character is *, so I just replaced the MQTT wildcard characters with *, and did the final pattern matching in Node.js.

Benchmarks

I was benchmarking the change with 1 million retained hash keys, and it looks like HKEYS method is about 258% slower than preprocessing with Lua. The results are listed below so you can compare the results of the current HKEYS method vs the lua script:

created 1000000 hash keys

real  0m23.453s
user  0m11.763s
sys  0m11.694s

--------------------------------------  HKEYS  --------------------------------------
retrieved 1000000 hash keys via HKEYS
JS matched topic: 'topic/0.09308264800347388/999997'
got value via HGET: 0.3918832363560796
result {"topic\/0.09308264800347388\/999997":"0.3918832363560796"}

real  0m3.460s
user  0m2.729s
sys  0m0.194s

----------------------------------------  LUA  ---------------------------------------
retrieved 1 hash keys via Lua
JS matched topic: 'topic/0.09308264800347388/999997'
result {"topic\/0.09308264800347388\/999997":"0.3918832363560796"}

real  0m1.337s
user  0m0.116s
sys  0m0.024s

The pull request to mosca can be found here: mcollina/mosca#379