Messaging with Redis and Node.js Using Node_Redis Client

1 Comment

Now that I have connectivity between client HTML 5 pages and a node.js web socket server (see creating an HTML web socket client for details), the next step in the process is to integrate a message broker to distribute messages to the scalable middleware.

In my architecture post I covered the proposed architecture of the Kraken Office system. In order to create a scalable, distributed system I am using messaging to break dependencies between components and to enable scalability of the middleware components. I have chosen the Redis data store technology as the message broker for the system, partially as Redis will also be used for data storage and also because it is extremely fast.

I had initially selected the Redback library to assist in integrating my Node.js server with Redis. On closer inspection, however, it turns out that this library is suitable for Redis persistence requirements but does not provide integration with the Redis publisher / subscriber messaging feature. I therefore looked for an alternative integration library that did provide access to the messaging features and eventually selected the node_redis project, which includes all of the features necessary for sending and receiving messages with Redis in a node.js context.

Installing And Starting Redis On Window

As I am working with Windows the first step is to download, install and start the Redis server. Redis is written for the Unix platform although several Windows ports of the Redis project exist. After looking around I eventually used the Windows downloads available here.

Redis doesn’t actually require any installation. In order to start the service you simply need to navigate to the download location and then double-click the redis-server.exe file. This starts the Redis service in a console window. Projects exist on the web for hosting Redis within a Windows Service – a quick search on the web will find them – although initially at least I am content to run Redis within a console window as it allows me to quickly and easily view the state of the server and any errors.

Redis can be configured by changing the setting in the redis.conf file. I won’t go through the various settings in here now other than to say that this is where the port settings can be found, enabling you to run multiple instances of Redis on different ports.

Installing The Node_Redis Library

Installing the Node_Redis library, enabling connectivity between Node.js and Redis, is straightforward. The library can be installed using the NPM package manager. Simply open a command window and use the CD command to navigate to the root installation of your node.js installation. Once you have done this you then use the following NPM command to install the library:

npm install redis

If you execute your node.js javascript driver files from the installation directory (usually Program Files) then the Node_Redis library is ready for use. If you execute your node.js scripts from a different location you will need to copy the node_modules folder from the installation directory to the directory in which your javascript node.js driver files reside. If you do not do this node.js will not be able to locate the correct library files.

Sending Messages To Redis From Node.js

Before using the Node_Redis library in your own node.js javascript driver files you will need to add a require statement to the top of your javascript file. This should be done as follows:

var redis = require("redis");

This ensures that node.js can locate the correct library files.

The next thing is to create an instance of the Redis client as follows:

var REDIS_URL = 'localhost';
var REDIS_PORT = 6379;

var redisClient = redis.createClient( REDIS_PORT, REDIS_URL );

Obviously the Url and Port variables will need to be set to those of your own Redis server. You can find the port on which your Redis server is running by referring to the Redis start-up trace information displayed in the console window.

To send a message to the broker you can now simply call:

redisClient.publish( 'channel_name','message_body' );

The ability to specify a channel enables multiple messaging channels to be open at any one time. A subscriber can the decide from which channel they are interested in receiving messages.

A real-world example would be:

redisClient.publish( "application_layer", "{client logon request message}" );

This shows how easy it is to publish messages to a Redis server from node.js.

Building on our echo web socket server described in setting up a web socket service with node.js on windows, below is code for an amended server which doesn’t echo messages back to the client but instead sends the message to a channel in Redis.

In order to run this web socket server in node.js you should:

  1.  Save the code to a file with a .js extension in the root of your node.js installation
  2. Open a command window and use the CD command to navigate to the node.js installation directory
  3. Use the node <filename> command to start the node.js server
#!/usr/bin/env nodevar WebSocketServer = require('websocket').server;
var http = require('http');
var redis = require("redis");

var REDIS_URL = 'localhost';
var REDIS_PORT = 6379;
// create a redis connection
try
{
 var redisClient = redis.createClient( REDIS_PORT, REDIS_URL );
}
catch (err)
{
 console.log( "ERROR => Cannot connect to Redis message broker: URL => " + REDIS_URL + "; Port => " + REDIS_PORT );
 console.log(err);
}

var server = http.createServer(function(request, response) {
 console.log((new Date()) + ' Received request for ' + request.url);
 response.writeHead(404);
 response.end();
});
server.listen(8080, function() {
 console.log((new Date()) + ' Server is listening on port 8080');
});

wsServer = new WebSocketServer({
 httpServer: server,
 // You should not use autoAcceptConnections for production
 // applications, as it defeats all standard cross-origin protection
 // facilities built into the protocol and the browser. You should
 // *always* verify the connection's origin and decide whether or not
 // to accept it.
 autoAcceptConnections: false
});

function originIsAllowed(origin) {
 // put logic here to detect whether the specified origin is allowed.
 console.log(origin);
 return true;
}

wsServer.on('request', function(request) {
 if (!originIsAllowed(request.origin)) {
 // Make sure we only accept requests from an allowed origin
 request.reject();
 console.log((new Date()) + ' Connection from origin ' + request.origin + ' rejected.');
 return;
 }

var connection = request.accept('kraken-protocol', request.origin);
 console.log((new Date()) + ' Connection accepted.');

connection.on('message', function(message) {

 if (message.type === 'utf8') {
 console.log('Received Message: ' + message.utf8Data);

// post the message to the redis message broker
 var channelName = 'application';
 redisClient.publish(channelName, message.utf8Data);

}
 });
 connection.on('close', function(reasonCode, description) {
 console.log((new Date()) + ' Peer ' + connection.remoteAddress + ' disconnected.');
 });
});
Subscribing To Messages From Redis In Node.js

Receiving messages from Redis in node.js is also straightforward.

Firstly you need to subscribe to a particular channel and then handle the message received event. The code below shows a simple example of how this works:

var redis = require("redis")

// create client on required port
var REDIS_URL = 'localhost';
var REDIS_PORT = 6379;
var redisClient = redis.createClient( REDIS_PORT, REDIS_URL );
client.on("message", function (channel, message) {

// message received - output to console window
 console.log("client channel => " + channel + "; message => " + message + ";");

}
});
// subscribe to receive messages from a particular channel
client.subscribe("application_channel");

If you need to handle messages based upon channel you will need to check this in the message event.

Conclusions

Using Redis as a message broker can help remove tight coupling in systems by removing direct communications between components. Using node.js enables a client to pass a message directly to a web socket server and then to distribute this via Redis to any number of subscribed middleware components, enabling scalable systems to be created.

The next step is to look into receiving messages from Redis in my .NET middleware components so this will be the focus of my next blog.

In the meantime – happy coding!

Advertisements

Creating an HTML Web Socket Client With JQuery.WebSocket

1 Comment

In my previous post Setting Up A Web Socket Service With Node.js on Windows I showed how a web socket server can quickly and easily be created using node.js. In the post was included a sample node.js javascript driver file which created an echo server, simply returning any received messages back to the client.

My aim in this post is to show how you can connect to the web socket server from HTML to create a basic echo application. This will prove that we can fully implement two-way communication between server and client using node.js and web sockets in the Google Chrome browser. Client side web sockets interfacing with a web socket server enable true duplex communication over the web, facilitating the creation of complex and responsive LOB applications.

This post will show just how easy it is to create a simple web socket client in HTML and how to send and receive messages.

Pre-requisites

In order to get started with the information in this post you will need the following:

  1. A running node.js instance, with the WebSocket-Node plugin, using the echo driver file from my previous post – Setting Up A Web Socket Service With Node.js on Windows
  2. JQuery, JQuery.WebSocket and JQuery.JSON javascript files
  3. A web socket compatible browser (i.e. Google Chrome 16)
Issues With JQuery.WebSocket

JQuery.WebSocket is a JQuery based helper file that simplifies the creation of WebSockets in Javascript. The code is elegant and very useful but unfortunately the latest published version has some bugs for which I could not find fixes on the official website.

As a result I fixed the bugs during my investigations and also added a web socket protocol parameter to the constructor (more on this later). Below is the fixed code – simply  copy and paste over the code in the jquery.websocket-0.0.1.js file.

/*
* jQuery Web Sockets Plugin v0.0.1
* http://code.google.com/p/jquery-websocket/
*
* This document is licensed as free software under the terms of the
* MIT License: http://www.opensource.org/licenses/mit-license.php
*
* Copyright (c) 2010 by shootaroo (Shotaro Tsubouchi).
*/
(function ($) {
    $.extend({
        websocketSettings: {
            open: function () { },
            close: function () { },
            message: function () { },
            options: {},
            events: {}
        },
        websocket: function (url, protocol, s) {
            var ws = WebSocket ? new WebSocket(url, protocol) : {
                send: function (m) { return false },
                close: function () { }
            };
            ws._settings = $.extend($.websocketSettings, s);
            $(ws)
			.bind('open', $.websocketSettings.open)
			.bind('close', $.websocketSettings.close)
			.bind('message', $.websocketSettings.message)
			.bind('message', function (e) {
			    var m = $.evalJSON(e.originalEvent.data);
			    var h = $.websocketSettings.events[m.command];
			    if (h) h.call(this, m);
			});
            ws._send = ws.send;
            ws.send = function (type, data) {
                var m = { command: type };
                m = $.extend(true, m, $.extend(true, {}, $.websocketSettings.options, m));
                if (data) m['data'] = data;
                try {
                    this._send($.toJSON(m));
                }
                catch (ex) {
                    alert(ex);
                    return false;
                }
                return true;
            }
            $(window).unload(function () { ws.close(); ws = null });
            return ws;
        }
    });
})(jQuery);
Web Socket HTML Client Code
The HTML for the web socket client is straightforward.
I have pasted my example HTML file below. You will need to ensure that you change any Javascript references at the top of the page (the script tags) to match the locations of your script files.
<html lang="en">
<head>
    <title>Node Based Echo</title>
    <script src="/Scripts/jquery-1.6.2.min.js" type="text/javascript"></script>
    <script src="/Scripts/jquery.websocket-0.0.1.js" type="text/javascript"></script>
    <script src="/Scripts/jquery.json-2.3.js" type="text/javascript"></script>
</head>
<body>
    <script type="text/javascript">

        // sends a message to the websocket server
        function sendToServer() {

            ws.send('krakenmsgA', '{ messageTextA: ' + $('#echoText').val() + ' }');
            ws.send('krakenmsgB', '{ messageTextB: ' + $('#echoText').val() + ' }');
        }

        // set-up web socket
        var ws = $.websocket("ws://localhost:8080/", "kraken-protocol", {
            open: function () { },
            close: function () { alert('websocket has been closed'); },
            events: {
                krakenmsgA: function (e) { $('#returnText').append(e.data + "<br/>"); },
                krakenmsgB: function (e) { $('#returnText').append(e.data + "<br/>"); }
            }
        });

    </script>
    <div>
        <div style="float: left; clear: left; padding-top: 2px;">
            Your text:
        </div>
        <div style="float: left; padding-left: 20px;">
            <input type="text" id="echoText" style="width: 150px;" required />
        </div>
        <div style="clear: left;">
            <input type="button" onclick="javascript:sendToServer();" value="Send" />
        </div>
        <div id="returnText" style="clear: left; height: 200px; padding-top: 30px;">
        </div>
    </div>
</body>
</html>

As you can see the code is very straightforward. The HTML renders a very simple form with a text box and button. When the button is clicked a Javascript call is made to send a message to the web socket server.

Opening a web socket connection is remarkably easy – this simple line of code manages the process:

var ws = $.websocket("ws://localhost:8080/", "kraken-protocol", { ... });

One important thing to note is the second parameter passed to the constructor which is one of the extensions I made to the JQuery.WebSocket script. The WebSocket-Node node.js plugin used to host the web socket server requires a protocol contract to exist between the server and client. If you view my previous post Setting Up A Web Socket Service With Node.js on Windows, the node.js script file creates a server web socket with the protocol as ‘kraken-protocol’. The client must also specify the same protocol otherwise the server will not allow the client to connect. It is unimportant what the actual protocol is – the important thing is that the server and client match. Obviously the port on which the server is running must be specified in the first parameter to the constructor as part of the URL of the web socket server.

Sending a message to the web socket server is also very straightforward in our example:

ws.send('krakenmsgA', '{ messageTextA: ' + $('#echoText').val() + ' }');

We are using JQuery in this example to retrieve the value specified by the user in the text box, which has the id echoText.

Behind the scenes the JQuery.WebSocket script sends a message to the web socket server in a JSON wrapper, with a command property set to the value of the first parameter of the send method. In this case the command value would be set to ‘krakenmsgA’.

The nice thing about JQuery.WebSocket is that it binds events that correspond to the type of message that is sent, simplifying the message handling process. For example, the line

ws.send('krakenmsgA', '{ messageTextA: ' + $('#echoText').val() + ' }');

sends a message to the web socket server of type ‘messageTextA’. The return message event corresponding to this type can then be handled by simply creating an event handler, as shown in the following code.

var ws = $.websocket("ws://localhost:8080/", "kraken-protocol", {
            open: function () { },
            close: function () { alert('websocket has been closed'); },
            events: {
                krakenmsgA: function (e) { $('#returnText').append(e.data + "&lt;br/&gt;"); },
                krakenmsgB: function (e) { $('#returnText').append(e.data + "&lt;br/&gt;"); }
            }
        });

This code both opens the connection and handles the required events. To handle a new message type, simply add a new handler to the websocket declaration.

Conclusions

Connecting to a web service from HTML and creating a simple echo application is remarkably trivial. My experiences of network programming with raw sockets in the .NET world led me expect that the process would prove problematic but this appears not to be the case. The web socket implementation in Google Chrome nicely handles the opening and closing of web sockets, including when communication with the server is lost, and looks resilient during testing. Open and close events are raised by the web socket, enabling clean handling of connection issues.

The use of JQuery.WebSocket makes life easier and provides a well structured event interface implementation, although you will need to use the amended script from this post if you wish to use these examples.

I can see many potential pitfalls, however, that will lead to such an implementation being problematic in a large LOB application. Code structure and abstraction are the obvious issues and I’m not yet sure of the best way to structure a large application based on HTML 5 and Javascript. This will definitely be a major consideration as I move further into the development of Kraken Office and I’m sure will be dealt with in future posts.

I am expecting Javascript structuring and code management to be one of the trickiest elements of the project, with Javascript acting as our controller between the UI elements and middleware. Without careful planning and code organisation the client side of the project will quickly become unmaintainable.

These are issues for another day, however.

For my next challenge I will be looking into the Redis implementation and how I can use the message broker functionality to loosely couple communication between system components.

Till next time – happy coding!

Setting Up A Web Socket Service With Node.js on Windows

2 Comments

In my previous post on selecting techologies for the Kraken Office project I explained that I have decided to use the node.js framework to create the required web socket server infrastructure.

There are a couple of existing web tutorials aimed at setting up web sockets with node.js but I could find none that took you through the process from the standpoint of no existing experience with the node.js framework.

Node.js is a lightweight framework used to create services. It is single threaded and can be easily scaled. Node.js scripts are written in Javascript and there are a number of plug-ins available.

Installing Node.js

To run node.js on Windows you should get the installer from here – node.js Windows installer. I am working with version 0.6.9 – installation of node.js is a simple one click process. Once node.js is installed you will need to get the WebSocket-Nodeplug-in to enable support for web sockets. There are other plug-ins available to enable the creation of web sockets but I have found WebSocket-Node stable and easy to use.

Using NPM to install WebSocket-Node

NPM is a package manager for node.js. It enables you to publish and manage node.js installations and plug-ins. You do not need to separately download NPM code as it is already included in the node.js installation.

Using NPM you can download WebSocket-Node and install it in one easy action. There is currently an issue (at the time of writing) with the latest version of WebSocket-Node on Windows – see this issue for further details. A previous version (v1.0.3) installs and works with no problems so I have used this to set-up node.js and WebSocket-Node on my local server.

In order to use NPM to install WebSocket-Node simply follow these steps:

  1. Open a command prompt window
  2. Navigate to the root of your node.js installation folder using the CD command – this will probably be in your Program Files directory
  3. Execute npm install websocket@1.0.3 in the command prompt window
  4. WebSocket-Node will be download from the npm repository and integrated into your existing node.js installation
Getting Started With Node.js and WebSocket-Node
Once you have successfully installed node.js and the WebSocket-Node extension you can now very easily create a web socket server.
The following is an example of how to create a simple echo web socket server taken directly from the WebSocket-Node website:
var WebSocketServer = require('websocket').server;
var http = require('http');

var server = http.createServer(function(request, response) {
    console.log((new Date()) + ' Received request for ' + request.url);
    response.writeHead(404);
    response.end();
});
server.listen(8080, function() {
    console.log((new Date()) + ' Server is listening on port 8080');
});

wsServer = new WebSocketServer({
    httpServer: server,
    // You should not use autoAcceptConnections for production
    // applications, as it defeats all standard cross-origin protection
    // facilities built into the protocol and the browser.  You should
    // *always* verify the connection's origin and decide whether or not
    // to accept it.
    autoAcceptConnections: false
});

function originIsAllowed(origin) {
  // put logic here to detect whether the specified origin is allowed.
  return true;
}

wsServer.on('request', function(request) {
    if (!originIsAllowed(request.origin)) {
      // Make sure we only accept requests from an allowed origin
      request.reject();
      console.log((new Date()) + ' Connection from origin ' + request.origin + ' rejected.');
      return;
    }

    var connection = request.accept('kraken-protocol', request.origin);
    console.log((new Date()) + ' Connection accepted.');
    connection.on('message', function(message) {
        if (message.type === 'utf8') {
            console.log('Received Message: ' + message.utf8Data);
            connection.sendUTF(message.utf8Data);
        }
        else if (message.type === 'binary') {
            console.log('Received Binary Message of ' + message.binaryData.length + ' bytes');
            connection.sendBytes(message.binaryData);
        }
    });
    connection.on('close', function(reasonCode, description) {
        console.log((new Date()) + ' Peer ' + connection.remoteAddress + ' disconnected.');
    });
});

To start / stop the web socket server simply follow these steps:

  1. Paste the above example into a file called WebSocketV1.js
  2. Save the file into the root of your node.js installation (the simplest way to get started)
  3. Open a command window and navigate to your node.js installation direction using the CD command
  4. Execute node WebSocketV1 in the command prompt window
  5. The web socket server will be started on the specified port
  6. To stop the server simply close the command window

I hope this example shows just how easy it is to get a web socket server up and running using node.js and the WebSocket-Node extension.

Obviously this post only scratches the surface of node.js but hopefully gives you an overview of how to get your first node.js server up and running.There are a multitude of other resources available on the web, not least the node.js documentation, to help you move forward.

Next Steps

In the context of the Kraken Office project I am presently getting all of the components outlined in my previous architecture post up and running so that I can prove the architecture design is feasible and the necessary components are fit for purpose. Node.js appears to be easy to configure and set-up and I am happy to continue to use it for my web socket server requirements.

Further posts will deal with using node.js to create the web socket server to be used in the Kraken Office project and will cover integration with the Redis server’s message broker, which is the link between the front end web socket server and the middleware components. See the architecture diagram in the architecture post for full details.

The next step is to create a simple HTML 5 application in ASP.NET MVC 3 to get JQuery and the associated web socket libraries configured, set-up and interacting with the node.js web socket server. Once this is done I will have completed the initial proof of concept for the technologies selected for the front end of the system and can move on to the messaging architecture.

Selecting Technologies For Kraken Office

3 Comments

My last, long, post was concerned with the overall architecture of the Kraken Office project. I am now at a point where I am settled enough about the initial architecture of a duplex, message based web application that I can start selecting technologies for the project.

Most of my career has involved developing software for the Microsoft platform ( ASP, VB6, VB Script, ASP.NET, C#, SQL, Silverlight etc.) so I am naturally going to approach the problem with .NET as the technology for the development of the majority of the middleware, That is not to say that I won’t consider alternative technologies but this is to be my starting point.

Web Application Technologies

The client facing user interface will be a web application. This is to maximise the audience for Kraken Office.

Having recent experience with Silverlight and the MVVM methodology, this would be a logical choice. I am becoming increasing concerned, however, about Silverlight’s usage as a internet facing technology due to its reliance on a plug-in. The fact that there is no plug-in support for most mobile devices is especially disconcerting. Another issue is Microsoft’s lack of vocal support for the technology which is currently worrying the entire Microsoft development community (see Microsoft developers horrified).

I want Kraken Office to appeal to as large an audience as possible and I don’t think this goal would be advanced by setting system pre-requisites or by excluding the mobile market.

The obvious alternative for creating a web facing interface with rich LOB features is HTML 5. While this is a relatively new technology and not yet universally supported across all browsers, the development community has generally welcomed it and take up has been good. Microsoft appear to have invested heavily in its usage in Windows 8, although the full details are yet to emerge.

I have therefore decided upon HTML 5 as the main web technology for the application, as it currently appears to be the only viable alternative to Silverlight (Flash also appears to be going the way of Silverlight) for Kraken Office, which I am planning to develop into a fairly complex LOB application. In many ways this is a shame as the toolset for Silverlight and MVVM is impressive, easy to use and easy to debug. Moving back to HTML with Javascript, albeit with comprehensive Javascript libraries these days, does seem a bit daunting.

It is important to remember, however, that the end product of the development process is a useful product with the biggest audience possible. If HTML 5 is the best way of achieving this goal then I feel this is the approach to take.

While HTML 5 may be the technology I also need to decide on a development environment. For me this is pretty easy decision – I will use the Microsoft MVC 3 platform for the website framework and will work within Visual Studio with which I have a lot of experience.

I’m going to need to get more familiar with the available Javascript libraries, with the starting point being the ubiquitous JQuery. It has a small footprint and a large plug in community.

The last main issue is communication with the application server. I have used duplex communication via Duplex WCF services, but these utilise a long poll mechanism rather than a real push mechanism. Web sockets are the new HTML 5 technology for duplex communication and offer support for real duplex communication between server and client. Web sockets currently have limited browser support but are currently being implemented in all major new browser releases. For further information on web sockets refer to this interesting comparison of the pros and cons of long polling vs web sockets

Unfortunately web sockets are not yet supported in Microsoft’s WCF platform, although support is on the way at some stage (WCF WebSocket preview).

In view of the lack of WCF support my first thought was to write a web socket server in C#. I did some initial prototyping and developed a working web sockets server but found the solution verbose and complex. I also have reservations about network based coding in general as I have always found it one of the most difficult types of component to get stable and reliable. In addition to this I want to re-use as many existing frameworks and toolkits as possible, especially when it comes to project infrastructure, so that I can concentrate on developing business functionality.

I came across a new project called XSockets which has only just been released at the time of writing and which looks like it could offer WebSockets hosted in .NET. It was a brand new release, however, with very little documentation so has been discounted for now.

The other framework that caught my eye was the node.js project. After reading this great node.js websocket introduction I was immediately intrigued. Node.js is a javascript based framework for creating scalable, distributable network servers. Although it does not directly support web sockets, a plugin, WebSocket-Node, is available which appears to render the creation of web socket servers a relatively trivial task. Allied with excellent scalability and a dedicated community I believe node.js is an ideal framework for hosting my web socket servers and will relieve me of the task of creating my own server application.

Client-side web sockets will be handled using Javascript. JQuery does not currently support easy scripting of web sockets but there is a JQuery plugin, jquery-websocket, which is designed specifically for client-side interaction with web socket servers.

In conclusion the following sums up my initial web platform technology choices:

  1. HTML 5 developed in Visual Studio with MVC 3
  2. Javascript scripting with JQuery
  3. Node.js hosted web socket server using WebSocket-Node plugin
  4. Client-side javascript web socket interaction using the jquery-websocket JQuery extension
Middleware / Data Storage Technologies

My initial system designs for Kraken Office, which can be found at Designing A Scalable, Resilient System, assumed that I would be creating a .NET based web sockets server. With my decision to implement the web socket server with node.js, this will now not be the case. What is clear, however, is that the node.js server will be the bridge between the client web application and the middleware. In order to ensure that the system remains scalable any communication between node.js and the middleware must use a message broker of some description.

While investigating message brokers, Enterprise Service Buses (ESBs) and in-memory data stores I came across a range of options. ESBs seem too heavy-weight for the system in mind, although at a later date such power may be required. Currently, however, I don’t want to spend all of my precious time resources on trying to configure a system that is not adding value to the project.

One of the most recommended, and fast, in-memory data caches is Redis. A Windows port of the project is available here. In addition to being immensely fast, it provides support for publisher / subscriber queues. Redis is open source and would enable me to kill two birds with one stone; to provide both in-memory data storage and message broker requirements in one system. This would minimise the amount of new technology I need to learn and also the amount of software installation and configuration.

Redis clients exist for both node.js (Redback) and C# (ServiceStack).

Using Redis and node.js will necessitate a change to the architecture of the system. The diagram below documents v2 of the architecture.

The system now uses a scalable node.js layer as the initial application layer. This layer will manage web socket instances for the client web applications. The node.js layer will relay messages to the rest of the system via the Redis messaging layer. A .NET application layer will receive messages from the node.js layer and will decode and act upon the messages. If a message requires data interaction then the application layer will again submit this to the Redis messaging layer, in a different message channel. The data layer (another .NET based layer) will subscribe to data channel messages in Redis and will action any relevant messages. Data will be returned from the data layer via the Redis messaging layer back to the application layer and then onto the node.js layer.

This should prove to be a scalable and resilient system.

In conclusion the following sums up my initial middleware technology choices:

  1. .NET middleware layers (application and data) written in Visual Studio using C#
  2. Messaging provided by Redis and accessed by node.js and .NET clients using Redback and ServiceStack respectively
  3. Fast in-memory data caching using Redis
Data Persistence

While data cached in Redis can be persisted to disk using a variety of configurable persistence strategies, I do not consider it suitable for a permanent data storage solution for two main reasons:

  1. Persisting data on a regular basis to disk, which would be necessary for data integrity, will degrade the performance of Redis
  2. The Kraken Office system will need to archive certain types of data but retain the ability to retrieve the data on demand, for which Redis would be unsuitable

As a result I am going to stick with my decision to persist data to a RDBMS for permanent storage, in addition to retaining data in Redis for real-time application usage. As I am most familiar with Microsoft SQL Server I will stick with this as an initial database, although I will investigate MySQL in the near future due to its Open Source status.