A multi-participant Paint in HTML5

Installing NodeJS

Go to http://www.nodejs.org and follow the installation procedure.

If you are running Linux, look carefuly at nodeJS home page, there is a link to precompiled builds :

Update your PATH variable so that the node executable is found when you type "node --version" in your favorite terminal/dos command window (note : C:\Program Files(x86)\nodejs by default on windows).

Test now NodeJS using a small application

• Create a test.js file with this content :

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(8124, "127.0.0.1");
console.log('Server running at http://127.0.0.1:8124/');

Save it somewhere. Then run from the command line :

node test.js

You just ran a nodeJS server that runs an application at this URL : http://localhost:8124

Open it using your web browser !

You should get :

This was our first NodeJS application ;-)

Adding node modules for WebSockets

There are numerous ways to add modules to NodeJS. If you are using Linux, it is pretty straightforward, normally it should go fine if you have a Mac, and if you are running Windows it may be a bit more complicated for some modules like NowJS that require VC++ redistribuable to be installed.

Most of the time we install modules using the npm command (Node Package Manager). And most of the time we run this command in the directory of our projet. This will create a node_modules subdir in the project.

The chat application we will run now comes in this archive : ChatNowJs.zip (1.7 Mbytes)

Note that the archive contains already a node_module directory, but it will not suffice as the installation of modules may also compile native parts of some modules.

Unzip the archive somewhere, go in the ChatNowJS directory and run the following commands :

• If on windows : install VC++ runtime (3Mo) before going any further !
   
• Then cd in the project dir, and run the following commands :
  
  1. npm install socket.io
  2. npm install node-proxy
  3. npm install now
  4. npm install express

Run the simple chat application

Run the following command : "node simpleChat_server.js", then open http://localhost:8080 in your browser. Open two tabs at the same URL, you will see an input field. Type in something, and it should be sent to the other tab (as both are clients connected to the same WebSocket server).

In case you have no errors when you launch"node  simpleChat_server.js", and the chat is not working, open the JavaScript debugger console. If you get a  "now.distributeMessage no such object/function", then your browser is "too old but not so much", it supports WebSockets, but not the "good ones" ! Indeed some Firefox version from one year/one year and a half had an implementation for an older versions of the protocol. Update or change your browser. If it still does not work even with a recent browser (problem only encountered under windows and exotic antivirus/anti spywares), you may tell socket.io (used by nowJS) not to use HTML5 WebSockets but another  mean of transmission instead: modify one line in the simpleChat_server.js code :

• Put var everyone = nowjs.initialize(server, {socketio: {transports: ['xhr-polling', 'jsonp-polling', 'htmlfile']}}); instead of var everyone = nowjs.initialize(server);.

Study the simple chat application

Two files are involved in the simple chat application :

1. simpleChat_server.js
2. simpleChat.html

Open these files in your favorite editor : the application is ...wow, just a few lines of code... Let's talk about nowJS.... now !

NowJS defines a "magic pocket" called "now" : a JavaScript variable you can manipulate from client code running in your browser, that will hold variables and functions shared between the client and the server code.

The server code can use a "everyone.now" variable in order to define variables and functions shared with all clients.

For example, in the server side, defining a function distributeMessage in the everyone.now variable will make it callable by any client who will use it with his own "now" magic pocket.

Server :

everyone.now.distributeMessage = function(message){
  everyone.now.receiveMessage(this.now.name, message);
};

Example of call by a client (see the simpleChat.html file) :

$("#send-button").click(function(){
    now.distributeMessage($("#text-input").val());
    $("#text-input").val("");
});

So one clic on the send button executes now.distributeMessage() that will be run on the server, passing the message typed in the input field. The server gets the message in its everyone.now.distributeMessage function, that calls in turn everyone.now.receiveMessage, a function defined.... in the client ! The "everyone" parts means that it will call this function for all the connected clients !

Client :

now.receiveMessage = function(name, message){
    $("#messages").append("<br>" + name + ": " + message);
}

And this function, located on the client side, will just append to the chat log the message that just arrived !

The same way, a variable defined in the client code,

now.name = prompt("What's your name?", "");

May be used in the server code as a kind of "session" variable, using the this.now namespace :

Server :

everyone.now.distributeMessage = function(message){
  everyone.now.receiveMessage(this.now.name, message);
};

Using this simple paradigm, NowJS enables rapid development of collaborative logic over WebSockets. Notice that objects are send as is, as pure javascript objects, they are serialized/deserialized in JSON under the hood.

Adding the Chat application to our Paint Program

Go into the step 7 directory. Stop nodeJS if it is still running.

Look at the server.js file :

// Use the express module for nodeJS
// Express is a framework for implementing a more
// complete web server with nodeJS
var express = require('express');

var app = express.createServer();

// Indicate where static files are located
app.configure(function () {
    app.use(express.static(__dirname + '/client/'));
});
app.get('/', function (req, res) {
    //res.render('client/paint.html');
    res.redirect("/paint.html");
});
app.listen(80);

// Creation of a web socket server using nowJS
var nowjs = require("now");
var everyone = nowjs.initialize(app);


nowjs.on("connect", function () {
    console.log("Joined: " + this.user.clientId);
});

nowjs.on("disconnect", function () {
    console.log("Left: " + this.user.clientId);
});

// Function called by a chat client, passes the message to each client
// connected by calling its processIncomingChatMessage shared function
everyone.now.distributeMessage = function (message) {
    // send the message to everyone
    everyone.now.processIncomingChatMessage(this.now.name, message);
};

This server code is very similar to the one of the chat program, instead  this time we use the "express" NodeJS module to run the HTTP web server part (13 first lines). This enables to serve properly HTML pages that include CSS or JavaScript files. The rest is very similar to the simpleChat_server.js file.

Run the following command : "node server.js", and open "http://localhost" in your browser.

This time you should see the same paint program we had in step6, augmented by a chat application at the bottom of the user interface.

Open two tabs with the same URL, try to chat... Both the paint and the chat should work at the same time. Paint actions are not transferred however... we will add that in the next step !

You may look at the source code as well. We isolated the JavaScript code for the chat in a chat.js file while we added the input field and a div for handling log messages in the paint.html file.

Here are the additions in the paint.html file :

<head>
    ...
    <!-- NowJS is a High level lib built on top of socket.io. 
         Uses WebSockets if available. The page needs to
         be served by the NodeJS server -->
    <script src="js/chat.js"></script>
    <script src="/nowjs/now.js"></script>

    <script type="text/javascript">
        // Run when the DOM is ready
        $(document).ready(function () {
            // Create the pseudo object which will handle the main canvas
            paint = new PaintObject("canvasMain");
            // Bind events to the canvas
            paint.bindMultiplexEvents();

            // Init the chat system
            now.ready(function () {
                // We must be sure "now" exists. So we prompt the user only
                // when the shared variable "now" is ready !

                // prompts a dialog and asks for a nickname
                now.name = prompt("What's your name?", "");

                //Create the pseudo object which will handle the chat
                chat = new ChatObject();
            });
        });
    </script>
</head>

Notice that the paint.html file includes a now/now.js file that is served by NodeJS and that is not part of the paint project. It is located in the node_modules/now directory. This is why we had to install/configure the different modules for NodeJS.

We also create the chat object only when the nowJS is fully initialized. NowJS calls the now.ready() method once it is ready, so we create the chat object in the body of this method.

At the end of the paint.html file we added the HTML necessary for the chat :

<div id="chat">
    <div id="chatMessages"></div>
     <input type="text" id="chatTextInput">
     <button class="menuButton" type="button" id="chatSendButton">Send</button>
</div>

The code for handling a press on the send button (or the enter key in the input field), sending the message, and displaying received messages is in the chat.js file :

var chat;

function ChatObject() {
    // Handle message input for the chat
    // When the send button has been clicked... or when the enter key has been 
    // pressed -> send content to the chat
    // server
    $("#chatTextInput").keypress(function (e) {
        var code = (e.keyCode ? e.keyCode : e.which);
        if (code == 13) {
            sendChatMessage();
        }
    });

    $("#chatSendButton").click(function () {
        sendChatMessage();
    });

    function sendChatMessage() {
        if ($("#chatTextInput").val() != "") {
            // This function is defined on the JavaScript code that runs on the server
            now.distributeMessage($("#chatTextInput").val());
            $("#chatTextInput").val("");
        }
    }

    // This function is called when a chat message arrives. Called by the server !
    now.processIncomingChatMessage = function (username, message) {
        // appends the incoming message to the messageLogs
        $("#chatMessages").append("<p><strong>" + username + "</strong> : " + 
                                   message + "</p>");
        $("#chatMessages").get(0).scrollTop = $("#chatMessages").get(0).scrollHeight;
    }
};

Server side : broadcasting paint event to every client except the sender

This time, without touching anything from the chat code, we added one single method on the server side for broadcasting paintCommand objects that will hold a description of a paint event (like "drawing a line with this color, from this position to this position")... :

Server (see server.js file) :

everyone.now.distributePaintCommand = function (paintCommand) {
    // send the paint command to everyone except sender
    everyone.exclude(this.user.clientId).now.processPaintCommand(paintCommand);
};

This time we used another goodie from nowJS : the feature for excluding some clients when broadcasting. The everyone.exclude(this.user.clientId).now.processPaintCommand(paintCommand) part means "call the processPaintCommand of every client except myself !

Indeed, when one client draws a line, this line must be drawn on other clients' screens but not on on the sender's screen. This is the only difference on the server side, if we compare to the distributeMessage we used for broadcasting chat messages.

Sending paint events

In the client code, the sending of messages is done in each drawing tool. Let's have a look at one of them : the line tool (drawingtools.js file) :

// The Line Drawing Tool Object
setOfDrawingTools.line = function () {
    ...

    this.mousemove = function (event) {
        ...
        sendLineMove(mousePos);
    };

    // Send the line to other clients.
    function sendLineMove(mousePos) {
        var paintCommand = {};
        paintCommand.type = 'lineMove';
        paintCommand.previousMousePos = previousMousePos;
        paintCommand.currentMousePos = mousePos;
        paintCommand.properties = paint.getCurrentContextProperties();
        now.distributePaintCommand(paintCommand);
    }

    this.mouseup = function (event) {
        ...
        paint.sendOrderToDrawFrontCanvasOnMainCanvas();
    }
};

We added a sendLineMove method that is called at every mousemove event. This method builds a paintCommand object with a type attribute equal to 'lineMove', and that will hold the coordinates of the extremity of the line, and a field name 'properties' for holding the current stroke color, fill color, line width etc. These properties are returned by   paint.getCurrentContextProperties() method.

After this object is built, we send it to the server for broadcasting, by calling the now.distributeCommand(paintCommand) method, running on the server.

Processing incoming messages

First, we defined in the paint.html file a now.processIncomingMessage method. As it is defined as a method of the now object, it is callable by the server code.

<script type="text/javascript">
        // Run when the DOM is ready
        $(document).ready(function () {

            // prompts a dialog and asks for a nickname
            now.ready(function () {
                ...

                now.processPaintCommand = function (paintCommand) {
                    paint.processPaintCommand(paintCommand);
                };

               ...
            });
        });
    </script>

So, everytime the server calls processPaintCommand of the connected clients, this will in turn call paint.processPaintCommand(paintCommand), a method from the paint Object, located in the paint.js file :

this.processPaintCommand = function (paintCommand) {
    // save contexts on a stack, method provided by the canvas API
    mainContext.save();
    frontContext.save();

    // change current contexts so that they are same as sender contexts
    this.changeContextsProperties(paintCommand.properties);

    switch (paintCommand.type) {
        ...
        case 'lineMove' :
            // clear the content of the front canvas
            frontContext.clearRect(0, 0, frontCanvas.width, frontCanvas.height);

            // draw the line, using values received in the paintCommand...
            ...
            break;
            ...
        case 'drawFrontCanvasOnMainCanvas' :
            // Got the order to draw front canvas on main canvas
            this.drawFrontCanvasOnMainCanvas();
            break;
        }

        // restore contexts, current color, etc. From canvas API
        mainContext.restore();
        frontContext.restore();
}

This is very simple processing, we just had to take care of the contexts : when we get an incoming paint event, we need to :

1. save current contexts (i.e : draw in blue, wireframe, line width = 1)
2. switch to sender contexts (i.e : draw in red, fill shapes, fill color = green, line width = 5)
3. draw
4. restore contexts to previous values

We use the save() and restore() methods from the context object, from the canvas API. These methods save and restore from a stack so we can do multiple save/restores.

You may look at other drawing tools and see the way we processed the different type of paint commands.

Init the getUserMedia API

In the paint.html file, we have a call to a initWebCam() function :

<script src="js/webcam.js"></script>
...    
<script type="text/javascript">
        // Run when the DOM is ready
        $(document).ready(function () {
            ...
            // init web cam, works only in last test versions of chrome and opera.
            initWebCam();
        });
    </script>

The initWebCam function is located in the webcam.js file :

function initWebCam() {
    var n = navigator,
    is_webkit = false;

    function onSuccess(stream) {
        var output = document.getElementById('output'),
            source;

        output.autoplay = true;

        if (!is_webkit) {
            source = stream;
        }
        else {
            source = window.webkitURL.createObjectURL(stream);
        }

        output.src = source;
    }

    function onError() {
        // womp, womp :(
    }

    if (n.getUserMedia) {
        // opera users (hopefully everyone else at some point)
        n.getUserMedia({video:true, audio:true}, onSuccess, onError);
    }
    else if (n.webkitGetUserMedia) {
        // webkit users
        is_webkit = true;
        n.webkitGetUserMedia('video, audio', onSuccess, onError);
    }
}

As you may notice, we've got here several tests for checking if we are running opera or a webkit based browser. Indeed, the getUserMedia is still in beta implementation stages and API implementations may differ.

The principle is :

1. Make a call to navigator.getUserMedia(params, onSuccess, onError), this is done at the end of the code and we take care of Opera/Webkit differences.
2. in the onSuccess callback, notice that a stream object is passed as a parameter. This is the live video stream coming from the webcam. We will need to set the src attribute of a <video> element with this stream...
3. To do so : get a reference to a video element :
   /div[12]/div/ol/li[3]/pre, reference to undefined name 'document'
4. Set the src attribute :
   /div[12]/div/ol/li[4]/pre, line 1, column 12: EOF expected
5. Or if you are running a WebKit based browser :

output.src = window.webkitURL.createObjectURL(stream);

and you are done !

Now, the live video stream should be displayed in your application (at bottom right of the GUI, in our paint program)

Painting with the live images

Like we did earlier for other tools, we added a 'webcam' drawing tool in the drawintools.js file :

setOfDrawingTools.webcam = function() {
    var mousePos, previousMousePos, x, y;
    // ref to the video element that displays webcam real time content
    var video =document.getElementById('output');
    ...

    this.mousedown = function (event) {
        previousMousePos = getMousePos(paint.getFrontCanvas(), event);
        paint.started = true;
    }

    this.mousemove = function (event) {
        mousePos = getMousePos(paint.getFrontCanvas(), event);
        // Draw only if we clicked somewhere
        if (paint.started) {
            // clear the content of the front canvas
            paint.getFrontContext().clearRect(0, 0, paint.getFrontCanvas().width, 
                                              paint.getFrontCanvas().height);

            // Size and pos of the elastic rectangle with video snapshot inside
            var imageProperties = computeProperties(previousMousePos, mousePos);

            // Draw video content on front canvas
            paint.getFrontContext().drawImage(video,imageProperties.x,imageProperties.y, 
                  imageProperties.width,imageProperties.height);
        }
    }

    // Compute the coordinates of the top left corner and the size of the image drawn.
	function computeProperties(previousMousePos, mousePos){
	    var properties = {};
        properties.x = Math.min(previousMousePos.x, mousePos.x);
        properties.y = Math.min(previousMousePos.y, mousePos.y);
        properties.width = Math.abs(previousMousePos.x - mousePos.x);
        properties.height = Math.abs(previousMousePos.y - mousePos.y);
        return properties;
    }

    ...

    this.mouseup = function (event) {
        paint.started = false;
        paint.drawFrontCanvasOnMainCanvas();
        ...
    }
};

We just draw the live video stream current displayed image with a call to the drawImage method of a context object. In that cas, we draw elastic rectangles filled with a live image using the frontContext object.

Sending live images as base64 pictures

The trick here is to draw the current video frame into an offscreenCanvas of the size of the rectangle we are drawing, then get its content as a base64 image, broadcast the image with position, size, etc. to other clients...

We use the offScreenCanvas.toDataURL("image/png") method in order to get the canvas content as a png imaghe encoded in base64. WebSockets are HTTP based so we can only send text ! And base64 is a text String!

So... we added several lines in our webcam drawing tool in order to grab and send the live pictures :

setOfDrawingTools.webcam = function() {
    var mousePos, previousMousePos, x, y;
    // ref to the video element that displays webcam real time content
    var video =document.getElementById('output');

    // An off screen canvas for getting webcam data
    var offScreenCanvas= document.createElement('canvas');
    var offScreenContext = offScreenCanvas.getContext('2d');

    this.mousedown = function (event) {
        previousMousePos = getMousePos(paint.getFrontCanvas(), event);
        paint.started = true;
    }

    this.mousemove = function (event) {
        mousePos = getMousePos(paint.getFrontCanvas(), event);
        // Draw only if we clicked somewhere
        if (paint.started) {
            ...

            // Size and pos of the elastic rectangle with video snapshot inside
            var imageProperties = computeProperties(previousMousePos, mousePos);

            // Draw video content on front canvas
            paint.getFrontContext().drawImage(video,imageProperties.x,imageProperties.y, 
                                     imageProperties.width,imageProperties.height);

            // draw in the offscreen canvas a snapshot of current picture 
            // displayed in the video element
            offScreenCanvas.width = imageProperties.width;
            offScreenCanvas.height = imageProperties.height;
            offScreenContext.drawImage(video, 0,0, imageProperties.width,
                                                   imageProperties.height);
            // Get this snapshot as a base64 picture
            var imageData = offScreenCanvas.toDataURL("image/png");

           // send image data through websocket
           sendWebcamMove(imageData,imageProperties);
        }
    }

   ...

    // Send the rectangle to other clients.
    function sendWebcamMove(imageData,properties) {
         var paintCommand = {};
        paintCommand.type = 'webcamMove';
        paintCommand.imageData = imageData;
        paintCommand.x = properties.x;
        paintCommand.y = properties.y;
        paintCommand.width = properties.width;
        paintCommand.height = properties.height;
        now.distributePaintCommand(paintCommand);
    }

    this.mouseup = function (event) {
        ...
        paint.sendOrderToDrawFrontCanvasOnMainCanvas();
    }
};

Processing images received as base64 data objects

We added some lines in the paint.js file, where we process incoming paint events. Notice the way we build an image from the received data :

this.processPaintCommand = function (paintCommand) {
       ...

        switch (paintCommand.type) {
            
            case 'webcamMove' :
                // clear the content of the front canvas
                frontContext.clearRect(0, 0, frontCanvas.width, frontCanvas.height);
                // build a temporary image of the right size
                var img = new Image();
                img.width = paintCommand.width;
                img.height = paintCommand.height;

                // Listener called when the image is ready to be drawn
                img.onload = function () {
                    // draw the received picture
                    frontContext.drawImage(img, paintCommand.x, paintCommand.y);
                };

                // will call the onload listener
                img.src = paintCommand.imageData;
                break;
            ...
        }

        ...
};