[Research] Websocket and Processing
Last edit: 19-Jan-2015
Some time ago I participated in a user study where I was given a mobile phone and carried out some tasks by moving it around. I was quite impressed by the study itself, but more importantly, the technology behind. Turns out the phone was just running a browser, which sent periodic updates of the phone's motion to a computer running Unity.
The great thing about this is no 3rd-party app is required. And to make this work two components are involved: 1) a mobile browser that captures the device's motion, and 2) a transmission channel called Websocket.
Mobile devices nowadays are pretty impressive. They have all kinds of sensors measuring all kinds of environmental variables. Some of the most common ones are motion sensors (angle, acceleration) and touch sensors (screen-touch). Equally impressive is the fact that many of the mobile browsers (e.g., chrome, safari, opera) can read those data. This by itself allows great interactivity to be provided within a webpage when accessed using a mobile device.
Now imagine this data can be sent wirelessly to a computer, by using Websocket.
Without going into the details about this Websocket thing, I'd just say it's a communication protocol similar to http (the way your browser requests a webpage from a web server), and can be initiated within a webpage. The browser in the mobile device first sends a special request to a computer, then the computer will respond, and a two-way communication channel is then established (both sides can send messages to each other).
The tricky part, however, is to make the computer understand the special request, respond, and carry out the rest of the communication. There are in fact quite a few plug-in's available, but they typically depend on other software, for example, an Apache server, or a Unity project. Having been using Processing for sometime, I decided to write my own code to do just that. What I wanted to achieve was I just need to run my Processing sketch and it will handle all the communication with a mobile device.
Turns out it's a matter of bit-shifting and some encrypting-decrypting (Websocket calls it "unmasking"), and since Processing supports native Java, I could use some Java libraries to do just that. The result can be seen in the embedded video below:
Some time ago I participated in a user study where I was given a mobile phone and carried out some tasks by moving it around. I was quite impressed by the study itself, but more importantly, the technology behind. Turns out the phone was just running a browser, which sent periodic updates of the phone's motion to a computer running Unity.
The great thing about this is no 3rd-party app is required. And to make this work two components are involved: 1) a mobile browser that captures the device's motion, and 2) a transmission channel called Websocket.
Mobile devices nowadays are pretty impressive. They have all kinds of sensors measuring all kinds of environmental variables. Some of the most common ones are motion sensors (angle, acceleration) and touch sensors (screen-touch). Equally impressive is the fact that many of the mobile browsers (e.g., chrome, safari, opera) can read those data. This by itself allows great interactivity to be provided within a webpage when accessed using a mobile device.
Now imagine this data can be sent wirelessly to a computer, by using Websocket.
Without going into the details about this Websocket thing, I'd just say it's a communication protocol similar to http (the way your browser requests a webpage from a web server), and can be initiated within a webpage. The browser in the mobile device first sends a special request to a computer, then the computer will respond, and a two-way communication channel is then established (both sides can send messages to each other).
The tricky part, however, is to make the computer understand the special request, respond, and carry out the rest of the communication. There are in fact quite a few plug-in's available, but they typically depend on other software, for example, an Apache server, or a Unity project. Having been using Processing for sometime, I decided to write my own code to do just that. What I wanted to achieve was I just need to run my Processing sketch and it will handle all the communication with a mobile device.
Turns out it's a matter of bit-shifting and some encrypting-decrypting (Websocket calls it "unmasking"), and since Processing supports native Java, I could use some Java libraries to do just that. The result can be seen in the embedded video below:
Hurdles
The "bit-shifting and some encrpyting-decrypting" might sound straight-forward, but it's not. There is indeed a whole RFC describing the Websocket protocol, including how the bits in the packet are used and how the masking-unmasking is done. But it's up to the programmer to implement it. There are surprisingly very few resources explaining how it is done, and most of the time they just repeat the RFC, and throw you some code that "should work" (some didn't). So I had to read through all those (you can tell by my Google search results -- all visited at some point) and do some reconstruction, so to speak.
I think the most annoying part, however, is the occasional dropping of the packets. There are times when the header of the packet claims it has a certain size of data, but turns out there is none. I have to actually check the size of the buffer and return immediately if it's zero. I don't think it's caused by some programming bug as the same code successfully receives data at other times.
The "bit-shifting and some encrpyting-decrypting" might sound straight-forward, but it's not. There is indeed a whole RFC describing the Websocket protocol, including how the bits in the packet are used and how the masking-unmasking is done. But it's up to the programmer to implement it. There are surprisingly very few resources explaining how it is done, and most of the time they just repeat the RFC, and throw you some code that "should work" (some didn't). So I had to read through all those (you can tell by my Google search results -- all visited at some point) and do some reconstruction, so to speak.
I think the most annoying part, however, is the occasional dropping of the packets. There are times when the header of the packet claims it has a certain size of data, but turns out there is none. I have to actually check the size of the buffer and return immediately if it's zero. I don't think it's caused by some programming bug as the same code successfully receives data at other times.
How to use
As seen in the video, the outcome is that a mobile device can connect wirelessly to a Processing sketch, and send orientation values to it afterwards (the rotating block is to imitate the rotation of the device). Other values (e.g. tap) can be sent as well by piggy-packing to the message sent, and the sketch can also send data back (e.g. change the background color of the device). Though care should be taken so delay and possible packet loss can be accounted for (I have a feeling that the complete packet will eventually arrive because of the supposedly reliable TCP underlying transmission, but just not always timely).
I've made the source code a Processing library so you can use it just like any other Processing libraries. For more details please check out this [Link]. Just like many Processing code this will work in both Windows and OSX (and possibly Linux). In most cases it will just work (can't guarantee it's error free... so use it at your own risk, however minimal it is) . However, if there is a network policy that two wireless devices cannot form a Server-Client relationship (happens to be my school's policy), you'll have to connect the computer to the internet through a wire. Of course, your computer should also be set to receive connection from the mobile device.
With the help of a colleague we eventually made a multiplayer version of it and implemented a very simple Brick-out game. We sort of shifted the focus so it became a mechanism towards overcoming barriers of using large interactive displays for more "research value" (afterall we didn't invent anything new technologically). Yet we frequently use this as a demo as almost all visitors can just take out their phone and try it out. Frankly, there are more stable plug-ins for other platforms so I might just use one of those (e.g. Unity3D). But I really enjoy the fact that I can do everything with Processing.
As seen in the video, the outcome is that a mobile device can connect wirelessly to a Processing sketch, and send orientation values to it afterwards (the rotating block is to imitate the rotation of the device). Other values (e.g. tap) can be sent as well by piggy-packing to the message sent, and the sketch can also send data back (e.g. change the background color of the device). Though care should be taken so delay and possible packet loss can be accounted for (I have a feeling that the complete packet will eventually arrive because of the supposedly reliable TCP underlying transmission, but just not always timely).
I've made the source code a Processing library so you can use it just like any other Processing libraries. For more details please check out this [Link]. Just like many Processing code this will work in both Windows and OSX (and possibly Linux). In most cases it will just work (can't guarantee it's error free... so use it at your own risk, however minimal it is) . However, if there is a network policy that two wireless devices cannot form a Server-Client relationship (happens to be my school's policy), you'll have to connect the computer to the internet through a wire. Of course, your computer should also be set to receive connection from the mobile device.
With the help of a colleague we eventually made a multiplayer version of it and implemented a very simple Brick-out game. We sort of shifted the focus so it became a mechanism towards overcoming barriers of using large interactive displays for more "research value" (afterall we didn't invent anything new technologically). Yet we frequently use this as a demo as almost all visitors can just take out their phone and try it out. Frankly, there are more stable plug-ins for other platforms so I might just use one of those (e.g. Unity3D). But I really enjoy the fact that I can do everything with Processing.
Related publication:
[1] Cheung, V., Watson, D., Vermeulen, J., Hancock, M., Scott, S.D. (2014). Overcoming Interaction Barriers in Large Public Displays Using Personal Devices. Extended Abstracts of ITS 2014: ACM Conference on Interactive Tabletops and Surfaces, November 16-19, 2014, Dresden, Germany.
[1] Cheung, V., Watson, D., Vermeulen, J., Hancock, M., Scott, S.D. (2014). Overcoming Interaction Barriers in Large Public Displays Using Personal Devices. Extended Abstracts of ITS 2014: ACM Conference on Interactive Tabletops and Surfaces, November 16-19, 2014, Dresden, Germany.
