Posts

MQTT vs Websockets vs HTTP/2: The Best IoT Messaging Protocol?

While doing any sort of development for an Internet of Things (IoT) connected device such as an Arduino, Raspberry Pi, or other embedded platform, the question inevitably comes up as to what is the best messaging protocol to use? Assuming your application can’t use straight up web pages, there are really two viable options at the moment and one up-coming.

MQTT

MQ Telemetry Transport (MQTT) from IBM is the bad boy of small device messaging. It uses TCP, is lightweight, and has features beneficial to IoT devices. It is mature and there are a lot of client and server implementations, making it easier to develop upon.

MQTT works on a pub/sub architecture. A client subscribes to a channel on a server, and when a server receives new information for that channel, it pushes it out to that device.

Another great feature of MQTT is that you can set a priority or Quality of Service (QoS):

  • 0: The client/server will deliver the message once, with no confirmation required.
  • 1: The client/server will deliver the message at least once, confirmation required.
  • 2: The client/server will deliver the message exactly once by using a handshake process.

Each level uses more bandwidth but will give you varying assurance of deliverability.

With MQTT your application must have a library to talk to the MQTT server and handle publish/subscribe methods.

Websockets

Now that browser websockets are standardized, they make a very compelling option for reading and writing data to IoT devices. Websockets are the best way to achieve full push/pull communications to a web server, which is not possible over basic HTTP protocol.

Websockets are great if you have a full web client! However, with many IoT devices, it is a lot of overhead which might not even be an option.

Another downside of using Websockets is that you would need to come up with your own protocol for the transmission of data. This isn’t too difficult (for example, using JSON) but does create a non-standard protocol.

MQTT over Websockets

A frequent use of Websockets is to actually use MQTT over Websockets. With this configuration, you can use the Paho JS MQTT client to make the connections and to read/write data. This makes for very easy development, but again you must be able to run the full browser stack on the device, so this can be very limiting. Since it does use MQTT, it may be beneficial to use this for any webpage MQTT reading of data. There is also some configuration to be done in the background to create the proxy between MQTT and Websocket, however there are plenty of services such as HiveMQ to do this for you.

HTTP/2

The new HTTP protocol, soon to be seen everywhere, usually a totally different structure than HTTP/1.1. Where HTTP/1 was based on frames/packets, HTTP/2 is a streaming protocol. It supports a few features useful in this situation such as the ability for the client to suspend data streaming, however also has some drawbacks. HTTP/2 was designed to have the client remain connected to the server for the remainder of the browsing session. If using HTTP/2 as a messaging protocol, it would mean leaving this connection open indefinitely. You also miss out on some of the features of MQTT such as QoS. HTTP/2 is so new that some of these features could in theory be added to the protocol and libraries, however it is not yet mature enough to say whether that will happen.

For more discussion on this, check out Can HTTP/2 Replace MQTT from @kellogh.

What’s the Best?

Right now for most applications, I would say MQTT is the best protocol to use for most IoT devices. Since it is so widely adopted, even newer solutions like Websockets support MQTT in some respects so your device will be able to communicate its data effectively to other devices over the internet.

I’m excited to see where HTTP/2 takes us with regards to the Internet of Things. The protocol is better suited for lightweight communication and could definitely be a contender in the future.

Photo courtesy NodeMCU Wikipedia

Firesheep Should Be A Call To Arms For System, Network & Web Admins

Firesheep by Eric Butler has just been released to the world. This Firefox plugin does a few things that have already been fairly easy to do for a while, but rolled up in one easy to use package:

  1. Sniffs data on unencrypted Wireless Networks
  2. Looks for unencrypted login cookies sent to known popular insecure sites
  3. Allows you to login to that account with ‘One Click’

So what sites are impacted by default? Amazon.com, Basecamp, bit.ly, Cisco, CNET, Dropbox, Enom, Evernote, Facebook, Flickr, Github, Google, HackerNews, Harvest, Windows Live, NY Times, Pivotal Tracker, Slicehost, tumblr, Twitter, WordPress, Yahoo, and Yelp are among the few. A plugin system allows anyone to add their own sites (and cookie styles) to the plugin.

Yikes! It goes without saying that this is a major security problem for anyone who uses unencrypted wireless networks. Includes on this list are many universities and companies such as Starbucks.

It is a bit funny, because just last night I was talking with my friend Jorge Sierra about this very problem. My university in fact is one of those which uses unencrypted wifi. I installed the unencrypted password extension for Chrome to let me know when I am submitted an unencrypted password to a site. I was surprised how often this little box was popping up!

Why Open WiFi?

I am not sure – my undergrad university requires that any traffic going over wifi goes through their VPN which encrypts the traffic and prevents this program from working. Is open wifi still the ‘poison of choice’ for network admins because setting up a VPN-style system is too much for some organizations? Maybe – but it is clearly the wrong answer.

The other clear reason is that it is easier to use, and this is a valid complaint from a user experience perspective. I’ve seen plenty of folks have a hard time even with a simple WPA password. A shared password makes it even harder for a user to sign in. Hotels and coffee houses across the world opt for open wifi because it is simply the easiest for consumers to use. This is a problem us tech people need to solve.

Even if it is encrypted via WEP or WPA (1) these are very insecure protocols and still can be hacked with relative ease. This plugin could in fact be modified to include the cracking as well and cover an even wider range of wireless networks. This brings me to my second point.

Web Developers Must Encrypt All Login Forms

If you run ANY consumer facing app you should be passing any and all login information via an SSL secured website.

For hosts on a static IP address you simply need to purchase an SSL certificate. They are seriously under $20 these days (my cost as a reseller is $12) and are simple to install. Your code should be set up to always use this site and to never allow username and password to be sent unencrypted over the network. This is important not only at the end user’s connection (possibly over open wifi) but also for end-to-end encryption of this data.

Let’s say you are running a site on a shared IP address. You usually still have options. Most hosts I know of offer on SSL connection via the shared site – eg: https://server.name/~username/. This URL can be used to access your site’s information via an SSL certificate and it is normally included with the service.

Ideally every site would have an SSL certificate. But we need a few things for that to happen. People who buy web hosting are almost always looking for the cheapest deal. They will not be getting SSL at these bottom level prices. Hosting needs to have a paradigm shift so that people who run websites need to know that it is better to have people who know what they are doing from a security standpoint have configured and are running their servers, and that paying $10 a year for hosting isn’t sustainable. Some say that there is a significant overhead to running SSL on websites. It will, in fact, add some processing and bandwidth overhead. However this is necessary to provide security of services to the end users.

In my opinion, you either host your website on a large provider who use set up to have a secure infrastructure, or you pay more for an expert to host your website in a secure manner.

Another roadblock is the end of the free source of IPv4 address blocks. Web hosts need to move to IPv6 to free up IP addresses, and every website should be on its own IP address. That will allow SSL certificate installations much easier.

Back to Reality

What can you do, right now, about this problem? If you have to use an unencrypted wireless network, you should be running some sort of VPN to encrypt your traffic over the air as this is the most likely place it would be sniffed. You can get a cheap VPS at < $10 a month and proxy all of your traffic over SSH. Not the fastest method, but it will secure your data.

You can also install the Chrome Extension to warn you if you are about to submit form information via an unencrypted website. It isn’t the prettiest extension but it does get the job done.

Hopefully network, web and system administrators will get their acts together and push for a solution to this problem. It is a big one and one that isn’t apparent to the end user until their data, financial details and/or identity is stolen. We can fix this.

/via TechMeme

Secure Your Google Apps (Gmail, GDocs, GCal, etc)

If you are a Google user – meaning Gmail, Google Docs, Google Calendar, Google Reader, etc – then you should know that by default, once you log in your sessions are typically not encrypted between your browser and the Google servers.

For some more technical information on this, check out dmiessler’s post on the subject.

He mentions using bookmarks to make force your browser to use Google’s secure connections – however I’ve noticed that occasionally depending on how you arrive to your Google services that you will switch to an unencrypted session without warning.

For that reason, If you are using Firefox and greasemonkey, I highly recommend installing the “Google Secure Pro” userscript. It automatically switches you from using http:// to https:// to ensure your data is encrypted to Google’s servers. This will increase your security greatly from using the unencrypted connections, which is good if you transfer confidential data over e-mail – like most normal people do!

Force HTTPS SSL Access for a URL with Apache

The situation is: you have an web application or URL that you would like to force your users (or yourself) to use the secure https protocol rather than the unencrypted http protocol. This is easy to do with Apache and .htaccess.

Create or add to the .htaccess file in the root of the web directory you would like to force redirect for. Add the following lines:

1
2
3
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

This says that if https is off, reload the page at the same location using HTTPS instead.

Google Gmail, Docs, Apps and Reader Secured

Google Secure

For Firefox and Opera only: Use the Google Secure Pro Greasemonkey script to force these Google programs to use SSL in your browser. It changes any accesses via http to https which is supported by Google, this secures your connection between your browser and Google’s servers.

If you do any type of work using Google’s services where private data is exchanged, I would highly recommend this greasemonkey script to secure your communications!

Apache’s mod_proxy with Exchange’s Outlook Web Access

Apache’s mod_proxy module is simply one of the best Apache modules out there. With it, you can do all sorts of things that you usually would not be able to do if you are behind a firewall or other limited network situations.

A problem that recently came up for me was how Microsoft Outlook Web Access (OWA) needs to run on an exchange server, however my linux server is the one that faces the internet (I have the firewall forward the ports to this server). I also purchased an SSL certificate for one domain, so I wanted to use this certificate to access OWA with a proper validating certificate.

All sounds well and good. Using this mod_proxy configuration should work:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
ProxyPreserveHost On

#OWA % character in email subject fix
RewriteEngine On
RewriteMap percentsubject int:escape
RewriteCond $1 ^/exchange/.*\%.*$
RewriteRule (/exchange/.*) ${percentsubject:$1} [P]

#OWA
ProxyPass /exchange https://exchangserver.example.com/exchange
ProxyPassReverse /exchange https://exchangeserver.example.com/exchange
ProxyPass /Exchange https://exchangeserver.example.com/exchange
ProxyPassReverse /Exchange https://exchangeserver.example.com/exchange
ProxyPass /exchweb https://exchangeserver.example.com/exchweb
ProxyPassReverse /exchweb https://exchangeserver.example.com/exchweb
ProxyPass /public https://exchangeserver.example.com/public
ProxyPassReverse /public https://exchangeserver.example.com/public
ProxyPass /iisadmpwd https://exchangeserver.example.com/iisadmpwd
ProxyPassReverse /iisadmpwd https://exchangeserver.example.com/iisadmpwd

Problem – it works ok – except in IE it will prompt you for the password indefinately and not allow you in. In Firefox (Mozilla) it rejects your password, until you hit cancel, then enter your password and it finally allows you in.

To fix this issue, you need to disable “Integrated Windows Authentication”. In the IIS administration panel, go to the website for your exchange server (“Default site” by default) and find the exchange share (This is most likely “Exchange” and “Public”). From there, right click, go to Properties->Directory Security->Anonymous Access and Authentication Control. Make sure “Basic Authentication” is checked while “Integrated Windows Authentication” is unchecked. Do this for any other Exchange shares. This allows authentication to work OK.

Second problem… in OWA, in Internet Explorer only, when you try to view your inbox the “Loading…” message appears indefinately. Microsoft’s Knowledgebase Article 280823 has a few workarounds for this problem, none of which worked for me. OWA apparently has two modes that it runs in, “rich” and “reach” modes. The “rich” client, which it uses for Internet Explorer, can have issues when running behind a firewall. It uses http-dav components which are not passed through correctly.

Now a fix, let’s make sure all clients run in “reach” mode! Using apache, we can hard-code the User agent that will hit the Exchange server. We use the mod_header module of apache, so make sure you compile it in with –enable-headers. Note: this only works with Apache 2.0. Once you have this compiled in, let’s set the User agent:

1
RequestHeader set User-Agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7)"

You can use whatever you’d like in the user-agent string, as long as Outlook Web Access does not think it is IE, then it will serve the “reach” client.

After correcting all of the above issues, Outlook Web Access finally works in both Internet Explorer and Firefox.