MQTT Promotes Standards, Interoperability on IoT

Patricio Robles
Apr. 18 2014, 08:17AM EDT

Last week MQTT (Message Queuing Telemetry Transport) provided developers with a signal that it may be emerging as a de facto protocol for the IoT. This validation comes in the light of positive MQTT Interoperability Test Day results that were published by the host of the event: The Eclipse Foundation and the Eclipse IoT Working Group. The Eclipse Foundation and the Eclipse IoT Working Group announced the results of a MQTT Interoperability Test Day held in March during which 15 companies, including large software vendors like IBM, Software AG and RedHat, tested how well various implementations of MQTT adhere to the draft specification and how interoperable they are with each other. Players in the IoT space have been moving slowly on standards but The Eclipse Foundation’s Ian Skerrett, expects the recent developments with MQTT will help change that. “I think we are already seeing more adoption of MQTT. Developers want and need some of the basic building blocks for creating IoT applications. They are tired of reinventing it themselves. Demonstrating that MQTT is indeed interoperable will help with future adoption,” Skerrett explains.

Leading the Drive for Interoperability: MQTT

According to Skerrett, interoperability is crucial to the burgeoning IoT market. “The current state of the IoT industry has resulted in a lot of siloed, proprietary solutions,” he told me. “Many of these solutions will offer an ‘open’ API but that API only works for the proprietary solutions. This makes it very difficult for the users to integrate solutions from different providers. If we really want an ‘Internet of Things’ it needs to be easy to integrate like we can with the real Internet.” One of the early protocols is Message Queuing Telemetry Transport, or MQTT, developed by IBM and Eurotech and designed to provide publish/subscribe messaging transport. While still early, the test results paint an encouraging picture for developers looking for standards to emerge. “Overall, more than 50% of the test pairs were considered successful. At this stage of the standardization process, this demonstrates a good level of interoperability between MQTT implementations and points to the ease of creating interoperable IoT solutions based on MQTT,” the Eclipse Foundation stated in a press release. Another test day is planned for later this year and Skerrett believes that even more progress will be evident then: “The next test day will be focused on the final specification for MQTT. Right now it is still in draft form so not all the providers have updated their implementation. I will expect at the next test day we would see a lot more participants and more instances of successful testing.”

Future Development

The Internet of Things (IoT) is booming, and as more and more companies look to connect the world using sensors and smart devices, establishing protocols that will serve as the backbone of the IoT is one of the most important steps in ensuring the IoT realizes its trillion-dollar potential. HTTP is the foundation of the web we use on a daily basis, but the IoT has different needs. For one, the IoT will be home to a lot more devices. Billions of smart devices, including sensors, will be a part of the IoT network. Additionally, many of them will need to communicate with each other and operate in environments where bandwidth and computing resources are constrained, which is one of MQTT’s sweet spots. There is still work to be done, both on MQTT and other protocols that are being adopted. As Holger Reinhardt of Layer 7 Technologies has noted, “each protocol has weaknesses.” MQTT, he suggests, “appears to be weak in security.” Skerrett acknowledges that security is an important subject and believes it is one of the biggest issues facing the IoT industry as a whole. For many of standards developing around protocols, finding a balance between usability and security could be key. “I think the challenge for MQTT will be to retain its simplicity and ease of use. All too often you see ‘feature-creep’ emerge in successful standards. Right now it does a very good job with a simple specification,” Skerrett says. Ultimately, a number of key standards and protocols, such as MQTT, will be broadly adopted across various IoT applications, says Skerrett, but because the IoT is so diverse and there are so many varied applications, there is also room for application-specific protocols, many of which will likely be built on top of open standards like MQTT.

Patricio Robles Follow me on Google+

Comments

Comments(9)

User HTML

  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.

How does one "Promote Standards" by oneself saying one "may be emerging as the de facto protocol for the IoT"? That's what many major proprietary solutions use to say also. "De facto" in this case, is a subjective term meaning that since many use it, it must therefore be a standard, which it of course is not.

Can you provide a link with more information on how exactly MQTT is "Leading the Drive for Interoperability"? MQTT is a binary protocol without any support for interoperability at all.

MQTT only supports the publish/subscribe pattern of communication, which hardly covers all use cases within IoT. To say that MQTT is a de facto protocol for IoT is a grave misrepresentation, even though it has an important role to play.

Compare this to XMPP [1], which it standardized by IETF, secure, reliable and extensible, and which has an open standards organization [2] where everybody can participate and promote new extensions to the protocol. Apart from supporting the publish/subscribe pattern, it also supports a wide array of other communication patterns, important for IoT. XMPP also has approved extensions for interoperability in IoT [3] [4] [5] [6] [7], and work is being done to enhance this even further [8]. Furthermore, XMPP allows extensions of the Semantic Web onto secure XMPP-based IoT networks [9] [10]. Nothing of this exists in MQTT. Since MQTT can be bridged in its entirety into XMPP networks [11], but not the other way around, one can see that MQTT provides a subset of functionality compared to what XMPP can provide. For a more detailed comparison between XMPP &amp; MQTT see [11].

Best regards,

Peter Waher

[1] XMPP Technologies Overview, http://xmpp.org/about-xmpp/technology-overview/

[2] XMPP Standards Foundation, http://xmpp.org/about-xmpp/xsf/

[3] XEP-0323: Internet of Things - Sensor Data, http://xmpp.org/extensions/xep-0323.html

[4] XEP-0324: Internet of Things - Provisioning, http://xmpp.org/extensions/xep-0324.html

[5] XEP-0325: Internet of Things - Control, http://xmpp.org/extensions/xep-0325.html

[6] XEP-0326: Internet of Things - Concentrators, http://xmpp.org/extensions/xep-0326.html

[7] XEP-0347: Internet of Things - Discovery, http://xmpp.org/extensions/xep-0347.html

[8] Internet of Things - Interoperability: http://htmlpreview.github.io/?https://github.com/joachimlindborg/XMPP-Io...

[9] XEP-0332: HTTP over XMPP transport, http://xmpp.org/extensions/xep-0332.html

[10] Extending the Semantic Web to Peer-to-Peer-Like Sensor Networks Based on XMPP, https://www.dropbox.com/s/mn7gflkgillomxw/Extending%20the%20Semantic%20W...

[11] Bridging MQTT &amp; XMPP Internet of Things networks, https://www.dropbox.com/s/vhnnk2wg7if62dc/Bridging%20MQTT%20%26%20XMPP%2...

XMPP can be used in constrained devices also. However, you need a client library that only implements the extensions you need, instead of using one that implements as many extensions as possible. (You can also reduce an existing library so it only implements the features you're interested in.)

Ronny Klauck &amp; Michael Kirsche present an example of such a library in [12] and [13].

Best regards,

Peter Waher

[12] Chatty Things – Making the Internet of Things Readily Usable for the Masses with XMPP:

https://www-rnks.informatik.tu-cottbus.de/content/unrestricted/staff/mk/...

[13] Unify to Bridge Gaps: Bringing XMPP into the Internet of Things:

https://www-rnks.informatik.tu-cottbus.de/content/unrestricted/staff/mk/...

Peter,

I think it is fair to say protocols like XMPP, CoAP, MQTT and others will be used in IoT. A lot will depend on the specific use case. This was referenced in the article by Holger Reinhardt. I think the advantage of protocols like MQTT and CoAP is that they are very lightweight so can target constrained devices.

Regardless, there is still lot of work to be done for all protocols.

Peter,

I would also add that you seem to be misinformed about the standardization efforts around MQTT. MQTT is currently being standardized at OASIS, an effort which I believe is expected to be finalized quite soon.

Peter,

I think I don't really get your point. While I understand your criticism on phrases like "de facto" are subjective, I don't understand your whole criticism on MQTT.

Sure, MQTT isn't as feature rich as XMPP, but I doubt that this is a disadvantage. MQTT was designed to be as simple as possible and only provide the features which are needed frequently. Everything MQTT is today came from the practical experience and concrete projects and were not designed in the ivory tower. All additional stuff MQTT brokers implement are completely vendor dependant.

And this is the whole point why I think MQTT gains that much traction at the moment: MQTT is super easy to grasp and the current 3.1 spec has &lt; 30 pages which makes it almost trivial to implement basic MQTT clients and brokers.

While I don&#039;t like the XMPP vs MQTT discussion, here are my 2 cents: The disadvantage I see with XMPP is, that the features supported by different servers are completely fragmented. See [1] for the list on Wikipedia. I don&#039;t see why XMPP would be more interoperable than MQTT, just because someone adds even more features as XEP which are only implemented by a subset of servers vendors. Note, that the extensions you proposed aren&#039;t even listed on the Wikipedia XEP list yet and I didn&#039;t find expressive information who is already implementing these additions.

It&#039;s also worth to note that all the IoT XMPP extensions you linked are A) experimental and B) written by yourself. Also, I don&#039;t feel that the comparison between XMPP and MQTT you linked (and wrote yourself) is objective and unbiased and I respectfully disagree with some paragraphs while I agree with others (SASL would indeed be a good extension to MQTT). The key thing about MQTT is, that it relies on other standards like TLS and doesn&#039;t try to do everything on its own. From a practical point of view, I disagree with the argument that TLS may be too heavy for constrained devices. This has been done for years and transport encryption is IMHO important, regardless of the application layer protocol.

I really don&#039;t see a point in arguing why XMPP or MQTT is &#039;better&#039;, I&#039;d argue that both of the protocols play a significant role in IoT. Both protocols have a completely different focus and this is good. I like XMPP and AMQP for general messaging and I like MQTT for lightweight messaging. If you only have a hammer every problem seems like a nail.

(Disclosure: I may also be biased in favor of MQTT)

[1] http://en.wikipedia.org/wiki/Comparison_of_XMPP_server_software

Hello Dominik (&amp; Community)

I’ll try to respond to your comments in an orderly and constructive manner. Sorry if it’s long, but there were of lot of concerns raised.

1. I did not write a critique on MQTT, but a critique on the statement that MQTT promotes standards &amp; interoperability for the IoT. Keyword here perhaps should be Internet, as in Internet of Things, and standards (plural). I like the MQTT protocol, for the same reasons I guess many like it, and we support it also. But we don’t recommend it for IoT, for various reasons listed in [11], but also implicitly in [10]. In our view, MQTT is a good back-end protocol where the publish/subscribe pattern is warranted. We differ between M2M and IoT (more on this below), and consider MQTT (and AMQP, etc.) to be well suited for M2M (as XMPP does too), but not for the Internet, and not in the general case, since it doesn’t support all required communication patterns (more on this below).

2. This ties into a comment made by Mike above, regarding the standardization process of MQTT. I am familiar with OASIS being the standardization body for MQTT. The question is, whether OASIS (which some call a promotional organization more than standardization body, even if that might be going too far) is the organization you want to go through when you want to standardize things for the Internet (which obviously IoT is part of). Some would say, IETF is the standardization body for that. There are many “standardization organizations” for various M2M protocols, OASIS is one. But Internet have one such organization (IETF) and the web an additional (W3C) even though the latter doesn’t produce standards, only recommendations.

Having said that, it is not even this point that is important, but the statement says it “promotes standards” (plural). By going through another organization than IETF you don’t promote standards for the Internet (of Things), rather you weaken the position of the body that produces standards for the Internet (of Things). It gets worse, since other efforts are not even mentioned so they can be compared.

3. The second part of the critique, is that MQTT is “Leading the Drive for Interoperability”. But it doesn’t say how this is so. Leading, means you’re aware of other attempts that you are ahead of. None are mentioned, and so the reader cannot compare. And Interoperability means what in this case? Interoperability in connecting to servers, or interoperability interchanging information across the network? It is not mentioned, so it is misleading. There are many attempts to “Drive Interoperability”, of which I mention one, the effort performed using the XMPP protocol in which I’m personally involved as you notice. This doesn’t make the argument invalid. In my view, MQTT is very far behind in this area, and not leading anything regarding Interoperability.

As of yet, I’ve not seen any such documentation on what actually is meant by interoperability in the MQTT case. I would very much like to know, and that is not said rhetorically, but out of actual interest.

4. You say that “MQTT was designed to be as simple as possible and only provide the features which are needed frequently. Everything MQTT is today came from the practical experience and concrete projects and were not designed in the ivory tower”. Not sure how to respond to this in short words. Let’s break it down:

It is true that MQTT is simple. But simple does not automatically mean it’s a good idea. People should know about the pros and cons of different methods and architecture patterns. It might be simple to start with MQTT, but since it has a limited architecture, you quickly get cornered in, and things stops being so simple anymore. Even though there are methods to go around things such as the request/response paradigm, these tend to make your topic tree a mess. Furthermore, since you don’t know who can listen to responses, and who made a request, it is very difficult to make context sensitive responses. “Dumb” simple responses might work, as long as you’re satisfied with that. But what do you do, when you want to respond differently depending on who made the request? Or the case when some should have access to only certain functions? All these cases are not so simple to solve in MQTT, as they are using other protocols, which might have a somewhat higher threshold to begin with. If you want to make a secure provisioning platform on-top of MQTT, you end up with complex end-to-end encryption methods and PKI. I wonder who thinks it is still simple then… The answer is of course “nobody”, which is why “everybody” will skip it (“” meaning “except a few exceptions”), which is why MQTT solutions will have large security problems. And this is not even the beginning. After having solved all these things, you want to have actual Interoperability between manufacturers regarding these things. I would like to see anybody try to solve that problem (I actually do – however I hope I’m invited to the discussion.).

Furthermore, “needed frequently” is kind of circular reference. It is needed frequently, because it’s the only pattern provided. MQTT covers the publish/subscribe, which is only one of many communication patterns in M2M, and IoT. Even though it is important, in my view it is not the most important.

What to do of your second comment? I’ve worked with M2M since 1994, and publish/subscribe has only become popular lately. I’m not sure what ivory tower you reference.

5. Regarding the “disadvantage I see with XMPP is, that the features supported by different servers are completely fragmented. See […] for the list on Wikipedia” and “Note, that the extensions you proposed aren't even listed on the Wikipedia XEP list yet and I didn't find expressive”.

The benefit of these extensions, is that support for them are not required on the server. They are client extensions, meaning that only XMPP clients need to implement them. The XMPP server just acts as a broker, relaying messages. There are some exceptions, like the EXI compression extension, which require server implementation (which is underway). But this extension was not even mentioned above (XEP-0322).

MQTT has no means of relying information on how the content that is relayed should be interpreted. HTTP has a minimal content type, which tells the recipient how to interpret incoming binary data. XMPP accomplished the same using qualified names in XML (local name + namespace). It is sufficient that clients are aware of how to interpret content, not the server itself.

Furthermore, the Wikipedia article is not a principal source, when it comes to XMPP. For those that are interested, a list of approved extensions is available in [14].

6. Regarding “It's also worth to note that all the IoT XMPP extensions you linked are A) experimental”. This is correct. It is the first step in the standardization process followed by XSF [15]. An extension passes through an initial review by the community, and in particular the council, and finally they decide if a proposal is to be accepted and published, or rejected. Any important objections must be addressed, and all discussions are open. The Experimental state is used for exactly this purpose: Time of experimentation. (More or less the same state MQTT is in.) When sufficient experience has been gathered, and issues mended, an extension can move to draft. Everybody is allowed to participate, and also contribute.

7. Regarding “B) written by yourself.” Also correct. I promote the work we do, and partake in any promotion of the ideas and arguments allowed. Or do you mean to say, that one shouldn’t be allowed to discuss things one has written about? Sounds like an ad hominem argument.

8. Regarding “I don't feel that the comparison between XMPP and MQTT you linked (and wrote yourself) is objective and unbiased and I respectfully disagree with some paragraphs while I agree with others”. Unless you provide any concrete arguments, it’s difficult to respond to such ad hominem arguments.

9. Regarding “I disagree with the argument that TLS may be too heavy for constrained devices”: The point you make is not the point I made in the document. The point I made, is that MQTT, in the simplest form, send password in clear text. To resolve this, they enforce devices to use TLS (instead of leaving it as an option) to encrypt the entire communication, including certificate validation if you want to remove MIML attack possibility, which is contra-productive if the main goal is to provide a solution for constrained devices. It would have been much better to redesign the protocol (which you obviously agree to). How many devices do actually use TLS with certificate validation to make authentication secure? SASL, together with SCRAM-SHA-1 authentication would have been a much better choice. It is completely unnecessary to perform TLS and certificate validation in constrained environments, if all you want to protect is the clear text password and don’t bother with encryption.

10. Regarding “If you only have a hammer every problem seems like a nail.”. I agree. See discussion above regarding communication patterns. If everything you have is publish/subscribe, you try to model everything using that pattern, which might not be the best choice to begin with. Sometimes, it is worth taking time in the beginning of a project to envision what you might want to do, before making architecturally important decisions on how to solve the problem.

Having said all the above, we still support the MQTT protocol, and like it, but don’t recommend MQTT communication to be published on the Internet. We use it as a back-end protocol, and bridge it to XMPP for publication on the Internet, according to [11], which allows for provisioning, i.e. better control of who can access which device, and in particular what data on those devices. It also allows for other interesting applications, such as using semantic technologies to access data [10], at no extra cost to the device.

Best regards,

Peter Waher

[14] XMPP Extensions:

http://xmpp.org/xmpp-protocols/xmpp-extensions/

[15] XEP-0001: XMPP Extension Protocols:

http://xmpp.org/extensions/xep-0001.html

Hi Peter,

thanks for the thorough response.

1) Regarding OASIS: While I agree that IETF is the right standardization body for all Internet relevant standards (and has always been), standardization organizations like OASIS standardized some important IoT relevant things recently. Also, OASIS is the home of many non-Internet related standards which play a role for 'Cloud' (as in Business Services over the Internet) and document management and security. Some important standards published by OASIS recently are: AMQP [1], PKCS11 [2], OData [3], SAML [4] and ODF [5]. I personally wouldn't have a problem seeing MQTT standardized by IETF, but I also don't see a problem with OASIS, as it seems that industry adoption doesn't suffer with either IETF and OASIS. For me it seems that OASIS standards get industry traction quickly after publishing a standard. This may of course be due to the fact that companies who are in the Technical Committees have interest in providing a standard conform solution quickly (see the current MQTT 3.1.1 spec. It's only a draft right now but many broker vendors already support it, as seen in the MQTT Interoperability Testing Day in San Francisco in March)

2) Regarding your point 4):

I agree, that you could get cornered in if you want to use the publish / subscribe pattern for things it wasn't designed for. A common pattern I've seen a lot in real world applications is using HTTP / CoAP *and* MQTT for IoT Platforms. RESTful API design and a well structured MQTT topic tree work great together. Of course this needs a proper design and if people get it wrong, then you can really get cornered. So I 100% agree with "MQTT covers the publish/subscribe, which is only one of many communication patterns in M2M, and IoT". My preference is having modularity (= using MQTT, CoAP, HTTP, XMPP or even SMS) and choose the right tool for the right job. I don't believe in silver bullets and none of the technologies I've seen so far is a silver bullet. And my preference is not to have all the modularity built into one protocol, I like the ability to combine technologies and protocols, especially in environments where removing the unneeded overhead is key.

I'm not sure if I understand your argument with context sensitive responses correctly. When you are using MQTT, most of the context is typically in the topic tree (Trivial example: {clientId}/status). Obviously it is important to restrict read / write access and provide a proper authentication and authorization mechanism for topics and clients.

While standard username / password over TLS is the most common approach, you could also use OTP and OAuth2 and other mechanisms, if your broker vendors supports that. But again, this is not built-in into MQTT. I could see value in adding things like SASL to the protocol at a later point but it is not necessary from my point of view, as this can always be added on top of the protocol. The argument that people don't use security because it's hard is applicable to every security discussion. (see REST API discussions. Most people tend to say that HMAC is a proper way of securing REST APIs but very people really do it because it's impracticable if you have many REST API consumers on different platforms).

I think it's worth writing up a blog post or paper about securing MQTT and common pitfalls. The security topic indeed irritates many people and there aren't too much good write-ups or papers about that. I'll try to find some time for doing that.

To clarify the comment with the ivory tower: It was meant, that the MQTT features that are in the 3.1 specification (which was the foundation for the upcoming 3.1.1 standard) were created by needs in concrete projects. I didn't mean to say that anyone builds systems in an ivory tower. This wasn't meant as an ad hominem argument to you.

3) MQTT doesn't have any mechanism for payload metadata. There is active discussion ongoing in the community (see [6] for the most recent mail thread and many other older mailing list threads). To overcome this issue, a popular approach is to use Apache Thrift, Google Protobuf, Msgpack or similar encoding mechanisms. I personally would like to see an MQTT extensions for Payload metadata on topics. The MQTT ecosystem is young and very active, so this will eventually will be solved. But again: This isn't needed for many project specific use cases (at least the one I have seen), but as you pointed out, can be very important for interoperability between IoT platforms.

And here comes my biggest concern when talking about interoperability and IoT in general. I don't think each application level protocol should implement its own 'interoperability' mechanism. From my point of view, true interoperability will only be possible if we agree on a standard for data syntax *and* semantic and which is detached from a concrete application layer protocol implementation like MQTT, XMPP or AMQP. I'm not aware that something like that exists and I'd be very happy if someone could point to standards and implementations which solve this challenge.

4) I apologize for my inappropriate ad hominem arguments as you pointed out in 7) and 8). Sorry for that.

To clarify what I actually meant to say: It is unclear to me if these XEPs are community efforts (and if the XMPP community is pushing these forward), because you are the sole author of many of those XEPs. I absolutely didn't mean to say that you shouldn't use your own writings for argumentation. The only thing I'm curious is if these XMPP IoT efforts have community traction.

I'm planning to do a writeup on your paper of the XMPP and MQTT comparison but this will be out of scope for the comment section of this blog post.

5) To summarize, while it seems we don't agree with certain points, we at least seem to agree that MQTT and XMPP have both their strengths and use cases where they excel.

I would like to invite you to join the MQTT mailing list[7], I believe these kind of discussions are very fruitful for the community and more people can get aware of discussions like this on these kind of mailing lists. It's good to see people stand up for such discussions and eventually we can move things forward!

[1] http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-complete-v1.0-os.pdf

[2] https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=pkcs11

[3] https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=odata

[4] https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=security

[5] https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office

[6] https://groups.google.com/forum/#!topic/mqtt/TOQ-34xgCsA

[7] https://groups.google.com/forum/#!forum/mqtt