MQTT goes free – a personal Q&A

There has been a lot of coverage over the past couple of days of some exciting announcements that I’ve been involved with at work. I’ve spent the past three days at EclipseCon Europe 2011, which doubled as the 10th birthday celebration for the Eclipse initiative. It was a funny feeling, because Eclipse started just a few weeks after I first joined IBM, and although I’ve used it and watch it “grow up”, I’ve never done EclipseCon before. The reason I’ve been out there for three days this time (as a WebSphere Messaging guy rather than a Rational tooling or build person, for example) was to get involved with activities around these announcements.

It’s all about machine-to-machine (or M2M) communications, Smarter Planet, and the Internet of Things.

Before I dive in to this, a few clarifications. First, I’m being described in a couple of news stories as “an IBM distinguished engineer”, and whilst I wish that was true, I’ve yet to ascend to those heights! Also, there are various numbers being quoted – note that the figures in the press release were not invented by IBM, the headline number of an expected 50 billion connected devices by 2020 comes from a recent study conducted by Ericsson AB. Oh, and this isn’t about a “new” protocol – MQTT has been in use since 1999.

The other clarification is that some articles seem to suggest that IBM is out to create some kind of new, alternative, Web – that’s not what has been announced, and I’m certainly not aware of any such plan! It’s about connecting “things” – sensors, mobile devices, embedded systems, even small appliances or medical devices for example – to the Web and the associated platform and ecosystem of technologies, not about reinventing or recreating them. I’m personally a huge fan of the Web as a platform🙂

Oh, and of course, the obligatory “all opinions expressed are my own” – this is my understanding of where things are going, although of course I’m talking about events I’m directly involved in!

So what is this all about?

Two things.

1. On Nov 2, IBM, Eurotech, Sierra Wireless and Eclipse formed a new M2M Industry Working Group at Eclipse. Sierra had already started the “Koneki” project at Eclipse to work on M2M tools, and the Working Group will look at a range of topics together, such as M2M tooling, software components, open communication and messaging protocols, data formats, and APIs.

2. On Nov 3, IBM and Eurotech announced the donation of their C and Java clients for MQTT to a new Eclipse project called “Paho” which is under proposal in the incubator – with code expected to hit the repository within the next couple of months. MQTT is being given to Eclipse to live within the M2M ecosystem that is emerging there, and to provide an avenue for adoption of the protocol as a more pervasive standard for connected devices.

How is that news? Isn’t MQTT already open / free?

Technically… kinda, sorta🙂

The MQTT specification has been published under a royalty-free license for some time, and that has led to a fantastic community contributing a range of different projects. IBM and Eurotech took this approach from early on, because it wouldn’t have been possible to compile and support code on every embedded platform that might come along – far simpler to set the protocol free.

Initially the specification was hidden away in the WebSphere Message Broker documentation, but last year it was republished, moved to a new home on developerWorks, and the license was clarified.

In August, IBM and Eurotech announced their intention to take MQTT to a standards organisation. The specific organisation has not yet been finalised, but this is also an important step in ensuring that MQTT is not “just” an IBM protocol, but something of general use which the community can feel comfortable with. If you’d like to join that discussion then there’s a Get Involved page on the mqtt.org community site.

The missing piece was code – a reference implementation, if you like. That’s one reason why the Eclipse Paho announcement is significant.

Why else is this significant?

Well, here are some of my musings on that one:

  • it shows IBM is serious, by committing code and open sourcing it (as with the original Eclipse donation in 2001);
  • the M2M Industry Working Group exists to foster the discussion in this space;
  • it makes high-quality reference Java and C client implementations freely available in source form, with a good Java implementation something that has been particularly lacking;
  • it creates an opportunity for Eclipse projects to use MQTT, and to develop tools on top of it.

The press release and Paho project proposals aren’t clear (to me) – what exactly is being donated?

IBM is seeding Eclipse Paho with C and Java client implementations of MQTT. Eurotech is donating a framework and sample applications which device and client developers can use when integrating and testing messaging components.

Why C and Java clients (aren’t they “dying” languages?) Where’s my Perl and Ruby code?!

IBM had previously made some C and Java code available in some SupportPacs, but those are outdated and the license for reuse was never clear.

It’s important to realise that this stuff came from the embedded world of 10 (and more) years ago, and continues to be applied in that industrial space. That category of device typically runs some kind of realtime Java-based OS, or a Linux-based or other runtime with a GCC toolchain for the CPU in question. C and Java are genuinely the most useful implementations to get out there. Oh, and on that “those old languages” thing – I think you’ll find they are very widely used (Android, iOS etc run variants of sorts, most non-web app development is likely to be in one or the other).

We’re very fortunate that clients libraries for a wide range of languages already exist thanks to the MQTT community – see the list at mqtt.org!

Hold on… don’t we need a broker / server / gateway?

Yes. But, one step at a time!🙂

There are brokers available for free today, either as precompiled binaries or as full Open Source implementations, so this is not a dead end from day one.

The Paho project scope outlines the intention to add a broker to the project in the future, and to host an M2M sandbox for developers as well. That is where we are today, and this position will evolve over time.

Why Eclipse?

10 years of Eclipse The Eclipse Foundation has been a fantastic success story (oh, and, Happy 10th Birthday, Eclipse!). As the scope of their mission has broadened beyond an IDE to the web, build environments, and all kinds of other tools, it was a good place for Sierra Wireless to kick off the Eclipse Koneki M2M tools project, and is now a natural place for this primarily M2M protocol to be hosted under Paho. As James Governor notes in his write-up of the news:

… the Eclipse Public License is designed to support derivative works and embedding, while the Eclipse Foundation can provide the stewardship of same. One of the main reasons Eclipse has been so successful is that rather than separate software from specification it brings them together – in freely available open source code – while still allowing for proprietary extensions which vendors can sell.

How quickly will the code donation happen?

The Paho proposal tentatively includes dates in November and December 2011 – there will need to be various approvals as code is accepted into Eclipse, so that may “flex” a little, but it is all in the pipeline.

OK… Why MQTT? Why not HTTP/XMPP/AMQP/PubSubHubbub/WebSockets/etcetcetc?

To answer this one adequately I’d probably end up addressing each individual difference between protocols in turn, and if you’ve heard me speak about MQTT I’ve covered some of this before – so I’ll keep this answer relatively brief. I will admit that I’ve been asked about all of these by journalists in the past couple of days.

There is space for a range of protocols to coexist, because they address different areas. In the messaging space, we’ve found over time that whilst efforts to create a single protocol have been made, that has often ended up as focused around a particular set of qualities of service, and not optimised to cover the the whole range of them.

For example, if we look at IBM’s own messaging protocols – there are several. There’s WebSphere MQ which is all about reliable, transactional, solid, clusterable, enterprise, JMS and other APIs, etc etc.. WMQ itself isn’t ideal for very high-speed in-memory or multicast scenarios, so there is also WMQ Low Latency (interoperable with the new multicast feature in WMQ 7.1, but a separate protocol). Neither WMQ LLM or WMQ scales down to unreliable device networks and embedded systems, so there is WMQ Telemetry (aka MQTT), which was specifically designed for constrained devices and networks, and that can interoperate with the main queue manager, too. Oh, and sometimes you want to deal with files (WMQ File Transfer Edition), or access message data via HTTP (WMQ HTTP Bridge). You need to address a range of requirements in a messaging story.

So why not those others? In this case, IBM believes that MQTT is ideally-suited to the Smarter Planet Instrumented->Interconnected layer – it’s tiny, not synchronous and brittle, isn’t specific to the web as it is all about data rather than documents, XML etc etc. In these scenarios, REST principles may add an overhead. Oh, and it has been around for over 10 years, and has been proven across a range of industries and in a range of extreme conditions. IBM’s commercial implementation is known to scale to hundreds of thousands of connected devices, and we know that is the direction that this space is heading.

Congratulations! / Thank you!

Thanks, but don’t congratulate or thank me! I’m familiar with this stuff, I’ve coded with this stuff, but I didn’t invent it and I didn’t write it. There are some amazing folks at both IBM and Eurotech (and some who have moved on) who started this all off in 1999, and who have helped to implement solutions using this protocol since then, and who have of course developed it. Several of them are on Twitter if you want to say hi! And huge thanks again to the community of folks that formed around mqtt.org and contributed client and server implementations – that absolutely helped to move things forward to this point.

HERE ENDS TODAY’S Q&A!

That, hopefully helps to clarify a few things and answers some of the questions I’ve seen via Twitter, forums, and mailing lists over the past few days. It has been something of a blur, to be honest, but a lot of fun. I’m looking forward to the next stage – working with the community more, working with our friends at Eurotech, Sierra Wireless and elsewhere, and making the M2M space much more real.

For more, here are a bunch of stories I’ve seen in the past couple of days… no particular order, just my cut-and-paste list!

4 responses to “MQTT goes free – a personal Q&A

  1. It is always a good thing to see investment and activity in the IoT space, but respectfully, I must disagree with some of your premises. I am always a bit put off when a technology is intentionally or unintentionally miscast in the press as being something that it is not. I am supportive of the move, but just want to provide some clarity and observations on the reality of the IoT.

    Regarding your comments on alternative technologies and that MQTT is all about “data” – the IoT isn’t just about “data”, either (and in fact, I’d argue that MQTT does very little in the context of “data” other than providing a transport – there’s nothing to describe/discover consistent data formats that I can see).

    Most of the real-world examples I’ve implemented include service invocation, event processing, file transfer, and a lot of things in addition to “data”. I also feel that a binary protocol is simple the wrong approach, even on lower capability devices. In fact, I highly doubt that there will be much standardization at that layer, ultimately. The standardization/interop will matter at a much higher level. This is where capabilities such as discovery, data semantics, and so on will be critical. It is also unlikely that most sensors will be directly connected to other applications, for a whole host of reasons (which I’ll touch on shortly). These sensors will be connected through gateways which can better manage security, caching, multi-modal communications routing, and so on. Plus, we cannot forget the critical importance of the billions of sensors that already exist. They need to be able to participate equally.

    More importantly, I feel that a few very important capabilities are fundamentally lacking in the protocol. Notably, the lack of a consistent timestamping mechanism in messages and a lack of any type of security. Security could (and perhaps should) be dealt with via a gateway, in which case the sensors (or actuators) wouldn’t be directly connected to any open network. But in that case, it really doesn’t matter what the “behind the gateway” protocol is anyway. The lack of a timestamp, however, is a more fundamental flaw that needs to be addressed. If we’re talking energy meter data or circuit breaker trips on the energy grid, coordinated (and high resolution) timestamps as a native part of the transport are very, very important. The current proposal essentially leaves that up to the implementor, which of course will lead to no interoperability.

    I understand that this is an early initiative and I am confident that many of the issues will be addressed by the community. But I just want to point out that MQTT “as is” doesn’t really bring a lot of immediate value. Once the other critical gaps in and around it are filled, then we can start to see some value. And I remain skeptical that a binary protocol is really the optimal approach, but look forward to watching and contributing to the discussion as it moves forward.

    FWIW, we do intend to support MQTT at multiple levels of our own technology stack – on the ThingWorx server(s), gateway(s), and agent(s) – among many, many other protocols of course.

    I applaud IBM for contributing this to the community and look forward to seeing it materialize into something of real utility.

    Best,

    Rick

    Like

    • Rick – thanks so much for your thoughtful response on this… and I have to say…. “yes”🙂

      You’re right. The Paho announcement was primarily about the protocol, and not so much about things like data formats, event processing etc.. Those are some of the things that the Eclipse M2M Industry Working Group charter talks about, in actual fact – and like you, I imagine that things like data formats between industries will continue to need translation and conversion. Koneki + Paho plus the overall M2M Industry Working Group thought at Eclipse is the broader story.

      To address a couple of your thoughts on timestamps and security… indeed, the messaging protocol itself doesn’t include timestamps, but that’s the kind of thing you might expect to include in a gateway application, or inside the data packet. When bandwidth is at a premium you might only want the device to emit the data value, for example. Some of these additional aspects are currently “exercises for the reader”🙂

      Security – the MQTT v3.1 specification includes the ability to provide a username/password on connection, with the server side expected to implement an authentication mechanism. For what it is worth, IBM’s WebSphere MQ Telemetry implementation provides both the ability to plug in any JAAS provider to do the authentication, and the ability to secure the connection using SSL. If you wanted to do things like data encryption, signing etc then that’s at the application level.

      Circling back to my point about where this protocol came from – it’s deliberately simple, and effective in that simplicity. Personally, I often think of it in terms of the UNIX philosophy of simple applications chained together to achieve a greater whole. You’re spot on with the observation that the M2M space has other requirements too, and I’m looking forward to those ongoing conversations!

      Like

  2. Pingback: IBM Java and C clients to be open source | Mosquitto

  3. Pingback: WebSphere MQ 7.1 is out – here’s why it’s cool… | The lost outpost

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s