MQTT: the Smarter Planet Protocol

I’m at the SHARE conference in Boston this week. Earlier today I gave a talk about one of IBM’s significant software announcements of this year: the forthcoming WebSphere MQ Telemetry feature.

Observant viewers / readers / followers will know that I’ve been at IBM for nearly 10 years now and throughout that time I’ve spent a lot of time working with our WebSphere MQ (yes, aka, MQSeries) family of products. That family includes WebSphere Message Broker, and more recent extensions like the WMQ File Transfer Edition. Now, it has been formally extended to include first-class support for MQ Telemetry Transport, otherwise known as MQTT.

I want to spend a little bit of time talking about this, partly because I haven’t posted all that much here on my blog for a while now, and mostly because I’m hugely excited by this direct and long-overdue convergence of two of our messaging technologies. It plays directly into the Smarter Planet vision that IBM has been talking about over the course of the past couple of years, so it is worth understanding how this all fits. I apologise in advance that this may be a longer blog post than my average! I’ll also warn that I may come across as just a teensy bit of a fanboy… :-) as usual, remember that all thoughts expressed here represent my own opinions and understandings, and are not necessarily also those of my employer.

The Smarter Planet vision

So let’s kick off by asking: what on earth is this Smarter Planet concept that IBM has been discussing, and which you’ve likely seen promoted on posters and billboards as you travel around and go about your daily life? Well, it’s a very simple but very exciting idea, founded on three pillars: Instrumented, Interconnected, and Intelligent.

What is that all about then? Well, here it is, broken down, in my own words.

Firstly, there are a heck of a lot of devices around these days… from teeny tiny sensors and RFID tags which may stand alone in a system, through smartphones, GPS location-aware devices, laptops, and embedded systems. These devices typically have enough computing power to at least gather and transmit data, and some of them have enough to respond to requests to modify their behaviour. This is where we recognise that the world is increasingly Instrumented. I won’t bore you with statistics but here’s one of the numbers that tends to blow people away: there are 1 billion transistors in the world for every one of us. Secondly, these devices are nearly all natively “online” to some extent, and most will have an Internet address of their own (even if the connections are not always super-high bandwidth, always-on, or reliable). So, they are becoming Interconnected, either directly to one another across local networks, or indirectly via clouds. This is what you may have heard referred to as The Internet of Things. Finally, to complete the puzzle consider this: what if we could gather all of that data being emitted by these small, medium, or even large devices in real time… route it to where it is best intepreted… and make use of the vast computational resources we have in our enterprises or cities to respond, adjust, and make our environment, well… better? Through Intelligent systems with advanced analytics, we can start to do just that.

Speaking personally, this is what has always excited me about technology: the potential to use it to improve the way that we live, work, collaborate, communicate, and exist. I realise I’m entirely biased, and I’m not sure I could have articulated it at the time that I joined the company… but the sheer breadth of talent, technology and reach that is wrapped up in the IBM brand and company, makes it an exciting place to be as we begin to address these ideas.

The technology foundations

OK, so here we are. We’ve got existing enterprise or municipal systems running on (frankly) a spaghetti of platforms, but which encapsulate fundamental business processes and applications which help to run our day-to-day lives – from the mainframes running critical financial transactions in IMS and CICS, to retail supply chains which ensure goods get to the right places at the right time, social security and medical processes which (in theory!) just make things “work” in our interactions with various authorities, utility companies which are continuously providing gas, water, and other scarce resources… the list is endless, but you name it, over the past 30-40 years there will have been a computer system built to support it.

IBM has been a leader in the enterprise messaging space for a long time. The space I’m specifically talking about here is what is often called message-oriented middleware or MOM. If you’re still looking blank, that’s OK… messaging middleware is, fundamentally, supposed to be invisible to you, so I’m not totally surprised. WebSphere MQ has been the reliable messaging transport used by many of the Fortune 500 companies for ~15 years now. It runs on a ridiculous number of platforms, has a number of language bindings, a stable API which has been backward-compatible the whole time, and it has become the de facto lingua franca and way of gluing disparate applications together. As an added bonus, the wider family of products also includes a high-performance, robust, exceptionally flexible, any-to-any connectivity, transformation and routing engine (Enterprise Service Bus) called WebSphere Message Broker. I don’t have space to talk about WMQ or WMB in detail here right now, but I guess I know some stuff (…!), so, ask me another time :-)

The piece that has been missing in all of this until recently is the ability to reliably connect the edges, the frontiers of the data network, to the existing systems that already understand what to do in various scenarios. In other words, we could start to take action based on some data, but we generally couldn’t collect that data in real time, and as a result of that our analysis also tended to be flawed and based on outdated information.

What next? Joining the dots

I kind of fibbed, just then. IBM has actually had a technology capable of living out on these little embedded devices for aaaaaages! It’s called MQTT, the MQ Telemetry Transport, and it has had a home over on mqtt.org for a number of years. In fact, if you take a look, you’ll find that a bunch of people have implemented their own language bindings and small footprint message brokers already. It has existed inside a number of IBM products, and it was supported in WebSphere Message Broker up until version 6.1 via what were called the SCADA nodes. The name has been fiddled with a few times, and it hasn’t been a massively prominent part of the portfolio, so I’ve often found that folks haven’t heard of it. It’s very, very cool though.

What’s so special about MQTT? Well, it was specifically designed for constrained environments, with limited processing capabilities, potentially very tiny memory capacities, and fragile, unreliable, high-latency, low-bandwidth networks. As a result, it doesn’t have to have the reliable transactional qualities of service you might find in an enterprise messaging solution (although it can support those, too). The design principles do however bring in some massive advantages: it’s incredibly simple, easy-to-learn, and can be very fast and effective, with many thousands of lightweight clients supported by a single server.

On July 6th, IBM announced that MQTT is being added to WebSphere MQ as a first-class protocol.

[warning: the next couple of paragraphs describe finer details which are
potentially subject to change prior to final release!]

Alongside the existing MQ channels, it will now be possible to define Telemetry channels to enable the connection of one or many MQTT clients (by many read, like, really, LOTS, but wait for the performance SupportPac for full details!). Not only that… here’s what I believe is the really sweet part: MQ and MQTT applications will be able to interoperate, so messages published to a topic by an MQ application will be able to be received and consumed by MQTT clients, and vice versa. Nearly instant interoperability between existing enterprise applications and the edge of the network, if you need it. We’ve got security via SSL and JAAS, and we’ve got some simply beautiful tooling integration with the WebSphere MQ Explorer, including a nice test client.

As the diagram shows, the new feature requires a recent version of WebSphere MQ – I’m currently running 7.0.1.2 or better, but you’ll need to check the final requirements at release time. The new telemetry channels enable simple MQTT clients to be connected directly to a queue manager via a component which is internally called MQXR (for MQ eXtended Reach). The WebSphere MQ Telemetry Daemon for Devices can act as a concentrator, connecting up to a queue manager via telemetry channels if required (think, multiplexing connections at the edge of the enterprise). If you are using one of the other, richer clients that already contain MQTT or an MQTT-based technology like Microbroker (e.g. Lotus Expeditor), they will seamlessly connect up, too. You could even use a tool such as the third-party mosquitto broker to connect into a full MQ network.

My friend James Governor posited not very long ago that “WebSphere MQ won’t ever be a pervasive play“. James is an extremely smart guy and I take his opinions very seriously, but with the extension of WMQ to the web via the HTTP bridge introduced in version 7, and with MQTT joining the product foundation, I’d say that WMQ absolutely has the scope to be pervasive. I’ve been working with the team adding MQTT to WMQ for the past few months, and I have to say that the integration is looking really, really nice. I don’t want to bore with details in this post – I spent an hour talking about them at a conference today! – but I can say that it’s trivial to build a lightweight MQTT client, publish data, have that delivered into WebSphere MQ, transformed from a simple and tiny bytes message into meaningful XML in WebSphere Message Broker, and then routed on to any number of backend systems. There are customers implementing very exciting solutions in healthcare, transport, and energy to name just a few, and they are using MQTT in those solutions today. Plus, our very own lovable propellerhead Andy Stanford-Clark runs his home automation and mouse-slaughtering system on the protocol that he invented (I have a similar, but significantly less impressive and less all-encompassing system that I’ve written about before). Now is the time for these pieces to come together.

Keep watching for more as the release approaches in the next few weeks. MQTT really could be called the Smarter Planet Protocol, and I’m looking forward to find out what kinds of things we can use it for as we collectively create more “smart” solutions.

Update: if you’re interested, I will be giving a similar talk for the Global WebSphere User Group webcast series on August 18th. Additionally, if you are a member of SHARE, my slides for this talk should be available via the schedule planner shortly.

About these ads

21 responses to “MQTT: the Smarter Planet Protocol

  1. Interesting article. I try to go through MQTT more.

  2. Are the slides from your talk available somewhere? (Externally or internal to IBM?)

    • They should be available via the SHARE conference site in the next couple of days but I’m not sure if that is open to non-partcipants. You’ll find them on cattail inside the firewall.

  3. Pingback: OS X mosquitto “bites”… | The lost outpost

  4. For heavy stuff there is XMPP. (security, authentication, access modes, extension). MQTT is nice for resource limited devices, would be great a bridge between both, keeping MQTT simple as it is.

    • Hi Salvador, and thanks for your comment. If you are referring to my “heavy lifting” comment I was really talking about enterprise messaging between applications, a space typically occupied by protocols and transports like WebSphere MQ and JMS. XMPP does indeed have the qualities you refer to but it’s more of a lightweight presence-based messaging protocol and typically more user-visible, in my opinion. I would not use it for transforming enterprise message data, or for integrating to databases, files and web service backends (that’s what you’d use an Enterprise Service Bus for). As for creating bridges – well – there are MQTT implementations out there, and I suspect that it shouldn’t be too hard… I see that someone is already using the Python mosquitto interface to do desktop notifications, for example…

  5. Hi Andy,
    Thanks for sharing this. I heard about this microbroker and MQTTa few months back. Now loved to try that. I have couple of doubts.
    What is the maximum number of clients one MQ queue manager can cope up, provided MQ is on AIX?
    Will there be any concepts for thread pooling like there in SCADA nodes of WMB?

    • Hi Manu, thanks for the comment, sorry for the delay in responding!
      Firstly, the Telemetry feature is initially only going to be on Windows and Linux (64-bit), no AIX to start out with, but other platforms are being prioritised for future consideration. A performance report is available and you’ll see that a range of numbers of clients have been tested – 50,000 clients attached to a Linux system is one metric quoted. One other point is that topologies are flexible, so you wouldn’t necessarily attached 50k clients to a single queue manager anyway, you may use one or more device daemons (small brokers equivalent to the Really Small Message Broker) to act as edge-of-network concentrators, so this is a slightly wider discussion.

      As far as thread pooling – MQXR, the component which supports Telemetry channels, is optimised to reduce the numbers of threads etc. used and there’s no user-configurable properties in respect to that.

  6. Andy & Dave

    Thanks for replies. I saw the performance report . That is just wonderful !!!
    Do we have this support pack released for MQ,?
    I could see only performance report.

  7. Pingback: Product updates and new releases | The lost outpost

  8. Hi Andy, Been having a look at RSMB and have a question about bridging from it to MQ. Is this possible in MQ lower than MQ 7? We use WMB at work (6.1) which I know has SCADA nodes ? Is it possible to bridge from RSMB to MB via a flow using these? If so I assume I have to set up some connection in RSMB (in the broker.cfg) file but not sure what exactly….Any help would be really useful…
    Pete

    • Hi Pete, thanks for the question. There are a few aspects to the answer, so bear with me!

      The Telemetry feature does pre-req WMQ v7.0.1.3 so no, you can’t go directly into MQ from an MQTT client or application like RSMB where MQ is at a lower level than that.

      The SCADA nodes are indeed in version 6.1 of WMB and they do support MQTT connections, however I’m honestly not sure whether it is technically possible to connect in to those from RSMB (although reading the doc it does appear that a bridge to WMB *is* available so I’m going to go with a tentative “yes”). One of the aspects of moving MQTT into MQ (and, in v7, out of WMB) is that by doing so the scalability has been dramatically increased, so in general I’d recommend going down a v7 route where possible. In that case the principle would be that you’d connect from your client app and potentially RSMB, to WMQ Telemetry on MQ, and then take regular MQ messages into WMB for processing as normal in a flow.

      I’ll explain the bridging connection briefly – but a) I’ve not personally tried this connecting in to SCADA nodes and b) again, I would not recommend that you do this in a production scenario as that is not the direction for the products. The essence here is that RSMB supports the ability to bridge to other messaging servers running MQTT. Search for “bridge” in the gettingstarted.htm file shipped in the doc/ folder of the RSMB distribution.

      The Really Small Message Broker has a bridge which enables the broker to connect to another MQTT-capable broker, and exchange messages with it, in both directions. For more sophisticated architectures, such as peer-to-peer networks, the bridge allows multiple concurrent connections to several brokers.

      You’ll want to look at the connection, addresses, and topic configuration settings in the broker.cfg file.

  9. Andy,
    Thanks for the info and I’ve got a test working. I’ve created a flow with a SCADAInput node that publishes any incoming message on the topic that comes in. This flow is listening on port 1885. I’ve started an RSMB (on std port 1883) with a broker.cfg file with following :-
    connection MB6Connection
    addresses 127.0.0.1:1885
    topic power out “” remote/

    NOTE:- the topic “power” will map to topic “remote/power” on MB.

    I then have used the RSMB stdinpub to publish a message to the RSMB on topic ‘power’. I’ve got a RSMB stdoutsub connected to the RSMB broker and subscribing to topic power and another one connected to the MB (flow on 1885) subscribing to topic “remote/power” (I’ve mapped these topics in the broker.cfg file). It’s all connecting and I can see the messages on both brokers! NEAT!
    Thanks for the help!

  10. Pingback: Awards season, and good causes | The lost outpost

  11. Pingback: A Smarter Planet needs lightweight messaging | The lost outpost

  12. Pingback: WebSphere MQ and MQ Telemetry | IBM Software Training

  13. Pingback: What a week for MQTT! | The lost outpost

  14. Pingback: MQTT goes free – a personal Q&A | The lost outpost

  15. Pingback: WebSphere MQ 7.1 is out – here’s why it is cool… | The lost outpost

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s