Tag Archives: telemetry

A Smarter Planet needs lightweight messaging

One of the primary things I’ve been working on this year has been IBM’s new WebSphere MQ Telemetry product. I say “new”, of course, but the underlying technologies – WebSphere MQ itself, and the MQTT protocol which takes the messaging infrastructure down to the edge of the network and into embedded devices – have both been around, and totally solid, for a number of years already, but they have only recently formally been brought together into a single package. MQTT is short for MQ Telemetry Transport, and I wrote about it a couple of months ago in a post where I referred to it as a Smarter Planet protocol.

I’ve done quite a bit of travelling and talking to IBM customers and communities this year, and that’s recently been recognised and formalised into (part of) my new role in Hursley where I’m the WebSphere Messaging Community Lead. What does that mean? Well, the product part is in the first two words – I’m looking at the MQ family of products (something I’ll be writing about a bit more in the near future). The most important part, though is that third word – Community. My primary focus is working with, listening to, and helping to develop the community – and linking that back in to what we do in our labs. Community, for me, means people: developers, administrators, architects, partners, and the overall ecosystem that surrounds WebSphere MQ. We’ve got some great third-party sites out there, some brilliant content that gets published through IBM Redbooks and developerWorks, and our support teams blog about the topic, but we can always do better and I’m looking forward to finding ways of socialising our content and the materials produced by others.

A great first step towards that is our new IBM Expert network on Slideshare, which Adam Christensen wrote about recently, enabling us to share content as IBMers in a more “social” manner. I had a bunch of presentations up there already, but it was high time that I contributed some material more from my specialist technical subject area. I’ve now done so 🙂

This was a quick presentation I gave at an Apache Retreat that was held at IBM Hursley a couple of months ago. It’s a cut-down version of the full talk I’d usually give on the topic, tailored to the audience to keep it relevant to that community rather than diving into the enterprise part of the story and listing out case studies etc.. This is an important point, as the MQTT ecosystem is very much about the developer community and the opportunity to embed a reliable and lightweight messaging protocol into devices like smartphones, sensors, routers and edge-of-network boxes. It’s one of the reasons why IBM has published the specification for royalty-free implementation, and we’re seeing some exciting (and sometimes unexpected!) things happening as folks build their own client APIs. If your application or library is implementing that specification then your devices or applicaitons could, ultimately, bridge up into an Enterprise Service Bus running on the MQ infrastructure, and all the backend power that exists in clouds and enterprise datacentres today can start to do clever things with the data. Predictive analytics, visualisations, better prioritisation of resources… the reliability of transport for the data and the ability to get down to the smallest devices is vital.

[ related aside – shout-out to Nick for the lovely visual on slide 18 of this deck which I failed to credit within the presentation where I originally put it together in a bit of a rush. It’s from his page for the Arduino library for MQTT ]

There’s always more to say in this space, but I hope the presentation provides an overview of how the Smarter Planet story bridges the ideas of Service Oriented Architecture and The Internet of Things, and the part that MQTT plays within that. In the future you can expect to hear me talking more in this space, and I should be giving talks at Home Camp 3 in London in a couple of weeks’ time, as well as at LinuxConf Australia in Brisbane in January. I look forward to meeting more people and discussing the whole messaging story in more detail!

Advertisements

MQTT: the Smarter Planet Protocol

I’m at the SHARE conference in Boston this week. Earlier today I gave a talk about one of IBM’s significant software announcements of this year: the forthcoming WebSphere MQ Telemetry feature.

Observant viewers / readers / followers will know that I’ve been at IBM for nearly 10 years now and throughout that time I’ve spent a lot of time working with our WebSphere MQ (yes, aka, MQSeries) family of products. That family includes WebSphere Message Broker, and more recent extensions like the WMQ File Transfer Edition. Now, it has been formally extended to include first-class support for MQ Telemetry Transport, otherwise known as MQTT.

I want to spend a little bit of time talking about this, partly because I haven’t posted all that much here on my blog for a while now, and mostly because I’m hugely excited by this direct and long-overdue convergence of two of our messaging technologies. It plays directly into the Smarter Planet vision that IBM has been talking about over the course of the past couple of years, so it is worth understanding how this all fits. I apologise in advance that this may be a longer blog post than my average! I’ll also warn that I may come across as just a teensy bit of a fanboy… 🙂 as usual, remember that all thoughts expressed here represent my own opinions and understandings, and are not necessarily also those of my employer.

The Smarter Planet vision

So let’s kick off by asking: what on earth is this Smarter Planet concept that IBM has been discussing, and which you’ve likely seen promoted on posters and billboards as you travel around and go about your daily life? Well, it’s a very simple but very exciting idea, founded on three pillars: Instrumented, Interconnected, and Intelligent.

What is that all about then? Well, here it is, broken down, in my own words.

Firstly, there are a heck of a lot of devices around these days… from teeny tiny sensors and RFID tags which may stand alone in a system, through smartphones, GPS location-aware devices, laptops, and embedded systems. These devices typically have enough computing power to at least gather and transmit data, and some of them have enough to respond to requests to modify their behaviour. This is where we recognise that the world is increasingly Instrumented. I won’t bore you with statistics but here’s one of the numbers that tends to blow people away: there are 1 billion transistors in the world for every one of us. Secondly, these devices are nearly all natively “online” to some extent, and most will have an Internet address of their own (even if the connections are not always super-high bandwidth, always-on, or reliable). So, they are becoming Interconnected, either directly to one another across local networks, or indirectly via clouds. This is what you may have heard referred to as The Internet of Things. Finally, to complete the puzzle consider this: what if we could gather all of that data being emitted by these small, medium, or even large devices in real time… route it to where it is best intepreted… and make use of the vast computational resources we have in our enterprises or cities to respond, adjust, and make our environment, well… better? Through Intelligent systems with advanced analytics, we can start to do just that.

Speaking personally, this is what has always excited me about technology: the potential to use it to improve the way that we live, work, collaborate, communicate, and exist. I realise I’m entirely biased, and I’m not sure I could have articulated it at the time that I joined the company… but the sheer breadth of talent, technology and reach that is wrapped up in the IBM brand and company, makes it an exciting place to be as we begin to address these ideas.

The technology foundations

OK, so here we are. We’ve got existing enterprise or municipal systems running on (frankly) a spaghetti of platforms, but which encapsulate fundamental business processes and applications which help to run our day-to-day lives – from the mainframes running critical financial transactions in IMS and CICS, to retail supply chains which ensure goods get to the right places at the right time, social security and medical processes which (in theory!) just make things “work” in our interactions with various authorities, utility companies which are continuously providing gas, water, and other scarce resources… the list is endless, but you name it, over the past 30-40 years there will have been a computer system built to support it.

IBM has been a leader in the enterprise messaging space for a long time. The space I’m specifically talking about here is what is often called message-oriented middleware or MOM. If you’re still looking blank, that’s OK… messaging middleware is, fundamentally, supposed to be invisible to you, so I’m not totally surprised. WebSphere MQ has been the reliable messaging transport used by many of the Fortune 500 companies for ~15 years now. It runs on a ridiculous number of platforms, has a number of language bindings, a stable API which has been backward-compatible the whole time, and it has become the de facto lingua franca and way of gluing disparate applications together. As an added bonus, the wider family of products also includes a high-performance, robust, exceptionally flexible, any-to-any connectivity, transformation and routing engine (Enterprise Service Bus) called WebSphere Message Broker. I don’t have space to talk about WMQ or WMB in detail here right now, but I guess I know some stuff (…!), so, ask me another time 🙂

The piece that has been missing in all of this until recently is the ability to reliably connect the edges, the frontiers of the data network, to the existing systems that already understand what to do in various scenarios. In other words, we could start to take action based on some data, but we generally couldn’t collect that data in real time, and as a result of that our analysis also tended to be flawed and based on outdated information.

What next? Joining the dots

I kind of fibbed, just then. IBM has actually had a technology capable of living out on these little embedded devices for aaaaaages! It’s called MQTT, the MQ Telemetry Transport, and it has had a home over on mqtt.org for a number of years. In fact, if you take a look, you’ll find that a bunch of people have implemented their own language bindings and small footprint message brokers already. It has existed inside a number of IBM products, and it was supported in WebSphere Message Broker up until version 6.1 via what were called the SCADA nodes. The name has been fiddled with a few times, and it hasn’t been a massively prominent part of the portfolio, so I’ve often found that folks haven’t heard of it. It’s very, very cool though.

What’s so special about MQTT? Well, it was specifically designed for constrained environments, with limited processing capabilities, potentially very tiny memory capacities, and fragile, unreliable, high-latency, low-bandwidth networks. As a result, it doesn’t have to have the reliable transactional qualities of service you might find in an enterprise messaging solution (although it can support those, too). The design principles do however bring in some massive advantages: it’s incredibly simple, easy-to-learn, and can be very fast and effective, with many thousands of lightweight clients supported by a single server.

On July 6th, IBM announced that MQTT is being added to WebSphere MQ as a first-class protocol.

[warning: the next couple of paragraphs describe finer details which are
potentially subject to change prior to final release!]

Alongside the existing MQ channels, it will now be possible to define Telemetry channels to enable the connection of one or many MQTT clients (by many read, like, really, LOTS, but wait for the performance SupportPac for full details!). Not only that… here’s what I believe is the really sweet part: MQ and MQTT applications will be able to interoperate, so messages published to a topic by an MQ application will be able to be received and consumed by MQTT clients, and vice versa. Nearly instant interoperability between existing enterprise applications and the edge of the network, if you need it. We’ve got security via SSL and JAAS, and we’ve got some simply beautiful tooling integration with the WebSphere MQ Explorer, including a nice test client.

As the diagram shows, the new feature requires a recent version of WebSphere MQ – I’m currently running 7.0.1.2 or better, but you’ll need to check the final requirements at release time. The new telemetry channels enable simple MQTT clients to be connected directly to a queue manager via a component which is internally called MQXR (for MQ eXtended Reach). The WebSphere MQ Telemetry Daemon for Devices can act as a concentrator, connecting up to a queue manager via telemetry channels if required (think, multiplexing connections at the edge of the enterprise). If you are using one of the other, richer clients that already contain MQTT or an MQTT-based technology like Microbroker (e.g. Lotus Expeditor), they will seamlessly connect up, too. You could even use a tool such as the third-party mosquitto broker to connect into a full MQ network.

My friend James Governor posited not very long ago that “WebSphere MQ won’t ever be a pervasive play“. James is an extremely smart guy and I take his opinions very seriously, but with the extension of WMQ to the web via the HTTP bridge introduced in version 7, and with MQTT joining the product foundation, I’d say that WMQ absolutely has the scope to be pervasive. I’ve been working with the team adding MQTT to WMQ for the past few months, and I have to say that the integration is looking really, really nice. I don’t want to bore with details in this post – I spent an hour talking about them at a conference today! – but I can say that it’s trivial to build a lightweight MQTT client, publish data, have that delivered into WebSphere MQ, transformed from a simple and tiny bytes message into meaningful XML in WebSphere Message Broker, and then routed on to any number of backend systems. There are customers implementing very exciting solutions in healthcare, transport, and energy to name just a few, and they are using MQTT in those solutions today. Plus, our very own lovable propellerhead Andy Stanford-Clark runs his home automation and mouse-slaughtering system on the protocol that he invented (I have a similar, but significantly less impressive and less all-encompassing system that I’ve written about before). Now is the time for these pieces to come together.

Keep watching for more as the release approaches in the next few weeks. MQTT really could be called the Smarter Planet Protocol, and I’m looking forward to find out what kinds of things we can use it for as we collectively create more “smart” solutions.

Update: if you’re interested, I will be giving a similar talk for the Global WebSphere User Group webcast series on August 18th. Additionally, if you are a member of SHARE, my slides for this talk should be available via the schedule planner shortly.