Tag Archives: smarter planet

MQTT: the Smarter Planet Protocol

I’m at the SHARE conference in Boston this week. Earlier today I gave a talk about one of IBM’s significant software announcements of this year: the forthcoming WebSphere MQ Telemetry feature.

Observant viewers / readers / followers will know that I’ve been at IBM for nearly 10 years now and throughout that time I’ve spent a lot of time working with our WebSphere MQ (yes, aka, MQSeries) family of products. That family includes WebSphere Message Broker, and more recent extensions like the WMQ File Transfer Edition. Now, it has been formally extended to include first-class support for MQ Telemetry Transport, otherwise known as MQTT.

I want to spend a little bit of time talking about this, partly because I haven’t posted all that much here on my blog for a while now, and mostly because I’m hugely excited by this direct and long-overdue convergence of two of our messaging technologies. It plays directly into the Smarter Planet vision that IBM has been talking about over the course of the past couple of years, so it is worth understanding how this all fits. I apologise in advance that this may be a longer blog post than my average! I’ll also warn that I may come across as just a teensy bit of a fanboy… 🙂 as usual, remember that all thoughts expressed here represent my own opinions and understandings, and are not necessarily also those of my employer.

The Smarter Planet vision

So let’s kick off by asking: what on earth is this Smarter Planet concept that IBM has been discussing, and which you’ve likely seen promoted on posters and billboards as you travel around and go about your daily life? Well, it’s a very simple but very exciting idea, founded on three pillars: Instrumented, Interconnected, and Intelligent.

What is that all about then? Well, here it is, broken down, in my own words.

Firstly, there are a heck of a lot of devices around these days… from teeny tiny sensors and RFID tags which may stand alone in a system, through smartphones, GPS location-aware devices, laptops, and embedded systems. These devices typically have enough computing power to at least gather and transmit data, and some of them have enough to respond to requests to modify their behaviour. This is where we recognise that the world is increasingly Instrumented. I won’t bore you with statistics but here’s one of the numbers that tends to blow people away: there are 1 billion transistors in the world for every one of us. Secondly, these devices are nearly all natively “online” to some extent, and most will have an Internet address of their own (even if the connections are not always super-high bandwidth, always-on, or reliable). So, they are becoming Interconnected, either directly to one another across local networks, or indirectly via clouds. This is what you may have heard referred to as The Internet of Things. Finally, to complete the puzzle consider this: what if we could gather all of that data being emitted by these small, medium, or even large devices in real time… route it to where it is best intepreted… and make use of the vast computational resources we have in our enterprises or cities to respond, adjust, and make our environment, well… better? Through Intelligent systems with advanced analytics, we can start to do just that.

Speaking personally, this is what has always excited me about technology: the potential to use it to improve the way that we live, work, collaborate, communicate, and exist. I realise I’m entirely biased, and I’m not sure I could have articulated it at the time that I joined the company… but the sheer breadth of talent, technology and reach that is wrapped up in the IBM brand and company, makes it an exciting place to be as we begin to address these ideas.

The technology foundations

OK, so here we are. We’ve got existing enterprise or municipal systems running on (frankly) a spaghetti of platforms, but which encapsulate fundamental business processes and applications which help to run our day-to-day lives – from the mainframes running critical financial transactions in IMS and CICS, to retail supply chains which ensure goods get to the right places at the right time, social security and medical processes which (in theory!) just make things “work” in our interactions with various authorities, utility companies which are continuously providing gas, water, and other scarce resources… the list is endless, but you name it, over the past 30-40 years there will have been a computer system built to support it.

IBM has been a leader in the enterprise messaging space for a long time. The space I’m specifically talking about here is what is often called message-oriented middleware or MOM. If you’re still looking blank, that’s OK… messaging middleware is, fundamentally, supposed to be invisible to you, so I’m not totally surprised. WebSphere MQ has been the reliable messaging transport used by many of the Fortune 500 companies for ~15 years now. It runs on a ridiculous number of platforms, has a number of language bindings, a stable API which has been backward-compatible the whole time, and it has become the de facto lingua franca and way of gluing disparate applications together. As an added bonus, the wider family of products also includes a high-performance, robust, exceptionally flexible, any-to-any connectivity, transformation and routing engine (Enterprise Service Bus) called WebSphere Message Broker. I don’t have space to talk about WMQ or WMB in detail here right now, but I guess I know some stuff (…!), so, ask me another time 🙂

The piece that has been missing in all of this until recently is the ability to reliably connect the edges, the frontiers of the data network, to the existing systems that already understand what to do in various scenarios. In other words, we could start to take action based on some data, but we generally couldn’t collect that data in real time, and as a result of that our analysis also tended to be flawed and based on outdated information.

What next? Joining the dots

I kind of fibbed, just then. IBM has actually had a technology capable of living out on these little embedded devices for aaaaaages! It’s called MQTT, the MQ Telemetry Transport, and it has had a home over on mqtt.org for a number of years. In fact, if you take a look, you’ll find that a bunch of people have implemented their own language bindings and small footprint message brokers already. It has existed inside a number of IBM products, and it was supported in WebSphere Message Broker up until version 6.1 via what were called the SCADA nodes. The name has been fiddled with a few times, and it hasn’t been a massively prominent part of the portfolio, so I’ve often found that folks haven’t heard of it. It’s very, very cool though.

What’s so special about MQTT? Well, it was specifically designed for constrained environments, with limited processing capabilities, potentially very tiny memory capacities, and fragile, unreliable, high-latency, low-bandwidth networks. As a result, it doesn’t have to have the reliable transactional qualities of service you might find in an enterprise messaging solution (although it can support those, too). The design principles do however bring in some massive advantages: it’s incredibly simple, easy-to-learn, and can be very fast and effective, with many thousands of lightweight clients supported by a single server.

On July 6th, IBM announced that MQTT is being added to WebSphere MQ as a first-class protocol.

[warning: the next couple of paragraphs describe finer details which are
potentially subject to change prior to final release!]

Alongside the existing MQ channels, it will now be possible to define Telemetry channels to enable the connection of one or many MQTT clients (by many read, like, really, LOTS, but wait for the performance SupportPac for full details!). Not only that… here’s what I believe is the really sweet part: MQ and MQTT applications will be able to interoperate, so messages published to a topic by an MQ application will be able to be received and consumed by MQTT clients, and vice versa. Nearly instant interoperability between existing enterprise applications and the edge of the network, if you need it. We’ve got security via SSL and JAAS, and we’ve got some simply beautiful tooling integration with the WebSphere MQ Explorer, including a nice test client.

As the diagram shows, the new feature requires a recent version of WebSphere MQ – I’m currently running 7.0.1.2 or better, but you’ll need to check the final requirements at release time. The new telemetry channels enable simple MQTT clients to be connected directly to a queue manager via a component which is internally called MQXR (for MQ eXtended Reach). The WebSphere MQ Telemetry Daemon for Devices can act as a concentrator, connecting up to a queue manager via telemetry channels if required (think, multiplexing connections at the edge of the enterprise). If you are using one of the other, richer clients that already contain MQTT or an MQTT-based technology like Microbroker (e.g. Lotus Expeditor), they will seamlessly connect up, too. You could even use a tool such as the third-party mosquitto broker to connect into a full MQ network.

My friend James Governor posited not very long ago that “WebSphere MQ won’t ever be a pervasive play“. James is an extremely smart guy and I take his opinions very seriously, but with the extension of WMQ to the web via the HTTP bridge introduced in version 7, and with MQTT joining the product foundation, I’d say that WMQ absolutely has the scope to be pervasive. I’ve been working with the team adding MQTT to WMQ for the past few months, and I have to say that the integration is looking really, really nice. I don’t want to bore with details in this post – I spent an hour talking about them at a conference today! – but I can say that it’s trivial to build a lightweight MQTT client, publish data, have that delivered into WebSphere MQ, transformed from a simple and tiny bytes message into meaningful XML in WebSphere Message Broker, and then routed on to any number of backend systems. There are customers implementing very exciting solutions in healthcare, transport, and energy to name just a few, and they are using MQTT in those solutions today. Plus, our very own lovable propellerhead Andy Stanford-Clark runs his home automation and mouse-slaughtering system on the protocol that he invented (I have a similar, but significantly less impressive and less all-encompassing system that I’ve written about before). Now is the time for these pieces to come together.

Keep watching for more as the release approaches in the next few weeks. MQTT really could be called the Smarter Planet Protocol, and I’m looking forward to find out what kinds of things we can use it for as we collectively create more “smart” solutions.

Update: if you’re interested, I will be giving a similar talk for the Global WebSphere User Group webcast series on August 18th. Additionally, if you are a member of SHARE, my slides for this talk should be available via the schedule planner shortly.

IMPACT – work, play, product announcements

I’m not going to run down all of the announcements that came out during the first day of IMPACT yesterday – you can take a look at Sandy Carter’s video summary for some of the key ones, like Smart Work, and the new WebSphere CloudBurst appliance (it’s very purple – and I want one!). There is a slew of cool new stuff being announced and coming out over the coming year.

Sitting in the keynote yesterday morning I was feeling as though several of the worlds that I live in / things I’m interested in were really coming together. It’s an exciting time. For example, we had:

As an “SOA event” it’s easy for me to see it as all to do with IBM’s WebSphere brand of software alone, since that’s what I’ve work been working on for almost a decade now. The truth is that IMPACT spans everything that IBM does, particularly in software. We are talking collaboration and social software (Lotus); monitoring and intelligent management (Tivoli); modelling and productive development (Rational); and sophisticated data analytics (Information Management). It’s a real showcase for the broad reach and range of IBM’s software portfolio.

Highlight of the day for me was absolutely nothing to do with IBM software – it was the opportunity to hear Jeremiah Owyang from Forrester speak on the Future of the Social Web. Jeremiah clearly understands this stuff in all the many facets that have been expressed by other people I follow, like Don Tapscott – I did recommend his book already, right…? 😉 I am glad to have made the connection with @jowyang, since I’ve read and followed much of his work in the past couple of years, so it was a bit of a treat to hear him speaking. I tried not to take over the Q&A completely 🙂

Oh, and the play part? Billy Crystal was the compere for the morning session and kept us amused despite the early start – and there were plenty of opportunities to catch up with friends and Twitter connections at the networking sessions. I call that a win. Oh, and wait a second, I almost forgot – we had a tweetup yesterday, with a couple more scheduled later in the week. Looking forward to them.

I’ll try to post another IMPACT-related update later in the week. Photos are on Flickr, and there is a very active ongoing discussion in the #ibmimpact hashtag on Twitter.

Blue Fusion, the 2009 edition

One of the first Hursley-related things I wrote about here and on the eightbar blog back in 2006 was how much I enjoy helping with our annual schools event for National Science and Engineering Week in the UK – Blue Fusion (the event website has gone AWOL at the moment but here’s a link to the press release).

This year was no exception, and referring back to my old blog entries it turns out that this is now the fifth year that I’ve been a volunteer. Unfortunately I only had room in my schedule to spend one day helping this time around, so I chose to host a school for the day rather than spending all day on a single activity (that way, I got to see all of the different things we had on offer).

So, yesterday I had the pleasure of hosting six intelligent and polite students from Malvern St James School and their teachers – they had travelled a fair distance to come to the event, but despite the early start I think they did really well.

I won’t go into too much detail and spoil the fun for people who might read this but have not yet taken part in this week’s event, but I think we had some great activities on offer. I twittered our way through a few of them. My own personal favourite was a remote surgery activity. You can’t see much in this image (it was a dark room) but the students basically had a “body” inside a box with some remote cameras to guide their hands around and had to identify organs and foreign objects.

img_3774

There was also some interesting application of visual technology / tangible interfaces – a genetics exercise using LEGO bricks and a camera which identified gene strands, and an energy planning exercise which used Reactivision-style markers to identify where power stations had been placed on a map (sort of similar to what we built in SLorpedo at Hackday a couple of years ago). We also had some logic puzzles to solve, built a, err… “typhoon-proof” (ahem) tower, simulated a computer processor, and commanded a colony of ants in a battle for survival against the other school teams.

Things I learned

  1. Facebook (not Bebo) is now where it’s at.
  2. If a tornado is coming, get out of the way or into a safe room.
  3. Girls are much better than boys at listening to multiple streams of conversation (actually I think I worked this out a long time ago!).

A now, some notes just for my team…

Here are links to a few of the other things we talked about during the day:

And most importantly, here’s the evidence that we started off in first place 🙂 and I think you were an awesome team throughout. Well done, it was brilliant spending the day with you.

img_3772

Smart Planet

IBM’s Chairman and CEO, Sam Palmisano, has been speaking to the Council of Foreign Relations in New York today. He’s been discussing how the planet is getting smarter:

These collective realizations have reminded us that we are all now connected – economically, technically and socially.  But we’re also learning that being connected is not sufficient.  Yes, the world continues to get “flatter.”  And yes, it continues to get smaller and more interconnected.  But something is happening that holds even greater potential.  In a word, our planet is becoming smarter.

In the speech, Sam talks about how the world has become instrumented, more interconnected, and devices more intelligent. And he talks about how the current world crises – ecological, financial, and others – represent an opportunity for change. The next step for the globally integrated economy is a globally integrated and intelligent economy and society.

Some of the problems and solutions that are being mentioned are interesting.

67 per cent of all electrical energy is lost due to inefficient power generation and grid management… utilities in the U.S., Denmark, Australia and Italy are now building digital grids to monitor the energy system in real time.

Congested roadways in the U.S. cost $78 billion annually in wasted hours and gas… Stockholm’s new smart toll system has resulted in 22 percent less traffic, a 12 to 40 percent drop in emissions and 40,000 additional daily users of the public transport system

This is exciting for me on many levels. Let me step up through them.

As regular readers will know, I’ve become increasingly interested in pervasive computing and home automation. The little “Current Cost craze” that has swept through my group of friends at work could be seen as a mark of the individual interest in applying technology in a smarter way. I’m excited that this has widened out to a group of folks who are supporting Chris Dalby’s Home Camp idea in London later this month.

Secondly, beyond this individual approach, it ties in to some of what I heard at the recent Web 2.0 Expo in Berlin… people talking about the opportunity for technology to change the way things work, from Tim O’Reilly’s keynote on the way forward for Internet technology and innovative thinking, to Tom Raftery’s brilliant GreenMonk pitch on Electricity 2.0.

 

STOP Studying the world. START Transforming it.

Finally, and most broadly, it’s a hopeful vision which resonates when lately, things do sometimes appear bleak.

 

Technology can help society. Let’s go and make it happen.

New York Times article on Sam Palmisano’s speech

YouTube Smarter Planet videos

Update: a couple more links, if you want to get involved…

Smarter Planet on FriendFeed

Smarter Planet on Tumblr