Tag Archives: open source

MQTT goes free – a personal Q&A

There has been a lot of coverage over the past couple of days of some exciting announcements that I’ve been involved with at work. I’ve spent the past three days at EclipseCon Europe 2011, which doubled as the 10th birthday celebration for the Eclipse initiative. It was a funny feeling, because Eclipse started just a few weeks after I first joined IBM, and although I’ve used it and watch it “grow up”, I’ve never done EclipseCon before. The reason I’ve been out there for three days this time (as a WebSphere Messaging guy rather than a Rational tooling or build person, for example) was to get involved with activities around these announcements.

It’s all about machine-to-machine (or M2M) communications, Smarter Planet, and the Internet of Things.

Before I dive in to this, a few clarifications. First, I’m being described in a couple of news stories as “an IBM distinguished engineer”, and whilst I wish that was true, I’ve yet to ascend to those heights! Also, there are various numbers being quoted – note that the figures in the press release were not invented by IBM, the headline number of an expected 50 billion connected devices by 2020 comes from a recent study conducted by Ericsson AB. Oh, and this isn’t about a “new” protocol – MQTT has been in use since 1999.

The other clarification is that some articles seem to suggest that IBM is out to create some kind of new, alternative, Web – that’s not what has been announced, and I’m certainly not aware of any such plan! It’s about connecting “things” – sensors, mobile devices, embedded systems, even small appliances or medical devices for example – to the Web and the associated platform and ecosystem of technologies, not about reinventing or recreating them. I’m personally a huge fan of the Web as a platform 🙂

Oh, and of course, the obligatory “all opinions expressed are my own” – this is my understanding of where things are going, although of course I’m talking about events I’m directly involved in!

So what is this all about?

Two things.

1. On Nov 2, IBM, Eurotech, Sierra Wireless and Eclipse formed a new M2M Industry Working Group at Eclipse. Sierra had already started the “Koneki” project at Eclipse to work on M2M tools, and the Working Group will look at a range of topics together, such as M2M tooling, software components, open communication and messaging protocols, data formats, and APIs.

2. On Nov 3, IBM and Eurotech announced the donation of their C and Java clients for MQTT to a new Eclipse project called “Paho” which is under proposal in the incubator – with code expected to hit the repository within the next couple of months. MQTT is being given to Eclipse to live within the M2M ecosystem that is emerging there, and to provide an avenue for adoption of the protocol as a more pervasive standard for connected devices.

How is that news? Isn’t MQTT already open / free?

Technically… kinda, sorta 🙂

The MQTT specification has been published under a royalty-free license for some time, and that has led to a fantastic community contributing a range of different projects. IBM and Eurotech took this approach from early on, because it wouldn’t have been possible to compile and support code on every embedded platform that might come along – far simpler to set the protocol free.

Initially the specification was hidden away in the WebSphere Message Broker documentation, but last year it was republished, moved to a new home on developerWorks, and the license was clarified.

In August, IBM and Eurotech announced their intention to take MQTT to a standards organisation. The specific organisation has not yet been finalised, but this is also an important step in ensuring that MQTT is not “just” an IBM protocol, but something of general use which the community can feel comfortable with. If you’d like to join that discussion then there’s a Get Involved page on the mqtt.org community site.

The missing piece was code – a reference implementation, if you like. That’s one reason why the Eclipse Paho announcement is significant.

Why else is this significant?

Well, here are some of my musings on that one:

  • it shows IBM is serious, by committing code and open sourcing it (as with the original Eclipse donation in 2001);
  • the M2M Industry Working Group exists to foster the discussion in this space;
  • it makes high-quality reference Java and C client implementations freely available in source form, with a good Java implementation something that has been particularly lacking;
  • it creates an opportunity for Eclipse projects to use MQTT, and to develop tools on top of it.

The press release and Paho project proposals aren’t clear (to me) – what exactly is being donated?

IBM is seeding Eclipse Paho with C and Java client implementations of MQTT. Eurotech is donating a framework and sample applications which device and client developers can use when integrating and testing messaging components.

Why C and Java clients (aren’t they “dying” languages?) Where’s my Perl and Ruby code?!

IBM had previously made some C and Java code available in some SupportPacs, but those are outdated and the license for reuse was never clear.

It’s important to realise that this stuff came from the embedded world of 10 (and more) years ago, and continues to be applied in that industrial space. That category of device typically runs some kind of realtime Java-based OS, or a Linux-based or other runtime with a GCC toolchain for the CPU in question. C and Java are genuinely the most useful implementations to get out there. Oh, and on that “those old languages” thing – I think you’ll find they are very widely used (Android, iOS etc run variants of sorts, most non-web app development is likely to be in one or the other).

We’re very fortunate that clients libraries for a wide range of languages already exist thanks to the MQTT community – see the list at mqtt.org!

Hold on… don’t we need a broker / server / gateway?

Yes. But, one step at a time! 🙂

There are brokers available for free today, either as precompiled binaries or as full Open Source implementations, so this is not a dead end from day one.

The Paho project scope outlines the intention to add a broker to the project in the future, and to host an M2M sandbox for developers as well. That is where we are today, and this position will evolve over time.

Why Eclipse?

10 years of Eclipse The Eclipse Foundation has been a fantastic success story (oh, and, Happy 10th Birthday, Eclipse!). As the scope of their mission has broadened beyond an IDE to the web, build environments, and all kinds of other tools, it was a good place for Sierra Wireless to kick off the Eclipse Koneki M2M tools project, and is now a natural place for this primarily M2M protocol to be hosted under Paho. As James Governor notes in his write-up of the news:

… the Eclipse Public License is designed to support derivative works and embedding, while the Eclipse Foundation can provide the stewardship of same. One of the main reasons Eclipse has been so successful is that rather than separate software from specification it brings them together – in freely available open source code – while still allowing for proprietary extensions which vendors can sell.

How quickly will the code donation happen?

The Paho proposal tentatively includes dates in November and December 2011 – there will need to be various approvals as code is accepted into Eclipse, so that may “flex” a little, but it is all in the pipeline.

OK… Why MQTT? Why not HTTP/XMPP/AMQP/PubSubHubbub/WebSockets/etcetcetc?

To answer this one adequately I’d probably end up addressing each individual difference between protocols in turn, and if you’ve heard me speak about MQTT I’ve covered some of this before – so I’ll keep this answer relatively brief. I will admit that I’ve been asked about all of these by journalists in the past couple of days.

There is space for a range of protocols to coexist, because they address different areas. In the messaging space, we’ve found over time that whilst efforts to create a single protocol have been made, that has often ended up as focused around a particular set of qualities of service, and not optimised to cover the the whole range of them.

For example, if we look at IBM’s own messaging protocols – there are several. There’s WebSphere MQ which is all about reliable, transactional, solid, clusterable, enterprise, JMS and other APIs, etc etc.. WMQ itself isn’t ideal for very high-speed in-memory or multicast scenarios, so there is also WMQ Low Latency (interoperable with the new multicast feature in WMQ 7.1, but a separate protocol). Neither WMQ LLM or WMQ scales down to unreliable device networks and embedded systems, so there is WMQ Telemetry (aka MQTT), which was specifically designed for constrained devices and networks, and that can interoperate with the main queue manager, too. Oh, and sometimes you want to deal with files (WMQ File Transfer Edition), or access message data via HTTP (WMQ HTTP Bridge). You need to address a range of requirements in a messaging story.

So why not those others? In this case, IBM believes that MQTT is ideally-suited to the Smarter Planet Instrumented->Interconnected layer – it’s tiny, not synchronous and brittle, isn’t specific to the web as it is all about data rather than documents, XML etc etc. In these scenarios, REST principles may add an overhead. Oh, and it has been around for over 10 years, and has been proven across a range of industries and in a range of extreme conditions. IBM’s commercial implementation is known to scale to hundreds of thousands of connected devices, and we know that is the direction that this space is heading.

Congratulations! / Thank you!

Thanks, but don’t congratulate or thank me! I’m familiar with this stuff, I’ve coded with this stuff, but I didn’t invent it and I didn’t write it. There are some amazing folks at both IBM and Eurotech (and some who have moved on) who started this all off in 1999, and who have helped to implement solutions using this protocol since then, and who have of course developed it. Several of them are on Twitter if you want to say hi! And huge thanks again to the community of folks that formed around mqtt.org and contributed client and server implementations – that absolutely helped to move things forward to this point.

HERE ENDS TODAY’S Q&A!

That, hopefully helps to clarify a few things and answers some of the questions I’ve seen via Twitter, forums, and mailing lists over the past few days. It has been something of a blur, to be honest, but a lot of fun. I’m looking forward to the next stage – working with the community more, working with our friends at Eurotech, Sierra Wireless and elsewhere, and making the M2M space much more real.

For more, here are a bunch of stories I’ve seen in the past couple of days… no particular order, just my cut-and-paste list!

TransferSummit 2011

One of the benefits of having attended OggCamp a few weeks ago was that I became aware of another event. Steve Lee, one of the speakers at OggCamp, is also involved with TransferSummit, and he was good enough to point it out to me. I’m grateful that he did.

TransferSummit bills itself as

… a forum for business executives and members of the academic and research community to discuss requirements, challenges, and opportunities in the use, development, licensing, and future innovation in Open Source technology.

Unlike a *Camp event, this wasn’t a self-organising unconference, and was much more business-focused. The sense I had was that it was far more about “getting down to work” than the more fun Open Source-oriented events that I otherwise attend. There were a range of fantastic speakers, and with my good friend James Governor giving the opening keynote it really didn’t take me long to decide that it was something that I should get to.

Not only that, but the event was held at my alma mater Oxford, in the rather lovely surroundings of Keble College – which I don’t remember ever having visited whilst I was at university – it was red brick, comparatively far up Parks Road away from my college, in the “science area”, and as a History student I simply never had much occasion to go up there! Have to say that I was very impressed by the college, accommodation, and service from students and staff. Fantastic.

You can explore more of my TransferSummit 2011 photos on Picasa.

I really enjoyed a number of elements of TransferSummit. Firstly, whilst there were a few folks I knew from my other networks, it was largely a group of people I’d not come across before directly, so it was a great opportunity to meet some new people in this space. It wasn’t too much of an echo chamber, and as Ross Gardler said during his introduction, it wasn’t a crowd of folks who already “get it” in terms of Open Source usage and adoption – there were a fair few organisations on the edge of making choices and I felt that the talks were more about how to go about making sensible ones, putting the right governance practices in place, and learning from the successes and mistakes of others.

I couldn’t cover all three tracks of the agenda in detail, but I’ll highlight a few particularly interesting sessions I did listen to (again, there’s more complete coverage on Lanyrd):

Another nice element of the event was the “gadget playtime” Open Source (and not-so-open) Hardware area, where I spent a lot of time talking to the folks from OSHUG and other projects.

One of the things that was negatively commented on via Twitter and other discussions was that Microsoft was the Platinum sponsor of the event. I found that very interesting, particularly where the commenters weren’t present at TransferSummit itself. To reassure those who may have stayed away or otherwise expressed concerns, I’ll just say that there was very definitely no “Microsoft agenda” being pushed, that my friend Steve Lamb was there very much in “listen, learn and interact” mode, but that others who attended and who I greatly respect did express other views about some elements of their participation (and I imagine it’s not hard to track those opinions down via hashtags etc.). Either way, having been involved with various conferences now, I fully support the idea that having a wide range of sponsors willing to help fund a professional conference and make is successful is important, so I thank Microsoft, HP and all of the sponsors (and in particular to the folks from OpenDirective) for enabling it to happen.

Definitely a worthwhile way to spend a couple of days of time – a well-run, informative event with great experiences shared, and some good contacts that I look forward to maintaining. My tip: look out for similar events and make an effort to mingle with the business, academic and government communities on Open Source. You might just learn something.

Disclosure: I was (unexpectedly) generously comp’d a ticket enabling me to attend, thanks to the organisers. My employer had no involvement and I attended on my own time.

The late, late OggCamp 11 write-up… and more UUPC

Forever Delayed

Oggs!It has been several weeks since OggCamp 11 now. I’ve been meaning to post a quick recap for a while.

I’ve written before about being friends with the crew from the Ubuntu UK Podcast (UUPC), so I’ve been following the progress of OggCamp over the past couple of years. Unfortunately I wasn’t able to attend either OggCamp 1 (2009) in Wolverhampton or OggCamp 2 in Liverpool last year.

Waitaminute… OggCamp?

Sounds like a weird name, huh?

Well… yeah ok, it is a bit odd. Breaking it down, there’s an audio file format called Ogg Vorbis which was intended to be a non-patent-encumbered, higher quality alternative to MP3. Many FLOSS (Free/Libre Open Source Software) supporters and audiophiles prefer it to MP3 and many podcasts aimed at these communities offer the .ogg format as an alternative to .mp3. The “Camp” idea is basically that of an unconference, popularised by events such as BarCamp – often a weekend-long gathering about nothing and everything in particular, with late night hacking and geekery. And being an unconference, the first rule of the event is that you definitely, definitely, have to talk about whatever you are ever passionate about, and participate.

So you smoosh together Ogg + Camp and you get…

… a very cool event populated by folks from the FLOSS and audio communities, often attracted by listening to podcasts like Linux Outlaws or the Ubuntu UK Podcast… the two teams that started the OggCamp events a few years ago as a kind of successor to the previous LugRadio Live. It’s not only about audio, although there tends to be some content on that subject, as well as some live podcast recordings, and other craziness.

Crew at #oggcampOggCamp 11 was “my first time”. It was held at the Farnham Maltings, a lovely venue that is very close to where I live (and also where we’ve held some Digital Surrey events in the past), so it would have been mad for me not to have attended, and just rude of me not to have offered to volunteer as part of the crew.

My excitement turned to a slight amount of trepidation a few days before things kicked off, when our now-legendary crew chieftan Les Pounder sent us out an email to check that everyone was “OK with heavy lifting”… 🙂 as it happened, that wasn’t too much of a problem! It was a pleasure to work with Les and the rest of the team actually – everyone was very laid back, happy, and just made things happen. I’d been wondering how onerous crew duties would be and whether they would prevent me from participating as an unconference attendee, but everything was shared around so I still found time for yet another talk on MQTT, and for some trademark heckling from the cheap seats during various other sessions.

You can explore my Flickr set from the event, but let me pick out a few small highlights:

  • meeting Roger Light for the first time, on the same day that Facebook mentioned their use of MQTT 🙂
  • hearing Ken Boak talk about his Nanode project from London Hackspace (and here’s one I made later!)
  • meeting Laura Czajkowski and hearing her talk about how to get involved in real world communities beyond IRC!
  • seeing a fantastic community that had formed around some great people from two podcasts I greatly enjoy.
  • a brief converation with Karen Sandler, the new lead of the GNOME Foundation.
  • winning a ChipKit Max32 and a Canonical goodie bag in the raffle 🙂
  • … and of course, watching Popey‘s demonstration of Extreme Ironing!

I hadn’t been to an event quite so specifically oriented towards freedom and Open Source for a while, and I’d forgotten how polarised some people can become around certain topics. In my career choices I’ve had to make some choices which make me a little more… shades of grey in my views about the technology landscape, so it is always good to have the challenging discussions and hear other views.

I’d definitely want to attend OggCamps in the future. A lot of fun, a great experience, and thanks to the organising team and sponsors. Recommended.

Even more talking

Following on from OggCamp, I was invited back to the UUPC Studio last week to cover for Alan – evidently I’ve not made too many slip-ups yet, since this is my third time as a guest presenter now. It’s really a fantastic experience and their production process and quality is always superb and well-planned and executed. Check out Episode 14 of Season 4 of UUPC “Revelations” to see how we got on with all the news, interviews, and listener feedback!

OggCamp approaches!

I’m getting quite ridiculously excited about an upcoming event…

I’m good friends with the team from the Ubuntu UK Podcast and have been privileged to be invited onto the show as a guest host twice now. They’ve partnered with the Linux Outlaws to create OggCamp for the past three years and this year, finally, I’m able to attend. I admit that it helps that the venue this year is reasonably local for me! I’ve also volunteered to assist as official Crew for the event, so I’ll either be very visible, or barely visible at all 🙂

I’ve blogged fairly frequently about my OSS, Linux, podcasting and social passions so I imagine it’s not a huge surprise to regular readers that I’m excited to finally have an opportunity to be involved. Laura has written about the rapid run-up to the event, and I hear that tickets may be becoming available from returns at the moment, so if you are interested it is worth checking back. I’ve also set up the OggCamp 11 page on Lanyrd if you want to add the event feed to your calendar. I’m also fairly certain that a good gaggle of MQTT geeks will be in attendance (the mosquitto project was born from the first OggCamp in 2009), so I’m looking forward to meeting folks!

Lightweight Messaging and Linux #lca2011

I’ve just delivered my talk at LinuxConf Australia 2011Lightweight Messaging for a Connected Planet. Unlike any of the other “spins” of the MQTT overview talk I’ve delivered before, this one really talks in more detail about the coding side, with shout outs to many of the members of the MQTT community and their projects. Special mentions here to Roger Light and the mosquitto project, and Nick O’Leary and the Arduino client. There has been a huge amount of discussion of Arduino and home automation (and sending Tux to the edge of space with an Arduino on a balloon… unrelated to MQTT but awesome) at LCA 2011 so these aspects turned out to fit really well.

Do let me know if you start to play around with MQTT as I believe there are some really funky things happening (some of which are highlighted in the slides). I’ve had some particularly great conversations with Andy Gelme who helped to run the Arduino Miniconf at LCA2011 and I hope to see things moving forward on his open hardware router project, too.