The story so far

There were two  M2M events on day one. The first was the M2M tutorial featuring some real hardware (Raspberry Pi and Arduino), using the Koneki Lua Development Tooling to deploy scripts to the embedded Mihini runtime on the Pi to drive and take input from sensors over Modbus. I cheated a little and mixed in a bit of MQTT using the Paho Lua client, to have my kit publishing temperature and light data to a broker instead It is early days for Eclipse Mihini, but it was a really slick demo and tutorial, and I’m looking forward to playing with this a lot more.

The evening Birds of a Feather session gave a group of us a couple of open hours to hack around with our Pi and Arduino kits. My own efforts were slightly thwarted by a European soldering iron and a 110V power supply, so I wasn’t able to assemble the add-on board I wanted for my Raspberry Pi, but I’ll get back to that in the future. For those looking to explore their new Raspberry Pis and hardware / GPIO interaction from Java, take a look at the pi4j project.

Building a community

One useful “war story” that was told today was in Benjamin Cabé’s Building the M2M Community talk. The slides will follow, but it was good to hear about the progress of the community and projects involved (yes, ok, I’m a Committer on one of them…) and also to hear about his “Community Manager Toolbox” for tracking and responding to community discussions. I use a very similar set of tools when I engage with the Cloud Foundry and MQTT communities (and others that I’m involved with).

Here’s a summary:

• Slideshare – monitor who favourites and downloads slides related to your project
• Twitter – monitor hashtags, engage complainers
• YouTube – screencasts, demos, build channel to aggregate content
• StackOverflow – answer questionss (even old ones), find answers that need to be complemented)
• Github – monitor forks, stars, look for people using your tech (include Gists)
• Google Alerts
• IFTTT – e.g. if Eclipse wiki changes get Gtalk notification recipe
• Google Analytics for your site
• LinkedIn – monitor groups, post news and relay blog posts

In short: Engage the community. Be very public about what you are working on and where the roadmap is going. Get management buy-in for your projects to persuade them of the value of opening them to the community (use metrics to demonstrate take-up).

Working on standards

The other thing that has been going on today “behind the scenes” at EclipseCon has been the first meeting of the OASIS MQTT Technical Committee. It is exciting to see that Richard Coppen and Raphael Cohn have been elected as co chairs of the group, with seven sponsoring organisations from across the technology and messaging market and 35 publically-visible members – lots of interest in MQTT!

There’s a LOT of content around the machine-to-machine space this year, and growing interest in how to use instrumented devices with an embedded runtime with lightweight messaging. If you’ve not been following the progress of the M2M community at Eclipse, we now have an M2M portal, along with nice pages for each of the three associated projects Koneki, Mihini, and Paho.

Almost the first thing I saw when I walked in yesterday was my buddy Benjamin Cabé assembling a bunch of electronics kits (Raspberry Pis and Arduino Unos) for today’s M2M tutorial which will use Eclipse Koneki and Mihini. This will be the first opportunity for many folks to play with the new Mihini runtime. Later this evening, we’ll have the chance to run a hackathon with things like Raspberry Pi and Orion and others parts as an extended Birds of a Feather.

What are some of the other M2M sessions to look out for?

• learn about building an open M2M community
• hear about how a 3D printer was hooked up to the web using MQTT
• listen to what Hitachi have been doing with M2M gateways
• understand how Koneki and Mihini can build M2M applications
• the TCF BoF may be of interest now that it can debug Raspberry Pi
• the IWG BoF will include folks from the M2M community

There’s also the first meeting of the OASIS TC for MQTT due this week, and a meeting of the Eclipse M2M Industry Working Group scheduled as well. Exciting times!

The corridor conversations and late night beer sessions are as always invaluable, and myself and many of the other project folks will be around – I’m always happy to talk about Paho in particular. At Paho we now have updated Java and C MQTT clients in Git (NB check the ‘develop’ branch for the latest Java updates), along with the Lua client, and proposed contributions of Objective-C, Javascript and Python clients are at various stages of review looking to join the project.

Oh, and if you are interested in MQTT, come and find me for some MQTT Inside stickers that you can use with your own hardware projects

The initial proposal and setup stages have taken a little while, but this week the initial availability of the C client code was announced on the Paho mailing list (Java will follow shortly).

Paho Quickstart

This is not intended to be a comprehensive guide – better documentation etc will emerge over time – but I thought I’d post a quick guide as a kickstart for anyone wanting to give it a look. I did this on 64-bit Ubuntu 11.10 – similar steps will apply on other Linux or UNIX platforms (note, the initial code contribution has a Makefile with rules which should work on UNIX, Windows, or z/OS).

Install the necessary packages to build code. NB git is for grabbing the source from Eclipse; build-essentials is a metapackage providing gcc etc on Ubuntu; and doxygen and optional graphviz are used for generating the documentation.

sudo apt-get install git build-essentials doxygen graphviz

Get the code from the git repository:

git clone git://git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.c.git

Quick build for the client library and documentation:

cd org.eclipse.paho.mqtt.c.git/src

make -f ../build/Makefile all

doxygen ../doc/DoxyfileV3ClientAPI

Once these commands complete, you should be left with subdirectories called <platform> and docs. In my case, <platform> was 64-bit Linux, so I had a binary at linux_ia64/libmqttv3c.so. There’s no “make install” rule at the moment, nor is there a rule to compile the docs so I had to run doxygen directly. In the future it would be nice to automate all of that, and also to build some test applications.

Opening docs/html/index.html in a browser reveals very nice documentation describing the client library, including some examples of how to use it. For example, in docs/html/pubasync.html there’s a complete listing for an asynchronous publisher application. I extracted that code into pubclient.c and decided to check that it worked!

gcc -Wall pubexample.c -L./linux_ia64 -lmqttv3c -lpthread -o pubexample

That command successfully built a binary called pubexample. All I needed to do was test it. The sample application assumes that an MQTT broker is available on localhost port 1883 – if you want to change that, simply modify the value of the static variable ADDRESS in pubexample.c – in my case I simply apt-get installed the mosquitto and mosquitto-clients packages onto my system, but I could equally have unzipped and run Really Small Message Broker – both start on port 1883 by default if not given alternative configuration.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./linux_ia64/

./pubexample

Waiting for publication of Hello World!
on topic MQTT Examples for client with ClientID: ExampleClientPub
Message with token value 1 delivery confirmed

It was trivial to test that a subscriber (mosquitto_sub in my case) also received the publication. Job done!

Getting involved, and other news on Paho

I mentioned that the Java client contribution should appear soon. One other piece of news this week is that the project’s sandbox broker implementation – based on mosquitto – has been spun up. That was posted on the Paho mailing list, and if you want to get involved you should definitely subscribe to that; start to track the Eclipse Bugzilla for Paho; watch the Paho wiki; keep an eye on the source repositories; etc.. I’m already thinking about getting an OS X build rule sorted out. If you want to test your sample code now, you’ve got the option of a local broker, the Eclipse Paho sandbox, the mosquitto sandbox, or various other implementations.

Oh – and please leave a comment on this post if you find this information interesting, or want to discuss where things are with Paho. I’ll be hanging out on the mailing list as well.

What about Bob? (or Andy, even!)

Well, although I’ve left IBM, I’m delighted that MQTT is now going Open Source – in fact that was one of the things that I really wanted to help to achieve before I moved on. I am really pleased that I will be able to continue to contribute to both Paho and the broader Eclipse M2M Industry Working Group. I’ll be helping to update the mqtt.org community site, and heading over to EclipseCon in Virginia in a couple of weeks’ time to talk about M2M and work with our friends from the Koneki project. If you are attending EclipseCon please come say hi to me – and you may be interested in Wes Johnson’s session on MQTT and Eclipse tools.

There are very cool times ahead!

It’s all about machine-to-machine (or M2M) communications, Smarter Planet, and the Internet of Things.

Before I dive in to this, a few clarifications. First, I’m being described in a couple of news stories as “an IBM distinguished engineer”, and whilst I wish that was true, I’ve yet to ascend to those heights! Also, there are various numbers being quoted – note that the figures in the press release were not invented by IBM, the headline number of an expected 50 billion connected devices by 2020 comes from a recent study conducted by Ericsson AB. Oh, and this isn’t about a “new” protocol – MQTT has been in use since 1999.

The other clarification is that some articles seem to suggest that IBM is out to create some kind of new, alternative, Web – that’s not what has been announced, and I’m certainly not aware of any such plan! It’s about connecting “things” – sensors, mobile devices, embedded systems, even small appliances or medical devices for example – to the Web and the associated platform and ecosystem of technologies, not about reinventing or recreating them. I’m personally a huge fan of the Web as a platform

Oh, and of course, the obligatory “all opinions expressed are my own” – this is my understanding of where things are going, although of course I’m talking about events I’m directly involved in!

So what is this all about?

Two things.

1. On Nov 2, IBM, Eurotech, Sierra Wireless and Eclipse formed a new M2M Industry Working Group at Eclipse. Sierra had already started the “Koneki” project at Eclipse to work on M2M tools, and the Working Group will look at a range of topics together, such as M2M tooling, software components, open communication and messaging protocols, data formats, and APIs.

2. On Nov 3, IBM and Eurotech announced the donation of their C and Java clients for MQTT to a new Eclipse project called “Paho” which is under proposal in the incubator – with code expected to hit the repository within the next couple of months. MQTT is being given to Eclipse to live within the M2M ecosystem that is emerging there, and to provide an avenue for adoption of the protocol as a more pervasive standard for connected devices.

How is that news? Isn’t MQTT already open / free?

Technically… kinda, sorta

The MQTT specification has been published under a royalty-free license for some time, and that has led to a fantastic community contributing a range of different projects. IBM and Eurotech took this approach from early on, because it wouldn’t have been possible to compile and support code on every embedded platform that might come along – far simpler to set the protocol free.

Initially the specification was hidden away in the WebSphere Message Broker documentation, but last year it was republished, moved to a new home on developerWorks, and the license was clarified.

In August, IBM and Eurotech announced their intention to take MQTT to a standards organisation. The specific organisation has not yet been finalised, but this is also an important step in ensuring that MQTT is not “just” an IBM protocol, but something of general use which the community can feel comfortable with. If you’d like to join that discussion then there’s a Get Involved page on the mqtt.org community site.

The missing piece was code – a reference implementation, if you like. That’s one reason why the Eclipse Paho announcement is significant.

Why else is this significant?

Well, here are some of my musings on that one:

• it shows IBM is serious, by committing code and open sourcing it (as with the original Eclipse donation in 2001);
• the M2M Industry Working Group exists to foster the discussion in this space;
• it makes high-quality reference Java and C client implementations freely available in source form, with a good Java implementation something that has been particularly lacking;
• it creates an opportunity for Eclipse projects to use MQTT, and to develop tools on top of it.

The press release and Paho project proposals aren’t clear (to me) – what exactly is being donated?

IBM is seeding Eclipse Paho with C and Java client implementations of MQTT. Eurotech is donating a framework and sample applications which device and client developers can use when integrating and testing messaging components.

Why C and Java clients (aren’t they “dying” languages?) Where’s my Perl and Ruby code?!

IBM had previously made some C and Java code available in some SupportPacs, but those are outdated and the license for reuse was never clear.

It’s important to realise that this stuff came from the embedded world of 10 (and more) years ago, and continues to be applied in that industrial space. That category of device typically runs some kind of realtime Java-based OS, or a Linux-based or other runtime with a GCC toolchain for the CPU in question. C and Java are genuinely the most useful implementations to get out there. Oh, and on that “those old languages” thing – I think you’ll find they are very widely used (Android, iOS etc run variants of sorts, most non-web app development is likely to be in one or the other).

We’re very fortunate that clients libraries for a wide range of languages already exist thanks to the MQTT community – see the list at mqtt.org!

Hold on… don’t we need a broker / server / gateway?

Yes. But, one step at a time!

There are brokers available for free today, either as precompiled binaries or as full Open Source implementations, so this is not a dead end from day one.

The Paho project scope outlines the intention to add a broker to the project in the future, and to host an M2M sandbox for developers as well. That is where we are today, and this position will evolve over time.

Why Eclipse?

The Eclipse Foundation has been a fantastic success story (oh, and, Happy 10th Birthday, Eclipse!). As the scope of their mission has broadened beyond an IDE to the web, build environments, and all kinds of other tools, it was a good place for Sierra Wireless to kick off the Eclipse Koneki M2M tools project, and is now a natural place for this primarily M2M protocol to be hosted under Paho. As James Governor notes in his write-up of the news:

… the Eclipse Public License is designed to support derivative works and embedding, while the Eclipse Foundation can provide the stewardship of same. One of the main reasons Eclipse has been so successful is that rather than separate software from specification it brings them together – in freely available open source code – while still allowing for proprietary extensions which vendors can sell.

How quickly will the code donation happen?

The Paho proposal tentatively includes dates in November and December 2011 – there will need to be various approvals as code is accepted into Eclipse, so that may “flex” a little, but it is all in the pipeline.

OK… Why MQTT? Why not HTTP/XMPP/AMQP/PubSubHubbub/WebSockets/etcetcetc?

To answer this one adequately I’d probably end up addressing each individual difference between protocols in turn, and if you’ve heard me speak about MQTT I’ve covered some of this before – so I’ll keep this answer relatively brief. I will admit that I’ve been asked about all of these by journalists in the past couple of days.

There is space for a range of protocols to coexist, because they address different areas. In the messaging space, we’ve found over time that whilst efforts to create a single protocol have been made, that has often ended up as focused around a particular set of qualities of service, and not optimised to cover the the whole range of them.

For example, if we look at IBM’s own messaging protocols – there are several. There’s WebSphere MQ which is all about reliable, transactional, solid, clusterable, enterprise, JMS and other APIs, etc etc.. WMQ itself isn’t ideal for very high-speed in-memory or multicast scenarios, so there is also WMQ Low Latency (interoperable with the new multicast feature in WMQ 7.1, but a separate protocol). Neither WMQ LLM or WMQ scales down to unreliable device networks and embedded systems, so there is WMQ Telemetry (aka MQTT), which was specifically designed for constrained devices and networks, and that can interoperate with the main queue manager, too. Oh, and sometimes you want to deal with files (WMQ File Transfer Edition), or access message data via HTTP (WMQ HTTP Bridge). You need to address a range of requirements in a messaging story.

So why not those others? In this case, IBM believes that MQTT is ideally-suited to the Smarter Planet Instrumented->Interconnected layer – it’s tiny, not synchronous and brittle, isn’t specific to the web as it is all about data rather than documents, XML etc etc. In these scenarios, REST principles may add an overhead. Oh, and it has been around for over 10 years, and has been proven across a range of industries and in a range of extreme conditions. IBM’s commercial implementation is known to scale to hundreds of thousands of connected devices, and we know that is the direction that this space is heading.

Congratulations! / Thank you!

Thanks, but don’t congratulate or thank me! I’m familiar with this stuff, I’ve coded with this stuff, but I didn’t invent it and I didn’t write it. There are some amazing folks at both IBM and Eurotech (and some who have moved on) who started this all off in 1999, and who have helped to implement solutions using this protocol since then, and who have of course developed it. Several of them are on Twitter if you want to say hi! And huge thanks again to the community of folks that formed around mqtt.org and contributed client and server implementations – that absolutely helped to move things forward to this point.

HERE ENDS TODAY’S Q&A!

That, hopefully helps to clarify a few things and answers some of the questions I’ve seen via Twitter, forums, and mailing lists over the past few days. It has been something of a blur, to be honest, but a lot of fun. I’m looking forward to the next stage – working with the community more, working with our friends at Eurotech, Sierra Wireless and elsewhere, and making the M2M space much more real.

For more, here are a bunch of stories I’ve seen in the past couple of days… no particular order, just my cut-and-paste list!

• Smarter Planet blog
• mqtt.org news
• Bob Sutor
• ReadWriteWeb
• PCWorld
• VDC Research
• WebWereld (NL)
• Technolenta (RU)
• ConnectedWorld
• Simon Phipps at ComputerWorld
• Complex.com
• Developer.com
• James Governor
• H-Online
• mosquitto blog
• Design Thinking Thoughts

Yesterday, Project Zero emerged.

We’re talking REST, PHP, Groovy and mashups.

Come and take a look. For full details, read the FAQ and follow the blog. Then, download the code and join the community.

Jazz is an extensible and scalable collaboration platform for the development cycle. It has a client/server architecture and runs on either an open source stack of Tomcat + Derby + Jabber, or an IBM one (potentially others I suppose, but that’s sheer speculation).

Several announcements were made around the middle of last year, and the Jazz site is out there in the open. There was some information about it at the RSDC talks last year. You’ll find analysts talking about it already.

The demo was extremely neat. Again, this is building REST, RSS/Atom, a rich web UI (as well as the Eclipse one) into the platform. I can’t say too much at this stage, but it would be worth getting involved in the Jazz community site if you want to know more.

Do I sound excited about it? It looked great. I can’t wait to see this start to appear in the open.

Technorati Tags: Eclipse, IBM, Jazz, development, Rational, WebSphere, wstc, wstc2007

It all looks very exciting indeed. Should arrive just in time for Christmas

In the meantime, take a look at all this other stuff:

• There are new SOAP nodes available in SupportPac IA9O, which will work with the new function in the toolkit.
• A new version of the WMB Administrator Eclipse plugin is available.
• SupportPac MR01 describes how to setup an HA Link from WMQ to the WebSphere SIBus.
• Check the new RedPaper on SSL with WMQ and WMB.

SupportPac IS02 was released today. From the text description, you might be lulled into suspecting that this is just another Eclipse plugin for MQ Explorer. In fact, this is a very cool extension that enables WebSphere Message Broker resources (brokers etc.) to be managed directly alongside your queue managers. It also means that WMB administrators no longer need to install the full Message Brokers Toolkit if all they want to do is perform basic administration tasks such as BAR file deployment, create/start/stop brokers, etc..

Congratulations to the development team. I got to take an early look at this in the lab, so I’m really keen to show it to our customers.

I’d post a screenshot but I don’t have the bandwidth to upload one at the moment. Watch this space.

• WebSphere Message Broker version 6, fixpack 6.0.0.1. Check out the release notes.
• WebSphere Application Server Community Edition, version 1.0.1.0. The list of new features includes web-tier clustering, simplified deployment, and improved portability to other members of the WebSphere Application Server family.

Technorati tags: IBM WebSphere WASCE Message Broker WMB fixpack

• Doug Tidwell on Eclipse
• Using Eclipse for building UNIX applications
• Migrating from Outlook Express to Lotus Notes
• DB2 on Linux HOWTO
• The $39 experiment

Technorati tags: IBM Notes Eclipse DB2 Lotus

Ensode.net has some screenshots of the new GTK look and feel that is due to come along in Mustang (aka Java 1.6), and it does seem that it is likely to be a much better match for my Linux GNOME desktop.

Of course, this isn't enough to sway me away from SWT, which always looks completely native, and as a programming framework it gets richer by the day. developerWorks has an article comparing AWT, Swing and SWT, and a tutorial on how to migrate your applications from Swing to SWT.

The admin console for this release has had a significant revamp, with some nice icons added to the GUI. It has some very useful functionality (I like the log viewer in particular).

Unfortunately (and foolishly) I ignored the warning not to install over the top of an existing installation, and pressed Next after the installer had told me to choose an alternative install path or uninstall first. As a result, I lost my existing server configuration. Fortunately, it was trivial to redeploy the sample JMS application I've been playing with:

C:\WebSphere\CE\bin> deploy --user [user] --password [pword] deploy ..\samples\jmssimple\sender.war

Sadly, I haven't had a chance to try out the Eclipse plugin yet… but this looks like a nice step forward.

Technorati tags: IBM Websphere WASCE geronimo Java

For example, in WebSphere Message Broker version 6, the Configuration Manager now uses Cloudscape instead of DB2 to store its information. This means that it is easily portable (it now runs on all of the WMB supported platforms), and you don't need to install and use DB2 if you don't want it.

Take a look at Cloudscape, also known as Apache Derby, if you are interested in a lightweight embeddable database. If you need something a bit bigger and don't mind some administration, DB2 Express-C is worth a look.

Technorati tags: db2 database derby mysql IBM websphere broker

Technorati tags: websphere mq ibm eclipse java

So instead I want to talk about the series of articles on The Register by Phil Howard of Bloor Research. The final entry in the series suggests that IBM has a problem with the SOA message – we just have too many products.

I heard this same statement from a customer earlier this week. Here are my thoughts on the matter:

• Sure, we have a number of products which fit in across the whole swathe of an SOA. Let's talk about at a few of the development tools, for example: Rational Software Architect, Rational Application Developer, WebSphere Integration Developer, WebSphere Business Modeler. These are all based on the Eclipse platform (as are all of our tools), and provide functionality appropriate to their target audience: architect, J2EE developer, ESB integration developer, business analyst. The look-and-feel is consistent. If necessary they can be combined into a single workbench. What's so scary about that? You can choose the products you want, and combine them as you wish.
• IBM is strongly behind open standards, and we go out of our way to ensure that our products conform to agreed open standards wherever possible. We don't go around evangelising a rip-and-replace strategy. We know that many customers have a technology soup already, and there are heritage applications and platforms that aren't going to be going away any time soon. I've been with IBM for 4 years, working with our WebSphere integration products, and literally every day of my time with the company to date has been about applying our technology to integration problems that customers face. By following a strategy based on open standards, the ability of our products to interoperate with those from other vendors is greatly increased. Again, you can pick and choose what you need from our portfolio to fit in with the needs of your business.
• What if we just had a single, "uber-product" for SOA? How much sense would that make? It just isn't reasonable, surely? And just how "simple" would such a product be? What we have is a set of software products which cover the challenges which customers are likely to face as they set about building an SOA. I also believe that we have a consistent message and that each of our software brands makes its own strong contribution as part of the SOA strategy. You need a development tool? Look at the Rational brand. You want to look at collaboration? That's Lotus. Monitoring, security, systems management? Tivoli products. We have excellent coverage; it doesn't matter which point you want to start from, we can help you to deliver an SOA.

The final point raised by the piece is that one of the really key aspects of implementing an SOA is that of the cultural impact, which I think we can talk about in terms of governance. Phil Howard argues that since this is more a business issue than an IT issue, it is outside the domain of IBM Software Group. I agree with him up to a point; but this is where we dovetail neatly (I hope!) back into the point about IBM Software Services and IBM Global Services. IBM Software Group may not be able to cause a cultural change simply through the software that we release*, but as a Software Services consultant I certainly go out of my way to talk about the business impact of SOA. It simply isn't going to work if the business decides to build an ESB and then the IT development groups fail to use it – you miss out on the benefits. Strong leadership and governance is critical. As a consultant part of my role is to not only transfer technical skills to our customers, but also some of our experience and understanding of the cultural impact of SOA on both business and IT people.

* unless… we came up with some kind of mind-control software… interesting… I'll have to talk to the guys in the labs…

Technorati tags: SOA ESB websphere IBM rational tivoli lotus Eclipse

Some people talk of me as "a CVS guru", although I'm not sure I'd go that far.

There has been a long-running issue with Eclipse not fully supporting CVSNT (or maybe it is the other way round!). CVSNT is a fork of the UNIX CVS code, integrated more closely with Windows, and it doesn't quite work the same as vanilla CVS. When an issue arose way back in the mists of time between Eclipse and CVSNT, the Eclipse project decided to stop supporting CVSNT. I'm not claiming to know the history here, but I do know that there were Eclipse bugs raised – go have a look in bugzilla at eclipse.org.

So we are now in a situation where, if you try to connect Eclipse 3.x (RAD, RSA etc.) to a CVSNT repository, you get told "CVSNT does not always properly communicate resource paths in this mode resuliting in failure of some specialized Eclipse CVS operations. The use of a repository prefix should be disabled if the full functionality of the Eclipse CVS client is desired."

However – it can be made to work – and I've personally found no issues.
There are three options (listed here in decreasing order of difficulty, so go with option 3 and work backwards!)

1. Obtain one of the earlier version of CVSNT. Until recently, I used 2.0.51d. There are archived versions at http://www.cvsnt.org/archive/ – but it looks like the earliest version now available is 2.0.58. If you run one of these versions you will still see the error, but there is a CVSNT option "Act like a Unix server". If you set that option, the error will disappear.
2. Instead of using the "Act like a Unix server" option, you can set the Repository name and location (both fields) to have the drive prefix (like "C:/cvsrepository"). CVSNT will issue a warning, but Eclipse will work fine. I believe that this works in both 2.0.51 and the latest, 2.5.0.1.1076.
   
3. Set the "Respond as cvs 1.11.2 to version request" option on the Compatibility tab in CVSNT 2.5.0.1.1076. You should also ensure that "Emulate '-n checkout' bug" is set (which it usually is by default).
   

Hope that this information helps someone out there.