overheating in an office with no comfort cooling or aircon. About to drink water.

—
Andy Piper (@andypiper) February 21, 2007

I remember discussing this exciting cool “new Web 2.0 site” with Kim Plowright @mildlydiverting in Roo’s office in Hursley a couple of days before, and before long he, Ian and I were all trying this new newness out. It was just before the 2007 SXSWi, where Twitter really started to get on the radar of the geekerati.

But wait a moment! It’s impossible to pull back more than just over the last 3,000 tweets using the API, so how was I able to get all the way back to 5 years ago and display that tweet when I’ve got over 33,000 of them to my name?

It’s a relatively little-known fact that you can ask Twitter to disclose everything they hold associated with your account – and they will (at least, in certain jurisdictions – I’m not sure whether they will do this for every single user but in the EU they are legally bound to do so). I learned about this recently after reading Anne Helmond’s blog entry on the subject, and decided to follow the process through. I first contacted Twitter on April 24, and a few days later faxed (!) them my identity documentation, most of which was “redacted” by me Yesterday, May 11, a very large zip file arrived via email.

I say very large, but actually it was smaller than the information dump that Anne received. Her tweets were delivered as 50Mb of files, but mine came in nearer to 9Mb zipped – 17Mb unzipped. I’d expected a gigantic amount of data in relation to my tweets, but it seems as though they have recently revised their process and now only provide the basic metadata about each one rather than a whole JSON dump.

So my entire archive of Tweets back to Feb 2007 is a ~8Mb text file (140 chars per tweet+metadata).

—
Andy Piper (@andypiper) May 11, 2012

So, what do you get for your trouble? Here’s the list of contents, as outlined by Twitter’s legal department in their email to me.

- USERNAME-account.txt: Basic information about your Twitter account.

- USERNAME-email-address-history.txt: Any records of changes of the email address on file for your Twitter account.

- USERNAME-tweets.txt: Tweets of your Twitter account.

- USERNAME-favorites.txt: Favorites of your Twitter account.

- USERNAME-dms.txt: Direct messages of your Twitter account.

- USERNAME-contacts.txt: Any contacts imported by your Twitter account.

- USERNAME-following.txt: Accounts followed by your Twitter account.

- USERNAME-followers.txt: Accounts that follow your Twitter account.

- USERNAME-lists_created.txt: Any lists created by your Twitter account.

- USERNAME-lists_subscribed.txt: Any lists subscribed to by your Twitter account.

- USERNAME-lists-member.txt: Any public lists that include your Twitter account.

- USERNAME-saved-searches.txt: Any searches saved by your Twitter account.

- USERNAME-ip.txt: Logins to your Twitter account and associated IP addresses.

- USERNAME-devices.txt: Any records of a mobile device that you registered to your Twitter account.

- USERNAME-facebook-connected.txt: Any records of a Facebook account connected to your Twitter account.

- USERNAME-screen-name-changes.txt: Any records of changes to your Twitter username.

- USERNAME-media.zip: Images uploaded using Twitter’s photo hosting service (attached only if your account has such images).

- other-sources.txt: Links and authenticated API calls that provide information about your Twitter account in real time.

Of these, let’s dig a bit more deeply into just a few of the items, no need to pick everything to pieces.

The “tracking data” is contained in andypiper-devices.txt and andypiper-ipaudit.txt – interesting. The devices file essentially contains information on my phone, presumably for the SMS feature. They know my number and the carrier. The IP address list tracks back to the start of March, so they have 2 months of data on what IPs have been used to access my account. I’ve yet to subject that to a lot of scrutiny to check where those are located, that’s another script I need to write.

I took a look at andypiper-contacts.txt and was astonished to find out how much of my contact data Twitter’s friend finder and mobile apps had slurped up. I mean, I don’t even have all of this in my address book… given the fact that the information contained the sender email addresses for various online retailer newsletters, I’m guessing that Google’s API (I’m a Gmail user) probably coughed up not just my defined contact list, but also all of the email addresses from anyone I’d ever heard from, ever.

Twitter has on record 981 phone numbers and 8741 email addresses from my contacts / when I've used Find Friends :-o deleting that...

—
Andy Piper (@andypiper) May 11, 2012

Fortunately, there’s a way to remove this information permanently, which Anne has written about. I went ahead and did that, and then Twitter warned me that the Who To Follow suggestions might not be so relevant. That’s OK because I don’t use that feature anyway – and in practice, I’ve noticed no difference in the past 24 hours!

I use DMs a lot for quick communication, particularly with colleagues (it was a pretty reliable way of contacting @andysc when I needed him at IBM!). That’s reflected in the size of andypiper-dms.txt, which is also a scary reminder to myself that I used to delete them, but since Twitter now makes it harder to get to and delete DMs, I’ve stopped removing them and there’s a lot of private data I wish I’d scrubbed.

Taking a peek at the early tweets in andypiper-tweets, I’m trying to remember when the @reply syntax was formalised and when Twitter themselves started creating links to the other person’s profile. Many of my early tweets refer to @roo and @epred and I don’t think they ever went by those handles. 5 years is a long time.

I mentioned that the format used to deliver the data appears to have changed since Anne made her request. She got a file containing a JSON dump of each tweet including metadata like retweet information, in_reply_to, geo, etc etc.. By comparison, I now have simply creation info, status ID (the magic that lets you get back to the tweets via web UI), and the text itself:

********************
user_id: 786491
created_at: Wed Feb 21 15:43:54 +0000 2007
created_via: web
status_id: 5623961
text: overheating in an office with no 
comfort cooling or aircon. About to drink water.

It’s a real shame that they have taken this approach, as it means the data is now far more cumbersome to parse and work with. However, using some shell scripts I did some simple slicing-and-dicing because I was curious how my use of Twitter had grown over time. Here’s a chart showing the numbers of tweets I posted per year (2012 is a “to date” figure of course). It looks like it was slow growth initially but last year I suddenly nearly doubled my output.

Still considering what other analysis I’d like to do. I can chart out the client applications I’ve used, or make a word cloud showing how my conversational topics have changed over time… now that all of the information is mine, that is. It is just a shame I have to do so much manual munging of the output beforehand.

Oh, and the email I received from Twitter Legal also said:

No records were found of any disclosure to law enforcement of information about your Twitter account.

So, that’s alright then…

Why did I do this? firstly, because I believe in the Open Web and ownership of my own data. Secondly, because I hope that I’ll now be able to archive this personal history and make it searchable via a tool like ThinkUp (which I’ve been running for a while now, but not for the whole 5 years). Lastly… no, not “because I could”… well OK at least partly because I could… because I believe that companies like Twitter, Facebook, Google and others should be fully transparent with their users and the data they hold, and that by going through this currently-slightly-painful procedure it will encourage Twitter to put in place formal tools to provide this level of access to everyone in a frictionless manner.

If you’ll excuse me, I’m off to dig around some more…

Then

On Saturday I went along to the Horizons 30th anniversary of the ZX Spectrum event, organised by Paul Squires and Leila Johnston and held at the BFI in London. The event ran on both days but I wasn’t able to stay on the Sunday, so I missed at least half of the fun!

Although I’m full of nostalgia for the 8-bit era, I have to confess I never actually owned a Speccy or any Sinclair hardware. My friends did, but I was primarily an Acorn enthusiast and our first home computer was an Electron (although the first computer I used at primary school was a Commodore PET).

Me: c.1985 Acorn Electron, A3000, Risc PC, Acorn A4, and then onto Dell PCs, homebuilt Linux PCs and Thinkpads. Apple from 2007. #sflzx

—
Andy Piper (@andypiper) May 05, 2012

I fondly remember some of the hacks I did on/with/to the Electron, including soldering a pair of headphones into the motherboard to avoid annoying my parents with the music from various Superior Software titles

Regardless of “allegiance”, Horizons was a really great day. Highlights for me included a fantastic history of computing by PJ Evans from The National Museum of Computing at Bletchley Park (if you haven’t been there yet, you should visit!); Spectranet, an Ethernet adapter for the Spectrum which had me wanting one for no good excuse that I can come up with; and the mind-blowing live composition of a chip tune by Matt Westcott which I saw, but I struggled to comprehend. Matt’s ability to reverse engineer a tune in his head was remarkable.

Oh, and if you haven’t downloaded or bought MJ Hibbett‘s Hey Hey 16k yet, or at least streamed it, you really should.

aside: since Horizons was part of SciFi London, I tried to get Micro Men director Saul Metzstein to drop some hints about his upcoming S7 Dr Who episodes. All he would say was that the western episodes were filmed in Spain (knew that), and that the script for the Christmas episode hasn’t been written yet (didn’t know that).

Now

After the event on Saturday evening, I found it a real struggle to avoid crazy, nostalgia-fuelled eBay purchases, but I did manage to resist! Instead, I resolved to finally get around to building the Fignition I’d picked up at the Hack to the Future event a couple of months ago.

For those who are not familiar with it, the Fignition is a credit card sized build-it-yourself 8-bit computer based around the ATMega chip (the same one used in the Arduino and Nanode Open Source hardware boards). It’s really a remarkable little device – I guess it took me about an hour to assemble and solder, although your mileage may vary. The build guide is excellent and very clear. After performing a couple of power on tests with and without the ICs inserted, it was time to connect up to the TV – and it worked first time. It boots into a simplified Forth environment, which was reminiscent of that BBC BASIC> prompt I am so familiar with from my childhood. The only real downside is that the keyboard – built from 8 clicker buttons – is a bit fiddly to get to grips with, but hey – I just assembled a complete 8-bit computer including video out and keyboard! It’s hard not to be excited.

The board I built was a RevD – the new RevE board has onboard audio in/out (get ready for some fun loading stuff from audio cassettes, again!), and is also slightly modified so that in principle, it is possible to add Arduino-footprint shields. That’s kind of cool, as it means that it might be possible to add a PS/2 keyboard or a network interface.

What’s “the point” of something so simple, by today’s standards? Well, actually – the simplicity. I went from a bag of components, to a fully working computer in the palm of my hand – no surface-mount components – to a programmable device. It’s “primitive” by the standards of today’s machines, but it’s not that hard to understand how an 8-bit “brain” works, in comparison to the 32 or 64-bit mulitcore CPUs and GPUs in modern laptops and mobile phones. In my opinion, the Fignition, Arduino and Nanode fulfil an important role in helping youngsters to understand the basic principles of electronics and computing.

Next

Last night I headed along to the fantastic Mozilla offices in London.

The main LJC event was Simon Maple from IBM showing off the new WebSphere 8.5 Liberty Profile running on a Raspberry Pi. I’d hooked Simon up with Sukkin Pang recently so that he could get one of the smart enclosures he provides for the Pi. It was pretty cool to see a full Java app server running on such a small computer – actually almost exactly the same size as the Fignition, only considerably more powerful of course.

The whole talk was live streamed on Mozilla Air – but if you missed it, there’s a video available (complete with semi-professional heckling from yours truly!)

What stole the evening for me, though, was two other glimpses of what lies ahead. First, Tom Banks from IBM Hursley came on stage after Simon and showed off the Liberty profile running on a mobile phone. Let me clarify – he was running Android 2.3 on a Nexus One (an “old” phone), running Ubuntu Linux as a virtual image inside of that, and WebSphere inside of that. Kind of mind-blowing! A proof-of-concept and arguably not very useful… not sure when I would want to put a full JEE app server in a phone… but extremely cool. Finally, @cyberdees let Tom and I have a play with Boot to Gecko – Mozilla’s new mobile play. B2G was something I’d heard about, but not touched. I have to say that even in an early form, it’s looking very slick, boots extremely fast – much more quickly than any Android or iOS device I’ve seen – and the device integration (GPS, camera, access to hardware settings, etc) was impressive.

With the Open Web as the platform, ubiquitous mobile devices, and increasingly sophisticated cloud-based backends to interact with, the future is looking pretty cool.

The “pre-shaved yak”, of course, is one aspect of what a polyglot open source PaaS is all about – delivering a ready-made, ready-to-host, application runtime environment. We shaved the yak, so you can just go ahead and get productive with your development tool of choice, be that vi or emacs, Notepad or TextMate, or Eclipse / a.n.other IDE. Grab a micro Cloud Foundry VM image and take your pre-shaved yak with you when you’re not connected!

I actually started to write this post in order to comment on something that’s a bit more hairy that, though! I’ve been playing around a little bit with MonoDevelop and ASP.NET (for reasons that will become apparent during this week, I suspect). I’m using the current stable Mono (2.10) and MonoDevelop (2.8) packages on Lion, and they seem to work well. I’ve also recently been learning about Sinatra, the lightweight web framework for Ruby, and one of the node.js equivalents called Express. It turns out that the .NET world has a bunch of Sinatra-wannabes, the most popular of which appears to be Nancy (see what they did there…? dive into the world of Sinatra-themed name-related web frameworks…!).

Nancy’s site recommends installation via NuGet, which is evidently really well integrated into Visual Studio (NuGet is the equivalent of gem in Ruby, or npm in node.js). Unfortunately there’s no MonoDevelop equivalent. Here’s where the yak shaving started! The NuGet FAQ claims that the command line NuGet.exe will run and can be compiled under Mono, but in my experience, that’s not quite true – I could not get the source to compile in MonoDevelop on OS X. I grabbed the pre-compiled version and followed the instruction to get it to update itself (basically you just run it, and it bootstraps and downloads the latest available)… that went fine, but after that, it would no longer work and produced a huge stack trace.

So here, after getting most of a yak’s fleece all over me, is the secret. The prebuilt NuGet.exe will work under Mono on OS X, but it does require a Windows .NET 4.0 DLL (Microsoft.Build.dll) to be in the same directory / locatable in the path – I grabbed mine from my Windows VM install. It also requires that you tell Mono to present a v4.0 runtime. So I whipped up a tiny script to avoid having to type a bunch of paths and switches each time.

Further results of this recent dalliance in .NET land will be coming soon…

(if you can hear any background noise on this one, it was because I was at the Scala Days event in London on the day we spoke, and not Michael’s fault at all! I don’t think it sounds too bad)

Coming up this week, there’s the big Cloud Foundry Open Tour London on Tuesday (based on the numbers I’m hearing about, it sounds like that is going to be busy). Many of us from the engineering and developer relations teams will be speaking at that one. The rest of the week, I’ll be at SourceDevCon in London where my head honcho Patrick will be speaking on Thursday afternoon.

To round the week off, there’s Horizons at the BFI on Saturday and Sunday, to celebrate the 30th anniversary of the ZX Spectrum. I was always more of an Acorn guy myself, but there’s no denying these classic machines really kickstarted my interest in this role space – and I can’t wait to hear MJ Hibbett perform “Hey Hey 16k” in person!

Related articles

• First week with VMware and Cloud Foundry (andypiper.co.uk)
• My next steps – joining the Cloud Foundry team (andypiper.co.uk)

I grabbed an ISO this morning (64-bit desktop version, natch), and quickly setup a new virtual machine to run on my Mac. A nice feature of VMware Fusion 4.0 is the “easy install” option which lets you rapidly pop in the basic information needed to setup the system, and the rest is taken care of for you.

In my case, I specified a user ID, selected that I wanted my home folder read/write shared into the VM, and then customised the machine to up the memory and add a CPU core. In a few minutes (my machine has an SSD…!) I had a new virtual machine, running full screen in a separate space on Mission Control. Fusion even took care of installing VMware Tools so it was able to do the file sharing, use the full screen resolution etc etc straight away.

So… first impressions? Much more slick than 11.10 which I was using on a daily basis until recently. In particular, the configuration options have been streamlined really nicely. I’m still struggling with discovery of applications in Unity but in general, it’s not bad at all.

Browsing through the available packages, I was interested to find that the Cloud Foundry tools are available in the default repositories:

That’s awesome! you can just run sudo apt-get install cloudfoundry-client and get the main tool for deploying apps and administering Cloud Foundry right from the repository using the regular apt method (actually this is simply a convenience package – under the covers it installs a package called ruby-vmc, which installs the vmc command-line Ruby gem… it’s nice that the Ubuntu guys have made it easier to discover, though).

So what’s the problem? Well – no big deal, but actually, if you want to keep up with Cloud Foundry as a developing platform, you might want to grab something slightly newer than what is available on tap in the repository. As I write this, the version of vmc available via the cloudfoundry-client package is 0.3.10 and the one we’re currently working with is at least 0.3.16.

My suggestion, therefore, is to do the following:

1. Install rvm (Ruby Version Manager). That way, you can have different versions of Ruby itself installed, manage gems for the different versions independently, and also – importantly – not require root privileges to do your development work and install additional gems. A handy guide to installing rvm on Ubuntu is here, and it still works fine on 12.04. Just follow the commands shown in the Installing RVM and Installing Ruby sections and you should be all set and rocking ruby-1.9.2 on your new 12.04 setup.
   (I’m using rvm and JewelryBox to manage Ruby versions on OS X, incidentally. Great tools)
2. Run either gem install vmc or gem install vmc --pre (the latter option will get you the very latest pre-release of vmc, if you like the bleeding edge). Note that, if you installed rvm and Ruby successfully, you should not need root permissions to install gems.
3. There is no step 3 — vmc target http://api.cloudfoundry.com and then vmc login and you should be good to go. Looking for a sample app to deploy? You could take a look at the Sinatra example I added to the cloudfoundry-samples repository on Github last week…

It’s fantastic that Ubuntu is moving towards strong desktop / development environment support for Cloud Foundry.

Oh, there’s another story with Precise Pangolin,  too – you can rapidly install the server side pieces to build your own cloud using the Juju Charms which provide Cloud Foundry support. But… that’s a story for another post, in another time and place….

Update 15/05: I raised bug #998111 against Ubuntu to ask for the ruby-vmc package to be updated, in case you feel like tracking progress via Launchpad.

Well – that was bracing!

I don’t expect to be posting “week notes” like this on a regular basis, but as a one-off it seems like a nice way to encapsulate just how much happened in the first week of my new role.

Tuesday

Joined VMware. Met new colleagues in London office. Started to look at the User Account and Authentication component in Cloud Foundry. Ran samples against cloudfoundry.com, modified the documentation. Issued first GitHub pull request

Wednesday

More hacking on samples. Updated the Cloud Foundry Google+ page. Agreed to present Cloud Foundry at the London Real-Time hack weekend for the RabbitMQ guys. Watched (and tweeted) the live webcast of the Cloud Foundry first birthday event in San Francisco – very exciting news! “more clouds, more community, more code“, including a broader range of partners, a new governance process around the cloudfoundry.org Open Source project, and the announcement of BOSH being released to the community, too – multi-cloud deployment, here we come!

Thursday

Trip to the VMware Frimley office for some HR stuff. First call with full Developer Relations team for event planning. Briefing with the two directors I work for. Nothing to see here, move along…

Friday

Setup new laptop (custom order from Apple so there was a small delay). Prepped for demo at London Real-Time. Started making a lot of noise on VMware Link (aka Socialcast, the social sharing/discussion platform) internally

Saturday

My first speaking gig – a Lightning Introduction to Cloud Foundry, taking Chris Richardson’s much more comprehensive Boot Camp presentation and cramming the essentials into ~15 min including a live demo, for a bunch of hackers at London Real-Time.

Oh, and it was caught on video.

Four or five of the hacks ended up running on Cloud Foundry, too, which I think was rather nice I was also interviewed on realtime and the importance of cloud at the event, but I’ve not seen that video appear just yet.

Monday

More laptop setup, HR stuff. Prep for Scala Days. Started to improve a sample app (Ruby/Sinatra) I’d used in the past by adding Twitter Bootstrap and restructuring the code.

Tuesday/Wednesday

Scala Days in London – helping to man the sponsor stand talking about Cloud Foundry, answering questions, and meeting many new colleagues from the US who were presenting on Spring, Scala, and Cloud Foundry (including an announcement that Play 2.0 framework support and standalone apps are coming to cloudfoundry.com Real Soon Now). Recorded a podcast interview with Uhuru about what a Developer Advocate does.

Summary

I’m pleased that I was able to be so productive so quickly. I’d had a little previous experience with Cloud Foundry but it’s a testament to how quick it is to learn the basics and get moving that I was able to rapidly start playing with a bunch of code. It was also exciting to be out on Github on my first day – not something I could have done in a former life… it’s nice to be working in an organisation that is innovating with Open Source at this level.

There’s much to learn, and to be honest, a couple of the key aspects of Cloud Foundry actually make it more challenging (and interesting) for me to get to grips with. It’s open, and with BOSH, can potentially target different IaaS offerings (initially vSphere and the beginnings of AWS support; a hackathon yesterday aimed at adding OpenStack to the list) – so suddenly I need to know about those. It’s a polyglot platform, which means I need to broaden my language knowledge – I’m already making a start on Ruby and node.js, to complement existing Java and PHP knowledge.

It’s also exciting to learn more about what VMware does, the layers of technology that they offer, and their vision. My previous experience has primarily been with the desktop virtualisation technology, but there’s a huge and vibrant community around the server-side virtualisation tools, and products like Socialcast, Sliderocket and Zimbra in the collaboration space too.

There’s a lot of exciting stuff happening in this space. It’s thrilling to be here. Thanks to all of my new colleagues for a warm welcome and support – looking forward to working with you!

This is a thrilling opportunity for me for a number of reasons.

• from a technology perspective: Cloud Foundry is very, very, very cool. In my opinion, it really comes from a different set of thought processes than the other Platform-as-a-Service offerings out there, which make it unique and compelling.
  • the operating system stuff gets out of the way (why should it matter?), but multiple language runtimes and backend resources are available for easy scaling. Seriously, the first time I walked through the command-line tutorial and scaled a Ruby app to 6 load balanced instances with a single command, I was instantly impressed.
  • it is Open Source. The code is on Github. You can run your own cloud if you like. You can add support for your own languages and frameworks, much as AppFog have done for PHP, Tier 3 and Uhuru have done with .NET in Iron Foundry, and so on. This provides a huge amount of flexibility. Oh, and of course mobile and cloud go hand-in-hand, so last week’s announcement of FeedHenry providing tools to develop HTML5 apps to deploy on Cloud Foundry was really significant, too.
  • you can take your cloud with you using Micro Cloud Foundry – so the development and deployment model remains the same whether you are online or offline. I love this idea.

• for me, personally: it’s a natural evolution of much of the work I’ve been doing over the past few years – focusing on developer communities and promoting technology adoption, as much as top-down solution selling. As my good friend James Governor is fond of saying and as his colleague Steve O’Grady wrote, developers are the new kingmakers – and with trends like mobile, cloud, and devops, nurturing those communities is more important than ever. You don’t impose technology on a community – you explain it and earn your place and reputation.
• I’m looking forward to more speaking, more writing, more mentoring, and more online community building. These are things I’ve grown to enjoy (and in the case of the latter, appear to do naturally).
• I’ve followed Patrick Chanezon, the Senior Director of the team, since he was setting up the developer advocacy programme back at Google – I have a lot of respect for what he’s achieved and the way he operates, so I’m delighted to have the chance to work closely with him. I’m excited to join everyone in the team, of course – I have spoken with most of the group already and I’m really looking forward to learning from their diverse range of experiences and backgrounds.

Between now and April 10th, I have a few things planned including a vacation (!), heading to EclipseCon to talk about MQTT and M2M topics, and some other speaking engagements. After I start the new role, I expect I’ll join in on the Cloud Foundry Open Tour and start to meet folks. I’ll also be on the team for the GOTO conference in Aarhus in October – exciting times ahead!

The initial proposal and setup stages have taken a little while, but this week the initial availability of the C client code was announced on the Paho mailing list (Java will follow shortly).

Paho Quickstart

This is not intended to be a comprehensive guide – better documentation etc will emerge over time – but I thought I’d post a quick guide as a kickstart for anyone wanting to give it a look. I did this on 64-bit Ubuntu 11.10 – similar steps will apply on other Linux or UNIX platforms (note, the initial code contribution has a Makefile with rules which should work on UNIX, Windows, or z/OS).

Install the necessary packages to build code. NB git is for grabbing the source from Eclipse; build-essentials is a metapackage providing gcc etc on Ubuntu; and doxygen and optional graphviz are used for generating the documentation.

sudo apt-get install git build-essentials doxygen graphviz

Get the code from the git repository:

git clone git://git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.c.git

Quick build for the client library and documentation:

cd org.eclipse.paho.mqtt.c.git/src

make -f ../build/Makefile all

doxygen ../doc/DoxyfileV3ClientAPI

Once these commands complete, you should be left with subdirectories called <platform> and docs. In my case, <platform> was 64-bit Linux, so I had a binary at linux_ia64/libmqttv3c.so. There’s no “make install” rule at the moment, nor is there a rule to compile the docs so I had to run doxygen directly. In the future it would be nice to automate all of that, and also to build some test applications.

Opening docs/html/index.html in a browser reveals very nice documentation describing the client library, including some examples of how to use it. For example, in docs/html/pubasync.html there’s a complete listing for an asynchronous publisher application. I extracted that code into pubclient.c and decided to check that it worked!

gcc -Wall pubexample.c -L./linux_ia64 -lmqttv3c -lpthread -o pubexample

That command successfully built a binary called pubexample. All I needed to do was test it. The sample application assumes that an MQTT broker is available on localhost port 1883 – if you want to change that, simply modify the value of the static variable ADDRESS in pubexample.c – in my case I simply apt-get installed the mosquitto and mosquitto-clients packages onto my system, but I could equally have unzipped and run Really Small Message Broker – both start on port 1883 by default if not given alternative configuration.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./linux_ia64/

./pubexample

Waiting for publication of Hello World!
on topic MQTT Examples for client with ClientID: ExampleClientPub
Message with token value 1 delivery confirmed

It was trivial to test that a subscriber (mosquitto_sub in my case) also received the publication. Job done!

Getting involved, and other news on Paho

I mentioned that the Java client contribution should appear soon. One other piece of news this week is that the project’s sandbox broker implementation – based on mosquitto – has been spun up. That was posted on the Paho mailing list, and if you want to get involved you should definitely subscribe to that; start to track the Eclipse Bugzilla for Paho; watch the Paho wiki; keep an eye on the source repositories; etc.. I’m already thinking about getting an OS X build rule sorted out. If you want to test your sample code now, you’ve got the option of a local broker, the Eclipse Paho sandbox, the mosquitto sandbox, or various other implementations.

Oh – and please leave a comment on this post if you find this information interesting, or want to discuss where things are with Paho. I’ll be hanging out on the mailing list as well.

What about Bob? (or Andy, even!)

Well, although I’ve left IBM, I’m delighted that MQTT is now going Open Source – in fact that was one of the things that I really wanted to help to achieve before I moved on. I am really pleased that I will be able to continue to contribute to both Paho and the broader Eclipse M2M Industry Working Group. I’ll be helping to update the mqtt.org community site, and heading over to EclipseCon in Virginia in a couple of weeks’ time to talk about M2M and work with our friends from the Koneki project. If you are attending EclipseCon please come say hi to me – and you may be interested in Wes Johnson’s session on MQTT and Eclipse tools.

There are very cool times ahead!

The Christmas 2011 Big Finish special subscriber-only release, The Four Doctors, contained characters named “Lady Cowen” and “Whitmore” – a lovely little reference to Laura and Tony, also huge fans of Dr Who (as well as hosts of the Ubuntu UK Podcast[1], and some of the folks behind OggCamp). Incidentally – I reckon The Four Doctors is probably one of the best and cleverest Big Finish plays, and certainly it’s my favourite. If you’re into Dr Who, then it is worth a listen.

A subsequent Companion Chronicle for Big Finish – Ferril’s Folly – contained a brilliant line which referenced another friend, my mentor Dr Andy Stanford-Clark (yes indeed, he of MQTT, mousetraps and ferries fame).

I’ve just listened to Peter’s new Blake’s 7 audio play, Counterfeit, performed by Gareth Thomas and Paul Darrow. I’d been tipped off via some tweets that there should be a nod to me in this one… initially, I’d jokingly whinged that it was only a concealed reference in the dialogue rather than a namesake character, but then I heard the line itself:

… but then he twittered on, about chronon bridgebuilding, and deep hyper messaging connections…

Genius! Thank you, Peter – very nice, and I’m honoured

[1] this also gives me a handy opportunity to mention that I was a guest presenter on the first episode of season 5 of UUPC, which was live-broadcast on the interwebs on Tuesday, and released as a download yesterday. Check it out!

Background

At January’s London Internet of Things meetup, I had the privilege to hear Haiyan Zhang speak with passion about various topics, including how she had collaborated with hackspaces in Japan in the aftermath of last year’s earthquake and subsequent nuclear disaster.

It was only the first time I’d come across Haiyan, so I was surprised but delighted that she invited me to the OpenIDEO Make-a-thon this past weekend, after tweeting about events like the London Green Hackathon.

I had only a vague idea what to expect of the Make-a-thon. When I saw some of the project briefs being published ahead of the day, I knew that it would be a little different to hackathons and other tech events I’d been to in the past. The briefs spanned issues such as improving local communities, bike safety, and several supporting campaigns by Amnesty International to use technology to support human rights activities.

My initial impression that it would not be a “run of the mill” tech event was reinforced when I arrived at the IDEO offices in Clerkenwell on Friday afternoon – it was a very different crowd to the ones I typically encounter – full of product designers, makers, human factors specialists, as well as web coders and developers. I rocked up with a bunch of Nanodes and other electronics with a vague thought of doing something hardware-related, but in the event I didn’t get that far!

IDEO’s typical approach to design revolves around prototyping and directed brainstorming, and in the event we divided into 8 teams of around 6 each, with diverse skills but with common interests around the briefs on offer. Friday afternoon was spent first understanding and exploring the brief, and then rapidly prototyping a rough idea before presenting it to the rest of the group. Saturday was spent refining the idea and producing an “experience prototype” which was intended to have been tried out “in the real world” if possible.

Evolution of the “Karma Phone”

Several of the briefs interested me, but I joined the team focused on the concept of Postcode Gangs – how could we build something to develop and improve community facilities within a postcode – essentially an arbitrarily-delineated area – in London? We spent some time brainstorming ideas around what “makes” a community before needing to rapidly decide on something to build for our rough prototype.

“What if” – there was a ringing phone in the middle of the street – and on the end of that phone, someone who knew something, had something to offer, or who was needed something? “What if” – we could create a new local hub with current and historical information about an area, enabling people to explore and meet their neighbours?

So the first prototype of what came to be called Karma Phone involved a lamppost (named, erm, Dan!) with a phone on it, which would randomly ring as people passed by – people could call it with a need and that others could then try to address. On the other side of the lamppost (also known as, Dan’s back) we imagined a large touch display with information about current events, realtime information, historical maps, and so on.

Karma Phone – The Outcome…

The team changed overnight, as Hayley and James were not able to stay for Saturday, Lydia joined us, and Victoria could only be involved for a short time on Saturday. We weren’t all convinced that a ringing phone would be answered, that the system wouldn’t be abused, that there wasn’t a social barrier around providing home address and asking for help, etc. How could we get an actual phone into the street and ringing, too? So, Saturday morning involved some rapid rethinking of what we wanted to build!

We settled on what turned out to be a subtle evolution of the original idea – a public phone which could act as a hyperlocal information service and skills exchange.

While we were brainstorming how to hack a physical phone, run a long cable into the street, use a mobile chained to a metal box, etc etc I remembered Twilio, which I’d been following for a long time, but never had the chance to hack on in anger. Within about 30 minutes I’d demonstrated the ability to initiate calls between two users from a web page, to the rest of the team.

Steve and Dan set about implementing the web UI; Tim started working on a physical enclosure; Victoria and Lydia managed to source a real “traditional” phone handset; and I remained hard at work writing PHP to talk to Twilio.

A couple of minor wrinkles along the way:

• network issues meant that I had to use Tim’s phone to tunnel through to my webserver’s console, since it was apparently impossible via the event wifi. Evidently IDEO had just had a network provider change, so it was just an awkward time, but I lost some time fiddling with hosting in the early part of Saturday.
• at a certain point on Saturday afternoon, I realised that attempting to call from the Twilio web client on the iPad was never going to work… since it requires Flash. I thought of a number of workarounds, but the one that finally stuck was that we were able to use Skype on the iPad, and use the skype:// URI scheme to launch the app from the web client. It wasn’t seamless as we needed Skype credit, and also had to tap an extra “call” button in order to start the call, but it was good enough for a prototype.
• I’d wanted to make the web app, a standalone launchable web app on iOS. Weirdly, adding the usual meta tags to the page header to instruct iOS to treat the app as standalone launchable, meant that it was no longer possible to invoke Skype from within the web UI… so I backed off from that idea. The only cosmetic issue that presented was an inability to hide Safari “furniture” like the header, but that wasn’t a big problem for a prototype.

Here’s how the final system hangs together:

Impressive Outcomes…

I spent so long coding and tweaking on Saturday (the commented and documented code is here – ignore how short it might seem – it was an intense few of hours!) that I missed most of the physical assembly. Tim and Dan did an amazing job of creating an enclosure for the iPad and handset. It might have been made from foam board, a box folder, and vinyl, but the final result was beautiful. And most importantly – it was fully functional!

We would have loved to get the prototype out on the street for public testing (I suspect none of us more than Steve and Lydia!), but time worked against us. The final experience prototype was presented as a live demo with willing audience volunteers – one example call going to an answering service, and the other redirected to the local expert on Scotch Eggs (Tim!).

I’m happy to say that Karma Phone won Best Digital Prototype at the event, and I was (apparently!) Best Tweeter. Nice accolades

So – conclusions? I really enjoyed the way we worked together as a team of very unique and different talents; and seeing the Karma Phone prototype realised so brilliantly. However, I also think the experience of the Make-a-thon was humbling… listening to the experiences of people illegally detained abroad, and seeing some truly brilliant ideas from all 7 of the other teams, was wonderful.

A huge thank you to everyone involved in the first IDEO Make-a-thon – a really unique hackday. The IDEO team in particular looked after us brilliantly, with superb facilities, a great welcome, and more-than-adequate quantities of the hacker staples (coffee, sweets, pizza and beer).

Read a full recap including information on all of the project briefs on the OpenIDEO site. There’s a gigantic set of photos from the IDEO team, and a much smaller one from me shot on an iPhone at lower quality.

Tim, Dan, Hayley, Victoria, Steve, James, Lydia – Thank You. It was a pleasure!

All of the other teams – you rocked. You did great things. I salute you!

Event reports:

• London Green Hackathon
• Monkigras
• Hack to the Future
• bcs Oxford talk on Connected Planet
• IDEO Make-a-thon

Other “stuff”:

• what I’m doing next (!)
• Project Nanode
• Using MQTT for 2-way device control
• In defence of the Nintendo 3DS
• Defining the Empty Room Problem

As I’ve repeatedly written over the past few years – IBM has been a company I always aspired to work for, and once I had the chance, one that I’ve been immensely proud to represent. It’s a company that has endured over a century, and one that I was able to spend time with for a tenth of its existence – it was really the age of both WebSphere and the rise of IBM Software Group, and I’m grateful for the opportunity to have been there.

I have brilliant memories of the past decade. IBM is an amazing company and I will always value the chance to be a part of it, particularly in a wonderful location like the Hursley Lab. The people I’ve worked with, and with whom I’ve formed what I believe will be enduring friendships, have been simply outstanding. There were so many opportunities to do great things, not only in “the day job” but also as a BlueIQ Ambassador and social collaboration advocate, with IBM developer communities, in the universities programme representing IBM at careers fairs and as a guest lecturer in degree programmes, and the schools and community programme as a BlueFusion volunteer and mentor to kids at schools in deprived areas. I’ve also loved the chances to learn from others formally and informally, and to act as a mentor to others.

This will sound like a total paean, but it’s very true that there are amazing talents around IBM. In 7 years in IBM Software Services, and more than 3 years representing the development, strategy and product management teams in the lab back out to the field, I amassed a list of friends and colleagues from across continents, business units, and brands. It’s amazing to think of the broad reach of my network and I can’t help but be grateful for that.

My next steps are still forming; but I’m looking forward to spending more time with Open Source communities, with developers, with new technology, with connected systems and the Internet of Things, and as a speaker and writer. I’m also grateful to a range of friends for their support, particularly in taking over initiatives like eightbar, and in enabling me to remain involved in strands like Eclipse and MQTT.

Thanks for following me, reading my blog, sharing my thoughts, and joining the journey. I hope what comes next will be a continuation of the path I’ve been on; and an exciting next step in developing the direction I’ve been headed in.

In the last week I’ve had a couple of interesting experiences related to IBM.

First of all, I visited the Oxford University Careers in Computing Event.  I’d been up to Oxford in November for the wider University careers event, but I enjoyed the opportunity to talk with science students about what the company is all about. IBM helped to invent modern computing; to put man on the moon; it invented the PC, the floppy disk, various storage advances; it helped decode the human genome; it built the machines that defeated humans at chess and at Jeopardy; it is helping to build a Smarter Planet. It’s a great place to be.

Secondly, I helped to host some US colleagues in our UK lab at Hursley. I love Hursley and I’ve been enormously privileged to work there for the past few years. I remember my first experience of visiting IBM there as a customer in ~2000 – seeing the wonderful Wedgewood Room, the IBM consultant I was working with dropped the thought that one day I could work there into my head, and I’ve spent a long time wanting to work there, getting to work there, and then learning the history and showing it to others. Wonderful place.

I’m proud to have had the chance to work with an organisation that has helped to reshape and change the world. The quality of the people, the history of the organisation, and the amazing technology, has transformed my life.

Tagged: deep blue, history, hursley, IBM, oxford, University of Oxford, watson ]]> http://andypiper.co.uk/2012/02/04/reflections-on-ibm/feed/ 0 andyp http://andypiper.co.uk/2012/01/17/new-podcast-games-at-work/ http://andypiper.co.uk/2012/01/17/new-podcast-games-at-work/#comments Tue, 17 Jan 2012 19:10:57 +0000 Andy Piper http://andypiper.co.uk/?p=2522 Continue reading →]]>

The undead corpse of @dogearnation, rising again as new Games at Work podcast! bit.ly/AtPY2Z Braaains... I mean: Gaaaames!

—
Mikael Haglund (@0xdeadbeef) January 15, 2012

I’ve  been podcasting fairly regularly for the past few years, primarily with my good friends Michael Martine and Michael Rowe over in Durham, NC on a weekly show called Dogear Nation.

As I’ve travelled more, and as we’ve all got more busy, it became harder to keep that momentum up. I know I, for one, was tired and looking for some fresh inspiration. When we reached episode 200 last year, we announced a hiatus.

This year, we’re starting to ramp up something new. Same presenters, different format – going back to basics, if you will. Probably not weekly, more likely every two weeks… but continuing to explore some of the themes we’d been looking at around how gaming technology and concepts can influence business, work and productivity.

We’re still working on branding etc but you can grab the first episode of Games at Work right now via the existing site. We’d be interested to hear your thoughts.

These are misconceived and threatening to the free and open operation of the Internet – and they have far-reaching implications that go far beyond the shores of the United States (got to love asymmetric extradition…). They need to be stopped.

One of my earliest memories is of kneeling on the floor at the back of my bedroom making LEGO cars – it was in version 1.0 of my bedroom as I grew up, before new furniture and decoration. I must have been about 4, or 5. I had a castle, knights, some space stuff including base boards with little moulded “craters”… lots of fun as a child.

When I became a man, I put childish ways behind me.

I’d long known that many of my friends and colleagues have remained huge LEGO fans (Cerys has just blogged about her interest; Ben made some fun timelapse videos of building his Christmas present). For me, a key moment was Roo‘s 3 minute masterpiece of a paean to the medium at Interesting in 2008, embedded here for your enjoyment. Listen to the audio slidecast – closest you can get to having been there, and Roo did a wonderful (and amusing!) job.

Also, a memorable talk at the CRIM Crystal Ball Conference in Montreal in April 2010 (at which I also spoke) came from then Professor of Innovation at LEGO Group, David Robertson – a tale of Rebuilding LEGO, and how the company had saved itself from bankruptcy by refocusing on its core values and customer needs. It was a fantastic story and I was rapt.

More recently, I went along to the Internet of Things meetup in London last month, and was delighted to see Ken “monsonite” Boak – creator of the Nanode, a fantastic UK-grown prototyping platform akin to Arduino – use LEGO as his metaphor for a talk exploring Open Source electronics. Ken was kind enough to pop his slides up on Slideshare today, so you can take a look. He’d just been out to get some LEGO the previous weekend…

I want some LEGO. I own a single minifig. This is not good enough. Where can I get LEGO in Guildford.

—
Andy Piper (@andypiper) January 14, 2012

That talk was more-or-less the moment when I realised – I needed some LEGO. I wanted some. Both as a way of seeing where things had gone to, and to help me to prototype things, and just… well… just because! I’d already started to use dioramas featuring minifigs in a couple of presentations recently and had good feedback, so I figured that was another excuse

So, on Saturday I decided to dip back into my passion for LEGO. It started with a bucket of bricks from the nearest toy shop… but then I noticed the LEGO Star Wars sets with slight discounts[1]… and I figured well, obviously I’d need some wheels of some kind so picked up some City sets… and some of the foil-bag Minifigures…

The splurge quickly developed into a binge via a @darachennis-inspired trip to the LEGO store in Westfield White City on Sunday… picking-and-mixing bricks from the back wall, and signing up for the VIP program. There may be no hope left for me…

So what have I learned?

• Minifigs are brilliant. The aforementioned David Robertson gave me his business card, his details printed on a minifig resembling him, in Montreal in 2010 and that reawakened my interest. When I was a kid they all had the same pair of staring eyes and identical pleasant non-threatening smile, but the range of looks and expressions now available make them as much fun to customise as the full sets.
• People talk about the beauty of Apple’s designs – both inside and outside of the product (not that I’ve ever cracked open an iPhone to look inside). LEGO is blocky and “harsh”… but the designs and assembly process is beautiful. Assembling little cars and other sets on Saturday evening, following simple pictorial instructions, I realised that every piece had a place and it all fitted together wonderfully, perfectly. That (re)discovery had me as delighted as an adult, with a more architectural and design-oriented brain, as I was as a kid with the sheer enjoyment of being able to build and modify things.
• In my opinion, all kids should be given some LEGO, and allowed to build the models from the boxes themselves (much though I’m sure as an involved adult I’d be itching to take over!). I’ve blogged recently about my excitement for the maker culture, and this is really where it can all begin.
• I need to keep an eye on my bank balance, and a check on my excitement. I love it, but I bought it for “professional” reasons…

"children don't need to spend a year learning PowerPoint, and get a GSCE for it... They need digital Meccano" @ian_livingstone #futureteched

—
Andy Piper (@andypiper) January 12, 2012

Last week, the UK Government announced that ICT courses would be replaced with Computer Science, including a programming element (one of the campaigns I’ve been passionate about). At an event from The Education Foundation in London the next day – The Future of Technology Education – I was privileged to hear one of my personal heroes Ian Livingstone (of Fighting Fantasy books, Games Workshop and Eidos fame) speak and refer to “digital Meccano” – and I owned Meccano as a child too.  He also highlighted the need to combine science and art to push the digital boundaries.

Here’s what I think: we should be giving children a choice of physical LEGO, Meccano, and other toys; encouraging their creativity and building skills; and helping them to bridge between both the digital and physical worlds. No child should be excluded, and none should be pushed down a particular path. We should be supporting and helping every child to discover their passions and explore them; recognising that not every individual will want to program, or draw, paint, build, or write – but never belitting anyone for their talents or interests.

I’ve rarely been as excited about the future than I have been right now!

[1] as a child in in the 1980s I owned significant numbers of the Palitoy Star Wars figures and vehicles[2]. Whoever thought of combining LEGO and Star Wars is a genius – so much MORE FUN than the original, inflexible, non-customisable toys. So much more interactive, and through the video games, adding a humorous new twist on the Star Wars saga. LOVE.

[2] … I never had the Millennium Falcon or the Death Star, though… always wanted those…

WMB v8.0 was announced back in early October and has just arrived ready for  download in versions for distributed platforms, System z mainframes, and as a Hypervisor Edition for Linux and AIX (to be provisioned via the IBM Workload Deployer appliance).

As I did with WMQ last month, I wanted to take a moment to break out and highlight some of the key things in this release that you may have missed from the announcement letter. This won’t be a comprehensive list of everything, but I  want to point out some of the cooler features that you’ll want to be aware of. So, here we go…

(I’ve included a few screenshots to whet your appetite, click for larger versions!)

A simpler development experience

Version 8 brings a number of enhancements to the development experience, but one worth highlighting is what we call “Apps and Libs” – the idea that sets of message flows may be grouped into a unit called an Application which can be deployed, stopped and started as a whole. With Libraries, there are also truly re-usable assets like .esql files, or sub-flows, which can be deployed and updated separately, and invoked dynamically at runtime. This is a key change in the way that the Broker works – previously, sub-flows were compiled into the main flow and changing one required redeployment of all flows using it… they are now dynamically linked when needed, so they can be deployed and replaced more easily.

A new standards-based parser and message modeler

A new Data Format Description Language (DFDL, which you’ll sometimes hear called “daffodil”) enables any text or binary data to be understood within the message model. The Broker has had the “MRM” for many years, so of course could already do this, but DFDL is a new industry standard which can supersede the MRM (of course, you can continue to use your existing flows and message formats – you’re not forced to use DFDL). There’s a new mapper, too.

More importantly, coming along with DFDL and the mapper is a really, really nice set of utilities for testing message models inside the Toolkit – you’ll now be able to confirm that the model matches the test data without having to go through a full model->deploy-> test-at-runtime cycle. I saw this demo’ed at the WebSphere Technical Conference in Berlin during October and was blown away by it – it would have saved me a lot of time back in my consulting days!

Comprehensive .NET support

If you have .NET applications, assemblies, or services on the Windows platform, and you want to access those from your message flows – you can. If you want to write your message flow logic using C# or VB.NET or any .NET 4.0 CLR-supported language, using Visual Studio – you can.

If you don’t know how to get started with this stuff, the Toolkit has a new .NET Pattern to lead you by the hand and get you going quickly, and project wizards for Visual Studio.So, if you want a high-performance ESB platform that connects “anything to anything”, with minimal need to learn new skills, and run it on Windows with deep .NET integration – this release is going to cover your requirements.

Web administration

Delivered in version 8 is a first stage in making the Broker more easy to administer from a lightweight client – a web browser. Whilst power users and existing administrators can continue to use the Message Broker Explorer GUI, there is now an easy way to enable an optional web interface for basic administration tasks. Continuing the theme of simplicity the product has followed for a while, no additional moving parts (app or web servers) are required! Version 8.0 provides read-only views of running Applications and access to the log – more capabilities will be rolled into this interface in the future.

Record and Replay

Sometimes, when you are dealing with a set of end-to-end flows of data between applications, you may want the capability to record what is going on, and to replay specific scenarios and sets of events. This could be the case in audit, test, and many other scenarios. Another of the massive enhancements in version 8 is the Broker’s response to this requirement – again delivered using the same simple, lightweight interface offered by the web administration tool.

This also builds on technology around monitoring that has been progressively built into the Broker over the past couple of releases, so there are some really solid foundations and it is straightforward to set up.

Richer, yet easier to use

Just as I highlighted in my piece about WebSphere MQ 7.1, the Hursley teams have been strongly focused on “consumability” (translation for non-IBM-speakers = UX) for a number of years now. WMB continues to add capabilities that make it a richer, stronger integration platform, but also smooths out rough edges seen in earlier releases and is just… well… more productive to use. There’s even a drive to reduce the jargon and make the Broker logs more easy to understand, with new Activity Logging which aims to explain what a flow is doing in plain language (“GET message queue X”, “Update DB table Z”, and so on).

Taken together, the new wizards, web interfaces, integrated testing tools, message modelling tools, reduced dependencies, lightweight deployment with apps and libs… the combination just makes it a much more enjoyable experience for developer and administrators. And there’s a new installer, too.

The “papercuts” and node additions lists are huge: new JMSReceive node; new options for the File nodes; new Connect:Direct nodes; WS-ReliableMessaging support in the SOAP nodes; ability to install without root privileges; dynamic configuration of services without the need to restart execution groups… the list just goes on! Check out the product Information Center for more details on all of the features I just don’t have space to list.

… and finally…

Huge congratulations to some hard-working development teams in Hursley, Toronto and Bangalore in getting this release out there. As I’ve said before, I’ve been using the Broker for 10 years now and it just keeps getting better, and better. These guys are a very strong set of developers who turn out a fantastic, high quality product every time. Special thanks to MGK, @mqmatt, and @domstorey for some of the screenshots in this post

Footnote: version 8.0 is friendly to developers who use Ubuntu, too! Anton (my go-to guy on all things Debian – listen to him!) has some good advice about running WMB or WMQ on Ubuntu and Debian.

Related articles

• WebSphere MQ 7.1 is out – here’s why it’s cool… (andypiper.co.uk)
• European WebSphere Technical Conference 2011 (andypiper.co.uk)
• WebSphere MQ and Ubuntu (and other developer resources) (andypiper.co.uk)

I’ve spent most of October and November travelling to speak with customers and present at a couple of conferences around Europe. In that time, I generally had very few problems with network access.

On one occasion though, I realised just how tricky things are becoming, as “social” elements become increasingly baked in to the fabric of the Web. I was in Switzerland, and the plan was for me to present locally during the morning, and then to host and facilitate a conversation with a number of my colleagues in the Hursley lab during the afternoon. The hosts arranged guest wifi network access for me, so that we could make this work. I’d be able to use Sametime to receive files to present locally (we couldn’t access LotusLive), to clarify questions with the remote team, and to coordinate other team members to join the conversation as we went along.

This plan was initially all looking good, until I found that the VPN connection I was using to tunnel in to the corporate network would suddenly and apparently randomly, drop in the middle of a conversation.

After a while these VPN disconnections became more frequent, I became more frustrated, and the meeting became less productive.

… and that’s when I looked at the piece of paper I’d been given with my guest network credentials. To summarise, it said that guests would be subject to all of the same restrictions as employees regarding network access and specific sites were disallowed including “Personal email: Hotmail, Gmail etc; IM: Skype, Google Talk, etc; Social networking: Twitter, Facebook, etc”.

The penny dropped that my browser was sitting there with tabs open on sites like Gmail and Twitter. I shut them, reconnected, reconnected to the VPN, and things…. were better…. well, better, for a while.

I still wanted to use the Internet, of course, so I continued to do so – searching Google for relevant issues when questions were asked in the workshop. That’s when the VPN started flaking out again…. and that’s when I realised that with the Google redesign, the +1 features in the header bar were accessing Google+ when I loaded the Google page, treating that as a “social network”, and silently dropping my wifi connection.

This was a case where a heavy-handed filter, no doubt designed to “protect” the users from themselves and the organisation from inappropriate behaviour, actually impaired real work getting done. Either this technology needs to get a lot, lot smarter; companies need to reconsider these blocking rules, and trust an increasingly savvy workforce to behave responsibly; or the Web just needs to stop getting so darned social and… troublesome. Which option do you prefer?

I’m 35 years old. More than 20 years ago, I was at school, studying for my GCSEs and later my A-levels. One of the subjects I studied was English Literature. I love reading. I love literature. I love imaginative, creative writing.

There was, obviously, a set curriculum of texts I was expected to read, learn, and internalise. Shakespeare, Dickens, Hardy, and others. I’m glad I have that grounding. I was also allowed to read anything I wanted, from an early age – and I gravitated towards novelisations of Star Trek, of the Neverending Story, the Lion the Witch and the Wardrobe, and other science fiction and fantasy stories.

Around the age of 12 or 13 I stumbled upon the Dragonriders of Pern series. At the time, I was interested in Games Workshop and Warhammer… a fantasy world involving dragons and, ultimately, a rediscovery of technology, was an obvious step.

So, I started to read the Pern chronicles. I remember reporting on them in my “reading diary” aged around 12 or 13 – a series of books about a near-mediaeval planet where dragonriders saved the population from the deadly Thread. It wasn’t until I read The White Dragon that I really appreciated that this wasn’t just a trash teen fantasy series – themes of erotic passion, love, independence, adventure, and intelligence were involved (and would connect to science fiction, computing and other directions beyond that tale).

I’m deeply saddened to learn that Anne McCaffrey has passed. Her tales and her books truly did light up my early teenage years. I loved the Dragonriders of Pern stories and I hope that others will connect with them in the same way in the future. Thank you, Anne.

I’m going to spend this post talking about WMQ 7.1, which became available in electronic download form for the distributed platforms last Friday (z/OS will follow shortly). I’ll return to talk about all the (über)-coolness in Message Broker a little closer to the release date for that product.

So what is the big deal in this release?

It brings parallel / multi-version install

From version 7.1 onwards, there is now the capability to install more than one copy of WMQ on a system, for Windows and UNIX platforms. This includes installing alongside WMQ v7.0.1.6 (fixpack 6 on v7.0.1, the minimum level for multi-version install to work) – you can have one copy of v7.0.1.6, and multiple copies of 7.1, for example – and future versions will also be able to be installed in parallel, should the need arise. This should make migration and testing simpler. Applications can now point to their “own” install of WMQ if required. The GSKit installation, which provides some of the security functions for the queue manager, now gets installed “inside” the main installation as well, to make the whole thing more self-contained, and potentially easier to embed into other solutions if needed.

Here’s a teaser image from a Windows system that my colleague “mqjeff” sent me earlier today he has 7.0.1.6 and 7.1 on the same machine.

It’s (even more) secure

WebSphere MQ has always had a number of strong security capabilities, including SSL for channel authentication and encryption, and fine-grained access control of queue manager objects via the Object Authority Manager. It has also been possible to add transparent, per-message / per-queue / per-policy on-disk encryption and signing of message data via the Advanced Message Security feature. In v7.1, a renewed focus on end-to-end security adds the ability to authorise on a per-IP/user connection basis, as well as adding more crypto algorithms and additional authorisation options, and making much more of that security function available via the MQSC administration tool. T-Rob has a much more complete post about these changes so I won’t go into any more detail here.

It runs better, on bigger systems

Bigger systems… like the z196 mainframes? Well, that’s one example, yes, but WMQ v7.1 has been more optimised for big and multicore systems in general. On the mainframe, there are a bunch of great enhancements such as increased resilience in dealing with shared queues in a coupling facility, and the introduction of Shared Message Data Sets (SMDS) to significantly improve performance there as well. Let’s just say that the performance numbers for z/OS are looking really, really good… which brings me on to…

It continues to push the performance envelope

A major focus on performance in the v7.1 cycle has produced some fantastic results, and when the performance reports appear (as SupportPacs, within the next few weeks), you’ll see the “fastest WMQ ever”. This theme runs throughout everything: not just the base runtime messaging, but also things like making the WMQ Explorer tooling significantly snappier to operate as well (oh, and that’s now 60% smaller, and more sleek!)

There is also a new option for publish/subscribe applications – the ability to publish on a topic via multicast. This re-uses some of the technology from the WebSphere MQ Low Latency product so that it can run very fast. After the initial application startup, it means that applications can also operate when the queue manager is not available.

It adds Telemetry to the base install

No surprise that I’d highlight this one (it is also an important part of the overall story, per the next heading!) – I’ve been talking about the IBM implementation of MQTT, the open protocol which is being standardised and which it was just-announced will be part of the Eclipse Paho M2M project, for the past couple of years.

In WMQ v7.1, there is no longer a separate installation to run in order to add this support. On the platforms where the Telemetry feature is supported – Windows, Linux IA64, and (new in v7.1) AIX – this is now an optional part of the base installation. That means it is very easy to try out. Oh, and as well as being integrated with WMQ Explorer, the full range of Telemetry objects can now also be administered via the MQSC command line.

It brings the family together

This is a big one, in my opinion. I’ve mentioned that WMQ “base” can now interoperate with WMQLLM via the multicast publish-and-subscribe support; and the WMQ Telemetry functionality is “in the box” as part of the installer on the relevant platforms.

Why do these things that matter? Well, as I mentioned in my recent MQTT FAQ, something that IBM has observed over a number of years of building and delivering production-ready messaging middleware is that one size does not fit all. There’s the fundamental transactional messaging backbone (WMQ base) which needs to be solid, reliable, and easy to administer through comprehensive scripted and graphical tools… but beyond that, there are some additional qualities of service that need to be considered. There’s the very high speed, low latency use case which may be very specialised (WMQLLM), and there’s the need to deal with small and constrained devices and less-reliable networks (WMQ Telemetry / MQTT). Of course, you may also want to perform file transfer over that infrastructure (WMQ File Transfer Edition), secure your messaging (WMQ AMS), or route and transform your data and connect with “foreign” systems via different protocols (WebSphere Message Broker). I’ve been talking about this as part of IBM’s Messaging Vision for a number of years and it is really showing through in this release of WebSphere MQ. It’s a complete story.

It addresses many “papercuts”

On top of all of that… the team has really tried to address many of the common papercut issues, by which I mean the gotchas, annoyances, and the “wouldn’t it be so much better if….”s. Things like, gosh, I wish I knew what version of WMQ that client is using to connect to me? (yep, you can find out now).  How about “bind on group” for messages in a cluster? The ability to backup / dump and restore the configuration of a queue manager without needing to use a SupportPac? There’s a real sense of “fit and finish”, and I believe that shows that the development team have been listening to feedback and making the tweaks that users have been asking for where possible.

So – all-in-all, there’s a lot in this release that makes it worth a look, either from the perspective of users who are looking at an upgrade to gain performance, security and usability benefits; or for those looking for a solid, dependable messaging platform which can support modern applications. There’s a lot of excitement and innovation going on in the “traditional Message Oriented Middleware” space at the moment and WMQ and the related protocols like MQTT are right at the heart of those trends.

To learn more about the features I’ve talked about, and some that I haven’t, check out the online Infocenter. You can also check out the “What’s New in WMQ v7.1″ presentation from the WebSphere Technical Conference, via T-Rob’s blog.

It’s all about machine-to-machine (or M2M) communications, Smarter Planet, and the Internet of Things.

Before I dive in to this, a few clarifications. First, I’m being described in a couple of news stories as “an IBM distinguished engineer”, and whilst I wish that was true, I’ve yet to ascend to those heights! Also, there are various numbers being quoted – note that the figures in the press release were not invented by IBM, the headline number of an expected 50 billion connected devices by 2020 comes from a recent study conducted by Ericsson AB. Oh, and this isn’t about a “new” protocol – MQTT has been in use since 1999.

The other clarification is that some articles seem to suggest that IBM is out to create some kind of new, alternative, Web – that’s not what has been announced, and I’m certainly not aware of any such plan! It’s about connecting “things” – sensors, mobile devices, embedded systems, even small appliances or medical devices for example – to the Web and the associated platform and ecosystem of technologies, not about reinventing or recreating them. I’m personally a huge fan of the Web as a platform

Oh, and of course, the obligatory “all opinions expressed are my own” – this is my understanding of where things are going, although of course I’m talking about events I’m directly involved in!

So what is this all about?

Two things.

1. On Nov 2, IBM, Eurotech, Sierra Wireless and Eclipse formed a new M2M Industry Working Group at Eclipse. Sierra had already started the “Koneki” project at Eclipse to work on M2M tools, and the Working Group will look at a range of topics together, such as M2M tooling, software components, open communication and messaging protocols, data formats, and APIs.

2. On Nov 3, IBM and Eurotech announced the donation of their C and Java clients for MQTT to a new Eclipse project called “Paho” which is under proposal in the incubator – with code expected to hit the repository within the next couple of months. MQTT is being given to Eclipse to live within the M2M ecosystem that is emerging there, and to provide an avenue for adoption of the protocol as a more pervasive standard for connected devices.

How is that news? Isn’t MQTT already open / free?

Technically… kinda, sorta

The MQTT specification has been published under a royalty-free license for some time, and that has led to a fantastic community contributing a range of different projects. IBM and Eurotech took this approach from early on, because it wouldn’t have been possible to compile and support code on every embedded platform that might come along – far simpler to set the protocol free.

Initially the specification was hidden away in the WebSphere Message Broker documentation, but last year it was republished, moved to a new home on developerWorks, and the license was clarified.

In August, IBM and Eurotech announced their intention to take MQTT to a standards organisation. The specific organisation has not yet been finalised, but this is also an important step in ensuring that MQTT is not “just” an IBM protocol, but something of general use which the community can feel comfortable with. If you’d like to join that discussion then there’s a Get Involved page on the mqtt.org community site.

The missing piece was code – a reference implementation, if you like. That’s one reason why the Eclipse Paho announcement is significant.

Why else is this significant?

Well, here are some of my musings on that one:

• it shows IBM is serious, by committing code and open sourcing it (as with the original Eclipse donation in 2001);
• the M2M Industry Working Group exists to foster the discussion in this space;
• it makes high-quality reference Java and C client implementations freely available in source form, with a good Java implementation something that has been particularly lacking;
• it creates an opportunity for Eclipse projects to use MQTT, and to develop tools on top of it.

The press release and Paho project proposals aren’t clear (to me) – what exactly is being donated?

IBM is seeding Eclipse Paho with C and Java client implementations of MQTT. Eurotech is donating a framework and sample applications which device and client developers can use when integrating and testing messaging components.

Why C and Java clients (aren’t they “dying” languages?) Where’s my Perl and Ruby code?!

IBM had previously made some C and Java code available in some SupportPacs, but those are outdated and the license for reuse was never clear.

It’s important to realise that this stuff came from the embedded world of 10 (and more) years ago, and continues to be applied in that industrial space. That category of device typically runs some kind of realtime Java-based OS, or a Linux-based or other runtime with a GCC toolchain for the CPU in question. C and Java are genuinely the most useful implementations to get out there. Oh, and on that “those old languages” thing – I think you’ll find they are very widely used (Android, iOS etc run variants of sorts, most non-web app development is likely to be in one or the other).

We’re very fortunate that clients libraries for a wide range of languages already exist thanks to the MQTT community – see the list at mqtt.org!

Hold on… don’t we need a broker / server / gateway?

Yes. But, one step at a time!

There are brokers available for free today, either as precompiled binaries or as full Open Source implementations, so this is not a dead end from day one.

The Paho project scope outlines the intention to add a broker to the project in the future, and to host an M2M sandbox for developers as well. That is where we are today, and this position will evolve over time.

Why Eclipse?

The Eclipse Foundation has been a fantastic success story (oh, and, Happy 10th Birthday, Eclipse!). As the scope of their mission has broadened beyond an IDE to the web, build environments, and all kinds of other tools, it was a good place for Sierra Wireless to kick off the Eclipse Koneki M2M tools project, and is now a natural place for this primarily M2M protocol to be hosted under Paho. As James Governor notes in his write-up of the news:

… the Eclipse Public License is designed to support derivative works and embedding, while the Eclipse Foundation can provide the stewardship of same. One of the main reasons Eclipse has been so successful is that rather than separate software from specification it brings them together – in freely available open source code – while still allowing for proprietary extensions which vendors can sell.

How quickly will the code donation happen?

The Paho proposal tentatively includes dates in November and December 2011 – there will need to be various approvals as code is accepted into Eclipse, so that may “flex” a little, but it is all in the pipeline.

OK… Why MQTT? Why not HTTP/XMPP/AMQP/PubSubHubbub/WebSockets/etcetcetc?

To answer this one adequately I’d probably end up addressing each individual difference between protocols in turn, and if you’ve heard me speak about MQTT I’ve covered some of this before – so I’ll keep this answer relatively brief. I will admit that I’ve been asked about all of these by journalists in the past couple of days.

There is space for a range of protocols to coexist, because they address different areas. In the messaging space, we’ve found over time that whilst efforts to create a single protocol have been made, that has often ended up as focused around a particular set of qualities of service, and not optimised to cover the the whole range of them.

For example, if we look at IBM’s own messaging protocols – there are several. There’s WebSphere MQ which is all about reliable, transactional, solid, clusterable, enterprise, JMS and other APIs, etc etc.. WMQ itself isn’t ideal for very high-speed in-memory or multicast scenarios, so there is also WMQ Low Latency (interoperable with the new multicast feature in WMQ 7.1, but a separate protocol). Neither WMQ LLM or WMQ scales down to unreliable device networks and embedded systems, so there is WMQ Telemetry (aka MQTT), which was specifically designed for constrained devices and networks, and that can interoperate with the main queue manager, too. Oh, and sometimes you want to deal with files (WMQ File Transfer Edition), or access message data via HTTP (WMQ HTTP Bridge). You need to address a range of requirements in a messaging story.

So why not those others? In this case, IBM believes that MQTT is ideally-suited to the Smarter Planet Instrumented->Interconnected layer – it’s tiny, not synchronous and brittle, isn’t specific to the web as it is all about data rather than documents, XML etc etc. In these scenarios, REST principles may add an overhead. Oh, and it has been around for over 10 years, and has been proven across a range of industries and in a range of extreme conditions. IBM’s commercial implementation is known to scale to hundreds of thousands of connected devices, and we know that is the direction that this space is heading.

Congratulations! / Thank you!

Thanks, but don’t congratulate or thank me! I’m familiar with this stuff, I’ve coded with this stuff, but I didn’t invent it and I didn’t write it. There are some amazing folks at both IBM and Eurotech (and some who have moved on) who started this all off in 1999, and who have helped to implement solutions using this protocol since then, and who have of course developed it. Several of them are on Twitter if you want to say hi! And huge thanks again to the community of folks that formed around mqtt.org and contributed client and server implementations – that absolutely helped to move things forward to this point.

HERE ENDS TODAY’S Q&A!

That, hopefully helps to clarify a few things and answers some of the questions I’ve seen via Twitter, forums, and mailing lists over the past few days. It has been something of a blur, to be honest, but a lot of fun. I’m looking forward to the next stage – working with the community more, working with our friends at Eurotech, Sierra Wireless and elsewhere, and making the M2M space much more real.

For more, here are a bunch of stories I’ve seen in the past couple of days… no particular order, just my cut-and-paste list!

• Smarter Planet blog
• mqtt.org news
• Bob Sutor
• ReadWriteWeb
• PCWorld
• VDC Research
• WebWereld (NL)
• Technolenta (RU)
• ConnectedWorld
• Simon Phipps at ComputerWorld
• Complex.com
• Developer.com
• James Governor
• H-Online
• mosquitto blog
• Design Thinking Thoughts

I’ll be in Berlin next week 10th-14th October, participating in at least one panel, speaking about MQTT, and also covering the latest on IBM MQ messaging technologies as they relate to cloud and web. There’s a Lanyrd event page where I’ll try to collate information relating to the individual talks.b

I have a feeling that by this time next week there could be quite a lot to talk about…

Related articles

• WebSphere MQ and Ubuntu (and other developer resources) (andypiper.co.uk)

I’d previously explained to the organisers that I hadn’t spent so much time exploring virtual worlds lately as I was doing three or four years ago at the height of IBM’s involvement with platforms such as Second Life and our own internal Metaverse. Having said that, I have spent more time with gaming platforms such as XBox and the Nintendo 3DS since then, and more recently also Minecraft. Naturally I did have that business perspective and story to share… and, as the closing keynote I had the interesting task of pulling together the threads we’d covered during the breakout sessions at the conference, as well as attempting to look ahead to what trends might be important in the future.

The video is online via the Open University website and the talk with Q&A lasted for about an hour. More coverage of ReLIVE 11 is aggregated on Lanyrd.

Summary

As I noted in the opening and closing sections of the talk – predictions of the future are a hit-and-miss affair. We may now have tablet computers arguably even cooler than the Star Trek padds and communicators, but I’m still waiting on my hoverboard. Nevertheless, I tried to frame the story of IBM’s exploration of virtual worlds and 3D environments with some discussion of trends. It also gave me an excuse to talk about Back to the Future, and a cool ad that Nike recently released tying back in to the movie.

I want to reiterate (as it may not have been clear from tweets that emerged during the event) that these were very much my own thoughts and not the views of my employer – in fact, I was attending the event in a personal capacity. So, per the presentation, my thoughts on trends to watch in the next five years:

1. 3D Printing: I’ve seen RepRap and other 3D printers more often in the past couple of months than ever before, and it is clear that prototyping and fabrication are coming within financial and technical reach of more than just the early adopting minority. That’s not to say this is something I see going “mainstream” – but as access opens up, expect to see many more interesting things happening here.
2. Social broadcast: I think “TV” is rapidly giving way to a more generalised broadcast media that is being consumed across multiple devices, remixed, shared, etc. I also think that social streams are adding to the experience of how these media are being consumed, as evidenced by hashtags broadcast on BBC programmes, and the ways in which conversations form online around events and video streams.  A nod to my friend Roo Reynolds too, a man constantly way ahead of his time…
3. Touch and Gesture: we already know that the ways in which we interact with technology is evolving fast. Watch any child approach a large screen and attempt to press the screen, expecting their cartoon hero to become interactive. This is not going to stop – Microsoft have some amazing technology in this space with Kinect and we should get used to and embrace the changes as they happen if we want to evolve.
4. Big Data: a nod to my own organisation’s Smarter Planet story, and an acknowledgement that every one of the major tech firms is investing in ways to store, mine, slice and analyse the increasing amounts of data flowing in from the environment and our personal signals. This is just a continuing story, but we’re at a point where it is a red hot topic. It would have been a good point to mention Watson, if I’d thought on my feet quickly enough!
5. Identity: this is not so much something where we will see technical progress necessarily, as an area I think will be a threat, and difficult to resolve. The nymwars of Google+ are one edge of the issue. I believe that there is a real tension between the freewheeling days of the earlier Internet, the desire of individuals to make their own choices about identity (often for valid social reasons, other times for vanity), and corporations and political entities that want to close this situation down. This is going to be a tricky one.

So what of virtual worlds? Three words: Not Gone Away. They may have morphed, lost their early shine, the bubble burst – but we have a range of immersive experiences (and social, but not necessarily immersive ones) through which we interact. I mentioned Minecraft and how that is being used for teaching. I talked through IBM’s work with serious gaming. I spoke about the IBM Virtual Center briefly, and that’s online and used today – in fact Jack Mason just posted a nice deck on that which carries some statistics, if you want to learn more.

Thoughts on education

I clearly was not the most experienced individual in the room when it came to discussions about teaching and education, and I particularly enjoyed hearing different presenters at ReLIVE11 talk about how they are using OpenSim, OpenWonderland and other platforms. However – after my recent post on Raspberry Pi and my exploration of the Brighton Mini Maker Faire I’ve been thinking increasingly about Maker culture and how we could bring technology teaching back around to practical matters.  I was disappointed to read the Government’s (lack of) response to John Graham-Cumming’s recent letter on the same subject, though.

One of the things that I called out as a barrier to the adoption of immersive worlds and new technologies at work is something I’m calling The Empty Room Problem – the fact that unless you build it and then populate it, they will not necessarily come. I’ll be writing about this some more shortly, prompted by Derek Jones’ great blog post.

During the Q&A session I gave an answer to one of the questions which contained some ideas I’ve had on a possible curriculum – I’ll try to expand on those in the near future as well.

Of course, thanks to the changes Apple have made to the way in-app content is sold, with 30% of that sale going straight to them through iTunes, sellers like Amazon have decided to stop allowing digital content to be ordered in-app on the iOS platform. I can’t really blame them, although it’s interesting to see Microsoft rumoured to be following the now-established “30% App Store rule” for Metro apps Windows 8, with other content gatekeepers likely to follow suit, one would assume.

So I get a polite message telling me that rather than buying Jeff’s book, I can go ahead and add it to my wish list, change to the website via Safari (how many iOS users actually realise the browser is called “Safari”, incidentally?), and purchase it there.

The first thing the Amazon website wants to  do is entice me to download the Amazon App for iPhone. I’m using an iPhone, so why wouldn’t I want it? I smirk to myself and continue. It’s actually just as quick to repeat the product search on the website as it would have been to add the item to my wish list, find the wish list, and open the item.

Once it gets to the part where taking the money is involved, of course, Amazon have that just as well sorted as they ever did – one click and I am, as they say, done… well, apart from the part where the website assumed I wanted to book sent to my iPhone, since I was shopping from that device. Anyway… looking forward to reading Jeff’s new book on my Kindle later tomorrow!

No wonder Amazon want to just go out and build an all-in-one content and physical goods purchasing tablet.

Related articles

• Amazon Bypasses Apple’s New iOS Rules With Kindle Cloud Reader [Amazon] (gizmodo.com)
• Amazon Kindle Tablet Could Shake Up Tablet Wars, And Here’s How (pcworld.com)

I’ve been blogging here for a number of years now and it’s always fascinating to see what search terms lead people in. For the first few years it was a post on the UK direct.gov car tax renewal site, because people seemed to be typing the URL into Google and Yahoo (instead of the address bar) and hitting my site rather than the actual service.

From the results over the past 12 months it seems that people usually are looking for me, or for something on MQTT. Some of the other search terms, though, are quite surprising… Visio? VMWare? iMovie 09? it has been a while since I wrote about those.

On another note, I’ve now got the new Google Authorship markup working, so hopefully search results should be linked to my Google Profile along with my happy smiling face…

Related articles

• What Does Google Think Your Site Is About? (blindfiveyearold.com)