Tag Archives: Coding

First week with VMware and Cloud Foundry

Hello, VMware.

Well – that was bracing!

I don’t expect to be posting “week notes” like this on a regular basis, but as a one-off it seems like a nice way to encapsulate just how much happened in the first week of my new role.

Tuesday

Joined VMware. Met new colleagues in London office. Started to look at the User Account and Authentication component in Cloud Foundry. Ran samples against cloudfoundry.com, modified the documentation. Issued first GitHub pull request 🙂

Wednesday

More hacking on samples. Updated the Cloud Foundry Google+ page. Agreed to present Cloud Foundry at the London Real-Time hack weekend for the RabbitMQ guys. Watched (and tweeted) the live webcast of the Cloud Foundry first birthday event in San Francisco – very exciting news! “more clouds, more community, more code“, including a broader range of partners, a new governance process around the cloudfoundry.org Open Source project, and the announcement of BOSH being released to the community, too – multi-cloud deployment, here we come!

Thursday

Trip to the VMware Frimley office for some HR stuff. First call with full Developer Relations team for event planning. Briefing with the two directors I work for. Nothing to see here, move along…

Friday

Setup new laptop (custom order from Apple so there was a small delay). Prepped for demo at London Real-Time. Started making a lot of noise on VMware Link (aka Socialcast, the social sharing/discussion platform) internally 🙂

Saturday

My first speaking gig – a Lightning Introduction to Cloud Foundry, taking Chris Richardson’s much more comprehensive Boot Camp presentation and cramming the essentials into ~15 min including a live demo, for a bunch of hackers at London Real-Time.

Oh, and it was caught on video.

[vimeo 40379801]

Four or five of the hacks ended up running on Cloud Foundry, too, which I think was rather nice 🙂 I was also interviewed on realtime and the importance of cloud at the event, but I’ve not seen that video appear just yet.

Monday

More laptop setup, HR stuff. Prep for Scala Days. Started to improve a sample app (Ruby/Sinatra) I’d used in the past by adding Twitter Bootstrap and restructuring the code.

Tuesday/Wednesday

Scala Days in London – helping to man the sponsor stand talking about Cloud Foundry, answering questions, and meeting many new colleagues from the US who were presenting on Spring, Scala, and Cloud Foundry (including an announcement that Play 2.0 framework support and standalone apps are coming to cloudfoundry.com Real Soon Now). Recorded a podcast interview with Uhuru about what a Developer Advocate does.

Summary

I’m pleased that I was able to be so productive so quickly. I’d had a little previous experience with Cloud Foundry but it’s a testament to how quick it is to learn the basics and get moving that I was able to rapidly start playing with a bunch of code. It was also exciting to be out on Github on my first day – not something I could have done in a former life… it’s nice to be working in an organisation that is innovating with Open Source at this level.

There’s much to learn, and to be honest, a couple of the key aspects of Cloud Foundry actually make it more challenging (and interesting) for me to get to grips with. It’s open, and with BOSH, can potentially target different IaaS offerings (initially vSphere and the beginnings of AWS support; a hackathon yesterday aimed at adding OpenStack to the list) – so suddenly I need to know about those. It’s a polyglot platform, which means I need to broaden my language knowledge – I’m already making a start on Ruby and node.js, to complement existing Java and PHP knowledge.

It’s also exciting to learn more about what VMware does, the layers of technology that they offer, and their vision. My previous experience has primarily been with the desktop virtualisation technology, but there’s a huge and vibrant community around the server-side virtualisation tools, and products like Socialcast, Sliderocket and Zimbra in the collaboration space too.

There’s a lot of exciting stuff happening in this space. It’s thrilling to be here. Thanks to all of my new colleagues for a warm welcome and support – looking forward to working with you!

Advertisements

Eclipse Paho gets started…

Since the announcement of Eclipse Paho (an Open Source project under the Machine-to-Machine umbrella at Eclipse) there has been a fair amount of excitement in the MQTT community about the availability of IBM’s C and Java client code under an Open Source license.

The initial proposal and setup stages have taken a little while, but this week the initial availability of the C client code was announced on the Paho mailing list (Java will follow shortly).

PahoHALF

Paho Quickstart

This is not intended to be a comprehensive guide – better documentation etc will emerge over time – but I thought I’d post a quick guide as a kickstart for anyone wanting to give it a look. I did this on 64-bit Ubuntu 11.10 – similar steps will apply on other Linux or UNIX platforms (note, the initial code contribution has a Makefile with rules which should work on UNIX, Windows, or z/OS).

Install the necessary packages to build code. NB git is for grabbing the source from Eclipse; build-essentials is a metapackage providing gcc etc on Ubuntu; and doxygen and optional graphviz are used for generating the documentation.

sudo apt-get install git build-essentials doxygen graphviz

Get the code from the git repository:

git clone git://git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.c.git

Quick build for the client library and documentation:

cd org.eclipse.paho.mqtt.c.git/src
make -f ../build/Makefile all
doxygen ../doc/DoxyfileV3ClientAPI

Once these commands complete, you should be left with subdirectories called <platform> and docs. In my case, <platform> was 64-bit Linux, so I had a binary at linux_ia64/libmqttv3c.so. There’s no “make install” rule at the moment, nor is there a rule to compile the docs so I had to run doxygen directly. In the future it would be nice to automate all of that, and also to build some test applications.

Opening docs/html/index.html in a browser reveals very nice documentation describing the client library, including some examples of how to use it. For example, in docs/html/pubasync.html there’s a complete listing for an asynchronous publisher application. I extracted that code into pubclient.c and decided to check that it worked!

gcc -Wall pubexample.c -L./linux_ia64 -lmqttv3c -lpthread -o pubexample

That command successfully built a binary called pubexample. All I needed to do was test it. The sample application assumes that an MQTT broker is available on localhost port 1883 – if you want to change that, simply modify the value of the static variable ADDRESS in pubexample.c – in my case I simply apt-get installed the mosquitto and mosquitto-clients packages onto my system, but I could equally have unzipped and run Really Small Message Broker – both start on port 1883 by default if not given alternative configuration.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./linux_ia64/
./pubexample
Waiting for publication of Hello World!
on topic MQTT Examples for client with ClientID: ExampleClientPub
Message with token value 1 delivery confirmed

It was trivial to test that a subscriber (mosquitto_sub in my case) also received the publication. Job done!

Getting involved, and other news on Paho

I mentioned that the Java client contribution should appear soon. One other piece of news this week is that the project’s sandbox broker implementation – based on mosquittohas been spun up. That was posted on the Paho mailing list, and if you want to get involved you should definitely subscribe to that; start to track the Eclipse Bugzilla for Paho; watch the Paho wiki; keep an eye on the source repositories; etc.. I’m already thinking about getting an OS X build rule sorted out. If you want to test your sample code now, you’ve got the option of a local broker, the Eclipse Paho sandbox, the mosquitto sandbox, or various other implementations.

Oh – and please leave a comment on this post if you find this information interesting, or want to discuss where things are with Paho. I’ll be hanging out on the mailing list as well.

What about Bob? (or Andy, even!)

Well, although I’ve left IBM, I’m delighted that MQTT is now going Open Source – in fact that was one of the things that I really wanted to help to achieve before I moved on. I am really pleased that I will be able to continue to contribute to both Paho and the broader Eclipse M2M Industry Working Group. I’ll be helping to update the mqtt.org community site, and heading over to EclipseCon in Virginia in a couple of weeks’ time to talk about M2M and work with our friends from the Koneki project. If you are attending EclipseCon please come say hi to me – and you may be interested in Wes Johnson’s session on MQTT and Eclipse tools.

There are very cool times ahead!

Podcasting with UUPC (and podcats)

On Monday I was surprised and delighted to be asked to join the team from the Ubuntu UK Podcast (aka @uupc) as a guest presenter. I’ve been a listener and friend of the show for a long time, I knew several of the regular team, and since I work not far from “Studio A” it was fairly straightforward to jump in at short notice.

I had a blast recording with Tony, Laura and Mark! The amount of work and polish that goes into each show is fantastic, I can tell you. I was also very impressed with their studio setup. My own podcast is recorded using Skype, and although I do own a small mixer, it’s really nothing compared to Tony’s much larger desk. Each of us had a proper mic, too. Oh, and there was a Very Large Clock. There was also a small experiment in streaming to a small audience (the magic-fu was served up on Twitter and in the #ubuntu-uk-podcast IRC channel, so you should hang out in those places in case it happens again!).

Most importantly, we were supported by the UUPC PODCATS.

I gave a brief intro to myself at the start of the show. If you’ve just discovered me via my guest appearance on UUPC then feel free to follow me on Twitter or read more about me here on my site.

Below, I’ll just recap and expand on my Linux credentials, because it’s not something I’ve really blogged about before.

My Linux experience

I started out using Linux with some early SuSE version in the late 90s just after leaving university. Then I switched to RedHat, which was in the phase of spinning out the Fedora project, and I did some packaging for a few projects there for a while. I helped out on a bunch of projects around that time, like OpenUT (the initial Unreal Tournament port to Linux – there’s a special thanks to me in the credits for the Linux version!), the brilliant Anjuta IDE, and the Bible software GnomeSword (which is now known as Xiphos). I spent a lot of time helping to do things like triage bugs, coordinate releases, polish the UI for GNOME apps like Anjuta, and basically to some extent “project managing” alongside the actual project leaders, freeing them up to code on their projects while I took on a more technical coordination role communicating to different developers, helping with test, etc.. At the time I was a middleware developer for a large company so I had an appreciation of how things like CVS worked, how to do releases and release notes, working with users etc. – things that are sometimes missed on OSS projects, although all of these things have become better over time. It was a great way for me to deepen my UNIX skills and hone my development abilities too. Plus, I built some great relationships and friendships from working with the community.

Times change, and I had to take a step back from all of that for quite a long time. Although I had a Linux box at home as a server (on dial-up for a long time… yikes!), I otherwise wandered the wastelands of Windows XP and then got a Mac. Fundamentally I believe I see the good and bad aspects in most systems; I believe it’s important to at least try something, and not dismiss it; I did go through a strongly anti-Microsoft phase but with things like XBox 360 Live, a few items in the “Live” family of software, and their new phone operating system, I do have respect for what they’ve done. I struggle a little on the whole “openness” thing – my tendency and first preference is absolutely towards open standards, open source and free collaboration, but then along comes Apple with stuff that just… works… and is so… shiny… and… I’m almost willing to suspend that view. And then I smell coffee and come back to my senses 🙂

So what am I doing now? Well, last year I switched to Ubuntu on my work laptop, a Lenovo Thinkpad, and I’ve been through Jaunty, Karmic, Lucid and now Maverick. I do NOT have a Windows partition, I run my whole work life in Ubuntu. I’ve got a Viglen MPC-L (previously featured on UUPC, of course!) running home and weather monitoring with some software called MQTT which I’m involved with as part of my job. I run Linux on my “set-top box”, an Acer Aspire Revo inspired by Popey, with Boxee and other bits. I have a netbook, a first-gen Acer Aspire One running UNE Lucid until they sort out Unity to a point where I feel it’s usable. And, as I mentioned on the show, I’ve recently picked up a cheap (actually, total bargain) Android handset as a development platform- my main phone is one of the Apple devices, sorry! The new phone is an Orange San Francisco, which is a rebadged ZTE Blade – I’ve flashed it to run Android 2.2 and moved a bunch of apps into SD storage, and it’s a lovely little device. Apart from that, I help to do a few things for internal Linux apps we use at work on Ubuntu, and I buzz around Launchpad largely helping to improve quality via bug reports etc. Oh, and I’m lined up to speak at LinuxConf AU in January! (very excited about that!)

So that’s me. I do Linux, I do Ubuntu, and I do a whole bunch of other stuff. Thanks again to the UUPC gang for not throwing me out of the studio! And thank you, dear reader, for listening…

OS X mosquitto “bites”…

In my post last week about the new MQTT support coming to WebSphere MQ I very briefly mentioned that there are some third-party tools that already implement MQTT. One of those I pointed to is the very neat mosquitto broker, a project started by Roger Light.

mosquitto has been around for a while now and is aiming to replicate the functionality of the Really Small Message Broker that is on IBM alphaWorks. One of the neat things about it, from my point of view, is that it there is an Ubuntu PPA repository, so with a couple of apt commands, I can install a running MQTT broker and build my own applications independently (NB there are packages for other Linux distros too, as well as Windows). When I want to do some “heavy lifting” or share data with my ESB, I connect up my local mosquitto broker to pass messages across to WebSphere MQ through the new telemetry channels – because MQTT supports a concept of bridges, and both RSMB and mosquitto both include support for bridging.

I noticed that there wasn’t yet a version available for Mac OS X but figured that it shouldn’t be too difficult to compile and run it on that platform. As it happens, it did turn into a bit of an adventure for a couple of reasons, but at least I learned from the experience. If you’re desperate to build yourself a version to try some MQTT development on the Mac, here’s what I had to do to get it going on Snow Leopard:

  1. Installed mercurial, and a GUI for it called Murky (which requires the hg command line tool from the base mercurial package). The sources for mosquitto are in bitbucket, a Mercurial repository… this is optional of course as I could have just used a source tarball.
  2. Grabbed the latest mosquitto source from bitbucket.
  3. Modified the Makefiles throughout the mosquitto tree to build libraries with a .dylib instead of a .so extension (the default on OS X), and also changed the -soname parameter to -install_name which the OS X version of gcc understands.
  4. At this point the compile was starting to show progress… but failing due to missing symbols… the offender being one from sqlite, _sqlite_enable_load_extension. Turns out that the version of sqlite shipped in OS X 10.6.x is 3.5.4 but it does not have extension loading functionality built in, as evidenced by nm -g /usr/lib/libsqlite3.dylib | grep 'sqlite3_enable'
  5. Downloaded sqlite3.8.0, configured it to install to /usr/local (to avoid overwriting the default OS X shipped version), and built and installed it with no issues.
  6. At this point the compile was pretty smooth, once I modded Makefile link and include lines to point to the new version of sqlite in /usr/local. The only thing that failed was documentation, but that was “optional” 🙂
  7. Trying to start the broker failed… because it was trying to load the sqlite3-pcre extension.
  8. Installed git (the source for the sqlite3-pcre extension is in a git repository).
  9. Grabbed the source for sqlite3-pcre and built and installed it using:
    gcc -shared -o pcre.so -L/usr/local/lib -lsqlite3
              -lpcre -Werror pcre.c -I/usr/local/include
    sudo mkdir /usr/local/lib/sqlite3
    sudo cp pcre.so /usr/local/lib/sqlite3
  10. The final issue was that the path to the pcre extension is hard-coded into mosquitto/src/conf.c so I modded that to point at the version in /usr/local and recompiled. I’m assuming that this would not generally be required, but it worked as a hack to get me going!
    D’oh. Just realised that this is precisely what the ext_sqlite_regex variable in the mosquitto.conf file is for. Shouldn’t have bothered!

So that was it. Being fair, if I hadn’t been feeling my way through that, I would have installed git and mercurial, grabbed all the lib sources for sqlite3 and pcre and built them, built mosquitto, and been good to go. At this point, the broker and test clients are runnable (assuming the library paths are set up appropriately):

DYLD_LIBRARY_PATH=/usr/local/lib ./mosquitto
DYLD_LIBRARY_PATH=../lib ./mosquitto_pub -t test/andy
        -m "hello world"

If you are interested in seeing this in action, here’s a short (and silent, but annotated) screencast:

The Java GUI application you see in the screencast is the test client shipped in an old IBM SupportPac, IA92, a Java implementation of MQTT. The final release of the WebSphere MQ Telemetry component for WebSphere MQ will contain something similar, considerably enhanced and integrated into WebSphere MQ Explorer.

In other news, Roger recently announced version 0.8 of mosquitto, which now has slightly different packaging and includes C, C++ and Python clients. I hope to give these a test drive shortly!

Experiments with PHP and MQTT

Over the past few days I’ve been playing around with combining lightweight messaging and PHP. There are a couple of reasons for this, but the primary one is that I’d like to extend my prototype iPhone CurrentCost monitoring web application to display more up-to-date information about the state of my home energy usage. I’d planned to do this for a while, but recently Mark Taylor created his own version of the iPhone interface (PDF link) and he has got current readings on the front page. Clearly, I have to compete 🙂

Actually, in my system, I’d like to do things a different way. The heart of my setup is a Really Small Message Broker. At the moment, data from the CurrentCost meter comes in over the USB connection and is then published in pieces, or on topics, to the RSMB (temperature and energy readings are separate). These published messages are then read by a script which is subscribing to the topics and squirrelling the historical data into an rrdtool database; and also being pushed up to our IBM broker “in the interweb cloud” via an MQTT broker bridge connection.

So in theory, having the up-to-date information in the web UI should be a simple case of grabbing the MQTT publications on each topic and displaying them. The way I’ve coded things (and would prefer to do things), this involves having the ability to subscribe to MQTT publications from PHP.

I’m not at the end of the road yet, but I do have a starting point.

howitworks.png

I’ve got a front-end test page which currently uses Prototype to send an Ajax request to a server-side PHP script (yes, I have had jQuery recommended to me, and I may well look into that instead of Prototype, but this works).

The server-side script uses the Simple Asynchronous Messaging PHP library. SAM is a wrapper which enables a variety of messaging transports to be supported in PHP, such as MQTT, WebSphere MQ or WebSphere Platform Messaging. Just one thing: I found that in order to get the most recent SAM release to work on Ubuntu on my MPC-L, I had to install IBM’s XMS client SupportPac (for some reason, it won’t build without it, even though it is “optional”) and I also had to delete a spurious empty line from the end of /usr/share/php/SAM/php_sam.php to prevent header issues. Other than that, it was all good.

The script is really simple and basically uses all of the defaults to create a connection to my local RSMB over MQTT. The advantage of this being server-side is that I don’t have to open my RSMB to the Internet, the PHP code can connect via localhost. Once that’s done, it creates a subscription on the topic I’ve asked for, and receives the first data that comes along, then echoes it back to the front-end. I could make it auto-updating with Ajax.PeriodicalUpdater too, but there’s no need to put a load on my server.

Wanna see a quick demo? 😉

I’m quite pleased with the way this is working. There’s some more plumbing to do, and I’ll almost certainly extend the server-side piece to allow two-way communications (publish as well as subscribe) as well as finer-grained control over the options. As a proof-of-concept though, I think this is looking good.