Tag Archives: Linux

OS X mosquitto “bites”…

In my post last week about the new MQTT support coming to WebSphere MQ I very briefly mentioned that there are some third-party tools that already implement MQTT. One of those I pointed to is the very neat mosquitto broker, a project started by Roger Light.

mosquitto has been around for a while now and is aiming to replicate the functionality of the Really Small Message Broker that is on IBM alphaWorks. One of the neat things about it, from my point of view, is that it there is an Ubuntu PPA repository, so with a couple of apt commands, I can install a running MQTT broker and build my own applications independently (NB there are packages for other Linux distros too, as well as Windows). When I want to do some “heavy lifting” or share data with my ESB, I connect up my local mosquitto broker to pass messages across to WebSphere MQ through the new telemetry channels – because MQTT supports a concept of bridges, and both RSMB and mosquitto both include support for bridging.

I noticed that there wasn’t yet a version available for Mac OS X but figured that it shouldn’t be too difficult to compile and run it on that platform. As it happens, it did turn into a bit of an adventure for a couple of reasons, but at least I learned from the experience. If you’re desperate to build yourself a version to try some MQTT development on the Mac, here’s what I had to do to get it going on Snow Leopard:

  1. Installed mercurial, and a GUI for it called Murky (which requires the hg command line tool from the base mercurial package). The sources for mosquitto are in bitbucket, a Mercurial repository… this is optional of course as I could have just used a source tarball.
  2. Grabbed the latest mosquitto source from bitbucket.
  3. Modified the Makefiles throughout the mosquitto tree to build libraries with a .dylib instead of a .so extension (the default on OS X), and also changed the -soname parameter to -install_name which the OS X version of gcc understands.
  4. At this point the compile was starting to show progress… but failing due to missing symbols… the offender being one from sqlite, _sqlite_enable_load_extension. Turns out that the version of sqlite shipped in OS X 10.6.x is 3.5.4 but it does not have extension loading functionality built in, as evidenced by nm -g /usr/lib/libsqlite3.dylib | grep 'sqlite3_enable'
  5. Downloaded sqlite3.8.0, configured it to install to /usr/local (to avoid overwriting the default OS X shipped version), and built and installed it with no issues.
  6. At this point the compile was pretty smooth, once I modded Makefile link and include lines to point to the new version of sqlite in /usr/local. The only thing that failed was documentation, but that was “optional” 🙂
  7. Trying to start the broker failed… because it was trying to load the sqlite3-pcre extension.
  8. Installed git (the source for the sqlite3-pcre extension is in a git repository).
  9. Grabbed the source for sqlite3-pcre and built and installed it using:
    gcc -shared -o pcre.so -L/usr/local/lib -lsqlite3
              -lpcre -Werror pcre.c -I/usr/local/include
    sudo mkdir /usr/local/lib/sqlite3
    sudo cp pcre.so /usr/local/lib/sqlite3
  10. The final issue was that the path to the pcre extension is hard-coded into mosquitto/src/conf.c so I modded that to point at the version in /usr/local and recompiled. I’m assuming that this would not generally be required, but it worked as a hack to get me going!
    D’oh. Just realised that this is precisely what the ext_sqlite_regex variable in the mosquitto.conf file is for. Shouldn’t have bothered!

So that was it. Being fair, if I hadn’t been feeling my way through that, I would have installed git and mercurial, grabbed all the lib sources for sqlite3 and pcre and built them, built mosquitto, and been good to go. At this point, the broker and test clients are runnable (assuming the library paths are set up appropriately):

DYLD_LIBRARY_PATH=/usr/local/lib ./mosquitto
DYLD_LIBRARY_PATH=../lib ./mosquitto_pub -t test/andy
        -m "hello world"

If you are interested in seeing this in action, here’s a short (and silent, but annotated) screencast:

The Java GUI application you see in the screencast is the test client shipped in an old IBM SupportPac, IA92, a Java implementation of MQTT. The final release of the WebSphere MQ Telemetry component for WebSphere MQ will contain something similar, considerably enhanced and integrated into WebSphere MQ Explorer.

In other news, Roger recently announced version 0.8 of mosquitto, which now has slightly different packaging and includes C, C++ and Python clients. I hope to give these a test drive shortly!

Today, I (hope I) helped people

Today, I…

This is what I do. Most days, it is good fun. Some days, it is a little full on.

Fedora 9 on a USB stick

Although you’ll most commonly hear me waxing lyrical about OS X these days, I’m a long-time Linux user. I’ve been running various flavours of Linux at home since Redhat 5.0 days.

Why Redhat and Fedora? Well, a good friend of mine noted a long time ago that it was the distribution likely to get the blessing of enterprise vendors as time went on. I’m not here to invite a flame war, and I’ve been impressed with a lot of other distros over the past couple of years in particular (while I keep meaning to give Ubuntu a “proper” run, I use it for development in VMWare Fusion on the Mac). I’ve run Fedora on a server and a workstation at home for a while now, and I’m always pretty keen to see what a new version has to offer.

Enter Fedora 9. I use my MacBook Pro pretty exclusively these days, so I just wanted a quick and easy way to see what Fedora 9 was like. I considered the Fusion option, but then read about the “Live USB” option. This is really nice… there’s a (currently Windows-based, sadly) desktop app that you run to select the “spin” of Fedora that you want, point it at an inserted USB memory key, and away you go… I chose Fedora 9 and let the Thinkpad download the image and then install it on my 1Gb USB stick. I also asked for a 200Mb “persistent overlay”, i.e. space that I could use for persistent storage of data like (I assume) my home directory. This is a far nicer option than a Live CD, as I can take my data with me.

A quick reboot, choosing a temporary boot device, pointing at the USB stick. The boot process was not all that promising, as it initially reported what looked like errors about inability to assign USB identifiers (or something), but it did all boot fine.

[click for a larger view]

In fact, it booted more than fine. I was pleased to find that Fedora 9 picked the “right” (i.e. max) resolution for my display straight away. The only customisation I needed to do to make the desktop more pleasant was to reduce the font size, but I can see why they went with the default size that they chose.

The next thing was to get myself online, or at least onto my home network. Based on past experience I went into the Network config under system preferences and started fiddling with the NIC settings. Didn’t work – although it could see the wireless card it didn’t want to let me join the network. Then I spotted the little wireless icon in the system tray at the top right of the screen, and clicking there let me join my home network immediately – with OS X levels of ease. Very impressive stuff.

Sound worked straight out of the box too… if I come across as surprised, remember I’ve been using Linux since RH 5 and I’m well aware of how flaky much of this stuff has been over the years.

Firefox worked fine, Pidgin let me configure my Google Talk account within seconds, taking a screenshot and editing in Gimp was no problem… this was a lovely experience overall. I was even able to install the Flash plugin for Firefox (although I had to download and install the RPM via sudo, rather than it just working via the Firefox addons installer).

All-in-all I was extremely impressed with the ease-of-installation and use. I’m not sure how often I’ll want to use this, but the fact that I have a fully-usable Linux distro on a bootable stick is just brilliant.

Linux box progress

Back in November, I made a hopelessly failed attempt to upgrade my main Linux workstation from Fedora Core 4 to Fedora Core 6. This is a machine that I built myself – 2GHz AMD64 CPU, dual 160Gb SATA drives with software RAID 1 (mainly to protect my photos), NVIDIA graphics. The problem was that the upgrade process rendered the box unbootable – unable to find a kernel, and once I’d hacked at grub it was unable to load the relevant modules… I had to give up on it through lack of time.

This week I finally had some time to play. I was surprised to find that I hadn’t really missed the machine in 5 months, apart from the lack of access to my address book and quite a large number of photos. I knew that the data was at least safe, I just couldn’t get to it.

On Thursday I managed to get the system booting again. There had been two fundamental issues with the upgrade. The first was that the bootloader was broken. I had to boot into rescue mode, reinstall grub on both SATA drives, and hack the menu.lst file like this:

title RAID partition 1 Fedora Core (2.6.18-1.2798.fc6)
kernel (hd0,0)/vmlinuz-2.6.18-1.2798.fc6 ro root=/dev/md3 rhgb quiet
initrd (hd0,0)/initrd-2.6.18-1.2798.fc6.img

Now, this has the disadvantage that it is targetting a particular RAID drive – I need to look at making it disk-independent as it was before, but at least this is letting me load the kernel.

The second problem was that once it found the kernel image, it failed to load various modules (like via_sata, for example, which was fairly critical to the whole boot process). To fix this, I had to rebuild the initial RAM disk using mkinitrd… in the process discovering that I had obsolete options in modprobe.conf, like stuff telling scsi_mod to scan max_scsi_luns of 127, so I removed those.

Once I got the thing to boot, I had to clean up the networking, and then install months of Fedora updates.

The most important thing (!) about the whole process was that I wanted to get Beryl running for full GL-enhanced desktop goodness. I managed to do that this morning, although I had to reinstall the NVIDIA driver in order to prevent it from complaining about the GLcore module not being found, and then install the glx-utils package to get hold of all the glx programs like glxinfo and glxgears.

The final problem I’m faced with is overheating. It turns out that if the CPU runs at 1.6GHz or 2GHz it immediately heats up to ~75C and the system shuts itself down. This is despite having a fan+heatsink on the processor itself, and several fans in the machine… clearly not good enough. I’m currently running at 800MHz, which at least works. I need to find a better method of getting the machine cooled.

Posted whilst at White Leaf House [ plazes.com

Technorati Tags: , , ,

Thank goodness for the command line

I’m trying to sort out some letters – trying to beat the Christmas posting dates (which we’ve failed to do, due to our holiday).

Unfortunately my Linux box is still dead, since I’ve not had time to tend to it properly and get it up-and-running. My address book is held in Evolution.

The FC6 boot CD, the “linux rescue” option, and some command-line tools came to my rescue.

Evolution stores the address book in Berkeley DB format. So, I booted into rescue mode, enabled my network interfaces, chroot’ed to /mnt/sysimage, and ran:

db_dump -p  /home/andyp/.evolution/addressbook/local/system/addressbook.db > /tmp/addr.out

Once I’d done that, I was able to FTP the resulting text file to my laptop, and use vi to search for the address entries (vcards) that I wanted. Primitive, but effective.