Tag Archives: WebSphere

Inside the Sphere

One of the things I’ve taken on this year is a regular writing assignment for the Global WebSphere Community. If you’re a member of the community (and if you use or are interested in WebSphere products, you probably should join – it’s free to sign up – here’s a link to my profile), depending on your profile preferences you may have just received the first edition of The Sphere Journal, a new online newsletter from GWC.

As the editor, Bruce Lynch, writes:

Welcome to the premier issue of The Sphere Journal Online. We will use this space to bring you opinion, news, and technical information on how to use your current WebSphere products more effectively and help you make more informed decisions about WebSphere products you should be deploying in your organization.

My monthly column is the WebSphere Deep Diving Instructor where I’ll be sharing news from inside the labs about the “hot” areas practitioners might want to explore more deeply, and the areas where I’m hearing the most interest or difficult technical questions from customers and the community. My first column does focus on my key area of messaging, but we’ll certainly broaden out from that over the course of the year.

[somewhat confusingly, there’s also a section called The Message Queue, but that’s a news section rather than being specific to WebSphere MQ!]

Links to places you can find and follow the GWC:

Advertisements

A Smarter Planet needs lightweight messaging

One of the primary things I’ve been working on this year has been IBM’s new WebSphere MQ Telemetry product. I say “new”, of course, but the underlying technologies – WebSphere MQ itself, and the MQTT protocol which takes the messaging infrastructure down to the edge of the network and into embedded devices – have both been around, and totally solid, for a number of years already, but they have only recently formally been brought together into a single package. MQTT is short for MQ Telemetry Transport, and I wrote about it a couple of months ago in a post where I referred to it as a Smarter Planet protocol.

I’ve done quite a bit of travelling and talking to IBM customers and communities this year, and that’s recently been recognised and formalised into (part of) my new role in Hursley where I’m the WebSphere Messaging Community Lead. What does that mean? Well, the product part is in the first two words – I’m looking at the MQ family of products (something I’ll be writing about a bit more in the near future). The most important part, though is that third word – Community. My primary focus is working with, listening to, and helping to develop the community – and linking that back in to what we do in our labs. Community, for me, means people: developers, administrators, architects, partners, and the overall ecosystem that surrounds WebSphere MQ. We’ve got some great third-party sites out there, some brilliant content that gets published through IBM Redbooks and developerWorks, and our support teams blog about the topic, but we can always do better and I’m looking forward to finding ways of socialising our content and the materials produced by others.

A great first step towards that is our new IBM Expert network on Slideshare, which Adam Christensen wrote about recently, enabling us to share content as IBMers in a more “social” manner. I had a bunch of presentations up there already, but it was high time that I contributed some material more from my specialist technical subject area. I’ve now done so 🙂

This was a quick presentation I gave at an Apache Retreat that was held at IBM Hursley a couple of months ago. It’s a cut-down version of the full talk I’d usually give on the topic, tailored to the audience to keep it relevant to that community rather than diving into the enterprise part of the story and listing out case studies etc.. This is an important point, as the MQTT ecosystem is very much about the developer community and the opportunity to embed a reliable and lightweight messaging protocol into devices like smartphones, sensors, routers and edge-of-network boxes. It’s one of the reasons why IBM has published the specification for royalty-free implementation, and we’re seeing some exciting (and sometimes unexpected!) things happening as folks build their own client APIs. If your application or library is implementing that specification then your devices or applicaitons could, ultimately, bridge up into an Enterprise Service Bus running on the MQ infrastructure, and all the backend power that exists in clouds and enterprise datacentres today can start to do clever things with the data. Predictive analytics, visualisations, better prioritisation of resources… the reliability of transport for the data and the ability to get down to the smallest devices is vital.

[ related aside – shout-out to Nick for the lovely visual on slide 18 of this deck which I failed to credit within the presentation where I originally put it together in a bit of a rush. It’s from his page for the Arduino library for MQTT ]

There’s always more to say in this space, but I hope the presentation provides an overview of how the Smarter Planet story bridges the ideas of Service Oriented Architecture and The Internet of Things, and the part that MQTT plays within that. In the future you can expect to hear me talking more in this space, and I should be giving talks at Home Camp 3 in London in a couple of weeks’ time, as well as at LinuxConf Australia in Brisbane in January. I look forward to meeting more people and discussing the whole messaging story in more detail!

Product updates and new releases

I don’t have time to post an in-depth update on the latest announcements from IBM Hursley today but will hopefully have a chance to dig deeper on some of these later in the week. My colleagues in Development have been working hard on new and updated software offerings in the WebSphere Connectivity space, and today was “the big reveal” of a slew of them. For now, here are the links to the announcements… I’ll try to fill in more detail on some of the areas in which I’ve been particularly interested, soon.

  • WebSphere MQ Advanced Message Security (AMS) version 7.0.1, also available for z/OS of course! This is a new product providing significant enhancements to MQ’s security story by encrypting data at rest with no need to re-code your applications. This is one I’ll definitely be coming back to in a future post… it’s very cool indeed, particularly since it’s non-invasive and transparent to the user.
  • WebSphere MQ Low Latency Messaging V2.5 includes major updates to self-management and additional message delivery styles. Incidentally, I’ll be talking about WMQLLM at the European WebSphere Technical Conference in Düsseldorf next week (and of course I also have other sessions at the event on topics like Telemetry!)
  • WebSphere MQ File Transfer Edition V7.0.3 adds some nice web and REST features, as well as ad-hoc transfers and sweeter integration with WebSphere Message Broker (which itself gained new FTE nodes recently). There’s a fantastic story developing around enterprise managed file transfer interoperating with an ESB, here. Oh yes, and this version also works with AMS if you need to thoroughly encrypt your FTE data, both on disk as well as the existing wire capabilities using SSL channels.
  • WebSphere Message Broker Hypervisor Edition enables WMB to live happily in a virtualised environment on Red Hat Enterprise Linux, and also to be used with the WebSphere Cloudburst Appliance.
  • WebSphere Enterprise Service Bus Registry Edition V7.0 puts the SOA registry at the heart of the ESB (which is quite honestly where it belongs!). There have been great improvements in WSRR and WebSphere ESB lately, and again I should come back to point some of these out soon.

Phew. Busy developers. If you follow me online you’ll know I’m a techie so it should come as little surprise that I’m excited, and dare I say it, “pumped”, about some of these updates. Looking forward to playing with them in more detail.

    OS X mosquitto “bites”…

    In my post last week about the new MQTT support coming to WebSphere MQ I very briefly mentioned that there are some third-party tools that already implement MQTT. One of those I pointed to is the very neat mosquitto broker, a project started by Roger Light.

    mosquitto has been around for a while now and is aiming to replicate the functionality of the Really Small Message Broker that is on IBM alphaWorks. One of the neat things about it, from my point of view, is that it there is an Ubuntu PPA repository, so with a couple of apt commands, I can install a running MQTT broker and build my own applications independently (NB there are packages for other Linux distros too, as well as Windows). When I want to do some “heavy lifting” or share data with my ESB, I connect up my local mosquitto broker to pass messages across to WebSphere MQ through the new telemetry channels – because MQTT supports a concept of bridges, and both RSMB and mosquitto both include support for bridging.

    I noticed that there wasn’t yet a version available for Mac OS X but figured that it shouldn’t be too difficult to compile and run it on that platform. As it happens, it did turn into a bit of an adventure for a couple of reasons, but at least I learned from the experience. If you’re desperate to build yourself a version to try some MQTT development on the Mac, here’s what I had to do to get it going on Snow Leopard:

    1. Installed mercurial, and a GUI for it called Murky (which requires the hg command line tool from the base mercurial package). The sources for mosquitto are in bitbucket, a Mercurial repository… this is optional of course as I could have just used a source tarball.
    2. Grabbed the latest mosquitto source from bitbucket.
    3. Modified the Makefiles throughout the mosquitto tree to build libraries with a .dylib instead of a .so extension (the default on OS X), and also changed the -soname parameter to -install_name which the OS X version of gcc understands.
    4. At this point the compile was starting to show progress… but failing due to missing symbols… the offender being one from sqlite, _sqlite_enable_load_extension. Turns out that the version of sqlite shipped in OS X 10.6.x is 3.5.4 but it does not have extension loading functionality built in, as evidenced by nm -g /usr/lib/libsqlite3.dylib | grep 'sqlite3_enable'
    5. Downloaded sqlite3.8.0, configured it to install to /usr/local (to avoid overwriting the default OS X shipped version), and built and installed it with no issues.
    6. At this point the compile was pretty smooth, once I modded Makefile link and include lines to point to the new version of sqlite in /usr/local. The only thing that failed was documentation, but that was “optional” 🙂
    7. Trying to start the broker failed… because it was trying to load the sqlite3-pcre extension.
    8. Installed git (the source for the sqlite3-pcre extension is in a git repository).
    9. Grabbed the source for sqlite3-pcre and built and installed it using:
      gcc -shared -o pcre.so -L/usr/local/lib -lsqlite3
                -lpcre -Werror pcre.c -I/usr/local/include
      sudo mkdir /usr/local/lib/sqlite3
      sudo cp pcre.so /usr/local/lib/sqlite3
    10. The final issue was that the path to the pcre extension is hard-coded into mosquitto/src/conf.c so I modded that to point at the version in /usr/local and recompiled. I’m assuming that this would not generally be required, but it worked as a hack to get me going!
      D’oh. Just realised that this is precisely what the ext_sqlite_regex variable in the mosquitto.conf file is for. Shouldn’t have bothered!

    So that was it. Being fair, if I hadn’t been feeling my way through that, I would have installed git and mercurial, grabbed all the lib sources for sqlite3 and pcre and built them, built mosquitto, and been good to go. At this point, the broker and test clients are runnable (assuming the library paths are set up appropriately):

    DYLD_LIBRARY_PATH=/usr/local/lib ./mosquitto
    DYLD_LIBRARY_PATH=../lib ./mosquitto_pub -t test/andy
            -m "hello world"

    If you are interested in seeing this in action, here’s a short (and silent, but annotated) screencast:

    The Java GUI application you see in the screencast is the test client shipped in an old IBM SupportPac, IA92, a Java implementation of MQTT. The final release of the WebSphere MQ Telemetry component for WebSphere MQ will contain something similar, considerably enhanced and integrated into WebSphere MQ Explorer.

    In other news, Roger recently announced version 0.8 of mosquitto, which now has slightly different packaging and includes C, C++ and Python clients. I hope to give these a test drive shortly!

    MQTT: the Smarter Planet Protocol

    I’m at the SHARE conference in Boston this week. Earlier today I gave a talk about one of IBM’s significant software announcements of this year: the forthcoming WebSphere MQ Telemetry feature.

    Observant viewers / readers / followers will know that I’ve been at IBM for nearly 10 years now and throughout that time I’ve spent a lot of time working with our WebSphere MQ (yes, aka, MQSeries) family of products. That family includes WebSphere Message Broker, and more recent extensions like the WMQ File Transfer Edition. Now, it has been formally extended to include first-class support for MQ Telemetry Transport, otherwise known as MQTT.

    I want to spend a little bit of time talking about this, partly because I haven’t posted all that much here on my blog for a while now, and mostly because I’m hugely excited by this direct and long-overdue convergence of two of our messaging technologies. It plays directly into the Smarter Planet vision that IBM has been talking about over the course of the past couple of years, so it is worth understanding how this all fits. I apologise in advance that this may be a longer blog post than my average! I’ll also warn that I may come across as just a teensy bit of a fanboy… 🙂 as usual, remember that all thoughts expressed here represent my own opinions and understandings, and are not necessarily also those of my employer.

    The Smarter Planet vision

    So let’s kick off by asking: what on earth is this Smarter Planet concept that IBM has been discussing, and which you’ve likely seen promoted on posters and billboards as you travel around and go about your daily life? Well, it’s a very simple but very exciting idea, founded on three pillars: Instrumented, Interconnected, and Intelligent.

    What is that all about then? Well, here it is, broken down, in my own words.

    Firstly, there are a heck of a lot of devices around these days… from teeny tiny sensors and RFID tags which may stand alone in a system, through smartphones, GPS location-aware devices, laptops, and embedded systems. These devices typically have enough computing power to at least gather and transmit data, and some of them have enough to respond to requests to modify their behaviour. This is where we recognise that the world is increasingly Instrumented. I won’t bore you with statistics but here’s one of the numbers that tends to blow people away: there are 1 billion transistors in the world for every one of us. Secondly, these devices are nearly all natively “online” to some extent, and most will have an Internet address of their own (even if the connections are not always super-high bandwidth, always-on, or reliable). So, they are becoming Interconnected, either directly to one another across local networks, or indirectly via clouds. This is what you may have heard referred to as The Internet of Things. Finally, to complete the puzzle consider this: what if we could gather all of that data being emitted by these small, medium, or even large devices in real time… route it to where it is best intepreted… and make use of the vast computational resources we have in our enterprises or cities to respond, adjust, and make our environment, well… better? Through Intelligent systems with advanced analytics, we can start to do just that.

    Speaking personally, this is what has always excited me about technology: the potential to use it to improve the way that we live, work, collaborate, communicate, and exist. I realise I’m entirely biased, and I’m not sure I could have articulated it at the time that I joined the company… but the sheer breadth of talent, technology and reach that is wrapped up in the IBM brand and company, makes it an exciting place to be as we begin to address these ideas.

    The technology foundations

    OK, so here we are. We’ve got existing enterprise or municipal systems running on (frankly) a spaghetti of platforms, but which encapsulate fundamental business processes and applications which help to run our day-to-day lives – from the mainframes running critical financial transactions in IMS and CICS, to retail supply chains which ensure goods get to the right places at the right time, social security and medical processes which (in theory!) just make things “work” in our interactions with various authorities, utility companies which are continuously providing gas, water, and other scarce resources… the list is endless, but you name it, over the past 30-40 years there will have been a computer system built to support it.

    IBM has been a leader in the enterprise messaging space for a long time. The space I’m specifically talking about here is what is often called message-oriented middleware or MOM. If you’re still looking blank, that’s OK… messaging middleware is, fundamentally, supposed to be invisible to you, so I’m not totally surprised. WebSphere MQ has been the reliable messaging transport used by many of the Fortune 500 companies for ~15 years now. It runs on a ridiculous number of platforms, has a number of language bindings, a stable API which has been backward-compatible the whole time, and it has become the de facto lingua franca and way of gluing disparate applications together. As an added bonus, the wider family of products also includes a high-performance, robust, exceptionally flexible, any-to-any connectivity, transformation and routing engine (Enterprise Service Bus) called WebSphere Message Broker. I don’t have space to talk about WMQ or WMB in detail here right now, but I guess I know some stuff (…!), so, ask me another time 🙂

    The piece that has been missing in all of this until recently is the ability to reliably connect the edges, the frontiers of the data network, to the existing systems that already understand what to do in various scenarios. In other words, we could start to take action based on some data, but we generally couldn’t collect that data in real time, and as a result of that our analysis also tended to be flawed and based on outdated information.

    What next? Joining the dots

    I kind of fibbed, just then. IBM has actually had a technology capable of living out on these little embedded devices for aaaaaages! It’s called MQTT, the MQ Telemetry Transport, and it has had a home over on mqtt.org for a number of years. In fact, if you take a look, you’ll find that a bunch of people have implemented their own language bindings and small footprint message brokers already. It has existed inside a number of IBM products, and it was supported in WebSphere Message Broker up until version 6.1 via what were called the SCADA nodes. The name has been fiddled with a few times, and it hasn’t been a massively prominent part of the portfolio, so I’ve often found that folks haven’t heard of it. It’s very, very cool though.

    What’s so special about MQTT? Well, it was specifically designed for constrained environments, with limited processing capabilities, potentially very tiny memory capacities, and fragile, unreliable, high-latency, low-bandwidth networks. As a result, it doesn’t have to have the reliable transactional qualities of service you might find in an enterprise messaging solution (although it can support those, too). The design principles do however bring in some massive advantages: it’s incredibly simple, easy-to-learn, and can be very fast and effective, with many thousands of lightweight clients supported by a single server.

    On July 6th, IBM announced that MQTT is being added to WebSphere MQ as a first-class protocol.

    [warning: the next couple of paragraphs describe finer details which are
    potentially subject to change prior to final release!]

    Alongside the existing MQ channels, it will now be possible to define Telemetry channels to enable the connection of one or many MQTT clients (by many read, like, really, LOTS, but wait for the performance SupportPac for full details!). Not only that… here’s what I believe is the really sweet part: MQ and MQTT applications will be able to interoperate, so messages published to a topic by an MQ application will be able to be received and consumed by MQTT clients, and vice versa. Nearly instant interoperability between existing enterprise applications and the edge of the network, if you need it. We’ve got security via SSL and JAAS, and we’ve got some simply beautiful tooling integration with the WebSphere MQ Explorer, including a nice test client.

    As the diagram shows, the new feature requires a recent version of WebSphere MQ – I’m currently running 7.0.1.2 or better, but you’ll need to check the final requirements at release time. The new telemetry channels enable simple MQTT clients to be connected directly to a queue manager via a component which is internally called MQXR (for MQ eXtended Reach). The WebSphere MQ Telemetry Daemon for Devices can act as a concentrator, connecting up to a queue manager via telemetry channels if required (think, multiplexing connections at the edge of the enterprise). If you are using one of the other, richer clients that already contain MQTT or an MQTT-based technology like Microbroker (e.g. Lotus Expeditor), they will seamlessly connect up, too. You could even use a tool such as the third-party mosquitto broker to connect into a full MQ network.

    My friend James Governor posited not very long ago that “WebSphere MQ won’t ever be a pervasive play“. James is an extremely smart guy and I take his opinions very seriously, but with the extension of WMQ to the web via the HTTP bridge introduced in version 7, and with MQTT joining the product foundation, I’d say that WMQ absolutely has the scope to be pervasive. I’ve been working with the team adding MQTT to WMQ for the past few months, and I have to say that the integration is looking really, really nice. I don’t want to bore with details in this post – I spent an hour talking about them at a conference today! – but I can say that it’s trivial to build a lightweight MQTT client, publish data, have that delivered into WebSphere MQ, transformed from a simple and tiny bytes message into meaningful XML in WebSphere Message Broker, and then routed on to any number of backend systems. There are customers implementing very exciting solutions in healthcare, transport, and energy to name just a few, and they are using MQTT in those solutions today. Plus, our very own lovable propellerhead Andy Stanford-Clark runs his home automation and mouse-slaughtering system on the protocol that he invented (I have a similar, but significantly less impressive and less all-encompassing system that I’ve written about before). Now is the time for these pieces to come together.

    Keep watching for more as the release approaches in the next few weeks. MQTT really could be called the Smarter Planet Protocol, and I’m looking forward to find out what kinds of things we can use it for as we collectively create more “smart” solutions.

    Update: if you’re interested, I will be giving a similar talk for the Global WebSphere User Group webcast series on August 18th. Additionally, if you are a member of SHARE, my slides for this talk should be available via the schedule planner shortly.