Tag Archives: open source

Some random reflections on 2014

As time has moved on, I’ve lavished less attention on my blog, which is a shame… “back in the day” I enjoyed writing for it, and gained a lot of value from doing so. It’s of no particular surprise to me that I’ve spent less time writing here in 2014 than in any of the previous years; but it is a regret. I blame my schedule, a general change in the way I interact online, and a lack of inspiration. Actually, that last one isn’t quite true: I’ve often been inspired, or felt the need to blog, but have found myself mentally blocked. I need to get over that!

Anyway… 2014, looking back… a little bit of a year in review.

LEGO AndyThe major life change this year was my move to Twitter, which has been very exciting and energising. I’m thrilled to have been invited to work with a team of exceptional people under Jeff Sandquist. In particular, this year I’ve had the short-lived opportunity to work with three brilliant and talented guys I want to say “thank you” to, for making my transition to my new role such a pleasure: Taylor Singletary, Sylvain Carle and Isaac Hepworth. A special shout-out too to my close friend and colleague based in London, Romain Huet, without whom I would have found the past nine months much less fun or easy-to-navigate! The whole team has been just amazing to work with, as have all my wonderful colleagues at the Twitter office in London #gratefulpipes

The work we’re doing on the Developer and Platform Relations team at Twitter is something I’m incredibly passionate about. Connecting with the third-party community and acting as the face and voice of Twitter with those developers, listening to them and responding to their concerns, is the reason I joined the company.

I’ve been involved in the launch of a couple of APIs (most notably the Mute API), and I’m getting to work on much of the external API surface, which plays well with my background and developer experience. We’ve completely relaunched our developer-facing website and forums in the past few months, which the whole team has worked hard on. I’m happy to see the focus of discussion on the developer forums substantially improved now that we’ve moved to the Discourse platform – the user experience is far better than we had with the previous solution.

Most importantly, this past quarter we launched Fabric, our new free mobile SDK and platform for iOS and Android, and delivered a swathe of improvements to the developer experience for mobile enthusiasts. We also ran our first mobile developer conference, Flight – I was excited to be there, and I’m looking forward to seeing that experience continue in 2015.

My background in the Internet of Things and MQTT space has partly carried over into my new life at Twitter, and I’ve had the opportunity to speak at a couple of events (including Flight) about how Twitter’s platform plays into that space. However, I’ve substantially stepped back from playing a major role in the MQTT community this year; a decision in part driven by the need to refocus on my new role, partly due to some personal hostility and “burnout” with a couple of specific issues, but mostly because – I’m no longer “needed”! It has been incredibly satisfying to see the MQTT community grow over the past few years. The standardisation of the protocol at OASIS, the large number of implementations, and the ability of many other much smarter people to pick up the kinds of speaking engagements I was previously doing as a matter of course – all of these things make me immensely proud to have helped to lay the foundations for the success of that community over the past six years or so.

I’ve also been very happy to see the success of the Cloud Foundry platform and the people involved – having devoted the previous two years of my career to that nascent Open Source community, it is just fantastic to see it take off and the Foundation get started. Nice work to everyone involved.

I’ve again thoroughly enjoyed my speaking opportunities this year, and the chance to broaden my range. Obviously that has included a lot about the Twitter API and developer platform, and lots again about IoT; but I’ve also spoken on wearables, developer advocacy, and API management. I’m very happy that I got to be a part of the first Twitter Flight conference – one of my speaking career highlights.

Personally, I’ve tried to stay healthy this year (no heart scares, no falls…!); although my travel schedule has been demanding again (TripIt tells me I covered 66613 miles in the air). That did at least include a couple of trips for fun, rather than being all about business :-)

The next year looks to be busy with more events to speak at (and organise!), and much more to do around the Twitter platform. As an historian, a sociologist and someone with a keen interest in the intersection of technology and people, I’m very excited to be a part of this wave of change.

Happy New Year – here’s to 2015!

Pivotal CF – the enterprise platform for software development

My boss and mentor, James Watters, just blogged about the launch of what we’ve been working on since before Pivotal was formed earlier this year – Pivotal One, powered by Pivotal CF (based on Cloud Foundry).

As I wrote back in April

Pivotal is bringing together a number of key technology assets – our Open Source cloud platform (Cloud Foundry), agile development frameworks like Spring, Groovy and Grails, a messaging fabric (RabbitMQ), and big, fast data assets like Pivotal HD.

What we’re announcing today delivers on that promise and our vision – the consumer-grade enterprise, enabling organisations to create new applications with unprecedented speed. The cloud – infrastructure clouds, IaaS like Amazon EC2, VMware vSphere, OpenStack, CloudStack, etc – can be thought of as the new hardware. It’s like buying a beige server box back in the 90s – the IaaS layer gives you a bunch of CPU, network, and storage resources, and for your application to use them, you need a layer in between – an operating system, if you like. We’ve spoken of our ambition for Cloud Foundry as “the Linux of the Cloud”, and it already runs on all of those infrastructures I’ve listed above – in the future, hopefully more.

Why is that important? Why should developers care about this Platform (PaaS) layer? A development team shouldn’t have to go through an 18 month delivery cycle to deliver an app! We’re putting an end to the whole cycle of calling up the infrastructure team, having new servers commissioned, operating systems installed, databases configured etc etc just to get an application deployed and running. When you first push an application to Cloud Foundry, and can then bind data services and scale out with simple individual commands, it really is a liberating experience compared to what traditionally has been required to get your application running. We’re making it quicker and easier to get going – a friction-free, turnkey experience. You should just be able to write your code and make something amazing.

We’re also delivering choice – of runtimes and languages, data services, and also importantly, a choice of “virtual hardware”. When Comic Relief ran in the UK this year, in order to avoid any risk of hardware failure (we all know there’s a risk that Amazon might go down), the applications were deployed on Cloud Foundry running on both Amazon EC2 with geographical redundancy, and on VMware vSphere – no lock-in to any cloud provider, and the developers didn’t have to learn all of the differences of operating different infrastructures, they just pushed their code. We’re happy to know that it was a very successful year for the Comic Relief charity, and that Cloud Foundry helped.

Pivotal One also includes some amazing data technologies – Pivotal HD (a simple to manage Hadoop distribution) and Pivotal AX (analytics for the enterprise). We recognise that as well as building applications, you need to store and analyse the data, so rather than just shipping a Cloud Foundry product, we roll up both the elastic scalable runtime, cutting-edge technologies like Spring.io, and and our big data offerings. That’s different from many of the others in the same market. We’ve been running our own hosted cloud, now available at run.pivotal.io, on AWS for over a year now, so we’ve learned a lot about running systems at scale and Pivotal One can do just that.

Above all, I wanted to say just how excited I am to be part of this amazing team. It is an honour to work with some incredibly talented engineers and leaders. I’m also personally excited that our commercial and our open source ecosystems continue to grow, including large organisations like IBM, SAP, Piston … it’s a long list. We took out an ad in the Wall Street Journal to thank them. I also want to thank our community of individual contributors (the Colins, Matts, Davids, Dr Nics, Yudais… etc etc!) many of whom, coincidentally for me, are in the UK – check out the very cool Github community where some of their projects are shared.

I’m convinced that this Platform is the way forward. It’s going to be an even more exciting year ahead.

A small selection of other coverage, plenty more to read around the web:

Busy times, but let’s talk Cloud Foundry!

Users of the existing beta Cloud Foundry hosted service cloudfoundry.com were sent emails this week explaining that we are almost ready to launch version 2 of the service. If you’re a current user, or if you have signed up in the past, dig through your inbox filters for the email (mine ended up under the “Promotions” label thanks to Gmail’s auto-filing magic).

Cloud Foundry v2, sometimes known as “next-gen” or ng, is a big set of updates. I wrote about some of them in my last blog post, and also noted there that we are going to run the new version on AWS.

Some of the things worth getting excited about are:

  • custom domains (the number one thing I’ve been asked for after every talk!)
  • buildpacks – the ability to use “any” language, framework or runtime that has a buildpack, not just Java, Ruby, or node.js (Matt and Brian seem to be competing to find interesting ones!). By the way, you should totally be trying your Spring and Groovy apps on v2! :-)
  • organisations and spaces – the ability to share apps with a team and collaborate
  • a web management console for your apps
  • a Marketplace, which we will be expanding over time, allowing you to bind third-party services in to your applications.

These are all big changes, and there are many more under the hood (Warden, a new staging process, a new router… it’s a very long list).

My colleague Nima posted a nice slide deck giving a more technical overview of some of the internal changes:

In addition, our demo ninja Dekel has shared a great video of some of the things you can expect from version 2:

Over the past 24 hours or so I’ve been doing my best to respond to questions on Twitter and elsewhere. The existing v1 version will go away on June 30th, so if you are using it now you’ll want to look at migrating apps in the next couple of weeks, and we’ll share more on that soon. The new version will have pricing attached, with a free trial period too. Of course, the code is always available on Github and you’re free to spin up your own CF instance running on AWS, OpenStack or vSphere.

I know folks will have many more questions about the specifics over the next week or so, and we will be looking out for them.

Supercharging the community

As we have grown closer to v2 release, there has been ever-increasing activity in the vcap-dev mailing list and around the community. We’ve had more and more code contributions (so much so that I recently wrote a blog post about how you can contribute to the CF core projects). Projects like the cf-vagrant-installer and cf-nise-installer are helping people get local environments running very quickly. Our friends from PistonCloud released their turtles project. Best of all, the super Dr Nic Williams recently set up a cloudfoundry-community organisation on Github to act as an umbrella for many of these community contributions (info on how to join is here).

Let’s talk! (in London)

Over the past year or so I’ve spent a lot of time out in the developer community in London, and it has become apparent that a lot of folks are interested, already contributing to the community, or in some cases, already running their own CF instances in production :-)

So, I thought it would be a good idea to do some bridge building and bring folks together to get to know one another. A brief unscientific Twitter poll suggested that other people liked the idea, so we’ve stuck a stake in the ground (evening of July 3rd) and I set up an Eventbrite page for a meet up. If you’d like to chat with people about Cloud Foundry over a drink, do sign up and come along. I’ll sort out a venue in the next couple of weeks, but I imagine it will be “around Shoreditch” or possibly over towards Waterloo, for purely selfish reasons! Totally informal, this is just a community meet up, so I’m not planning to do slides and talks and stuff – just come and share ideas or ask questions!


Cloud Foundry has gone Pivotal – so what’s new?

A few weeks ago I was privileged to be at the launch of Pivotal – a new organisation formed by VMware, EMC, and with investment from GE. You can read all about our new company at GoPivotal.com.

I am Pivotal

I am Pivotal

What does that mean for me, and for my role on the Cloud Foundry team? What is happening with Cloud Foundry right now? What about the Cloud Foundry community?

Well, as my über-boss James Watters recently wrote – we are a central part of the Pivotal business.

Our mission is to become the most popular platform for new applications and the services used to build them across public and private clouds.

That’s a pretty compelling mission statement, and I’d personally even add that we want to be the “best” platform, as much as “most popular”. One of the main reasons I wanted to spend a couple of weeks at the Pivotal office in San Francisco was really to immerse myself in the team and in the culture of Pivotal Labs, as well as to be at the launch event, and to get a strong handle on what is happening with Cloud Foundry, version 2…

Wait, what? Version 2?

In the middle of last year, the Cloud Foundry team started some major work to improve many of the features offered by the platform. Back then, it was written about on the Cloud Foundry blog. We initially started to refer to “ng” components like the Cloud Controller (“cc-ng”), and that’s what we now mean when we refer to “v2”. At the start of the year we published a roadmap which laid out a lot more detail in terms of what is coming. There’s some really great stuff in there – many bugs squashed; a new, high performance router; support for developers to collaborate on apps, via concepts of organisations and spaces; new containerisation via Warden; custom domains (yes, finally!); and most importantly, support for buildpacks. Buildpacks will bring a major change to our platform, replacing the former concepts of runtimes and frameworks (say, Java with Spring) with the ability to drop in whatever runtime or container you may choose, instantly making the platform more customisable. We’re pleased that the folks over at Heroku have allowed us to inherit the buildpack concept and having played with the new platform, I believe this gives us a really cool and solid way to support apps.

Deploying #cloudfoundry v2 on Amazon

Deploying Cloud Foundry v2 on Amazon

While I was in San Francisco, I used BOSH to deploy my own new Cloud Foundry v2 instance to Amazon EC2 (and also attended the AWS Summit, which was a bonus!). Right now the team is working on migrating our  hosted cloudfoundry.com platform to EC2, and when we officially boot up v2 for the public, it will be running right there. This is not new news – both James, and our CEO Paul Maritz, have repeatedly spoken about AWS.  The point of Cloud Foundry has always been that it is a platform that is Infrastructure-as-a-Service agnostic, even when it was started by VMware, and I’m seeing increasing interest from folks want to run it on OpenStack, AWS, and other infrastructures as well as vSphere (by the way, did you read about how Comic Relief 2013 ran on Cloud Foundry on vSphere and AWS? so cool!). There is no lock-in here – write once, deploy to cloudfoundry.com, to a partner running a compatible Cloud Foundry-based instance, or to your own private cloud on your on infrastructure, as you wish. The Open Source nature of the project is exactly why I jumped on board with the team a little over a year ago.

Talking of the update to cloudfoundry.com: it is also worth mentioning that when the beta period comes to a close we will have pricing plans, a nice web console for user, organisation and application management, and the start of a marketplace for partners to plug-in their own services for developers. I can’t give more details in this post, watch the official channels for news!

I felt very strongly that I wanted to write about version 2. It is a very big step in evolving the Cloud Foundry architecture, and I believe that it is important for the broader  community to understand that it is a significant change. If you are running an app on cloudfoundry.com today, we’ll shortly contact you with information about migration to the new platform, as some changes will be needed to adapt to the fact that runtimes and frameworks are now buildpacks, there will be some changes to the way services work, and you will need our new ‘cf’ gem to deploy to the new service. We have already “paused” new signups on the current platform. If you look at the new documentation, you will find that it now focuses on version 2 – we apologise for any confusion during the transition process.

We’ve been talking with ecosystem partners about version 2 as well. For instance, our friends at Tier 3 recently blogged about Iron Foundry plans, and I had the pleasure of meeting with Stackato folks in person in San Francisco recently. If you are working with your own Cloud Foundry instance privately (we know that many organisations are!) I strongly urge you to talk to us via the vcap-dev mailing list to learn how you can start to take advantage of what the new platform brings.

What else does Pivotal mean for Cloud Foundry? Well – we are more open than ever, and keen to work with the community on pull requests to add features via Github. I’ve just written a  post for the Cloud Foundry blog about how to participate in the Open Source project. In fact, I’ll be talking more about this at the Cloud East conference in Cambridge next Friday May 24. We’re always happy to talk more about how to collaborate.

These are exciting times!


M2M, IoT and MQTT at EclipseCon 2013

EclipseCon 2013 is here, and I’m in Boston with the great folks from around the community this week.

Koneki, Paho, Mihini

There’s a LOT of content around the machine-to-machine space this year, and growing interest in how to use instrumented devices with an embedded runtime with lightweight messaging. If you’ve not been following the progress of the M2M community at Eclipse, we now have an M2M portal, along with nice pages for each of the three associated projects Koneki, Mihini, and Paho.

M2M hardware kits

Almost the first thing I saw when I walked in yesterday was my buddy Benjamin Cabé assembling a bunch of electronics kits (Raspberry Pis and Arduino Unos) for today’s M2M tutorial which will use Eclipse Koneki and Mihini. This will be the first opportunity for many folks to play with the new Mihini runtime. Later this evening, we’ll have the chance to run a hackathon with things like Raspberry Pi and Orion and others parts as an extended Birds of a Feather.

What are some of the other M2M sessions to look out for?

There’s also the first meeting of the OASIS TC for MQTT due this week, and a meeting of the Eclipse M2M Industry Working Group scheduled as well. Exciting times!

The corridor conversations and late night beer sessions are as always invaluable, and myself and many of the other project folks will be around – I’m always happy to talk about Paho in particular. At Paho we now have updated Java and C MQTT clients in Git (NB check the ‘develop’ branch for the latest Java updates), along with the Lua client, and proposed contributions of Objective-C, Javascript and Python clients are at various stages of review looking to join the project.

Oh, and if you are interested in MQTT, come and find me for some MQTT Inside stickers that you can use with your own hardware projects :-)