Watzmann.Blog Varying amounts of fiber

• Home

TL;DR: have a look at sinatra-rabbit.

When we converted Deltacloud from Rails to Sinatra, we needed a way to conveniently write the controller logic for RESTful routes with Sinatra. On a lark, I cooked up a DSL called ‘Rabbit’ that lets you write things like

collection :images do
  description "The collection of images"

  operation :index do
    description "List all images"
    param :id,            :string
    param :architecture,  :string,  :optional
    control { ... controller code ... }
  end

  operation :show do
    description 'Show an image identified by "id" parameter.'
    param :id,           :string, :required
    control { ... show image params[:id] ... }
  end

  operation :destroy do
    description "Remove specified"
    param :id,    :string,    :required
    control do
      driver.destroy_image(credentials, params[:id])
      status 204
      respond_to do |format|
        format.xml
        format.json
        format.html { redirect(images_url) }
      end
    end
  end

end

That makes supporting the common REST operations convenient, and allows us to auto-generate documentation for the REST API. It has been very useful in writing the two frontends for Deltacloud.

The DSL has lots of features, for example, validation of input parameters, conditionally allowing additional parameters, describing subcollections, autogenerating HEAD and OPTIONS routes and controllers, and many more.

Michal Fojtik has pulled that code out of Deltacloud and extracted it into its own github project as sinatra-rabbit In the process, there were quite a few dragones to slay: for example, in Deltacloud we change what parameters some operations can accept based on the specific backend driver. For example, in some clouds, it is possible to inject user-defined data into instances upon launch. In Deltacloud, the logic of what routes to turn on or off is based on introspecting the current driver, which means that Deltacloud’s Rabbit knows about drivers. That, of course, has to be changed for the standalone sinatra-rabbit. Michal just added route conditions that look like

collection :images do
    operation :create, :if => lambda { complicated_condition(request) } do
        ...
    end
end

Hopefully, sinatra-rabbit will grow to the point where we can remove our bundled implementation from Deltacloud, and use the standalone version; there’s still a couple of features missing, but with enough people sending patches, it can’t be very long now ;)

Installing Deltacloud is work. Not a lot of work, in fact it is very easy, but it still involves installing a package/gem and starting a server. For simple development and test uses, even that is not necessary any more.

There’s two of them: one, https://api.deltacloud.org/ runs the latest stable release, currently release 0.4.1. The other, https://dev.deltacloud.org/, runs the bleeding edge code from the git repo.

Both use the same self-signed SSL certificate. Its SHA-1 fingerprint is D3:3D:13:73:37:88:59:F1:FE:08:51:70:A0:BA:60:99:F1:E9:DD:45.

If you’ve been scratching your head, wondering what all this Deltacloud business is about, just head over to one of these two servers and explore the API. There’s a friendly HTML interface for just that, or, of course, the obligatory XML and JSON, variants. The public servers run the EC2 driver as their default; when prompted for a username and password, just enter your Amazon AWS access key ID and secret.

We just released Apache Deltacloud 0.4.0, part of the Apache Incubator. The release contains a huge number of enhancements and additions. The full list can be found in the release announcement, but some of them bear highlighting separately.

The biggest new feature is probably a driver for VMWare’s vSphere. This makes it possible to turn any vSphere 4.0 installation into a simple cloud. To use the driver against a vSphere API at https://vsphere.example.com/sdk, start the Deltacloud server with

API_PROVIDER=vsphere.example.com deltacloudd -i vsphere -t 200

Besides the basics of image and instance management, the driver supports a few nifty features, in particular injection of user data.

A second new driver adds support for ‘condor-cloud’, a simple cloud implementation that uses the Condor grid manager as its backend. While it’s probably not enough to replicate EC2, it is certainly good enough to build a simple cloud out of a few machines, Condor, Deltacloud, and a few glue scripts.

We (well, Marios) added support for firewalls to the Deltacloud API. Since not all clouds offer firewalling, this is currently only supported for EC2 and Eucalyptus, via their security groups. The model that the API exposes though represents a fairly generic firewall with rules and sets of rules. We will expand that to other drivers in future releases.

We (well, Michal) reworked the entire HTML UI using jquery-mobile — if you want to explore the Deltacloud API from your smartphone or tablet, that just got a whole lot easier. I find the result a much cleaner UI; since the Deltacloud HTML interface follows the API very closely, as it is meant as a tool to test and explore the API, pages have always been fairly sparse, something that the mobile interface makes less annoying.

For more details, read the release announcement or download the release

As we all know by now, cloud computing is a veritable goat rodeo, an unseemly sight for anybody’s stomach. Disconcerted by these proceedings, Mark Shuttleworth lets his stomach have the better of him, and declares it over by picking the winner. That, of course, is not how you end a goat rodeo: you end it by pointing out that there are goats, not horses, involved.

The goat in question is the undisputable fact that, in North America, Amazon’s Web Services are the front runner in the IaaS cloud computing space. Which makes Amazon’s API the most used and most widely studied IaaS API. Mark’s recommendation for the OpenStack project, which currently has multiple API’s, but should, for a number of reasons, settle on one is to just adopt the AWS API. And does this with arguments that are simultaneously breathtaking and misleading. In particular, the discussion of HTTP belittles how crucial it was that the definition and evolution of HTTP was not controlled by a single vendor.

I completely agree with Mark that whether the AWS API is a good or bad API is completely besides the point for this discussion — where we disagree is his assertion that API’s aren’t a place where meaningful innovation is going to happen. Anybody who has followed the evolution of the API’s of any single cloud provider, let alone the wider cloud ecosystem, will immediately see the folly of this assertion. And there is no reason to assume that this API evolution will stop any time soon, there are still way too many important debates on modeling familiar concepts for cloud, and on streamlining cloud usage to be had.

The big problem with adopting the AWS API wholesale though is that it puts the future of OpenStack, and if Mark had his way, the whole IaaS space, into the hands of a single vendor. There is absolutely no way for anybody to get a feature into that API, unless they can get Amazon to agree with them and implement it in their version. Mark addresses this concern with a brusque

It’s true that those API’s would better be defined in a clean, independent forum analogous to the W3C than inside the boiler-room of development at any single cloud provider, but that’s a secondary issue. And over time, it can be engineered to work that way.

This is not a matter of ‘better’ or ‘cleaner’ — it is a question of how we, as a community, want to see the world of cloud to shape up. Especially from a North American viewpoint, that landscape looks disheartening right now, as if IaaS cloud will sooner or later be the domain of two or three large vendors; that is not the case in Europe, nor does it have to be the future. The much-used analogy with the early ISP market comes to mind, with AOL’s seemingly unstoppable march to world dominance. One of the reasons this never happened and we all can choose from a large number of ISP’s today of course is the dogged pursuit of truly open standards.

Sticking to one vendor’s offering is the exact opposite of an open standard; and there is no reason to believe that Amazon can be engineered into a more open process for API innovation. For one, I am not aware of a single standards effort, be it through an SDO or an open source project, that Amazon is involved in. Adopting AWS API’s wholesale and widely will reduce the incentive for Amazon to join any open API effort, not increase it.

Amazon’s API’s should be studied because Amazon is a pioneer, a first-mover in the IaaS space, who were in many cases the first ones to think through specific issues. Whatever the technical merits, their lack of participation in open processes makes their API a dangerous bet, no matter how benign they might be today.

P.S.: There’s some talk of Deltacloud in the comments; the analogy with JDBC is completely oblique, since we are talking about loosely coupled web services here, and not tightly coupled in-process API’s. Which also makes the second point, that Deltacloud is and has to be a lowest common denominator, moot. There is enough flexibility in a RESTful API to avoid that. Interestingly though, the comment directly contradicts Mark’s assertion that API’s aren’t a place to innovate — certainly, there is some innovation in all these different models.

Somehow, I find myself writing the same email to introduce people to git over and over again. But no more ! Now, I will only send out links to this blog entry.

Git can be intimidating at first, even though it is probably the most forgiving source control system out there. There are plenty of great tutorials on git these days. I assume that you have read enough of them to understand the absolute basics of cloning a repo, pulling to update your repo, and how to commit. In other words, you’ve run git clone, git pull and git commit before.

The key to working with git happily is to use branches liberally; if in doubt, branch. And the key to working with branches is understanding git rebase. In particular, git rebase -i will make you fall in love with git. It lets you not just edit committed patches, it also lets you combine patches, reorganize them etc. Once the initial excitement over interactive rebase wanes, try out interactive add (git add -i) to renew the bliss.

Once the basics are out of the way, you will want to implement some extension to whatever you’ve cloned and pulled, and then submit that back upstream for inclusion. That usually involves working on your own for a bit, and then generating and sending out patches of your work for review and merging upstream. Changes you make should always go onto private (‘topic’) branches; create a new branch for each piece of distinct work. The overall workflow for this is

    git checkout master
    git pull # make sure we have the latest bits
    git checkout -b dev/feature
    ... edit/add/commit until happy, with an eye towards having your
    branch constitute an easily reviewable patch series; when
    working on the branch for longer, pull master repeatedly and
    rebase your branch ...

Once your work is ready to be shared with the rest of the world, do the following to generate and mail out patches

    git checkout master
    git pull
    git rebase master dev/feature
    git format-patch -o /tmp/patches master
    git send-email --to=hackers@example.org --compose --subject 'Awesome feature' --thread /tmp/patches

When changes need to be made to address review comments, work them into your dev/feature branch, using interactive rebase to add them where needed in the patch series, then repost.

The initial Deltacloud project consisted of two components: the RESTful API and an inter-cloud broker, a web application. When we submitted the API part of the project to Apache, also under the name Deltacloud, it was clear that we’d have to find a new name and home for the cloud broker. The source repos had always been separate, the mailing lists were separated upon the acceptance of Deltacloud into the Apache Incubator, but separating the web site proved to be more work (that we repeatedly put off).

Not any longer. Thanks to the work of Chris Lalancette and others, the two projects now also have separate web sites: the cloud broker is now called Aeolus, and can be found at aeolusproject.org. The Deltacloud web site that had been available through Apache for a while now, is also available at deltacloud.org (via a redirect).

ApacheCon takes place next week in Atlanta. And Deltacloud will be there, too: we are having a Deltacloud meetup on Thursday, 11/4, at 8pm in Room 4. You can find more details about the meetup on the ApacheCon Wiki

The goal of the meeting is to bring everybody from the Deltacloud community together, be it seasoned contributors or novices just starting out with Deltacloud and discuss the state and the future of the Deltacloud project.

I am looking forward to meeting lots of people there; besides Deltacloud shop talk, we’ll also have beer (the free as in beer variety). Carl Trieloff, Deltacloud’s sponsor in the Apache incubator will also be there.

Watzmann.Blog by David Lutterkort is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

Generated with Jekyll