GoGrid locations support in libcloud

GoGrid added support for locations (US-West-1 and US-East-1) quite some time ago and I finally managed to implement this feature for libcloud's GoGrid driver. Fortunately, libcloud API already takes into account that cloud providers could have more than one location, so the most important part was there already.

Examples

libcloud

lc-tools

Some of the lc-tools support '-l' flag to specify location id. Here's how a typical sequence would look like:



Few notes here: you probably noticed that we filter images by locations but don't do it for sizes. Strictly speaking, sizes might differ for various providers and this is also accounted by libcloud API; but for GoGrid sizes are same for both locations, so I skipped this step.

And the second thing: all these features are available in svn version of libcloud and git version of lc-tools.

Also the thing I should have been mentioned for some time already: GoGrid provided me with an account and it is quite helpful for various testing.

New website version

For a long time my website was just a set of random links. Recently I decided to give it some structure and make it look like more like an extended visit card instead of just a collection of random links. So, here's the result: http://novel.evilcoder.org.

I used Hyde for static generation of the site and I've put the sources on github: http://github.com/novel/novel.evilcoder.org.

It helped a lot to look at Steve Losh's sources of his site which are hosted on github as well: http://github.com/sjl/stevelosh (actual site being: http://stevelosh.com).

Anyway, there is a lot of work ahead, I need add some more content and learn and work on html/css/design stuff, at which I'm not very good.

Installing Man Pages with Distutils

People who read my blog probably remember that I started small project lc-tools which is based on libcloud and provides facilities to control clouds. Since the whole project is basically several Python scripts, I though it would make sense to provide a manual page for each.

At first I had to choose format to use for manual pages. I remember dealing with groff several years ago and decided it's too bothersome for me and will require as much effort as writing all the other Python code. :]

Fortunately, I've spotted ronn which allows to use Markdown for formatting manual pages, and it's a great thing as it allows one to think about content, not the formatting.

Using ronn I created all the man pages quite quickly but then faced another problem: distutils doesn't have facilities to install man pages. That is.

I've managed to implement a temporary (will explain why it is temporary later) solution. The idea is to override default install target this way:



So we just execute install.sh after default 'install' target did its job. One important thing is to pass "PREFIX" to make sure that man pages will be installed in correct place.

The install.sh script is pretty trivial (19 lines length) and you can see it here.

Ok, so what's wrong with this solution? It forces users to have ronn installed to be able to install man pages. While it's OK for people who clone things from git, it's definitely not OK for people downloading the package from PyPI to force them installing ruby, ronn and friends. So the plan is to modify bdist targets to include generated manpages and modify install target to be able to install cached versions if ronn is not installed on the system.

That doesn't seem to be hard to do, though. I think I will implement it and release a new version.

BTW, ronn allows you create HTML man pages easily, so probably it's time to create a site for lc-tools with extensive documentation.

Update: I've asked this question on stackoverflow and haven't got any responses so far.

On Skytap

I've ranted about GoGrid a bit in my previous post and now it's Skytap's time to receive its fair amount of rant. OK, just kidding, will sure my experience of working with it.

Concept

The general concept is quite different from all the cloud providers I worked with. What I've usually seen is the main abstraction of the cloud providers was a 'Server' (or node, or vm, whatever) -- just a single VM with properties like RAM size, CPU power etc, and which is created based on some 'Template'.

Skytap does things differently: 'Server' model doesn't play the main role. The thing all the stuff's built around is 'Configuration', which is basically a collection of VMs. Configuration is created from 'Template' which is also actually a collection of VM templates. You cannot directly add a VM to configuration, you have to merge the template with a VM.


I haven't decided if I like this way of representing things or not. Surely, in some situation it's convenient to be able to group VMs and work with them as with single object and we've even implemented a similar scenario at work on top of other cloud provider. On the other hand, when you need to work on single VM level it's a bit confusing and cumbersome.

Web UI

It's ok but has some usability problems IMO. Well, I'm not a usability expert by any means, but these things just seem wrong to me.

For example: a way to API doc: Dashboard -> Support -> API -> 'How do I access API' solution -> link to developers page -> link to pdf guide. 5 steps from dashboard, not very good IMHO.

API

I have an impression that a lot of people treats any http service as REST service, that's obviously wrong. Skytap is good at this matter and their REST service is really RESTful. They even use all these fancy HTTP methods like PUT or DELETE.

Based on some error messages and headers, I assume API is powered by Rails and Mongrel. The tutorial suggests using XML, but it looks like the API supports JSON as well, but not for all resources though. Having full JSON support would be really nice. Moreover, I've been playing with Firebug on web ui and noticed that AJAX is actually using JSON to communicate with the API, so it's strange why JSON is not fully supported.

Also, API gives quite a low lever control to VMs. I.e. if you want to assign an IP address to a VM, you have to think not on VM level (like "assign address A to VM 'alice'") but on VM's OS network interface level (like "asign address A to the first network interface of VM 'alice'").

This forces you to make more API calls to get things done.

Let's look at the quite typical task: create a VM with some of public IP available on our account.

GoGrid	Skytap
find an unassigned IP create node from some template and use the IP from the previous step	create a configuration from template find available IP assign an IP from the pevious step manually start the configuration

So Skytap forces you to make twice more API calls than GoGrid. On the other hand, the feature of assigning public IPs wasn't documented at all in the API reference, so probably it will be changed in future and will become better. I've figured out how this works using Firebug. One more thing I haven't figured out how to handle better is the definition of term 'available IP' by Skytap. Basically, it treats IP as 'available' if it's not used by a running VM. Sooo... you can assign the same IP to several VMs. You will be able to start one of these VMs even, but other VMs with the same IP will fail to start.

There is one more thing that makes me feel quite worried: it looks like the credentials are stored in per template basis. So, if you create two configrations from the same, say, RHEL public template, the root passwords of VMs in both of your configurations will be same. I really really hope to be wrong on that or maybe that passwords are generated on per-account basis (which I cannot check as I have only one account).

Conclusion

Summarising things up I have a strong feeling that I probably don't quite understand philosophy behind Skytap and probably it's targeted for the use cases where API is not very important.

About GoGrid Support

Let's discuss GoGrid support today. As an example I will take recent '<h1>596 Service Not Found</h1>' error.

This error arises about every 15-20th request to the API, and it's quite often. The fact that this error is not well formatted JSON or XML with GoGrid API exception and therefore might not be handled well by some libs makes it even more annoying. Yeah, and this error was noticed by at least two more people on different accounts.

So the natural thing to do was to start a case and try to find out what's wrong (as it's obviously not a normal behaviour). I did that and I got an answer like "Yeah, we know about this error, if you spot it try waiting 5 minutes and try again". It's just ridiculous. Imagine an engine of your car stops after each hour of driving, you come to the service and being told like wait 5 minutes then start the engine again and your problem is fixed!

You'd unlikely want to visit such a service again and the same goes for support. Correct me if I'm wrong, but the first thing that customer wants from support is getting his problem fixed and if the problem cannot be fixed right away get estimates and details why it cannot be fixed. And who needs workarounds, especially for such obvious cases?

Announcing lc-tools: command line tools on top of libcloud

Looks like it's time to announce lc-tools! What is it? Basically, it's just a few command line tools to manage your clouds based on libcloud. What advantages does it give? Well, it supports as many cloud providers as libcloud supports (ec2, rackspace, gogrid and many others) and you can manage them all with a single tool.

Look and feel is pretty similar to gg's one as it seems to be quite comfortable and I used gg more than year on a regular basis and haven't had any issues with it.

Currently lc-tools covers almost all functionality provided by libcloud, but I've only tested it with GoGrid.

Some examples so you can check look and feel of the application:

Listing available node images

$ lc-image-list|grep -i centos
image CentOS 5.2 (32-bit) w/ RightScale (id = 62)
image CentOS 5.2 (64-bit) w/ RightScale (id = 63)
image CentOS 5.3 (32-bit) w/ None (id = 1531)
image CentOS 5.3 (64-bit) w/ None (id = 1532)


... and sizes

$ lc-sizes-list
size 512MB (id=512MB, ram=512, disk=30 bandwidth=None)
size 4GB (id=4GB, ram=4096, disk=240 bandwidth=None)
size 2GB (id=2GB, ram=2048, disk=120 bandwidth=None)
size 8GB (id=8GB, ram=8192, disk=480 bandwidth=None)
size 1GB (id=1GB, ram=1024, disk=60 bandwidth=None)

Creating a new node

lc-node-add -i 62 -s 1GB -n mynewnode

Pay attention to values 62 and 1GB -- they are ids of image and size respectively (from previous examples).

Listing nodes

$ lc-node-list
100xxx mynode1 173.204.xx.yy Running
100xxx mynode2 173.204.xx.zz Running

Destroying nodes

$ lc-node-do -i node_id destroy

Project's github page contains a README file with documentation enough to get started, but if you have any problems or suggestions feel free to contact me via github, blog or email.

Kharkov

Posting from a hotel in Kharkov. The trip to Kharkov is quite short, standing only for one night; so already going to check out soon. Bad thing is transportation that takes 2 nights (and two different trains) to get there, so actually have to spend 4 nights in trains for the sake of one night here.

The city looks clean and accurate, most buildings are in old classical style, at least at the central part and I have an impression that central part is quite large. There's a subway here also, though city is not very large (population is 1.5 millions).

Now it's time to pack my bags...