lc-tools 0.7.1.1: bugfix release

I have released a bugfix lc-tools version: 0.7.1.1.

It contains just two improvements:

• Support RAM printing for GoGrid nodes using %(ram)s format string, submitted by Kirill Popov
• Format string decoding using 'string-escape', submitted by Stanislav

As usual, new version is available on pypi:

easy_install lctools

I decided to stop uploading distfiles to github because most people use pypi anyway, so I doubt anybody needs it on github, but please let me know if I'm wrong.

lc-tools 0.7.1.0 released!

I'm happy to announce new lc-tools release: 0.7.1.0!

It doesn't have new features, but was updated to work with libcloud 0.7.x (earlier libcloud versions are no longer supported!). All the kudos go to Pavel Vozdvizhenskiy for noticing problems and testing.

You can install the new version using either easy_install:

easy_install lctools

or by downloading and installing it by hands:

lctools-0.7.1.0.tar.gz

SHA256 (lctools-0.7.1.0.tar.gz) = 7f65c29cda06d256c02f6544b4957ca984f34f1cfed5b24558c3b4338f87dd9c

lc-tools 0.5.0.0 released!

A lot of time passed since the latest lc-tools release, and now it's almost summer, so it's time for updates!

The first news is that libcloud is now an Apache Top Level Project (TLP) and new version 0.5.0 has been released with some new features, bug fixes and new support for working with storage and load balancers. And it's a perfect time for new lc-tools release!

There are not so many new features in this lc-tools release, most of the changes were to chase internal libcloud changes. Additionally, I've changed versioning scheme a little and now it looks like $libcloud_version$lctools_version, where $libcloud_version is the version of libcloud current release was targeted and tested with.

The only new major change is the addition of lb-* scripts to manage load balancers. You can get more information on them from Documnetation.

You can install the new version using either easy_install:

easy_install lctools

or by downloading and installing it by hands:

lctools-0.5.0.0.tar.gz

SHA256 (lctools-0.5.0.0.tar.gz) = f09629664a5209be687c56cfe07e123939e28b18070b719436307a74b823f10b

Happy hacking on the cloud stuff!

Managing OpenStack using libcloud and lc-tools

OpenStack is getting more and more popular and number of its installation increases day by day. We've got one as well. :-) While there are quite a number of bindings and tools to connecting to its API I prefer use familiar proven tools like libcloud and lc-tools. And since OpenStack supports both EC2-compatible and Rackspace-compatible APIs, there are two ways connecting to it.

EC2 API

To use EC2-compatible API, pick EUCALYPTUS driver like that:



You need to define EC2_ACCESS_KEY and EC2_SECRET_KEY variables to match your setup. Hostname, port and path arguments which are passed to constructor could be extracted from your EC2_URL environment variable.

Note: if you're not using trunk version of libcloud, you will need to adjust imports a little, i.e. strip compute. part from them.

Rackspace API

To use Rackspace-compatible API you will to check out trunk version of libcloud (please refer here to obtain it) which provides OPENSTACK driver, which could be used in a similar fashion:



Again, here you will need to redefine NOVA_API_KEY and NOVA_USERNAME for your setup and grab host and port from NOVA_URL.

Let me remind you once again that you need trunk version of libcloud to make it working. One more thing: OpenStack is not fully compatible with Rackspace API, so not everything will work smooth.

lc-tools

lc-tools is a set of command line tools on top of libcloud to manage your cloud servers. Git version of lc-tools supports working with OpenStack both through EC2 and Rackspace compatible APIs. Here's config samples for both:


[ec2compat]
driver = eucalyptus
access_id = CHANGE_ME
secret_key = CHANGE_ME
extra = { "host": "127.0.0.1", "secure": False, "port": 8773, "path": "/services/Cloud" }

[rscompat]
driver = openstack
access_id = nova_admin
secret_key = CHANGE_ME
extra = {"host": "127.0.0.1", "port": 8774}


And just check if it works by executing lc-node-list -p ec2compat and lc-node-list -p rscompat.

References

* libcloud
* lc-tools

libcloud load balancers feature status

Background

If you follow libcloud maillist, you probably aware that I've started working on adding load balancers support. If not, please give it a quick view.

Current status

At this point I have implemented almost all features I've planned, namely:

• Defined a common interface
• Implemented drivers for Rackspace and GoGrid
• Covered basic functionality with unit tests
• Implemented load balancer support in lc-tools and run-time tested it

Currently interface is pretty basic and supports only main operations, like: CREATE/READ/DELETE for balancers and ADD/LIST/REMOVE operations for their nodes. Obviously, there are far more operations on load balancers that could be useful such as balancing algorithm and session handing options, but I need some time to play around with these things to understand how to plug it better.

One more thing I'm considering is adding support for blocking operations. It's quite common situation when you find balancer in immutable state: for example, when you add new node to balancer it needs some time to perform initial configuration and during that period you cannot perform operations on it, such as removing or adding more nodes. My current implementation just throws and exception in such case so user can figure out that object is immutable and try after some time.

However, it might be not very convenient for user to wrap any call to catch such situations, so I'm thinking about adding support for blocking mode, where function will block until operation succeeds or timeout reaches; blocking/non-blocking mode could be specified at driver init time. Most likely it's the way things will be done.

Examples

Meanwhile, take a look at this example to understand how things look like at the moment:

example_lb.py

Also, as I've mentioned above, I've added basic support for balancer manipulation to lc-tools. Will cover this topic a bit later when things settle down a little, but here's a quick overview how it feels like:

Giving it a try

To give it a try checkout balancers branch of my libcloud fork on github:


git clone git://github.com/novel/libcloud.git --branch balancers


and then install it like you usually do. The example file mentioned above would be useful to get started. Additionally, it might be useful to look at lb-* scripts in lc-tools lb branch.

Further reading

• Original announce on the maillist
• GoGrid Balancers API (part of their main API) reference
• Rackspace Balancers API reference (PDF)

lc-tools 0.1.4 released!

I'm happy to announce release of lc-tools 0.1.4.

This release is based on libcould 0.4.2 which was released not so long ago. Changes from the previous version include support of the brand new SSL handling in libcloud, some custom features for Rackspace and GoGrid drivers and minor bugfixes. Please check NEWS file for a full list of changes.

There are some non-function changes as well: lc-tools got its own website and also a tutorial.

You can install the new version using either easy_install:

easy_install lctools

or by downloading and installing it by hands:

lctools-0.1.4.tar.gz

SHA256 (lctools-0.1.4.tar.gz) = 407ce1f29f0e869219ffff2f866fb6de226b693bf925817b6b43a612120b1ff7

Happy hacking on the cloud stuff!

GoGrid locations support in libcloud

GoGrid added support for locations (US-West-1 and US-East-1) quite some time ago and I finally managed to implement this feature for libcloud's GoGrid driver. Fortunately, libcloud API already takes into account that cloud providers could have more than one location, so the most important part was there already.

Examples

libcloud

lc-tools

Some of the lc-tools support '-l' flag to specify location id. Here's how a typical sequence would look like:



Few notes here: you probably noticed that we filter images by locations but don't do it for sizes. Strictly speaking, sizes might differ for various providers and this is also accounted by libcloud API; but for GoGrid sizes are same for both locations, so I skipped this step.

And the second thing: all these features are available in svn version of libcloud and git version of lc-tools.

Also the thing I should have been mentioned for some time already: GoGrid provided me with an account and it is quite helpful for various testing.

Installing Man Pages with Distutils

People who read my blog probably remember that I started small project lc-tools which is based on libcloud and provides facilities to control clouds. Since the whole project is basically several Python scripts, I though it would make sense to provide a manual page for each.

At first I had to choose format to use for manual pages. I remember dealing with groff several years ago and decided it's too bothersome for me and will require as much effort as writing all the other Python code. :]

Fortunately, I've spotted ronn which allows to use Markdown for formatting manual pages, and it's a great thing as it allows one to think about content, not the formatting.

Using ronn I created all the man pages quite quickly but then faced another problem: distutils doesn't have facilities to install man pages. That is.

I've managed to implement a temporary (will explain why it is temporary later) solution. The idea is to override default install target this way:



So we just execute install.sh after default 'install' target did its job. One important thing is to pass "PREFIX" to make sure that man pages will be installed in correct place.

The install.sh script is pretty trivial (19 lines length) and you can see it here.

Ok, so what's wrong with this solution? It forces users to have ronn installed to be able to install man pages. While it's OK for people who clone things from git, it's definitely not OK for people downloading the package from PyPI to force them installing ruby, ronn and friends. So the plan is to modify bdist targets to include generated manpages and modify install target to be able to install cached versions if ronn is not installed on the system.

That doesn't seem to be hard to do, though. I think I will implement it and release a new version.

BTW, ronn allows you create HTML man pages easily, so probably it's time to create a site for lc-tools with extensive documentation.

Update: I've asked this question on stackoverflow and haven't got any responses so far.

Announcing lc-tools: command line tools on top of libcloud

Looks like it's time to announce lc-tools! What is it? Basically, it's just a few command line tools to manage your clouds based on libcloud. What advantages does it give? Well, it supports as many cloud providers as libcloud supports (ec2, rackspace, gogrid and many others) and you can manage them all with a single tool.

Look and feel is pretty similar to gg's one as it seems to be quite comfortable and I used gg more than year on a regular basis and haven't had any issues with it.

Currently lc-tools covers almost all functionality provided by libcloud, but I've only tested it with GoGrid.

Some examples so you can check look and feel of the application:

Listing available node images

$ lc-image-list|grep -i centos
image CentOS 5.2 (32-bit) w/ RightScale (id = 62)
image CentOS 5.2 (64-bit) w/ RightScale (id = 63)
image CentOS 5.3 (32-bit) w/ None (id = 1531)
image CentOS 5.3 (64-bit) w/ None (id = 1532)


... and sizes

$ lc-sizes-list
size 512MB (id=512MB, ram=512, disk=30 bandwidth=None)
size 4GB (id=4GB, ram=4096, disk=240 bandwidth=None)
size 2GB (id=2GB, ram=2048, disk=120 bandwidth=None)
size 8GB (id=8GB, ram=8192, disk=480 bandwidth=None)
size 1GB (id=1GB, ram=1024, disk=60 bandwidth=None)

Creating a new node

lc-node-add -i 62 -s 1GB -n mynewnode

Pay attention to values 62 and 1GB -- they are ids of image and size respectively (from previous examples).

Listing nodes

$ lc-node-list
100xxx mynode1 173.204.xx.yy Running
100xxx mynode2 173.204.xx.zz Running

Destroying nodes

$ lc-node-do -i node_id destroy

Project's github page contains a README file with documentation enough to get started, but if you have any problems or suggestions feel free to contact me via github, blog or email.