fusefs-cloudstorage: accessing cloud storage using libcloud and fusefs

Few weeks after cloud storage support appeared in libcloud I started playing with it by creating a fusefs filesystem based on it. Finally the first version is almost ready, so it's time to write a few words about it.

So, for example, you have a cloud files account and want to mount it as it were a local filesystem. You do it this way:

novel@ritual:~/code/fusefs-cloudstorage/test %> cloudstorage.py -o driver=CLOUDFILES_US -o access_id=foo -o secret=acbd18db4cc2f85cedef654fccc4a4d8 ./test

where foo is your access id and this hash is the secret key. Filesystem will be mounted to test sub-directory of the current directory.

A list of directories at top level of this filesystem represents a list of containers on your account and looks like this:

novel@ritual:~/code/fusefs-cloudstorage/test %> ls -1
backups
cloudservers
event_logs
mir_package
novel@ritual:~/code/fusefs-cloudstorage/test %>

Listing each of these directories will (as you might have guessed already) show a list of objects that belong to a corresponding container. For example:

novel@ritual:~/code/fusefs-cloudstorage/test %> ls -1 backups
1345b96a-ae25-11e0-94a9-40401629a6e1.sql.gz
135dbf74-ae25-11e0-94a9-40401629a6e1.sql.gz
27ed5e4e-7faf-11e0-bbcc-40401629a6e1.sql.gz
4fbb5e06-ae14-11e0-aace-40401629a6e1.sql.gz
4fd6274a-ae14-11e0-aace-40401629a6e1.sql.gz
645e7cde-7f9e-11e0-8a43-40401629a6e1.sql.gz
b181ef6c-ae1c-11e0-a6b9-40401629a6e1.sql.gz
b19dcbba-ae1c-11e0-a6b9-40401629a6e1.sql.gz
c626402a-7fa6-11e0-bb13-40401629a6e1.sql.gz
f460c934-9b2f-11e0-89ce-40401629a6e1.sql.gz
novel@ritual:~/code/fusefs-cloudstorage/test %>

You can create containers using mkdir command, copy files in using cp, remove them with rm and so on. You might want to check a shell script test that tests basic features for more examples: https://github.com/novel/fusefs-cloudstorage/blob/master/test.sh.

However, it seems that it's not that easy and quick tasks to implement such a filesystem.

Why is it hard to make such kind of filesystem fast?

My implementation is pretty dumb -- it just does things in a straight-forward way that doesn't work well in this case because filesystem usage patters doesn't match cloud storage usage patterns well.

A couple of examples: cloud storage doesn't support doing writes or reads with offset, so if you want to append a line to a text file, you have to remove the old one and create a new one with the new content. This gets especially slow when we're modifying large files.

Other thing that it looks like vfs design generally assumes it's easy to obtain meta-information about the files, but with cloud storage every single simple operation like checking if file exists, involves issuing an API call, which is slow.

It looks like we can speedup things dramatically if we introduce caching for both meta-data and file contents. But the hard thing with cache is that we don't have an exclusive access to the API. I.e. we can cache things in our filesystem driver and some user at the same time might open up Web Dashboard and remove or upload new files and we will never know about it unless re-request all the data.

At this point I consider this project more like "just for fun" type of thing than something really useful, but probably I will implement caching to make it more or less fast if I decide it will be useful.

Links

• fusefs-cloudstorage on github
• Rackspace Cloud Files
• Amazon S3
• Libcloud

Managing OpenStack using libcloud and lc-tools

OpenStack is getting more and more popular and number of its installation increases day by day. We've got one as well. :-) While there are quite a number of bindings and tools to connecting to its API I prefer use familiar proven tools like libcloud and lc-tools. And since OpenStack supports both EC2-compatible and Rackspace-compatible APIs, there are two ways connecting to it.

EC2 API

To use EC2-compatible API, pick EUCALYPTUS driver like that:



You need to define EC2_ACCESS_KEY and EC2_SECRET_KEY variables to match your setup. Hostname, port and path arguments which are passed to constructor could be extracted from your EC2_URL environment variable.

Note: if you're not using trunk version of libcloud, you will need to adjust imports a little, i.e. strip compute. part from them.

Rackspace API

To use Rackspace-compatible API you will to check out trunk version of libcloud (please refer here to obtain it) which provides OPENSTACK driver, which could be used in a similar fashion:



Again, here you will need to redefine NOVA_API_KEY and NOVA_USERNAME for your setup and grab host and port from NOVA_URL.

Let me remind you once again that you need trunk version of libcloud to make it working. One more thing: OpenStack is not fully compatible with Rackspace API, so not everything will work smooth.

lc-tools

lc-tools is a set of command line tools on top of libcloud to manage your cloud servers. Git version of lc-tools supports working with OpenStack both through EC2 and Rackspace compatible APIs. Here's config samples for both:


[ec2compat]
driver = eucalyptus
access_id = CHANGE_ME
secret_key = CHANGE_ME
extra = { "host": "127.0.0.1", "secure": False, "port": 8773, "path": "/services/Cloud" }

[rscompat]
driver = openstack
access_id = nova_admin
secret_key = CHANGE_ME
extra = {"host": "127.0.0.1", "port": 8774}


And just check if it works by executing lc-node-list -p ec2compat and lc-node-list -p rscompat.

References

* libcloud
* lc-tools

Overview of GoGrid and Rackspace Load Balancing Services

Overview

Load Balancing is a technology to spread workload between several computers. These days one of the most popular application of this technology is load balancing for Web sites, such as balancing HTTP/HTTPS traffic across several Web servers (such as Apache httpd).

A lot of web deployments are moving to the cloud, so load balancers do. I'll give an overview of Load Balancing services provided by Rackspace and GoGrid.

GoGrid

Load Balancer service is a part of GoGrid's Cloud offering and was there from the version 1.0 of the API, so it's about 2008.

GoGrid uses F5 hardware Load Balancers, as stated in the documentation.

Rackspace

Rackspace's Load Balancer service is a separate stand alone service as opposite to GoGrid's one.

The service is relatively young: private beta was announced in November, 2010 and the final release happened in April, 2011, just few days ago at time of writing. Generally, I have started using it from the first private beta and it became quite stable already in the beginning of 2011.

Rackspace offering is based on Zeus software.

API

Like a theatre begins with hanger, services begins with API. Let's overview what API allows us to do with Load Balancers.

GoGrid

As it was mentioned above, Load Balancer API is just a subset of GoGrid API with all its pros and cons. It provides CRUD (create, read, update, delete) operations support for load balancers.

Actually, it's not very close to REST concept as:

* It doesn't use HTTP any method but GET, so e.g. to add a new balancer you make GET request on URL like 'loadbalancer/add' instead of POST on 'loadbalancer' and to delete it you call GET on 'loadbalancer/delete' instead of DELETE, etc
* It doesn't have a concept of element URIs, only collections. So, to get details on a balancer you request 'loadbalancers/get?id=bal_id' instead of 'loadbalancers/bal_id/'
* It uses GET arguments for passing options instead of serialised object in request body

It's not good or bad that it doesn't follow REST closely, REST is not a standard after all, and actually the fact that one doesn't need to bother with HTTP methods other than GET and serialisation might be beneficial for somebody. I will provide some analysis as API usability from programmers point of view later in the post.

So, what is it all about? You create a balancer and you have to specify IPs and ports of nodes to share the load. Important thing to know that you have to use IP addresses assigned to your GoGrid account, otherwise it will fail with not very descriptive internal error.

Besides specifying IP list, you can tweak some load balancer options, such as type and persistence. Currently GoGrid supports two types of load balancing:

* Round Robin (default)
* Least Connect

And for persistence, the options are:

* No persistence (default)
* SSL Sticky
* By source address

Once a balancer has been created, you can change only list of IPs and ports it balances load for. One caveat is that you have to pass a complete list of IPs, not just incrementally add or remove them one by one, so be careful not to run into race condition.

With GoGrid you can have up to 6 load balancers per account and up to 3 load balancers for a data center.

Rackspace

Rackspace Load Balancer API is, obviously, centered around CRUD operations for load balancer objects as well, though it supports far more than that.

It seems to follow REST quite precisely: uses collection and elements URI, correct HTTP semantics and content types.

When creating a balancer, you can specify not only IPs that belong to your Rackspace account, but basically any IP you want. Actually, I think it makes a lot of sense to provide such a flexibility if you're running a hybrid setup, and, say, have part of the nodes in your data center and part of them running on the cloud.

Additionally, you can tweak quite a lot of options. Let's check details for the most important ones, such as balancing type and persistence. As for balancing type, it provides a wide range of options:

* Round Robin
* Least Connect
* Random
* Weighted Least Connect
* Weighted Round Robin

For the last two options node weight (which you can assign to node) is taken into account. Type has to be specified at creation time, so no default value marked.

As for session persistence, only HTTP cookie persistence is supported.

Programming Specifics

There is a specific thing about building something on top of load balancing API. The thing is that unlike dealing with cloud servers, where you work mostly with atomic objects (like server itself), with load balancers you have a collection of nodes you want to balance between. And here two race conditions possible.

The first one is caused by the fact that load balancer needs to be reconfigured after you add or remove a node. And when you try to modify nodes list during the process of re-configuration, you'll get an error. Moreover, GoGrid doesn't have a special state which says that balancer is being re-configured at the moment. Rackspace has such a state, but race condition is still possible, image the following scenario:

* Balancer B is in 're-configuring' state
* Apps A1 and A2 want to add a new node to B
* Apps A1 and A2 see that B is immutable and start polling its status while it's not 'Ready'
* A1 becomes first
* A2 fails

This is not very serious problem, but it make coding a little more complicated.

The second possible race condition is GoGrid implementation specific because of its model of keeping IP list as a whole, without support of adding/removing individual IPs. Imagine a slightly modified version of the previous scenario:

* Balancer B is in 're-configuring' state
* App A1 wants to add IP I1 to B, so the IP list would be [B.ips] + [I1]
* App A2 wants to add IP I2 to B, so the IP list would be [B.ips] + [I2]
* Apps A1 and A2 tries to make a request until it succeeds
* App A1 becomes first, B.ips becomes [B.old_ips] + [I1]
* App A2 still fails because B turns 're-configuring' again
* App A2 finally succeeds with it's request and updates list of Ips to [B.old_ips] + [I2]
* As a result, I1 that should have been added is actually missing

Again, it's not like this is not solvable problem, but it's quite an effort to solve it.

And finally some numbers. I took Rackspace and GoGrid Load Balancer drivers from libcloud trunk which implement the same common interface and gathered these metrics:

* Driver's lines of code (loc)
* Unit-tests' lines of code (test loc)
* Lines of fixtures (i.e. content of responses)



You know that there are lies, damned lies, and statistics, so it's up to you to analyse these numbers. :-)

libcloud load balancers feature status

Background

If you follow libcloud maillist, you probably aware that I've started working on adding load balancers support. If not, please give it a quick view.

Current status

At this point I have implemented almost all features I've planned, namely:

• Defined a common interface
• Implemented drivers for Rackspace and GoGrid
• Covered basic functionality with unit tests
• Implemented load balancer support in lc-tools and run-time tested it

Currently interface is pretty basic and supports only main operations, like: CREATE/READ/DELETE for balancers and ADD/LIST/REMOVE operations for their nodes. Obviously, there are far more operations on load balancers that could be useful such as balancing algorithm and session handing options, but I need some time to play around with these things to understand how to plug it better.

One more thing I'm considering is adding support for blocking operations. It's quite common situation when you find balancer in immutable state: for example, when you add new node to balancer it needs some time to perform initial configuration and during that period you cannot perform operations on it, such as removing or adding more nodes. My current implementation just throws and exception in such case so user can figure out that object is immutable and try after some time.

However, it might be not very convenient for user to wrap any call to catch such situations, so I'm thinking about adding support for blocking mode, where function will block until operation succeeds or timeout reaches; blocking/non-blocking mode could be specified at driver init time. Most likely it's the way things will be done.

Examples

Meanwhile, take a look at this example to understand how things look like at the moment:

example_lb.py

Also, as I've mentioned above, I've added basic support for balancer manipulation to lc-tools. Will cover this topic a bit later when things settle down a little, but here's a quick overview how it feels like:

Giving it a try

To give it a try checkout balancers branch of my libcloud fork on github:


git clone git://github.com/novel/libcloud.git --branch balancers


and then install it like you usually do. The example file mentioned above would be useful to get started. Additionally, it might be useful to look at lb-* scripts in lc-tools lb branch.

Further reading

• Original announce on the maillist
• GoGrid Balancers API (part of their main API) reference
• Rackspace Balancers API reference (PDF)

Mocking libcloud using Mox

If you're using libcloud to interact with cloud providers, you probably would like to cover this code with tests if you haven't done so already.

However, most likely you don't want libcloud to do any real calls to the cloud provider API service as it could create additional noise on your account, add extra load and cost you money (esp. if tests are being run on a regular basis on a Continues Integration server).

One possible solution is to mock these things up using Mox mocking framework. Libcloud uses 2-step initialisation process for a connection class -- first you obtain driver class and then initiate it -- and I seem to always forgot how to mock it right, so decided to write it down.

For example, we have a simple class that returns a list of names of currently available nodes:



And we want to test it without making actual API calls. Let's see how the test code will look for this task. For the sake of simplicity default unittest framework will be used.



Basically, what we are doing here in the test:

• Create a mock for RackspaceNodeDriver class and record list_nodes() call for it; we also make it return two node objects. Actually, we create fake objects using type(), so we don't have to bother with creation of real Node objects which need a way more properties to be set when constructing that we don't care about


• Use mox's StubOutWithMock routine (please check mox docs for details on that) to stub out libcloud.providers module. Here, in get_driver() we pass lambda because get_driver() returns class itself, not an instance of the class, and using lambda here seems to be more clear than recording __call__() for a class


• All the standard ReplayAll/VerifyAll mox routines
  

For the sake of experiment, you can replace actual node_names() method of the Example class with just return ["node1", "node2"]. The assertEqual() will pass, but mox will complain that list_nodes() wasn't called and test will fail.

GoGrid locations support in libcloud

GoGrid added support for locations (US-West-1 and US-East-1) quite some time ago and I finally managed to implement this feature for libcloud's GoGrid driver. Fortunately, libcloud API already takes into account that cloud providers could have more than one location, so the most important part was there already.

Examples

libcloud

lc-tools

Some of the lc-tools support '-l' flag to specify location id. Here's how a typical sequence would look like:



Few notes here: you probably noticed that we filter images by locations but don't do it for sizes. Strictly speaking, sizes might differ for various providers and this is also accounted by libcloud API; but for GoGrid sizes are same for both locations, so I skipped this step.

And the second thing: all these features are available in svn version of libcloud and git version of lc-tools.

Also the thing I should have been mentioned for some time already: GoGrid provided me with an account and it is quite helpful for various testing.

Announcing lc-tools: command line tools on top of libcloud

Looks like it's time to announce lc-tools! What is it? Basically, it's just a few command line tools to manage your clouds based on libcloud. What advantages does it give? Well, it supports as many cloud providers as libcloud supports (ec2, rackspace, gogrid and many others) and you can manage them all with a single tool.

Look and feel is pretty similar to gg's one as it seems to be quite comfortable and I used gg more than year on a regular basis and haven't had any issues with it.

Currently lc-tools covers almost all functionality provided by libcloud, but I've only tested it with GoGrid.

Some examples so you can check look and feel of the application:

Listing available node images

$ lc-image-list|grep -i centos
image CentOS 5.2 (32-bit) w/ RightScale (id = 62)
image CentOS 5.2 (64-bit) w/ RightScale (id = 63)
image CentOS 5.3 (32-bit) w/ None (id = 1531)
image CentOS 5.3 (64-bit) w/ None (id = 1532)


... and sizes

$ lc-sizes-list
size 512MB (id=512MB, ram=512, disk=30 bandwidth=None)
size 4GB (id=4GB, ram=4096, disk=240 bandwidth=None)
size 2GB (id=2GB, ram=2048, disk=120 bandwidth=None)
size 8GB (id=8GB, ram=8192, disk=480 bandwidth=None)
size 1GB (id=1GB, ram=1024, disk=60 bandwidth=None)

Creating a new node

lc-node-add -i 62 -s 1GB -n mynewnode

Pay attention to values 62 and 1GB -- they are ids of image and size respectively (from previous examples).

Listing nodes

$ lc-node-list
100xxx mynode1 173.204.xx.yy Running
100xxx mynode2 173.204.xx.zz Running

Destroying nodes

$ lc-node-do -i node_id destroy

Project's github page contains a README file with documentation enough to get started, but if you have any problems or suggestions feel free to contact me via github, blog or email.

libcloud's GoGrid driver

We have to work quite often with GoGrid quite frequently and about a year ago I wrote gg-tools -- a set of CLI tools and a library in Python. We've used it for a while but later decided to switch to libcloud to have a better portability if we decide to move to other cloud provider.

However I found that some things in GoGrid driver could be improved. Specifically, there is quite annoying thing that gogrid doesn't assign an id to a node immediately after creation, but libcloud always expects to get an id for the image. As per discussion in libcloud maillist, create_node() was patched to block until the node gets ip assigned. Also, ex_create_node_nowait() method was added which simply adds node right away and returns id == None.

While the patches are being reviewed, I created a fork on github: http://github.com/novel/libcloud
with these changes applied. Feel free to check it out, GoGrid driver should be in a working state here as we use it on a daily basis and even if some error raises up it will be fixed quickly.