Roma's Blog

lc-tools 0.7.1.0 released!

2012-01-03T04:17:00.000-08:00

I'm happy to announce new lc-tools release: 0.7.1.0!

It doesn't have new features, but was updated to work with libcloud 0.7.x (earlier libcloud versions are no longer supported!). All the kudos go to Pavel Vozdvizhenskiy for noticing problems and testing.

You can install the new version using either easy_install:

easy_install lctools

or by downloading and installing it by hands:

lctools-0.7.1.0.tar.gz

SHA256 (lctools-0.7.1.0.tar.gz) = 7f65c29cda06d256c02f6544b4957ca984f34f1cfed5b24558c3b4338f87dd9c

Bug Prediction

2011-12-21T23:51:00.001-08:00

Recently I've read a very interesting post about Bug Prediction on Google Engineering Tools blog and decided to inspect FreeBSD source tree in the same manner.

The question is how to determine if the given commit is a bug fix. I decided to start with a security fixes first and filtered commit messages by "Security:" tag in them. The result is the following:

Filename	Score
crypto/openssl/ssl/s3_srvr.c	1.12807360699
crypto/openssl/ssl/s3_clnt.c	1.12546730725
crypto/openssl/openssl.spec	0.914805209201
crypto/openssl/NEWS	0.914805209201
crypto/openssl/CHANGES	0.914805209201
crypto/openssl/README	0.914805209201
crypto/openssl/crypto/opensslv.h	0.914805209201
crypto/openssl/Makefile	0.914805209201
crypto/openssl/FAQ	0.914805209201
crypto/openssl/apps/speed.c	0.707204148366
crypto/openssl/ssl/t1_lib.c	0.696806056187
crypto/openssl/ssl/s3_lib.c	0.663083892748
crypto/openssl/crypto/md32_common.h	0.578509821664
crypto/openssl/crypto/ocsp/ocsp_prn.c	0.578509821664
crypto/openssl/engines/e_chil.c	0.578509821664
crypto/openssl/ssl/d1_pkt.c	0.51304099663
crypto/openssl/crypto/pqueue/pqueue.c	0.51304099663
crypto/openssl/ssl/d1_both.c	0.51304099663
contrib/ntp/ntpd/ntp_crypto.c	0.505424173077
crypto/openssl/crypto/asn1/tasn_dec.c	0.482876843456
sys/compat/linux/linux_socket.c	0.477720395098
usr.bin/compress/zopen.c	0.462407652214
usr.bin/gzip/zuncompress.c	0.462407652214
sys/kern/uipc_usrreq.c	0.462407652214
crypto/openssl/ssl/ssltest.c	0.460574251046
crypto/openssl/ssl/s2_srvr.c	0.460574251046
crypto/openssl/apps/x509.c	0.460574251046
crypto/openssl/ssl/s3_pkt.c	0.456096100304
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c	0.44243544619
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c	0.44243544619

There results appeared not very interesting though. This kind of matches security advisories, but given that there are not very much of them (luckily!), it's not very representative.

I also tried to run the script with "[\t ]+[Bb]ug" regular expression instead of "Security:", and got the following result:

Filename	Score
sys/netinet/sctp_output.c	7.66251938944
sys/netinet/sctp_pcb.c	7.2065885975
sys/netinet/sctputil.c	6.80461378901
sys/dev/bge/if_bge.c	6.50257671046
sys/netinet/sctp_input.c	5.62524237411
sys/netinet/sctp_indata.c	5.25336437345
sys/netinet/sctp_usrreq.c	5.11155404498
sys/netinet/sctp_timer.c	3.67543247047
sys/netinet/sctp_structs.h	2.99517512907
sys/conf/files	2.89041302648
sys/netinet/sctp_asconf.c	2.79978400331
sys/netinet/sctp_constants.h	2.79452724525
sys/dev/bge/if_bgereg.h	2.62852430236
sys/dev/msk/if_msk.c	2.31771522145
sys/netinet/sctputil.h	2.19989952769
contrib/bind9/CHANGES	2.15204576684
contrib/bind9/version	2.15204576684
sys/kern/kern_fork.c	2.12687393752
sys/netinet/in.c	2.09501338006
sys/netinet/sctp_sysctl.c	2.06560623664
sys/netinet6/sctp6_usrreq.c	1.92839801197
sys/netinet/sctp_var.h	1.91990472478
sys/conf/NOTES	1.79166008565
sys/dev/mxge/if_mxge.c	1.74628407608
contrib/bind9/bin/named/query.c	1.74336559317
contrib/bind9/lib/dns/rbtdb.c	1.74336559317
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c	1.7287195258
sys/i386/xen/pmap.c	1.68724079063
sys/powerpc/aim/mmu_oea64.c	1.68644933665
contrib/bind9/lib/dns/validator.c	1.6676964412

This is somewhat more interesting result. It shows that most 'dangerous' sits in SCTP implementation, NIC drivers (bge, msk, mxge) and also some bits in zfs and bind. I wonder how accurate is that?

The script is available here. It's designed to use git repository. I used freebsd git mirror on github: https://github.com/freebsd/freebsd-head.

Apple keyboard on FreeBSD

2011-10-23T19:39:00.000-07:00

Recently I bought an iBook G4 just for fun and of course installed FreeBSD on it. Installation goes pretty smooth starting with 9.0: I've started with 9.0-BETA3 and later updated to 10-CURRENT. I used it as a toy mostly but on Thursday my main laptop (which also runs FreeBSD, but amd64/8-STABLE) died and I had to use iBook as my main system and some problems became noticable.

There are two annoying problems with Apple keyboards:

1. Function keys (F1, F2, ...) work as they supposed to work only when pressing them with FN key. I rely on virtual desktops a lot and I use Ctrl-Fx to switch between them, so you imagine how annoying is that to use Ctrl-Fn-Fx to switch a desktop.

2. Apple keyboard misses 'Insert' key completely, so you cannot use Ctrl-Insert for pasting, so it's extremely annoying also.

Fortunately, both problems could be easily solved.

Several days ago Nathan Whitehorn ⟨nwhitehorn@) committed a support for dev.akbd.%d.fn_keys_function_as_primary which allows you to configure F-keys behavior. In order to make the keys look as F1,F2,... without pressing FN, execute:

sysctl dev.akbd.0.fn_keys_function_as_primary=1

The Insert key problem could be solved on X level: I've used xmodmap (x11/xmodmap) to remap F11 to Insert, which is a good solution for me because I don't use F11 key and it's located quite close to a place where you'd expect to have 'Insert' key on a typical keyboard.

So I've created ~/.Xmodmap file:

keycode 95 = Insert

And added execution of "xmodmap ~/.Xmodmap" on X session start.

I'm not sure if F11 has keycode 95 on every Apple keyboard, it's easy to figure out keycode using xev (x11/xev).

the bike

2011-07-28T11:11:00.000-07:00

frame: blkmrkt three57
fork: marzocchi dj2 '09 100mm
handlebar: blkmrkt badaboom
stem: ns quark pro '11
seat: blkMrkt brass knuckles (--> leaf cycles symboll pivotal soon-ish)
seat post: blkmrkt riot stick (--> code pivotal 135mm)
sprocket: federal street 25t
pedals: ns aerial '11
cranks: crmo 3pc, 48 splined
chain: kmc
front hub: ns rotary 20 '11
rear hub: ns rotary single '11 10t
tires: kenda 26x2.10"
spokes: mach

photos are so-so and bike config is not perfect I guess, but it's good starting point.

fusefs-cloudstorage: accessing cloud storage using libcloud and fusefs

2011-07-12T06:53:00.000-07:00

Few weeks after cloud storage support appeared in libcloud I started playing with it by creating a fusefs filesystem based on it. Finally the first version is almost ready, so it's time to write a few words about it.

So, for example, you have a cloud files account and want to mount it as it were a local filesystem. You do it this way:


novel@ritual:~/code/fusefs-cloudstorage/test %> cloudstorage.py -o driver=CLOUDFILES_US -o access_id=foo -o secret=acbd18db4cc2f85cedef654fccc4a4d8 ./test

where foo is your access id and this hash is the secret key. Filesystem will be mounted to test sub-directory of the current directory.

A list of directories at top level of this filesystem represents a list of containers on your account and looks like this:


novel@ritual:~/code/fusefs-cloudstorage/test %> ls -1
backups
cloudservers
event_logs
mir_package
novel@ritual:~/code/fusefs-cloudstorage/test %>

Listing each of these directories will (as you might have guessed already) show a list of objects that belong to a corresponding container. For example:


novel@ritual:~/code/fusefs-cloudstorage/test %> ls -1 backups
1345b96a-ae25-11e0-94a9-40401629a6e1.sql.gz
135dbf74-ae25-11e0-94a9-40401629a6e1.sql.gz
27ed5e4e-7faf-11e0-bbcc-40401629a6e1.sql.gz
4fbb5e06-ae14-11e0-aace-40401629a6e1.sql.gz
4fd6274a-ae14-11e0-aace-40401629a6e1.sql.gz
645e7cde-7f9e-11e0-8a43-40401629a6e1.sql.gz
b181ef6c-ae1c-11e0-a6b9-40401629a6e1.sql.gz
b19dcbba-ae1c-11e0-a6b9-40401629a6e1.sql.gz
c626402a-7fa6-11e0-bb13-40401629a6e1.sql.gz
f460c934-9b2f-11e0-89ce-40401629a6e1.sql.gz
novel@ritual:~/code/fusefs-cloudstorage/test %>

You can create containers using mkdir command, copy files in using cp, remove them with rm and so on. You might want to check a shell script test that tests basic features for more examples: https://github.com/novel/fusefs-cloudstorage/blob/master/test.sh. However, it seems that it's not that easy and quick tasks to implement such a filesystem. Why is it hard to make such kind of filesystem fast? My implementation is pretty dumb -- it just does things in a straight-forward way that doesn't work well in this case because filesystem usage patters doesn't match cloud storage usage patterns well. A couple of examples: cloud storage doesn't support doing writes or reads with offset, so if you want to append a line to a text file, you have to remove the old one and create a new one with the new content. This gets especially slow when we're modifying large files. Other thing that it looks like vfs design generally assumes it's easy to obtain meta-information about the files, but with cloud storage every single simple operation like checking if file exists, involves issuing an API call, which is slow. It looks like we can speedup things dramatically if we introduce caching for both meta-data and file contents. But the hard thing with cache is that we don't have an exclusive access to the API. I.e. we can cache things in our filesystem driver and some user at the same time might open up Web Dashboard and remove or upload new files and we will never know about it unless re-request all the data. At this point I consider this project more like "just for fun" type of thing than something really useful, but probably I will implement caching to make it more or less fast if I decide it will be useful. Links fusefs-cloudstorage on github Rackspace Cloud Files Amazon S3 Libcloud

lc-tools 0.5.0.0 released! 2011-05-30T05:01:00.000-07:00 A lot of time passed since the latest lc-tools release, and now it's almost summer, so it's time for updates! The first news is that libcloud is now an Apache Top Level Project (TLP) and new version 0.5.0 has been released with some new features, bug fixes and new support for working with storage and load balancers. And it's a perfect time for new lc-tools release! There are not so many new features in this lc-tools release, most of the changes were to chase internal libcloud changes. Additionally, I've changed versioning scheme a little and now it looks like $libcloud_version$lctools_version, where $libcloud_version is the version of libcloud current release was targeted and tested with. The only new major change is the addition of lb-* scripts to manage load balancers. You can get more information on them from Documnetation. You can install the new version using either easy_install: easy_install lctools or by downloading and installing it by hands: lctools-0.5.0.0.tar.gz SHA256 (lctools-0.5.0.0.tar.gz) = f09629664a5209be687c56cfe07e123939e28b18070b719436307a74b823f10b Happy hacking on the cloud stuff! Dealing with pkg-config detection of FreeBSD's base OpenSSL 2011-05-18T03:39:00.000-07:00 Recently I spotted a software that uses pkg-config to find OpenSSL on the system. This way doesn't work on FreeBSD: it does have OpenSSL in base system, but it doesn't supply '*.pc' file, unfortunately. So, when configure script tries to execute pkg-config openssl it fails, obviously. There were a number of solutions suggested for this problem on #bsdports like replacing 'pkg-config openssl' with 'pkg-config pkg-config', depending on openssl from ports etc, but the cleanest solution is to define libssl_LIBS and libssl_CFLAGS environment variables, as was suggested by bapt@. In port's Makefile it would look something like this: USE_OPENSSL=yes CONFIGURE_ENV+=libssl_CFLAGS="-I${OPENSSLINC}" libssl_LIBS="-L${OPENSSLLIB} -lssl" Yes, configure will not call pkg-config if we define these 'libssl_*' variables. While it looks quite simple, this issue took surprisingly a lot time to figure everything out. :( Managing OpenStack using libcloud and lc-tools 2011-05-06T02:32:00.000-07:00 OpenStack is getting more and more popular and number of its installation increases day by day. We've got one as well. :-) While there are quite a number of bindings and tools to connecting to its API I prefer use familiar proven tools like libcloud and lc-tools. And since OpenStack supports both EC2-compatible and Rackspace-compatible APIs, there are two ways connecting to it. EC2 API To use EC2-compatible API, pick EUCALYPTUS driver like that: You need to define EC2_ACCESS_KEY and EC2_SECRET_KEY variables to match your setup. Hostname, port and path arguments which are passed to constructor could be extracted from your EC2_URL environment variable. Note: if you're not using trunk version of libcloud, you will need to adjust imports a little, i.e. strip compute. part from them. Rackspace API To use Rackspace-compatible API you will to check out trunk version of libcloud (please refer here to obtain it) which provides OPENSTACK driver, which could be used in a similar fashion: Again, here you will need to redefine NOVA_API_KEY and NOVA_USERNAME for your setup and grab host and port from NOVA_URL. Let me remind you once again that you need trunk version of libcloud to make it working. One more thing: OpenStack is not fully compatible with Rackspace API, so not everything will work smooth. lc-tools lc-tools is a set of command line tools on top of libcloud to manage your cloud servers. Git version of lc-tools supports working with OpenStack both through EC2 and Rackspace compatible APIs. Here's config samples for both: [ec2compat] driver = eucalyptus access_id = CHANGE_ME secret_key = CHANGE_ME extra = { "host": "127.0.0.1", "secure": False, "port": 8773, "path": "/services/Cloud" } [rscompat] driver = openstack access_id = nova_admin secret_key = CHANGE_ME extra = {"host": "127.0.0.1", "port": 8774} And just check if it works by executing lc-node-list -p ec2compat and lc-node-list -p rscompat. References * libcloud * lc-tools Trying DotCloud for Django/Python deployment 2011-04-28T15:12:00.000-07:00 About dotcloud The first time I heard about DotCloud was about two months ago, in the beginning of 2011. I was very tempted to try it as soon as possible but it seems there were a very lengthy queue of people like me so I had to wait for quite some time. But finally I've got my account activated few days ago. I was so happy about that so initially I didn't even know what application to try to develop on it, as I wanted to cover as much services as possible that DotCloud offers. After all, I decided to go with a frontend Django application and several workers communicating with each other using messaging as it seems to be a good basis for a lot of things I have to do these days. Originally I wanted to use RabbitMQ server, but unfortunately it's not working for some reason, which made me take a look at celery with redis backend. I have never used neither celery nor redis so it was even more interesting to learn these things as well. So I came up to a scheme like these. Sample Application * Django Web Application provides a user interface for issuing tasks and obtaining their results * MySQL is used to keep history of issued tasks * Celery handles all task-related operations. It passes tasks to Workers for execution using Redis as transport layer and results are stored also in Redis As for the task, I choose a really simple one: just sending 60 ICMP ECHO_REQUEST packets to determine if a host specified by user up or down. Use case is simple: user opens up a page, specifies a host to ping and then waits for result and it appears to him after some time. Workers Implementation I have started with workers implementation. I followed Tutorial on DotCloud web site to create my first Celery worker and it worked alright, but I had step from it several times for the following reasons: RabbitMQ service problems When I created a rabbitmq service it for some reason had empty ports list, like that: $ dotcloud info bublick.rabbitmq cluster: wolverine config: password: password rabbitmq_management: true user: root created_at: 1303573502.963711 name: foo.rabbitmq namespace: foo ports: [] state: running type: rabbitmq $ I thought it was some glitch, especially considering that this happened not long ago after EC2 outage (and DotCloud runs on top of EC2). So I tried to drop this service and deploy a new one. Alas, everything was same. I joined an IRC channel -- #dotcloud on freenode -- to ask what's wrong with that and it appeared that it's a known problem and is going to be fixed soon. It's not a big deal though, because Celery supports a lot of brokers in addition to RabbitMQ, so I decided to go with Redis and use it to store results as well. So I just changed celeryconfig.py to point to Redis instead of RabbmitMQ: Then I implemented actual task which pings a given host: As you see, it's pretty trivial. However, I've spotted a problem when I tried to push the code, basically it said that 'celeryd' could not be started. So I've logged in to VM running celery using the command 'dotcloud ssh foo.celery' (where foo.celery is a name of deployment) and tried executing 'celeryd' by hand, and it failed with 'Permission denied' error. I asked on IRC channel again and it was a known problem also, and it was quickly fixed for my deployment. After that my worker was fully functional, you can check sources and its celery configuration here: https://github.com/novel/dotcloud-sample-app/tree/master/worker. Then I moved to web application. Web Application As it was mentioned, I wanted to use Django framework for web application. Luckily, there's a tutorial on django deployment on dotcloud as well. I followed this tutorial step by step and sample django application deployed smoothly. Then I needed to add celery integration. As I deployed code separately I didn't want to import actual task code but call by name. Also, I wanted task status to be fetched only when user actually requests it, so I had to find tasks by id. The first steps is to configure django application to use celery. It's done by adding django-celery and redis dependencies to requirements.txt and defining celery stuff in settings.py: . And finally set of simple views to schedule tasks and report their status: You can also see a view which provides a list of active workers. Now, if you deploy your web application and the worker thing should work fine. Additionally, you can add extra workers seamlessly. For example, if you named your worker 'foo.celery', you can add more workers exactly the same way, just use e.g. foo.celery2 etc for dotcloud deploy and dotcloud push commands. Things to keep in mind * PYTHONPATH contains only top level project directory by default, so one either should add django application to PYTHONPATH or use full path for modules, e.g. pay attention I have my 'ping' application listed in settings.py as 'frontend.ping' for example, otherwise it won't work * You don't have to add database drivers like mysqldb-python to requirements.txt since they're available by default DotCloud first impressions * Documentation is good, fairly complete, I haven't seen an outdated or misleading information * DotCloud guys are very helpful, all the questions get addressed very quickly on the IRC channel * Great jobs done with the services. For example, I have never ever used redis and I have no idea how to set it up and configure. All I had to do is to run dotcloud deploy -t redis foo.redis and it's there, very good. The same goes for wsgi deployment -- it's a shame, but I've never did such type of deployment myself, I just use 'manage.py runserver' for local testing... it's good to have a deployment engineer on the team! :) So, it also went seamlessly, I don't even have to know where logs are located, I just do 'dotcloud logs foo.www'. * Some services are unstable or not working, like RabbitMQ (which is unusable) and Celery (which needs some manual actions from support team to work fine). That doesn't seem to be a serious problem to me though, considering that DotCloud is at beta stage currently, and more over they must have been very busy last few days recovering from EC2 outage. Things I'm curious about * Currently, there's no way to install system-level package (like, say, ImageMagick. It way mentioned on IRC that popular packages like that will be eventually included into base images based on users requests. I'm curious how it will be done without letting images spreading out. * I need to research how scaling is implemented and choosing scaling strategy looks like * I wonder how upgrade strategy looks like. FAQ says that upgrades are "prudent and thoroughly tested" (c) but everyone knows that things can always break in some totally unexpected ways and what are the procedures for rolling back, sticking to specific version of the components and so on. * I wasn't able to find if there are any monitoring facilities provided. Also it's not clear what's going to happen if some critical component or VM goes down. Conclusion It was fun to experiment with DotCloud and I'm going to spend some more time on it. It differs from services that I worked with before. I've got an impression that its abstraction level is somewhere higher that IaaS but lower that a typical PaaS. On the one hand, you don't have to worry about things you usually have to worry on IaaS, like installation and configuration of services like MySQL and so on. On the other hand, you still have to do a fair amount of deployment-related things, like creating databases, configuring users, manually running syncdb commands and other things you typically don't do on PaaS. Also, you can view almost everything you might need as you have ssh access to every VM, but you cannot change almost anything since it's not root access. So I'm definitely interested to see how DotCloud evolves and hope to spend some more time using and learning it. References Sample Application Sample application I've used as a sample could be found on github: https://github.com/novel/dotcloud-sample-app Clone it: git clone git://github.com/novel/dotcloud-sample-app.git All you need to run it now is to put credentials for you mysql and celery services and deploy it on DotCloud. Have fun! Further Reading * DotCloud FAQ * DotCloud Django Tutorial * DotCloud Celery Tutorial * Celery Documentation * Django-celery Documentation * Sample App A tiny client for lingvo.yandex.ru 2011-04-25T15:14:00.000-07:00 I've been using lingvo.yandex.ru translation service for several years now for Russian<->English translation, and was quite happy about it. In my opinion, it's one of the best online translation services and provides better quality translations (esp. for phrases) than e.g. Google Translate. The only thing I've been missing is an API so I, as a console geek, could have a cli tool to craft translations. Not so long they've added complete-as-you-type feature. So... firebug, 10 minutes and script is ready: I wouldn't typically post a blog entry about 50-lines Python script, but I've figured out that it became my one of most used CLI tool (not counting basic ones, obviously, like cd or ls). To try it just do: # easy_install yaslov and start using it! Like: $ yaslov gnome. Thanks to Yandex for a wonderful services and especially for the exposed suggestion script. PS If you liked that, you might also want to check my similar script for urbandictionary: py-urbandict. Overview of GoGrid and Rackspace Load Balancing Services 2011-04-21T10:02:00.000-07:00 Overview Load Balancing is a technology to spread workload between several computers. These days one of the most popular application of this technology is load balancing for Web sites, such as balancing HTTP/HTTPS traffic across several Web servers (such as Apache httpd). A lot of web deployments are moving to the cloud, so load balancers do. I'll give an overview of Load Balancing services provided by Rackspace and GoGrid. GoGrid Load Balancer service is a part of GoGrid's Cloud offering and was there from the version 1.0 of the API, so it's about 2008. GoGrid uses F5 hardware Load Balancers, as stated in the documentation. Rackspace Rackspace's Load Balancer service is a separate stand alone service as opposite to GoGrid's one. The service is relatively young: private beta was announced in November, 2010 and the final release happened in April, 2011, just few days ago at time of writing. Generally, I have started using it from the first private beta and it became quite stable already in the beginning of 2011. Rackspace offering is based on Zeus software. API Like a theatre begins with hanger, services begins with API. Let's overview what API allows us to do with Load Balancers. GoGrid As it was mentioned above, Load Balancer API is just a subset of GoGrid API with all its pros and cons. It provides CRUD (create, read, update, delete) operations support for load balancers. Actually, it's not very close to REST concept as: * It doesn't use HTTP any method but GET, so e.g. to add a new balancer you make GET request on URL like 'loadbalancer/add' instead of POST on 'loadbalancer' and to delete it you call GET on 'loadbalancer/delete' instead of DELETE, etc * It doesn't have a concept of element URIs, only collections. So, to get details on a balancer you request 'loadbalancers/get?id=bal_id' instead of 'loadbalancers/bal_id/' * It uses GET arguments for passing options instead of serialised object in request body It's not good or bad that it doesn't follow REST closely, REST is not a standard after all, and actually the fact that one doesn't need to bother with HTTP methods other than GET and serialisation might be beneficial for somebody. I will provide some analysis as API usability from programmers point of view later in the post. So, what is it all about? You create a balancer and you have to specify IPs and ports of nodes to share the load. Important thing to know that you have to use IP addresses assigned to your GoGrid account, otherwise it will fail with not very descriptive internal error. Besides specifying IP list, you can tweak some load balancer options, such as type and persistence. Currently GoGrid supports two types of load balancing: * Round Robin (default) * Least Connect And for persistence, the options are: * No persistence (default) * SSL Sticky * By source address Once a balancer has been created, you can change only list of IPs and ports it balances load for. One caveat is that you have to pass a complete list of IPs, not just incrementally add or remove them one by one, so be careful not to run into race condition. With GoGrid you can have up to 6 load balancers per account and up to 3 load balancers for a data center. Rackspace Rackspace Load Balancer API is, obviously, centered around CRUD operations for load balancer objects as well, though it supports far more than that. It seems to follow REST quite precisely: uses collection and elements URI, correct HTTP semantics and content types. When creating a balancer, you can specify not only IPs that belong to your Rackspace account, but basically any IP you want. Actually, I think it makes a lot of sense to provide such a flexibility if you're running a hybrid setup, and, say, have part of the nodes in your data center and part of them running on the cloud. Additionally, you can tweak quite a lot of options. Let's check details for the most important ones, such as balancing type and persistence. As for balancing type, it provides a wide range of options: * Round Robin * Least Connect * Random * Weighted Least Connect * Weighted Round Robin For the last two options node weight (which you can assign to node) is taken into account. Type has to be specified at creation time, so no default value marked. As for session persistence, only HTTP cookie persistence is supported. Programming Specifics There is a specific thing about building something on top of load balancing API. The thing is that unlike dealing with cloud servers, where you work mostly with atomic objects (like server itself), with load balancers you have a collection of nodes you want to balance between. And here two race conditions possible. The first one is caused by the fact that load balancer needs to be reconfigured after you add or remove a node. And when you try to modify nodes list during the process of re-configuration, you'll get an error. Moreover, GoGrid doesn't have a special state which says that balancer is being re-configured at the moment. Rackspace has such a state, but race condition is still possible, image the following scenario: * Balancer B is in 're-configuring' state * Apps A1 and A2 want to add a new node to B * Apps A1 and A2 see that B is immutable and start polling its status while it's not 'Ready' * A1 becomes first * A2 fails This is not very serious problem, but it make coding a little more complicated. The second possible race condition is GoGrid implementation specific because of its model of keeping IP list as a whole, without support of adding/removing individual IPs. Imagine a slightly modified version of the previous scenario: * Balancer B is in 're-configuring' state * App A1 wants to add IP I1 to B, so the IP list would be [B.ips] + [I1] * App A2 wants to add IP I2 to B, so the IP list would be [B.ips] + [I2] * Apps A1 and A2 tries to make a request until it succeeds * App A1 becomes first, B.ips becomes [B.old_ips] + [I1] * App A2 still fails because B turns 're-configuring' again * App A2 finally succeeds with it's request and updates list of Ips to [B.old_ips] + [I2] * As a result, I1 that should have been added is actually missing Again, it's not like this is not solvable problem, but it's quite an effort to solve it. And finally some numbers. I took Rackspace and GoGrid Load Balancer drivers from libcloud trunk which implement the same common interface and gathered these metrics: * Driver's lines of code (loc) * Unit-tests' lines of code (test loc) * Lines of fixtures (i.e. content of responses) You know that there are lies, damned lies, and statistics, so it's up to you to analyse these numbers. :-) libcloud load balancers feature status 2011-04-12T16:43:00.000-07:00 Background If you follow libcloud maillist, you probably aware that I've started working on adding load balancers support. If not, please give it a quick view. Current status At this point I have implemented almost all features I've planned, namely: Defined a common interface Implemented drivers for Rackspace and GoGrid Covered basic functionality with unit tests Implemented load balancer support in lc-tools and run-time tested it Currently interface is pretty basic and supports only main operations, like: CREATE/READ/DELETE for balancers and ADD/LIST/REMOVE operations for their nodes. Obviously, there are far more operations on load balancers that could be useful such as balancing algorithm and session handing options, but I need some time to play around with these things to understand how to plug it better. One more thing I'm considering is adding support for blocking operations. It's quite common situation when you find balancer in immutable state: for example, when you add new node to balancer it needs some time to perform initial configuration and during that period you cannot perform operations on it, such as removing or adding more nodes. My current implementation just throws and exception in such case so user can figure out that object is immutable and try after some time. However, it might be not very convenient for user to wrap any call to catch such situations, so I'm thinking about adding support for blocking mode, where function will block until operation succeeds or timeout reaches; blocking/non-blocking mode could be specified at driver init time. Most likely it's the way things will be done. Examples Meanwhile, take a look at this example to understand how things look like at the moment: example_lb.pyAlso, as I've mentioned above, I've added basic support for balancer manipulation to lc-tools. Will cover this topic a bit later when things settle down a little, but here's a quick overview how it feels like: Giving it a try To give it a try checkout balancers branch of my libcloud fork on github: git clone git://github.com/novel/libcloud.git --branch balancers and then install it like you usually do. The example file mentioned above would be useful to get started. Additionally, it might be useful to look at lb-* scripts in lc-tools lb branch. Further reading Original announce on the maillist GoGrid Balancers API (part of their main API) reference Rackspace Balancers API reference (PDF) Poor recruiting 2011-03-24T12:41:00.000-07:00 I receive invitations for job interviews for various positions time to time, as probably a lot of software engineers who have public profiles on linkedin (and similar) do, usually it's nothing interesting, but this time it was a new experience for me. This time it was Epam – an outsourcing company, almost nothing special. So they had some Python position at New-York with H1B visa. As you probably know from reading my blog that I do some Python programming and feel quite good about it and I'm quite interested in migration also, that's why I said I was generally interested. The first (and the most) surprising thing is that they didn't provide much info on the technical side of the vacancy. Yes, Python, yes, some company that does advertising on the internet, that's all. I thought, ok, I'll probably figure out more details during an interview. I had an non-tech interview with their HR and I wasn't able to find out anything new. On the next day after an interview he goes like 'hey, please skype to "some_skype_id" here, he'll provide more info, I made him aware that you'll contact him." I wondered why I should bother contacting anybody as I basically don't know anything about the vacancy and therefore not interested that much. So I asked if that guy could contact me himself on what HR replied something like "no, please contact him yourself" and was like "I've arranged this possibility so you could get more technical details." and it sounded like he made a huge favor for me already so I could talk to this person. At this point I decided I don't want to work at this company already. So I'm trying to understand point of view of such recruiting managers. They work in a company that doesn't have a reputation as a good place for engineers (unlike Google and similar). Moreover, they have an office in my hometown and even there it's not the best place to work at. Then, they try to recruit a person who has a job already and doesn't state anywhere that he's looking for job. And finally they offer a candidate (which is not really a candidate strictly speaking because at that point it's not clear if the job worth it or not) to drive himself through information gathering for the vacancy on his own. So I'm wondering who are they expect to find. A programmer who's only interested in obtaining H1B? A programmer who's not very interested what he's going to do (even if it is legacy Python 2.2 code support or maybe shell-like scripting in Python)? I don't get it. PS Don't think I'm such an asshole. Once I traveled to a different city on my own expense for an interview because I saw a potential in the job, more on that later I think. Treating security loosely 2011-03-11T10:11:00.000-08:00 I don't want to sound like a grumpy old man, but anyway sometimes I get really surprised how people treat security these days. I was going to try gitflow (which seems to be quite interesting thing to adopt by the way) today and was reading documentation on the installation process. Believe it or not, the suggested installation way on linux system is: $ wget --no-check-certificate -q -O - https://github.com/nvie/gitflow/raw/develop/contrib/gitflow-installer.sh | sudo sh Yes, you're right, it's like that: downloading a random script from the internet and executing it from 'root' without extra checks. Certainly, it's up to you if you trust the source and sure that it doesn't have any malicios code (generally, it's better not to trust). But anyway, such installation schema should be improved: SSL verification shouldn't be disabled, so you're sure that you're actually downloading your file from github, not from some source identifying itself as github Control checksums should be provided or the file should be signed with gpg to make sure you're downloading exactly the same thing as author uploaded Without it executing such a command seems to be plain dangerous. Some Python3 fun 2011-02-27T03:08:00.000-08:00 I wasn't feeling very productive today so decided to take a look at Python 3.2 and specifically understand how hard would it be to create a development environment not affecting Python 2.6 related stuff and also how much time would it take to port a simple 2.x script to 3.x. Luckily, Python 3.x is available in FreeBSD ports tree and, moreover, it doesn't conflict with python2.x stuff, so all I had to do is to cd /usr/ports/lang/python32 && sudo make install clean (actually I've spotted a minor problem with the port, but it doesn't affect anything, for me at least). Viola, it's already there. Then I decided to pick some simple script and see how much effort it will take to make it run with 3.2. A script that I created some time ago to parse urbandictionary's output seemed like a good fit for this task. I've ran 2to3 over my old script and it generated a patch which I successfully applied. Apart from that I had to fix bytes/string problem (not really hard and there are lots of information on it on the net) and my script just worked! So it looks like things are not very hard with 2-to-3 migration. Obviously, things are harder for the larger projects, but with larger projects everything is harder anyway. The only thing I haven't figured out yet is how to have different versions of easy_install for both 2.6 and 3.2 on FreeBSD, though I haven't spent much time digging into it. PS I went further with the urbandictionary script, created a package for it, uploaded on PyPI, improved formatting and made it easier to use from python. Here's a screenshot: Check it out on github if you're interested. ChangeLog / NEWS importance 2011-02-24T10:15:00.000-08:00 I was scanning portscout recently and noticed new version of xplanet released. As usual, I've grabbed new version tarball and started compilation, meanwhile checking ChangeLog and NEWS for list of changes in the new version. I was a bit disappointed when I wasn't able to find anything I was looking for. I thought I probably could find some announcement on the forums or project's website, but had no luck with this as well. Now it's not even funny. Of course, I can do diff -ruN against the old version, but it's not quite easy to understand what's been changed without knowing context and it won't work for all users. So, it's quite strange situation for a typical user as he has no clue why he should upgrade. While providing a list of changes for the release is a very simple thing, it's very important from user perspective and unfortunately keeps getting forgotten sometimes. PS I almost forgot to mention I've tried to check project's svn, but sourceforge is slow as hell or even slower, so I wasn't able neither checkout sources nor view them from the website. Argument-based methods synchronization in Python 2011-02-12T06:07:00.000-08:00 I've been implementing yet another client for a REST-like service and I've spotted a problem I typically haven't faced while working with REST services. I needed to modify some resource -- meaning issuing a PUT request on resource /api/resource/id -- for which changes didn't apply immediately but it rather went into 'building' state or something like that. I had a method like modify_bar(self, resource_id) and things worked great for a single thread, but caused problems with multiple active threads. Obviously, I checked that resource_id is ready for operations, but in the middle of the end of the check and starting of actual work another thread could get in and cause troubles. Let's check sample code: And its output: (14:46) novel@nov-testing2:~/sync %> ./sync.py working on bar id = 1 working on bar id = 1 working on bar id = 1 done with bar id = 1 done with bar id = 1 done with bar id = 1 (14:47) novel@nov-testing2:~/sync %> But the result I wanted to achieve: (14:46) novel@nov-testing2:~/sync %> ./sync.py working on bar id = 1 done with bar id = 1 working on bar id = 1 done with bar id = 1 working on bar id = 1 done with bar id = 1 (14:47) novel@nov-testing2:~/sync %> I've been thinking how to do it without class redesign and API changes and after some time came up with a synchronization solution for methods based on their arguments. Implementation looks like this: Basically, we keep a list of ids of objects we're working with. When we want to start working with some object we check the list first, if its id in the list it means it's currently being worked on and we have to wait, otherwise we push the id to the list and start working. And the result is: (15:09) novel@nov-testing2:~/sync %> ./sync2.py working on bar id = 1 done with bar id = 1 working on bar id = 1 done with bar id = 1 working on bar id = 1 done with bar id = 1 (15:10) novel@nov-testing2:~/sync %> For the sake of experiment let's start another thread with different bar_id value to make sure it's not blocked: And the output now: (15:13) novel@nov-testing2:~/sync %> ./sync2.py working on bar id = 1 working on bar id = 0 done with bar id = 1 working on bar id = 1 done with bar id = 0 done with bar id = 1 working on bar id = 1 done with bar id = 1 working on bar id = 1 done with bar id = 1 (15:14) novel@nov-testing2:~/sync %> As you can see, things are now working as expected. Mocking libcloud using Mox 2011-02-04T06:59:00.000-08:00 If you're using libcloud to interact with cloud providers, you probably would like to cover this code with tests if you haven't done so already. However, most likely you don't want libcloud to do any real calls to the cloud provider API service as it could create additional noise on your account, add extra load and cost you money (esp. if tests are being run on a regular basis on a Continues Integration server). One possible solution is to mock these things up using Mox mocking framework. Libcloud uses 2-step initialisation process for a connection class -- first you obtain driver class and then initiate it -- and I seem to always forgot how to mock it right, so decided to write it down. For example, we have a simple class that returns a list of names of currently available nodes: And we want to test it without making actual API calls. Let's see how the test code will look for this task. For the sake of simplicity default unittest framework will be used. Basically, what we are doing here in the test: Create a mock for RackspaceNodeDriver class and record list_nodes() call for it; we also make it return two node objects. Actually, we create fake objects using type(), so we don't have to bother with creation of real Node objects which need a way more properties to be set when constructing that we don't care about Use mox's StubOutWithMock routine (please check mox docs for details on that) to stub out libcloud.providers module. Here, in get_driver() we pass lambda because get_driver() returns class itself, not an instance of the class, and using lambda here seems to be more clear than recording __call__() for a class All the standard ReplayAll/VerifyAll mox routines For the sake of experiment, you can replace actual node_names() method of the Example class with just return ["node1", "node2"]. The assertEqual() will pass, but mox will complain that list_nodes() wasn't called and test will fail. REST service on top of Marcuscom Tinderbox 2011-01-30T09:15:00.000-08:00 Like many people who work on FreeBSD Ports I've been using great Marcuscom Tinderbox to verify my changes. Some time ago I started to realise that it's quite a boring task to schedule a lot of builds using web interface again and again and finally I've managed to make myself start making the process a little bit easier. It seems that building a REST service on top on the existing is a nice way to go, so that's what I've been doing last few days and finally I have a prototype now. As I didn't want to patch original tinderbox's webui sources, I just created a sub-directory in it called 'api' and started there. To my surprise, I wasn't able to find any ready to use solution to implement REST service in PHP (maybe I've missed something though), so ended up with a few php scripts and rewrite rules to make URLs feel RESTy. It was a quite hard part of the task since I haven't been coding in PHP for years and don't feel comfortable with it even after few days of practice. Simultaneously I've started coding a client for which I choose Python. Workflow The goal was to support minimum but most important (for me at least) scenario -- testing an update for a port. Let's assume the port being tested is security/gnutls-devel. The process looks like that: 1. Produce a patch and apply it to the tree (can be done via ssh, or nfs, whatever, and could be easily scripted up) 2. Check builds we have: (21:33) novel@fsol:~ %> tbc build id name status current port updated 1 8.x-FreeBSD IDLE None 2011-01-30 14:05:25 (21:40) novel@fsol:~ %> So, we see an idle build here with id 1. Not so much choices, so let's use it, but first let's see what's in the queue: (21:40) novel@fsol:~ %> tbc queue id username port build pri status enqueued completed 10 novel security/gnutls 8.x-FreeBSD 10 SUCCESS 2011-01-30 13:54:43 2011-01-30 14:05:27 (21:45) novel@fsol:~ %> Now adding a port to the queue: (21:46) novel@fsol:~ %> tbc queue add -b 1 security/gnutls-devel (21:46) novel@fsol:~ %> Let's check if new port build added successfully: (21:46) novel@fsol:~ %> tbc queue id username port build pri status enqueued completed 10 novel security/gnutls 8.x-FreeBSD 10 SUCCESS 2011-01-30 13:54:43 2011-01-30 14:05:27 11 novel security/gnutls-devel 8.x-FreeBSD 10 ENQUEUED 2011-01-30 21:46:20 None (21:46) novel@fsol:~ %> Cool, it's here! Now we can see that our build turned into PREPARE state: (21:48) novel@fsol:~ %> tbc build id name status current port updated 1 8.x-FreeBSD PREPARE None 2011-01-30 21:48:15 (21:48) novel@fsol:~ %> All we have to do now is wait and poll for changes. Now we can see it's building: (21:59) novel@fsol:~ %> tbc build id name status current port updated 1 8.x-FreeBSD PORTBUILD gnutls-devel-2.11.5 2011-01-30 22:00:25 (22:00) novel@fsol:~ %> And finally we see it's done: (22:07) novel@fsol:~ %> tbc queue id username port build pri status enqueued completed 11 novel security/gnutls 8.x-FreeBSD 10 SUCCESS 2011-01-30 21:46:20 2011-01-30 21:57:18 12 novel security/gnutls-devel 8.x-FreeBSD 10 SUCCESS 2011-01-30 21:50:54 2011-01-30 22:09:22 (22:15) novel@fsol:~ %> tbc queue 11 For everyone who's interested I've uploaded sources on github: service (README, API Reference) client (README) NOTICE: it's an early work in progress! The code is unstable and lacks a lot of features. Anyway, any feedback about an idea and implementation is welcome. lc-tools 0.1.4 released! 2011-01-22T05:45:00.000-08:00 I'm happy to announce release of lc-tools 0.1.4. This release is based on libcould 0.4.2 which was released not so long ago. Changes from the previous version include support of the brand new SSL handling in libcloud, some custom features for Rackspace and GoGrid drivers and minor bugfixes. Please check NEWS file for a full list of changes. There are some non-function changes as well: lc-tools got its own website and also a tutorial. You can install the new version using either easy_install: easy_install lctools or by downloading and installing it by hands: lctools-0.1.4.tar.gz SHA256 (lctools-0.1.4.tar.gz) = 407ce1f29f0e869219ffff2f866fb6de226b693bf925817b6b43a612120b1ff7 Happy hacking on the cloud stuff! GoGrid locations support in libcloud 2010-12-13T10:37:00.001-08:00 GoGrid added support for locations (US-West-1 and US-East-1) quite some time ago and I finally managed to implement this feature for libcloud's GoGrid driver. Fortunately, libcloud API already takes into account that cloud providers could have more than one location, so the most important part was there already. Examples libcloud lc-tools Some of the lc-tools support '-l' flag to specify location id. Here's how a typical sequence would look like: Few notes here: you probably noticed that we filter images by locations but don't do it for sizes. Strictly speaking, sizes might differ for various providers and this is also accounted by libcloud API; but for GoGrid sizes are same for both locations, so I skipped this step. And the second thing: all these features are available in svn version of libcloud and git version of lc-tools. Also the thing I should have been mentioned for some time already: GoGrid provided me with an account and it is quite helpful for various testing. New website version 2010-09-27T11:30:00.000-07:00 For a long time my website was just a set of random links. Recently I decided to give it some structure and make it look like more like an extended visit card instead of just a collection of random links. So, here's the result: http://novel.evilcoder.org. I used Hyde for static generation of the site and I've put the sources on github: http://github.com/novel/novel.evilcoder.org. It helped a lot to look at Steve Losh's sources of his site which are hosted on github as well: http://github.com/sjl/stevelosh (actual site being: http://stevelosh.com). Anyway, there is a lot of work ahead, I need add some more content and learn and work on html/css/design stuff, at which I'm not very good. Installing Man Pages with Distutils 2010-08-26T05:56:00.000-07:00 People who read my blog probably remember that I started small project lc-tools which is based on libcloud and provides facilities to control clouds. Since the whole project is basically several Python scripts, I though it would make sense to provide a manual page for each. At first I had to choose format to use for manual pages. I remember dealing with groff several years ago and decided it's too bothersome for me and will require as much effort as writing all the other Python code. :] Fortunately, I've spotted ronn which allows to use Markdown for formatting manual pages, and it's a great thing as it allows one to think about content, not the formatting. Using ronn I created all the man pages quite quickly but then faced another problem: distutils doesn't have facilities to install man pages. That is. I've managed to implement a temporary (will explain why it is temporary later) solution. The idea is to override default install target this way: So we just execute install.sh after default 'install' target did its job. One important thing is to pass "PREFIX" to make sure that man pages will be installed in correct place. The install.sh script is pretty trivial (19 lines length) and you can see it here. Ok, so what's wrong with this solution? It forces users to have ronn installed to be able to install man pages. While it's OK for people who clone things from git, it's definitely not OK for people downloading the package from PyPI to force them installing ruby, ronn and friends. So the plan is to modify bdist targets to include generated manpages and modify install target to be able to install cached versions if ronn is not installed on the system. That doesn't seem to be hard to do, though. I think I will implement it and release a new version. BTW, ronn allows you create HTML man pages easily, so probably it's time to create a site for lc-tools with extensive documentation. Update: I've asked this question on stackoverflow and haven't got any responses so far. On Skytap 2010-07-30T10:16:00.000-07:00 I've ranted about GoGrid a bit in my previous post and now it's Skytap's time to receive its fair amount of rant. OK, just kidding, will sure my experience of working with it. Concept The general concept is quite different from all the cloud providers I worked with. What I've usually seen is the main abstraction of the cloud providers was a 'Server' (or node, or vm, whatever) -- just a single VM with properties like RAM size, CPU power etc, and which is created based on some 'Template'. Skytap does things differently: 'Server' model doesn't play the main role. The thing all the stuff's built around is 'Configuration', which is basically a collection of VMs. Configuration is created from 'Template' which is also actually a collection of VM templates. You cannot directly add a VM to configuration, you have to merge the template with a VM. I haven't decided if I like this way of representing things or not. Surely, in some situation it's convenient to be able to group VMs and work with them as with single object and we've even implemented a similar scenario at work on top of other cloud provider. On the other hand, when you need to work on single VM level it's a bit confusing and cumbersome. Web UI It's ok but has some usability problems IMO. Well, I'm not a usability expert by any means, but these things just seem wrong to me. For example: a way to API doc: Dashboard -> Support -> API -> 'How do I access API' solution -> link to developers page -> link to pdf guide. 5 steps from dashboard, not very good IMHO. API I have an impression that a lot of people treats any http service as REST service, that's obviously wrong. Skytap is good at this matter and their REST service is really RESTful. They even use all these fancy HTTP methods like PUT or DELETE. Based on some error messages and headers, I assume API is powered by Rails and Mongrel. The tutorial suggests using XML, but it looks like the API supports JSON as well, but not for all resources though. Having full JSON support would be really nice. Moreover, I've been playing with Firebug on web ui and noticed that AJAX is actually using JSON to communicate with the API, so it's strange why JSON is not fully supported. Also, API gives quite a low lever control to VMs. I.e. if you want to assign an IP address to a VM, you have to think not on VM level (like "assign address A to VM 'alice'") but on VM's OS network interface level (like "asign address A to the first network interface of VM 'alice'"). This forces you to make more API calls to get things done. Let's look at the quite typical task: create a VM with some of public IP available on our account. GoGrid Skytap find an unassigned IP create node from some template and use the IP from the previous step create a configuration from template find available IP assign an IP from the pevious step manually start the configuration So Skytap forces you to make twice more API calls than GoGrid. On the other hand, the feature of assigning public IPs wasn't documented at all in the API reference, so probably it will be changed in future and will become better. I've figured out how this works using Firebug. One more thing I haven't figured out how to handle better is the definition of term 'available IP' by Skytap. Basically, it treats IP as 'available' if it's not used by a running VM. Sooo... you can assign the same IP to several VMs. You will be able to start one of these VMs even, but other VMs with the same IP will fail to start. There is one more thing that makes me feel quite worried: it looks like the credentials are stored in per template basis. So, if you create two configrations from the same, say, RHEL public template, the root passwords of VMs in both of your configurations will be same. I really really hope to be wrong on that or maybe that passwords are generated on per-account basis (which I cannot check as I have only one account). Conclusion Summarising things up I have a strong feeling that I probably don't quite understand philosophy behind Skytap and probably it's targeted for the use cases where API is not very important. About GoGrid Support 2010-07-17T07:47:00.000-07:00 Let's discuss GoGrid support today. As an example I will take recent '<h1>596 Service Not Found</h1>' error. This error arises about every 15-20th request to the API, and it's quite often. The fact that this error is not well formatted JSON or XML with GoGrid API exception and therefore might not be handled well by some libs makes it even more annoying. Yeah, and this error was noticed by at least two more people on different accounts. So the natural thing to do was to start a case and try to find out what's wrong (as it's obviously not a normal behaviour). I did that and I got an answer like "Yeah, we know about this error, if you spot it try waiting 5 minutes and try again". It's just ridiculous. Imagine an engine of your car stops after each hour of driving, you come to the service and being told like wait 5 minutes then start the engine again and your problem is fixed! You'd unlikely want to visit such a service again and the same goes for support. Correct me if I'm wrong, but the first thing that customer wants from support is getting his problem fixed and if the problem cannot be fixed right away get estimates and details why it cannot be fixed. And who needs workarounds, especially for such obvious cases? Announcing lc-tools: command line tools on top of libcloud 2010-07-14T10:49:00.000-07:00 Looks like it's time to announce lc-tools! What is it? Basically, it's just a few command line tools to manage your clouds based on libcloud. What advantages does it give? Well, it supports as many cloud providers as libcloud supports (ec2, rackspace, gogrid and many others) and you can manage them all with a single tool. Look and feel is pretty similar to gg's one as it seems to be quite comfortable and I used gg more than year on a regular basis and haven't had any issues with it. Currently lc-tools covers almost all functionality provided by libcloud, but I've only tested it with GoGrid. Some examples so you can check look and feel of the application:Listing available node images $ lc-image-list|grep -i centos image CentOS 5.2 (32-bit) w/ RightScale (id = 62) image CentOS 5.2 (64-bit) w/ RightScale (id = 63) image CentOS 5.3 (32-bit) w/ None (id = 1531) image CentOS 5.3 (64-bit) w/ None (id = 1532) ... and sizes $ lc-sizes-list size 512MB (id=512MB, ram=512, disk=30 bandwidth=None) size 4GB (id=4GB, ram=4096, disk=240 bandwidth=None) size 2GB (id=2GB, ram=2048, disk=120 bandwidth=None) size 8GB (id=8GB, ram=8192, disk=480 bandwidth=None) size 1GB (id=1GB, ram=1024, disk=60 bandwidth=None) Creating a new node lc-node-add -i 62 -s 1GB -n mynewnode Pay attention to values 62 and 1GB -- they are ids of image and size respectively (from previous examples). Listing nodes $ lc-node-list 100xxx mynode1 173.204.xx.yy Running 100xxx mynode2 173.204.xx.zz RunningDestroying nodes $ lc-node-do -i node_id destroy Project's github page contains a README file with documentation enough to get started, but if you have any problems or suggestions feel free to contact me via github, blog or email. Kharkov 2010-07-08T21:22:00.000-07:00 Posting from a hotel in Kharkov. The trip to Kharkov is quite short, standing only for one night; so already going to check out soon. Bad thing is transportation that takes 2 nights (and two different trains) to get there, so actually have to spend 4 nights in trains for the sake of one night here. The city looks clean and accurate, most buildings are in old classical style, at least at the central part and I have an impression that central part is quite large. There's a subway here also, though city is not very large (population is 1.5 millions). Now it's time to pack my bags... Motivation 2010-06-24T12:38:00.000-07:00 I'm analysing some things happening in my life and wondering how... well, subjective and isolated from environment certain things could be. A good example is a car. You buy a new car and you start adjusting it the way you like: you buy wheels, suspension, maybe audio etc etc, you have a long list of things you want to install like silicon hoses, gauges, consider buying some new wheels and so on. There are of course a lot of cars better then yours and you surely can afford some of them, but you don't think about that. But one day a simple thought kicks you in a head: "I NEED this car". So you decide to sell your car and buy a new one; probably not now, probably not even this month, but you're sure you'll do it soon. So... nothing has changed in the environment. Just a simple idea in your head kills your passion in a second without any obvious changes in the environment and you cannot do anything about that. It scares me a little when I think about it and subconsciously I apply these thought to my programming job: what if my attitude to the certain projects or people are caused more by some ideas in my head than the environment? Well, there is nothing new in this thoughts and this subject is covered in a psychology, but despite of that it's really hard to find own base sometimes. About commit messages 2010-06-14T23:51:00.000-07:00 I noticed that top 3 commit messages on a project I'm currently working on are following: 1. fixes / fix bug / fix bugs 2. typo 3. changes (hey, 2pac) Every time I spot such a commit message I literally see how the $EDITOR arises between the poor developer wanting to commit his changes and the repository like the magic door in front of Ali Baba and how the developer has to say 'Open sesame' to make this annoying thing go away. Anyway, I know how to fix that! If you're using git, add the following alias to your ~/.gitconfig: ci = commit -a -m "changes" and you're all settled up! Just use git ci to commit your stuff. For a reasonable price I can consult you how to exorcise the $EDITOR from other scms. libcloud's GoGrid driver 2010-06-03T12:04:00.000-07:00 We have to work quite often with GoGrid quite frequently and about a year ago I wrote gg-tools -- a set of CLI tools and a library in Python. We've used it for a while but later decided to switch to libcloud to have a better portability if we decide to move to other cloud provider. However I found that some things in GoGrid driver could be improved. Specifically, there is quite annoying thing that gogrid doesn't assign an id to a node immediately after creation, but libcloud always expects to get an id for the image. As per discussion in libcloud maillist, create_node() was patched to block until the node gets ip assigned. Also, ex_create_node_nowait() method was added which simply adds node right away and returns id == None. While the patches are being reviewed, I created a fork on github: http://github.com/novel/libcloud with these changes applied. Feel free to check it out, GoGrid driver should be in a working state here as we use it on a daily basis and even if some error raises up it will be fixed quickly. Changing partition type on FreeBSD 2010-05-18T09:53:00.000-07:00 I had an ufs partition that I've used as a swap device. I was pretty satisfied with this setup until I decided to use it as a kernel dump device. I've ran dumpon on it and figured out it's not possible to use it as dump device. So I decided to change partition type to swap. I executed bsdlabel and got such error: bsdlabel: Class not found. I've been googling and reading quite a lot of time, lots of such questions are not answered, but finally was able to find a solution. The solution is to use gpart tool. It appeared to be a handy tool and to change type of my ad0s2b partition I executed these commands: Works great. Introducing goocaa, or on the way to my perfect mutt setup 2010-05-02T10:11:00.000-07:00 I've already posted a guide how I configured mutt over gmail to leverage some of gmail useful things, esp. address completion. I've used goobook for query_command. However, I don't quite like its stability, the way of its configuration and generally I prefer to avoid using software written in Python (or Ruby or any interpreted language) on my workstations. So I wrote a small tool called goocaa (click on the link to check its github page). It's written in C and uses libxml2, glib2 and neon. It uses ~/.goocaarc configuration file and its schema looks this way: [default] email = me@gmail.com passwd = lalala To start using it, just add: set query_command = "/usr/local/bin/goocaa %s" to ~/.muttrc [1]. One more thing that I've implemented is multi-account support. I have my work mail hosted at google also, so I forward it to my gmail account and put to a separate mailbox. For this mailbox I apply special rules, like, set "mail from" to work address, etc. One more thing that makes sense to do is to search in work's contacts instead of gmail ones. It could be done with goocaa's multi-account support. First, add additional profile to ~/.goocaarc: [default] email = me@gmail.com passwd = lalala [mywork] email = me@mywork.com passwd = tatata And now use folder hooks in ~/.muttrc: folder-hook . 'set from = "me@gmail.com"; \ set query_command = "/usr/local/bin/goocaa -p default %s" folder-hook mywork 'set from = "me@mywork.com"; \ set query_command = "/usr/local/bin/goocaa -p mywork %s" All done! Now mutt will use your gmail contacts everywhere except 'mywork' folder where it will be using your work contacts. 1: documentation on mutt's $query_command Ingeneeriing 2010-04-26T10:24:00.000-07:00 Found some colleagues on Facebook. I think they rock: I think it's funny when an enginiir cannot spell word 'enginiir' right AND doesn't use a spell checker. My first magazine publication 2010-03-08T12:16:00.000-08:00 My small article about cijoe has been published in Russian journal Linux Format! It's a small article describing how to get cijoe up and running for a typical Django project. I am really happy to see my article published! It seems the hardest part about writing technical articles in Russian is translating not very popular technical terms into Russian. For example, I still don't know what would an adequate and elegant trasnlation for 'test runner', for example. Anyway, I want to continue writing articles and I already have an idea about my next article. Mocking/stubbing in Django tests again 2010-02-12T09:16:00.000-08:00 I've already wrote a bit abouttesting in python. Since that, I've adopted nose and django-nose and I'm quite satisfied with the result. However, I was still feeling frustrated as I wasn't able to find a way of mocking/stubbing certain modules in my code easily without having duplicate settings.py. Later I figured out that I'm missing very simple and natural solution for some reason: as views are just functions I could simply import them and do whatever I want with them! The only thing I have to do is to construct HttpRequest object. So, the typical example will look like: This, however, doesn't make much sense without injecting some mocked stuff. Mock object could be created in setUp like that: Obviously it's also possible to just imported stubbed versions as well. Anyway, I prefer to go with mocks if I don't need to implement some complicated logic for testing, and create them on fly. In order to inject this stuff, I modify func_globals property of the function: So we just use our mocked version of 'foo' instead of 'foo' instead and also override some global function that makes requests to some remote service. This approach doesn't look very clean to my opinion, but it works almost fine. There are some problems with it however: if the view is decorated, we have to figure out how to reach the original function. For standard decorators like login we know that view is accessible as 'view_func' property, but if there are several of them including third party ones, it could be a bit troublesome to figure out what is the original view; on the other hand I don't think it'd be hard to solve these problems. Anyway, I'm quite happy with this solution. Few words about December trip to CA 2010-01-09T02:15:00.000-08:00 Stayed 2.5 weeks in CA in December, 2009. It was surprisingly cold, esp. during the first week of my stay - from 6th to 12th. This time wasn't driving around much but spent few nights in San Francisco instead. One of the place visited was Tommy's, tasted some tequilas and I must say that tequila is quite nice. :) Also visited Ruby Skye and also liked it, despite of the fact that some techno DJ (forgot the name) was performing that night and I'm not much into techno. Made the picture of it on the other day: Also visited some other places and met some FreeBSD people -- happy about that. Some random pic of San Francisco: The more time I spend in SF, the more I like it. Bazylum -- a tool to track x11 windows activity 2010-01-02T02:59:00.000-08:00 I was always wondering how much time per day I spend in firefox or, say, pidgin. So I finally managed to implement a tool which would track that. I called it bazylum - "lazy bum" joined together and first chars swapped. Funny, but after such a transformation it still has quite sensible meaning. :-) According to urbandict, bazy means being doing nothing. Lum is more ambiguous word, check it if interested as well. Getting back to the tool itself. It basically reports how much time a certain window was active, here's the sample: (20:50) novel@dirtyvegas:~/code/bazylum[master] %> ./bazylum stat urxvt: 9952 sec Navigator: 3934 sec gvim: 1818 sec xv: 1515 sec pidgin: 121 sec desktop: 10 sec Dialog: 6 sec avg window active time: 25 sec longest active time: 658 sec (20:50) novel@dirtyvegas:~/code/bazylum[master] %> So, from that report I can see that the majority of time I've spent in urxvt (that is good), the second popular window is firefox (aka Navigator) and I spent only 2 minutes chatting in pidgin (also good). Also, it says that average time before window change is just 25 seconds and the longest time when window was active is only about 11 minutes. It means that my concentration is not very good, probably a sign that I suffer from AD/HD. If you like the idea, you might want to take a look at the project's github page. The tool is written in C and requires sqlite3 and x11 (you will need -dev packages or something like that on binary based distros). To build it, simply type 'make'. Then collector should be started using './bazylumd' -- it will fork into background and start collecting window activity information. You might pass '-f' arg to it and it will stay in the foreground. And use './bazylum stat' to view the report. The tool is obviously on its early stage of development but works alright on my desktop and laptop. Happy New... 2009-12-30T14:46:00.000-08:00 It's 31th of December, 2am. I cannot sleep, don't know why -- maybe because of jet lag or because of a bunch of not so pleasant thoughts in my head. Anyway, this year was not boring by any means. I'm not in a mood to write some lengthy general post, just want to touch professional aspect of my life. I wrote quite a lot of code this year (not very much though, but not bad at all I think). Unfortunately, most of the code ended up being a shelf-ware or ended up only on my computers with me being the only user as I cannot make the code go live because it's not usable for people other than me -- lacks docs, misses general, but not important to me features and so on. So I really hope to make a dramatic improvement in that sense in the new year. Happy Holidays to you. Train Girl 2009-12-21T21:01:00.000-08:00 I've boarded the train to Moscow sometime around 4-5 pm or something like that. The only other person in a coupe was a girl. A few minutes after train departed I've started talking to her discussing some gibberish that people usually talk about in trains/planes/etc and eventually the talk became very interesting. You know this feeling, when you're talking with somebody and you have so much things to say and you feel that the person you talk with also has a lot of things to say and basically that it would be fine to have a several discussions with this person in parallel at the same time. So we've been talking until about 1am and discussed lots of topics, including some personal ones, so, about 8 hours or so, quite a lot of time. I decided that it's definitely worth not to lose a connection with this person, however I was shy to ask a number. We've agreed to go to the subway (the one that under the ground, not food) together and I though I'd ask number there, but somebody meat her right near the train so I had no chance to ask the number. So I was feeling quite impressed. I was really really angry at myself that I haven't asked her number. I wasn't able to stop thinking about her for a 4 or 5 days (and it's quite a lot of time for me to think about a girl ;]). I even managed to figure out her last name and address. But now, a bit more than two weeks later I realised that I almost stopped thinking about her. So, now I'm not sure already if I should try to contact her... blah, on one hand I definitely won't loose anything by trying to contact her, on the other hand it doesn't to be a good situation when you're not sure if you like a person or not, at least I don't like it. Will see how the stuff goes. Doing Nothing Zen 2009-12-21T15:46:00.000-08:00 Looks like I've somehow managed to master 'doing nothing' skill. When I was flying from Moscow to SF two weaks ago (I guess it's about 12 hours + 7 hours) the time passed very quick, I wasn't even reading a book or watching movies etc, was just sitting and doing nothing. And that is quite unnatural to me, because I always feel a need to do something and sitting in the plane without an ability to move or do something interesting was a pain in a previous trips. I even thought that I have an ac/hd as it was really hard to not change things quite often. Moreover, being in bart also doesn't bother me much. I don't use laptop, I don't read books -- not because I don't bring them, I always have it in my bag, but just because I enjoy looking through the window (even though I've seen everything quite a lot of time there) and I feel totally OK about it. That's probably because I'm quite tired and my brain switched to some 'lazy' mode or something like that, or maybe I'm just getting old. :) I hope I will be feeling same on the long way home. One more thing I have to learn is how to sleep in a plane. About St. Petersburg trip 2009-11-01T03:51:00.000-08:00 Yesterday night I've returned from St. Petersburg trip I previously wrote about. It takes 25 hours on train to get there (and the same from there to home). Quite a lot of time, but really, not very painful. I've spend about 2 weeks there. Right from the train we went to the office, it's only 1 subway station away from the rails station. This part of the city is really beautiful. Here's a picture of the office building: The view of the street leading to the office: Very nice. In the evening we've checked in into the hotel. The room is small but accurate: The only problem with that was the furniture. I cannot even say it is uncomfortable, but it's strange, I mean, I wasn't able to sit or lay comfortably. Also, pillows were unusually small. Anyway, it's just a whim, stuff was generally good for a hotel. Initially, I thought I will visit various interesting places in SPb - museums, parks, palaces and stuff like that. Later on, I've realized that I've visited them lots of time before but I still don't have an overview of the city. So this time I was just walking on the streets. I've made some photos of the places I like. I guess they don't have any value from the photographer's point of view, but probably they will make you want visit SPb or at least find a better photos :) Great city, like it so much :-) One more good thing about it -- your shoes stays clean even if it's raining everyday. Obviously, it's because the city is very clean. Ah, I definitely want to visit it again. PS Here's my bag: I've managed to cram all my stuff into it and it looks like a giant pregnant bee. Looks even more funnier on my back: imagine a person being shagged by a giant bee, haha. Vacation and St. Petersburg 2009-10-19T11:47:00.000-07:00 Looks like last week I had a first real vacation in my life. By real I mean that I took a vacation not because I had to do something important, just because I wanted to have some rest. I've been really lazy and wasn't anything. :-) Yeah, a little bit of slacking is not bad at all. Anyway, the vacation is over and today I've arrived to St. Petersburg. Haven't been there for 7 years... yeah, it's a great city. And I like it's weather too. The only that disheartens me a bit is that I don't have my car here. Using mutt with Gmail IMAP: complete guide 2009-10-13T04:10:00.000-07:00 There are some guides around on the net, but I wasn't able to find a complete guide to make mutt as usable as gmail web interface, so I will try to put stuff together in one single post. I will also try to explain why each certain option is needed. Also, I don't pretend it to be the most optimal or correct way of configuration, but it works in practice. :-) Basic IMAP configuration So, the basic IMAP options follow: set realname = "Your Name" set imap_user = 'yourname@gmail.com' set spoolfile = imaps://imap.gmail.com:993/INBOX set smtp_url = "smtp://yourname@smtp.gmail.com:587/" set folder = "imaps://imap.gmail.com:993" set record="+[Gmail]/Sent Mail" set postponed="+[Gmail]/Drafts" set header_cache="~/.mutt/cache/headers" set message_cachedir="~/.mutt/cache/bodies" set certificate_file=~/.mutt/certificates These lines are quite obvious and could be seen in a number of guides. That's probably a minimal set of setting you need to be able to check your gmail mails using mutt. You might want to put your password to the config file in order to prevent mutt asking it all the time. Ah, yeah, and I forgot one obvious thing that you probably know about anyway, the config file is ~/.muttrc. sorting Gmail was the first app I've used that sorted threads by the date of the last message in the thread. This is quite nice, because in the times when I used mutt with POP, I sorted thread by the first message, so if somebody sends a message to some ancient thread few screens of scrolling back I could easily miss it. So, I've configured similar sorting in mutt, with the only difference that I prefer newer threads to be in the bottom, not on top. So I've added the following lines to my ~/.muttrc: set sort=threads set sort_aux=last-date-received The first line says that messages should be grouped in thread, the second one determines the order of the threads, so in our case we sort by newest message in the thread. You might want to consult muttrc(5) manpage for other sorting options. Address book When I've performed configuration steps described above, I've realized I really really really miss the built in address book of Gmail! The thing is that our company uses gmail to handle its mail and all the people who have an account automatically appear in the address book, which is quite handy and I've got used to this feature very very quickly. So, in order to use the address book from mutt, query_command feature can be used. It allows to use an external application as address book. There's a nice script which allows to be used as query_command in mutt, it's called goobook. In order to install it, create some directory, e.g. ~/opt, cd into it and execute the following command: svn checkout http://goobook.googlecode.com/svn/trunk/ goobook-read-only Note: you will have to have subversion client installed. Then cd to "~/opt/goobook-read-only" (or whatever dir you used), execute: cp settings_example.py settings.py, then edit settings.py and fill in your gmail credentials. Now while you're in the working dir of goobook, check if it's configured correctly by executing something like: ./contactos.py e Instead of 'e' you can type any letters which you sure will match somebody's name or address in your address book. If the script output something sensible then it works. Now, teach the mutt to use this script by adding these lines to ~/.muttrc: set query_command = '/home/user/opt/goobook-read-only/contactos.py "%s"' Now, in mutt you can type 'Q' and type some query to make sure stuff is working correctly. If it works nice, then it means you can use address book when composing new messages. For example, if you hit 'm' to create a new message, in 'To:' prompt you can type some stuff and click 'ctrl-t' and mutt will complete stuff from the address book. Yeah, one more thing about goobook. I had to insert empty 'print' right after arguments check in the contactos.py script as it seems that mutt use first line of the output to show in the status bar. So the scripts looks this way for me: ... 74 if __name__ == '__main__': 75 if len(sys.argv) < 2: 76 sys.exit(1) 77 print 78 79 try: ... I haven't had time to look at mutt sources to investigate this problem, so I'm not sure if it's an expected behavior. New messages info What else... the other very important option is 'imap_check_subscribed'. To make mutt automatically add all the folders to mailboxes (so you can see a number of new/unread messages in each folder, new mail confirmation in different mailboxes etc), the following setting should be made: set imap_check_subscribed=yes If you're moving from Gmail or maybe some GUI MUA, you might want to see a sidebar with list of folders and mail counts. It's possible to implement in mutt as well, however it requires applying third-party patches, sidebar patch specifically. I don't provide instructions how to apply it as in FreeBSD it's as simple as building mail/mutt-devel port with "-DWITH_MUTT_SIDEBAR_PATCH" flag. Configuration is as simple as adding these lines to ~/.muttrc: set sidebar_visible=yes set sidebar_width=30 Summary So, here's my complete ~/.muttrc file, only 50 lines long: set realname = "yourname" set imap_user = 'yourname@gmail.com' set spoolfile = imaps://imap.gmail.com:993/INBOX set smtp_url = "smtp://yourname@smtp.gmail.com:587/" set folder = "imaps://imap.gmail.com:993" set record="+[Gmail]/Sent Mail" set postponed="+[Gmail]/Drafts" set header_cache="~/.mutt/cache/headers" set message_cachedir="~/.mutt/cache/bodies" set certificate_file=~/.mutt/certificates set imap_check_subscribed=yes set move = no set delete = yes set edit_hdrs set include set reply_to set abort_nosubject=no set sig_dashes = no # don't append -- as I use only my name as a sig set attribution = " %n wrote:\n" # I don't like lengthy attributions as well alternates '(username|altusername)([-+].*)?@.*' set hdr_format="%4C %Z %{%m/%d} %-15.15F (%4c) %s" # format of the index ignore * unignore from subject to cc date x-mailer x-url user-agent hdr_order from to cc date subject x-mailer user-agent set query_command = '~/opt/goobook-read-only/contactos.py "%s"' set sort=threads set sort_aux=last-date-received # see beyond for hooks explanations folder-hook . 'set from = "yourname@gmail.com"; \ set smtp_url= "smtp://yourname@smtp.gmail.com:587/"' folder-hook my 'set from = "i@mywork.com"; \ set smtp_url = "smtp://i@mywork.com@smtp.gmail.com:587/";' folder-hook foobar 'set from = "yo@example.org";' # colors stuff outside source ~/.mutt/themes/mytheme # gpg source ~/.mutt/gpg.rc set crypt_autosign = yes # sidebar set sidebar_visible=yes set sidebar_width=30 Few words about the hooks. I have two mail accounts, work and personal, both hosted at gmail. As I don't want to check both accounts, I've configured my work account to forward all the mail to my personal account and configured personal account to place all such mail into a separate folder. So, the first default '.' hook just says that we generally use personal account. The second hook for 'mywork' says that when in 'mywork' folder, we change 'from' address to work address and use smtp from work account for sending (could be configured on gmail side as well to add working address for account, but I don't like this way for various reasons). The third hook is for an alias at example.org which I use for some maillists, so I change 'from' address to it so maillist software allows me to write there. In this case I've added this address to gmail account, so no special 'smtp_url' magic is required. So, it doesn't seem to be complex to configure mutt this way and I'm totally happy with such a configuration. Update @ 24th, May 2010: I've wrote a replacement for goobook and switched to it instead: http://empt1e.blogspot.com/2010/05/introducing-goocaa-or-on-way-to-my.html Social networks 2009-10-07T00:45:00.000-07:00 I've just realised that my behaviour pattern in social network looks that way: 1. Register 2. Give up on it: don't use, don't talk with people, don't add anybody, etc 3. Some people start adding me, I add them back (why not?). This step repeats several times. 4. When I have a plenty of people 'friended'/'connected' whatever, I realise it makes no sense as I don't really talk to anybody so... 5. I delete the account. I guess the only interesting social network was livejournal as in state it was few years ago, because it was interesting to know what's going with people you know (but not close enough) or interesting people you don't know. But for some reason, people don't write much interesting stuff to their journals on livejournal, just post some bullshit. Testing in Django and Python in gerenal 2009-09-30T12:32:00.000-07:00 When I've been programming in Java I got used to write various tests. It's good to be able to play with the API you just wrote and tests save you a lot of time as programmed tests take less time than manual testing. Especially it's handy when you have some external service your app talks to and this service is not very fast, e.g. a response could take several minutes. So if you're testing stuff manually, it can take you several minutes, and the other approach to stub this service out and use this stub in tests to test the logic and stuff. So, returning back to Java, I must admit that Java world has a plenty of great testing/mocking/etc frameworks. And the situation in the Python world really disheartens me. First of all, testing frameworks. Maybe I've missed something, but looks like the only used frameworks are doctest and unittest, both from standard library. I don't get this doctest framework at all. Putting code in the comments to your code seems to be awkward to me. I don't think inline code documentation is a good place to place some examples, if your code requires examples, it's better to write up a tutorial and place all the examples here. So, it's not the way to go (IMHO). The other framework is unittest -- a clone of some old JUnit with very limited feature set. Misses lots of vital features that could be found in TestNG for example. Specifically, I miss the following features: * Test groups/dependencies. I want to make one test depend on other. It's clear that if some basic thing fails (e.g. user auth) it makes no sense at all to test all the stuff on top of that (e.g. actions for logged in user). And all these 'EEEEE' for lots of failed tests just make it harder the reason of error * Lack of beforeSuite etc methods ... and many more, lazy to recall right now. Testing in Django itself is also quite cumbersome. First of all, the concept of keeping tests in app/tests.py is not very flexible. This app-centric approach is probably good when your project consists only from Django, but I really doubt it's possible in real life: most likely you will have some kind of helpers, misc. classes that don't fit into apps and so on, and all of them require to be tested. What's the good place to place these tests? If you won't place them into tests.py of one of the apps, you will most likely have to implement your own test runner. More examples: * You cannot run an individual test using django test runner. Imagine you managed to write, say, 100 tests and it takes about 3 minutes to run all of them. Then you find that one of the tests failed. Instead of 'try to fix -> run failing test' iterations you will have 'try to fix -> run all the tests -> wait N minutes'. * To plug some simple things, like code coverage reports, you have to override/implement own test runner. Now it's time to get back to stubbing again. Django doesn't provide any easy way to replace real libs with the stubbed ones. The only good thing about it is that it's quite easy to get separate database instance for tests and automatically upload fixtures into it. But I don't see a clean way to stub out an external lib, the trick with overriding sys.modules['modulename'] doesn't work if you do it in setUp, because the lib could be already improted, etc. sendxmpp!! 2009-09-01T05:19:00.000-07:00 sendxmpp tools is actually the worst and the most buggy program I've ever seen (considering its simplicity and size). Jet Lag 2009-08-02T16:04:00.000-07:00 ... gets me more serious than the last time. Fall asleep at 4pm yesterday and got up now at 2am. I don't think I will be able to sleep anymore this morning and looks like I'll be having hard times this evening trying to cope with sleep. US Trip is almost over 2009-07-24T23:04:00.000-07:00 My US trip is almost over - I fly back in the very early morning on Monday. Final accord would be driving to Hollywood for the weekend, then go back on the Sunday night, grab luggage and go to the airport. This trip was kinda fun and a lot of stuff happened so it wasn't boring by any means, and I feel tired and miss a lot of people I haven't seen for one month. So I look forward the Hollywood trip and getting back home then. paramiko vs scp 2009-07-16T20:22:00.000-07:00 I had a task to benchmark a server by measuring how much time would it take to downloaded file via ftp. It was boring to script that up in shell since the server to be benchmarked didn't support key-based auth, so I'd had to pass password to ssh and I don't know a clean way to do it. Just for fun I decided to try doing it in Python using Paramiko module, which is a pure-python implementation of SSH protocol. I though it would be slower than native scp tool, but anyway I could use it to track the dynamics of performance changes. Imagine my surprise when tests results showed that Paramiko performance is nearly equals to native 'scp' performance! I have to say it was a really pleasant surprise and makes Paramiko even more useful in my eyes. Lenovo X61S 2009-07-16T20:12:00.000-07:00 ... is just CRAP!!! At least for a *nix user. First of all, it has some clit instead of a touchpad. I don't like the idea to place a clitoris in a middle of laptop's keyboard. OK, since almost all apps I use are command line and I also use stuff like vimperator for non-cli apps, I don't need to use mouse often. BUT BUT BUT. The keyboard is HORRIBLE! It's totally non standard and offset between rows are not usual as well because I always type stuff with offset of 1 symbol to the left or to the right. Moreover, the Escape key is placed in a very strange place, to the top of F1 key, so I often confuse esc with f1. It has function key as a very left button of bottom row, so I confuse it with control key. It has sleep key bound to fn + f4. So if I want to switch to 4th virtual desktop I risk to press fn-f4 instead of ctrl-f4 and suspend the laptop. I don't know how much time will I need to get used to it and I see no point at all to make such a non traditional controls. I wish I just could throw it into the window... Las Vegas pwnz! 2009-07-08T20:33:00.000-07:00 I've been in Las Vegas last weekend. Well, what can I say, it's impressive! I've attended only two shows: Blue Man Group show and KA Theatre. Both shows are really great. I'd probably visit them both again. Unfortunately, I was kinda short on money and time and haven't had a chance to try out more shows or try girls ;] By the way, there are really lots of guys who offer girls on the Strip. They stay there any time, though there are less of them in the morning. But in the evening they stay in a row (literally) and there are LOTS of them. I wonder if that's effective, because they seem to give people all the same slides. I guess the efficiency would not change if their could would be reduced 5-10 times. Anyway, I really liked Las Vegas. That's the first place in US where I'd agree to live. Probably it would get annoying after some time though. In US again 2009-06-29T14:25:00.000-07:00 As the title says, I'm in US again. Flight and train wasn't very good this time and while I was on road I slept only for about 3-4 hours in total. But anyway, I'm here finally and have enough time to sleep. I will be staying here for quite a long time and I hope I will be able to find some fun stuff to do. I think I could start to annoy waitresses in cafeterias by not answering "Don't matter" when they ask about some options for the order, but asking them in details what each option means. Ok, just kidding, it'd be so cruel =) project's webpage updates automation 2009-06-13T05:44:00.000-07:00 Currently I'm reading "The Pragmatic Programmer: From Journeyman to Master" book. And while its contents seem quite obvious and natural to me, by repeating "DRY/automate things" stuff again and again it forces my mind to re-think about possible ways of automation of various aspects of my activity. I have a number of small projects, which are not very popular, but I prefer to keep a page for them just in case if they will be useful for somebody. However, I never liked updating web-sites because it's boring (like: ftp or ssh, edit files, re-check, edit again, you know). So after reading this book I decided to try to automate that too. Recently I moved my small pymgsrc project - a command line client for imgsrc.ru photo hosting - to github and wrote a simple script for the website update. The way it works is: - Build a release distfile - Load template for website (single html) page, replace %%VERSION%% with actual version (in order to update download links and info about the latest version available) and replace %%CHANGELOG%% with HTMLfied (using simple sed expression) ChangeLog entires - scp resulting distfile and webpage to the hosting - Viola, all done! Using only one command. I'm very pleased with the script and that I don't have to bother with HTML editing anymore, at least while I didn't decide to alter template. I guess the hardest part of the automation is to find pieces of work which could be automated, because sometimes you do stuff and don't understand that it could be automated, but when you finally automated it it starts to look pretty natural and you wonder why you haven't noticed it earlier. python subprocess module 2009-06-12T12:41:00.000-07:00 Recent Python versions show DeprecatedWarning when using various popen functions and encourages to use subprocess module. And it's frustrating because I don't quite like the subprocess module. Let's take a simple example: executing one app and piping other app, like "dmesg|grep hda". They even have this example in the subprocess module documentation so I just took it. So, take a look: output=`dmesg | grep hda` ==> p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) output = p2.communicate()[0] That looks lame to me. Look, original command has only 4 words used and is easy to understand. The second example has: 17 words, uses classes (who needs them here?) with some weird non intuitive constructor arguments and the final accord is "p2.communicate()[0]". What is "commmunicate()", can you guess what it does in details without looking at the docs? What does it return? What the hell is "[0]"? Seems really horrible to me. Generally, it seems quite inconvenient to execute external applications using Python. Some people call Python 'a better shell', but it doesn't seem quite true if you need to execute external apps and in _most_ of shell scripts you really do need such things. Well, probably it's good at some sense that it forces you to split your Python logic and shell-scripting logic into python and shell scripts respectively and call shell scripts from your Python scripts and vice versa, but anyway, I don't like it a lot. ObjC experiments 2009-06-05T12:31:00.000-07:00 After some experiments with iPhone development and ObjC I decided to take a look at ObjC and use it to implement some simple but useful app. As I don't use any kind of tray or docker whatever (I will elaborate on that topic) I decided to implement some app which would poll IMAP server for new messages and notify me using libnotify. If you're interested, you can check sources on github. Actually, it's more toy than real app atm, however it seem to work alright with my gmail account. And yeah, it's pretty simple and can only work with IMAPS (IMAP with SSL). Well... ObjC impression... surprise! Nothing is looking really impressive. What can I say. The language looks pretty natural and intuitive. I guess I like it more than C++, though I cannot be quite right in that since I don't know C++ and ObjC well enough, but I just like how ObjC feels. One more thing is that I haven't read any ObjC so I don't know best practices and design principles that should be used when wring in ObjC. I think it would worth to read some good book about ObjC. I didn't use GNUstep foundation classes on purpose. First reason is my philosophy of writing tools is that a tiny tool should not bring such huge dependencies like GNUstep. The second reason is that it would be better to learn pure language before moving to high-level frameworks. And my understanding is that using foundation classes changes ObjC programming dramatically. I think I'll play with this tool a little because it contains a lot of bugs and leaks (but it's more related to the C aspect of programming), will probably try to redesign it and move to playing with GNUstep classes, just need to come up with a project which would worth it. PS Yeah, the only thing that makes me curious is "+/-" in method signatures. IIRC, they're used to distinguish class and instance methods. In the most code I've seen or wrote, most of methods were actually instance methods, class methods appear not quite often, at least for me, so I don't really get why they deserved so much attention in ObjC. gmail generation 2009-05-27T03:33:00.000-07:00 Gmail got extremely popular about 2-3 years ago. That's probably because of kinda simple and at the same time powerful web interface it has. Even my mother has a gmail account these days and uses it all the time and has no problem with it at all. But there's a bad, very bad thing about it: it makes people format messages awfully. It's been written in various places that default quoting method is top-quoting, which is really annoying in most of the time. Moreover, people who read mails sent by normal mail clients like mutt, have problems with viewing them. Because text placed after a quoted message will by most likely formatted as a quotation as well by gmail. And inline quotation must be a real PITA for gmail users. Moreover, it forces people to over-quote because it leaves all messages from the thread, not the last one. As a result, most e-mail threads in which gmail users take part look like: several lines on message and then lots of lines of the previous messages quoted. It's really painful to read and I guess anti-gmail campaign would be more actual than ascii ribbon campaign these days. PS Actually not gmail, but it's formatting, because in general service seems to be OK. bike again 2009-05-25T07:18:00.000-07:00 It's raining for the second day in a row and looks like will be rainy all the week. So using car instead of a bike. Bike is a drug, really. bike 2009-05-14T09:13:00.000-07:00 So I've started biking more or less often: biked last weekend and started to ride at work on a bike instead of car. Actually, I have a pretty positive impression - it takes about same time as getting to work by car, maybe even faster. Road to work is almost all the way downhill, so when I came at work I don't feel tired etc. Obviously, all the way back is an uphill, but it's also nice because it trains me I guess. The only inconvenient thing about it is that I'm feeling kinda tired after getting home and I still have a jet-lag after US trip, so I go to bed early and don't have much time to do useful stuff. But I hope in a week or maybe I will adopt to uphill and cope with jet-lag. recent Xorg fashion 2009-05-10T18:21:00.000-07:00 I've been using free *nix systems for about 10 years now and almost always used X11 servers on them. Back then it was XFree86, not Xorg. First it was something like 3.3.x and then 4.x versions of XFree86. I always wondered how X was easy to configure. Actually, I've never had problems with X configuration. It was just a matter of generating a configuration file once and forgetting about it - it just kept working after upgrades etc. Probably only once I had problem with it (not sure "problem" is a right word though) - when "keyboard" driver was renamed to "kbd" driver. And I'm not even sure it actually happened in XFree, not Xorg. But with Xorg, I have problems all the time, esp. last few months when I switched to xorg with hal support. It's like... grab Ubuntu, install it, disable gnome, start openbox and figure out that shortcut for layout switching is "shift-alt" instead of "shift-ctrl". Ok, go to xorg.conf and see that you're configured "shift-ctrl", go "wtf?" and dig into /etc to find some gnome leftover scripts which probably re-configure xkb. Fail at that, google up the problem and figure out that xkb settings in some hal xml configs override the ones in xorg.conf. WTF? I totally don't get it why to make it possible to configure single item in two different places, esp. considering that in one place configuration doesn't make any sense. It's sooooooooo linuxish so it's even frustrating. Or another example - I've installed freebsd on my laptop today and installed Xorg. When X starts, I cannot get mouse moving etc... input basically doesn't work. I go to Xorg.log and see something like "AllowEmtpyInput is not set, disabling input devices". WTF??? What kind of reasoning process made it think that I don't need input devices? I don't even care what is "AllowEmtpyInput" and I don't know why is it so critical that I should be punished for not setting it in such cruel way. SO I've added it and got stuff working. I assume it's somehow related to hal. What I don't get is: I don't need hal. I guess a lot of people don't need hal in X (I don't know any who needs it). Why the hell stuff cannot just work in a way it worked before without hal? I think it would be more fair if people who need hal would fuck with it to enable it instead of people who don't need hal having to fuck around it to get things working like they're used to see. MacOS X first impressions 2009-05-08T11:23:00.000-07:00 By occasion I had to develop stuff for iPhone whole this week and therefore used MacBook as a primary computer. I've never used MacOS before except that once I've tried to install Hackintosh on my laptop but I could not used it because it didn't support neither wifi nor ethernet. So, well, MacOS X. It seems very hard to use in default configuration. Like, MacBook screensize is 13.3" (or something like that, lazy to check), considering that it's weird that the dock consumes so much screen space. I don't know how is it supposed to be usable, because the menu bar + window title + dock in default size consumes almost 1/4 of the screen space, it's weird. The only option is to set it's size to minimum and make it auto-hide. Window manager also seems to be kinda unusual. The first strange thing is how spaces (or whatever it's called) are implemented. I wonder why "Alt-Tab" shows me windows from all spaces instead of windows from the current space only. I was told it's possible to configure it, but I don't spot such option in the preferences. I also cannot find how to move a window from one space to another. Next disappointment is xcode, and first of all, it's editor. It feels like notepad with the only difference that it has syntax highlighting. For some reason, in the default configuration when you hit 'build and go' button it doesn't just to the first error/warning line in the sources. Seems to be kinda weird as well... Other stuff made me curious as well, for example, I wasn't able to find an easy way to rename a project. I don't know if xcode is positioned as IDE, probably is, but it looks like it's missing a lot of useful features available in other IDEs, for example I wasn't able to find any refactoring stuff there, while e.g. Idea has great support of various refactoring stuff. As for hardware, keyboard is weird in macbook. The thing that makes me feel uncomfortable is the "fn" key located in usual place of "ctrl" key. Space between the keys are also kinda large and it's unusual as well. Now I have an impression that OSX is not OS of my choice, I'm happy with freebsd/openbox or linux/openbox at least and the only interesting thing to me in osx is their API, esp. the UI part of it, because it looks like it's way better than for example QT or GTK+. ObjC is also quite interesting, I think I'll play with it in the future. Trip to US, part2 - food 2009-05-07T11:13:00.000-07:00 As promised, here's a quick post about my food-related experience in US. * First, the most impressive thing is a size of portions - they _veeeeeeeeeery_ large. Like, when you order a sandwich, they bring you two of them and bring some extra stuff like cakes or probably free. It's a lot of food, esp. for the breakfast. * Coffee - coffee is very tasty. I tasted coffee in peet's, starbucks and probably something else I forgot and it was pretty good everywhere. Muffins and cakes there are also pretty yummu. * The above two items lead me to conclusion that I don't really need a traditional breakfast, I'd better grab medium size coffee and a muffin or two and it's enough for me and cheaper. * As for various kitchens, Thai and Italian are pretty good. And due to size of portions you can order chicken or pizza, eat part of it, ask for the box and use it for lunch or even for two. * This item is closer to service than to food itself. When you order food you always get asked additional questions, like "what do you want it to be served with" etc, and it's pretty hard to tell, because I learned English from UNIX manpages and then IRC and don't know all these words =) It's kinda problematic, because "I don't care" almost never works. But it's ok, just have to learn more food-related words :) Trip to US, part 1 - general impression 2009-04-29T10:41:00.001-07:00 It's the 3rd week of my trip to US. It's my first time to US and actually it's the very first time when I've crossed the borders of motherland. I guess I will split my story in a few posts instead of making one lengthy post. In the first post I'll write up about general impressions. The first impressive thing is that it's very clean here on the streets - almost no garbage and no dust and it's the thing I really like here. The second thing about streets - they look all the same! I've stayed at Fremont and all the streets look same here - same houses, building of the same style and color, roads and tjunctions all look the same, so it's not very easy for me to navigate here. I've been in other small cities in CA and it's the same for them too. Well, landscapes might be different for different cities, but in one city it's very easy to miss streets. There's no explicit borders between cities. You walk on the street, walk over the road and you're already in a different city! It's unusual, because in Russia if you want to get from one city to other you'll have to spend at least 30 minutes riding on a car and you will see very large signs like "city X ends here", "city Y starts here". People here like to ask you stuff like "how are you doing?" and so on, while they don't really care and I dunno why do they ask that stuff, in Russia people just don't care about you and don't ask such things. It looks like that in small cities like Fremont people prefer to stay at home after work because there's few people on the streets after 6-7pm, though maybe they just ride somewhere or something like that, I dunno. I've noticed the same in Palo Alto as well. However, there are a lot of people in San Francisco, a lot of freaks here as well ;) and it's nice. In the centre of San Francisco I've seen a slowly riding old american car (dunno the model) on large chrome wheels and loud hip-hop sounds and serious looking persons inside of it. Looked impressive, just like in a movies =) Subject for the next post is food. ubuntu gutsy sources.list 2009-04-24T12:07:00.000-07:00 Decided to install some stuff on ubuntu gutsy server today and found that sources.list is outdated. It was pretty hard to find working sources.list and that's the one I've found that works fine: deb http://old-releases.ubuntu.com/ubuntu/ feisty main restricted universe multiverse deb http://old-releases.ubuntu.com/ubuntu/ feisty-updates main restricted universe multiverse deb http://old-releases.ubuntu.com/ubuntu/ feisty-security main restricted universe multiverse deb http://old-releases.ubuntu.com/ubuntu/ feisty-backports main restricted universe multiverse deb http://old-releases.ubuntu.com/ubuntu/ feisty-backports main/debian-installer deb-src http://old-releases.ubuntu.com/ubuntu/ feisty-backports main restricted universe multiverse Bike, Health and stuff 2009-04-07T10:01:00.000-07:00 I cannot say I was a sporty type from the childhood, but since the yearly age I've been spending a lot of time on the street and was playing soccer, basketball and other games all the time. It lasted until about 8th grade, then I changed school and a way of life a bit, I've started spending more time on computers and stuff and started to smoke and drink =) And I've started to attend gym as well. It's kinda strange combination and it resulted it gaining some strength and weight and decreasing overall stamina. Then I've entered the university and my lifestyle didn't change much: I was still spending a lot of time with computers and other time I was drinking (have tried almost every alco drinks during uni years ;]), though I quit smoking. And I was still attending gym. I did pretty well for an amateur, I did a bench press with about 105-110 kg while my own weight was about 95 kg or so. On the 3rd year of university I decided that I need more money than scholarship ;]. I've found a half-time job and was still attending most of the classes and gym, but it was kinda hard. Moreover, it turned out that halt-time job is really not very good due to various reasons, so I found other job, full time this time. So I've started working full time and was still attending most important classes. I've gave up on the gym, because I almost had no free time. However, I consider that's because I don't manage time well and it was still possible to find some room for such activities, but time management is a totally different subject and I will probably write about my problems with it later on. Oh, yeah, so I've left the gym. Then I bought a car, so I even separated myself from walking. Now I realize it wasn't very good, however the car allowed me to have some more free time. I've lived in such a mode for more than two years. I've started to realize that I feel bad without sport activities. I've gained some more weight (needless to say it wasn't a muscle weight ;]). The lack of physical activities made my stamina even more lower. Moreover, I was missing this wonderful pain that you feel after trainings. Last summer I've finally graduated. First few month I was really glad that I have a lot of free time and had no exams, tests and other annoying stuff so totally forgot about sport stuff. And just recently I've started to think about it. My first though was to start attending gym again. But then I decided that attending gym will just more increase weight (and I don't think it's good even it will be muscle weight this time) and lower stamina. I think it's possible to increase stamina by attending gym but it will be more effective to choose some more 'dynamic' physical loads. So, I've decided to by a bike. I had a bike when I was a child, it was a very simple bike, but I really enjoyed it, so I thought that bike is a great tool to get dynamic loads and increase stamina. Norco Bigfoot has became my bike of choice: it's pretty durable like most Norco bikes, has nice breaks and other stuff. First impression: it feels pretty good. My area is a slight downhill (or uphill, depends on how you look at it ;]). I prefer to start uphill, so as soon as I get tired I could just ride back. Currently, I can do about 2-3 km uphill and then my legs get tired I have to move back. I guess it's a payment for my not balanced lifestyle during last few years. Anyway, I think I will ride 3 or 4 times a week and will increase the distance a little each time. Will see how it will go. 2009-04-01T22:41:00.000-07:00 Anthill Pro CI server is a really strage thing. The tutorial is very lengthy, there are sooo much various links and buttons and menus in anthillpro UI so I still cannot figure out how to add a simple project. Seems waaayyyy too complicated. Maybe it's very powerful, however. Anyways. Hudson seemed much easier to understand and manage. my weekend of debugging 2009-03-22T11:57:00.000-07:00 I'm starting to learn C++. I've read 2 or 3 basic-level C++ books (really basic ones) and then decided to write some simple app. People who are more experienced with C++ suggest me to read books like EffC++ or smth like that and I will do it a bit later (as soon as I buy them) but I don't see how it should stop my experiments. About the app: it's a really basic one, a bit more than 2000 lines on code and about 10-15 classes, but it's the biggest thing I've ever done in C++. The app has a simple configuration file and uses libconfig to parse it. There's a wrapper class for libconfig I wrote to make it more simple. Last week I decided to convert it to singleton pattern and that I've started to see weird behavior: when I was reading the same config property more then once, it started to get corrupted (I was dumping config file after each read to stdout for debug purposes). I decided that I've somehow triggered some hidden libconfig problem so I wrote a simple standalone app to reproduce the problem. To my surprise, my standalone app was working like a charm. Then I've decided to try to debug this issue with gdb. The only command I've used in gdb before is 'bt' and really it was enough for me to catch problems in my or other people' C programs. So I learned how to set breakpoints and add watches, learned how to run apps in step-through mode and maybe other stuff. It turned out to be quite convenient and simple to use gdb so I'll use it in feature instead of using 'printf-technique' =) But, alas, gdb didn't help. I've stepped through the program, stepped through libconfig but it didn't help, I didn't noticed what's the reason of segfault. I've remembered that there's an app called valgrind and people say it's extremely useful. As I'm running FreeBSD, I looked for it in the ports collection. But, alas, valgrind port runs only on FreeBSD <8 (and I run -CURRENT) and only on i386 (I have amd64 here). So I had to ask a person using linux to help and he provided me an output quickly. I've read "getting started" guide on valgrind site and figured out what's wrong with my app in 5 minutes (it was a malloc problem, but in totally unrelated class o_O). So I've spent about 2 days and then fixed the bug in 5 minutes. The lessons learned are: 1. gdb is a nice and simple tool easy to use and it's better than "printf-technique", so it worth to read some basic guide on gdb 2. valgrid seem to be a powerful tool and it looks like if some non-obvious problem occurs it makes sense to try it's memcheck tool first and then go to gdb or other stuff. It' sad, however, that it doesn't work on FreeBSD/amd64 -CURRENT. I'm not sure how to workaround this issue since I don't want to change os on my desktop. OTOH, I have an i386 laptop, so I'll probably install FreeBSD/i386 7.0 there or maybe some Linux. first post, yay 2009-03-21T10:43:00.000-07:00 So I've created _another_ blog. I've had (and still have) a twitter account and realized it's fun to have such a thing because you can write various stuff which is only interesting for you and at the same time train an English. But twitter is not the best service for it because it limits post length to 140 chars or something like that, which is obviously not enough for a good rant. Other funny thing about twitter is that it looks like companies follow twitter results, probably they subscribe to search results on google, or something like that, but once you write something like "$product suck!" somebody who represents "$product" starts following you and I don't feel like I like it. So let's see what's going on here.