Mutt is Slow

Recently it started bothering me that Mutt is very slow.

I use it with IMAP and I have a number of mailboxes there, usually 1 mailbox per a mailing list. Each of these mail boxes contains relatively large amount of messages, for example, in FreeBSD's svn-ports-all mailbox I currently have 170.000+ messages.

It takes quite some time to open such a mailbox. The first measurements showed approx. 80-90 seconds. And it's not only for the first pass, it happens every time when I open the mailbox.

I noticed that mutt is using bdb to store the headers cache and decided to try the other backend: tokyo cabinet. It helped to improve things a little, but not significantly, only down to 60 seconds.

The other thing I tried is to use per-mailbox header cache file instead of a common one. If you set header_cache in your ~/.muttrc to a directory, it'll use per-mailbox files, otherwise, if it points to a file, it'd be a single cache. I didn't notice any performance changes after this though.

I was wondering if using some sort of memory-based filesystem would improve performance, but, unfortunately, the header cache size is approx. 2.5G and my total RAM is 4G, so it doesn't look like it'd make any sense.

I've made a call graph to get a better picture of what's going on:

Looking at this picture, it appears that the most time consuming operations of the mailbox opening are:

• Iterating through IMAP server response (15.19% out of 74.64%)
• Checking cache (33.32% out of 74.64%)
• Sorting headers (16.6% out of 74.64%)

So it looks like cache handling contribution to the overall time is not that huge as I thought, i.e. even if we remove that part completely, it'd still take tens of seconds to open mailbox and that's no good.

So, actually, I have an impression that the current design with blunt fetching of all the messages simply cannot cope with large amount of data.

Probably it should do something like that:

• Keep cache only for amount of messages visible on the screen (maybe x3-5 of that to be on the safe side, but still that would be under 1000)
• Display stuff from cache right away, then pull the newer messages and rebuild the view
• Lazy-load older messages when user is scrolling (not sure how doable that with ncurses though).

As even doing a PoC appears to be a quite time-consuming task, I'll probably take a look if there are some other terminal-based MUAs that could handle large amounts of email better.

Introducing goocaa, or on the way to my perfect mutt setup

I've already posted a guide how I configured mutt over gmail to leverage some of gmail useful things, esp. address completion. I've used goobook for query_command. However, I don't quite like its stability, the way of its configuration and generally I prefer to avoid using software written in Python (or Ruby or any interpreted language) on my workstations.

So I wrote a small tool called goocaa (click on the link to check its github page). It's written in C and uses libxml2, glib2 and neon.

It uses ~/.goocaarc configuration file and its schema looks this way:


[default]
email = me@gmail.com
passwd = lalala


To start using it, just add:

set query_command = "/usr/local/bin/goocaa %s"

to ~/.muttrc [1].

One more thing that I've implemented is multi-account support. I have my work mail hosted at google also, so I forward it to my gmail account and put to a separate mailbox. For this mailbox I apply special rules, like, set "mail from" to work address, etc. One more thing that makes sense to do is to search in work's contacts instead of gmail ones. It could be done with goocaa's multi-account support. First, add additional profile to ~/.goocaarc:


[default]
email = me@gmail.com
passwd = lalala

[mywork]
email = me@mywork.com
passwd = tatata


And now use folder hooks in ~/.muttrc:


folder-hook . 'set from = "me@gmail.com"; \
set query_command = "/usr/local/bin/goocaa -p default %s"

folder-hook mywork 'set from = "me@mywork.com"; \
set query_command = "/usr/local/bin/goocaa -p mywork %s"


All done! Now mutt will use your gmail contacts everywhere except 'mywork' folder where it will be using your work contacts.

1: documentation on mutt's $query_command

Using mutt with Gmail IMAP: complete guide

There are some guides around on the net, but I wasn't able to find a complete guide to make mutt as usable as gmail web interface, so I will try to put stuff together in one single post. I will also try to explain why each certain option is needed. Also, I don't pretend it to be the most optimal or correct way of configuration, but it works in practice. :-)

Basic IMAP configuration

So, the basic IMAP options follow:


set realname = "Your Name"
set imap_user = 'yourname@gmail.com'
set spoolfile = imaps://imap.gmail.com:993/INBOX
set smtp_url = "smtp://yourname@smtp.gmail.com:587/"
set folder = "imaps://imap.gmail.com:993"
set record="+[Gmail]/Sent Mail"
set postponed="+[Gmail]/Drafts"
set header_cache="~/.mutt/cache/headers"
set message_cachedir="~/.mutt/cache/bodies"
set certificate_file=~/.mutt/certificates


These lines are quite obvious and could be seen in a number of guides.

That's probably a minimal set of setting you need to be able to check your gmail mails using mutt. You might want to put your password to the config file in order to prevent mutt asking it all the time.

Ah, yeah, and I forgot one obvious thing that you probably know about anyway, the config file is ~/.muttrc.

sorting

Gmail was the first app I've used that sorted threads by the date of the last message in the thread. This is quite nice, because in the times when I used mutt with POP, I sorted thread by the first message, so if somebody sends a message to some ancient thread few screens of scrolling back I could easily miss it.

So, I've configured similar sorting in mutt, with the only difference that I prefer newer threads to be in the bottom, not on top.

So I've added the following lines to my ~/.muttrc:


set sort=threads
set sort_aux=last-date-received


The first line says that messages should be grouped in thread, the second one determines the order of the threads, so in our case we sort by newest message in the thread. You might want to consult muttrc(5) manpage for other sorting options.

Address book

When I've performed configuration steps described above, I've realized I really really really miss the built in address book of Gmail! The thing is that our company uses gmail to handle its mail and all the people who have an account automatically appear in the address book, which is quite handy and I've got used to this feature very very quickly.

So, in order to use the address book from mutt, query_command feature can be used. It allows to use an external application as address book.

There's a nice script which allows to be used as query_command in mutt, it's called goobook.

In order to install it, create some directory, e.g. ~/opt, cd into it and execute the following command:

svn checkout http://goobook.googlecode.com/svn/trunk/ goobook-read-only

Note: you will have to have subversion client installed.

Then cd to "~/opt/goobook-read-only" (or whatever dir you used), execute:

cp settings_example.py settings.py, then edit settings.py and fill in your gmail credentials. Now while you're in the working dir of goobook, check if it's configured correctly by executing something like:

./contactos.py e

Instead of 'e' you can type any letters which you sure will match somebody's name or address in your address book. If the script output something sensible then it works.

Now, teach the mutt to use this script by adding these lines to ~/.muttrc:


set query_command = '/home/user/opt/goobook-read-only/contactos.py "%s"'


Now, in mutt you can type 'Q' and type some query to make sure stuff is working correctly. If it works nice, then it means you can use address book when composing new messages. For example, if you hit 'm' to create a new message, in 'To:' prompt you can type some stuff and click 'ctrl-t' and mutt will complete stuff from the address book.

Yeah, one more thing about goobook. I had to insert empty 'print' right after arguments check in the contactos.py script as it seems that mutt use first line of the output to show in the status bar. So the scripts looks this way for me:


...

74 if __name__ == '__main__':
75 if len(sys.argv) < 2:
76 sys.exit(1)
77 print
78
79 try:

...


I haven't had time to look at mutt sources to investigate this problem, so I'm not sure if it's an expected behavior.

New messages info

What else... the other very important option is 'imap_check_subscribed'. To make mutt automatically add all the folders to mailboxes (so you can see a number of new/unread messages in each folder, new mail confirmation in different mailboxes etc), the following setting should be made:

set imap_check_subscribed=yes

If you're moving from Gmail or maybe some GUI MUA, you might want to see a sidebar with list of folders and mail counts. It's possible to implement in mutt as well, however it requires applying third-party patches, sidebar patch specifically. I don't provide instructions how to apply it as in FreeBSD it's as simple as building mail/mutt-devel port with "-DWITH_MUTT_SIDEBAR_PATCH" flag. Configuration is as simple as adding these lines to ~/.muttrc:


set sidebar_visible=yes
set sidebar_width=30


Summary

So, here's my complete ~/.muttrc file, only 50 lines long:


set realname = "yourname"
set imap_user = 'yourname@gmail.com'
set spoolfile = imaps://imap.gmail.com:993/INBOX
set smtp_url = "smtp://yourname@smtp.gmail.com:587/"
set folder = "imaps://imap.gmail.com:993"
set record="+[Gmail]/Sent Mail"
set postponed="+[Gmail]/Drafts"
set header_cache="~/.mutt/cache/headers"
set message_cachedir="~/.mutt/cache/bodies"
set certificate_file=~/.mutt/certificates
set imap_check_subscribed=yes
set move = no
set delete = yes
set edit_hdrs
set include
set reply_to
set abort_nosubject=no
set sig_dashes = no # don't append -- as I use only my name as a sig
set attribution = " %n wrote:\n" # I don't like lengthy attributions as well
alternates '(username|altusername)([-+].*)?@.*'
set hdr_format="%4C %Z %{%m/%d} %-15.15F (%4c) %s" # format of the index

ignore *
unignore from subject to cc date x-mailer x-url user-agent
hdr_order from to cc date subject x-mailer user-agent

set query_command = '~/opt/goobook-read-only/contactos.py "%s"'

set sort=threads
set sort_aux=last-date-received

# see beyond for hooks explanations
folder-hook . 'set from = "yourname@gmail.com"; \
set smtp_url= "smtp://yourname@smtp.gmail.com:587/"'

folder-hook my 'set from = "i@mywork.com"; \
set smtp_url = "smtp://i@mywork.com@smtp.gmail.com:587/";'

folder-hook foobar 'set from = "yo@example.org";'

# colors stuff outside
source ~/.mutt/themes/mytheme

# gpg
source ~/.mutt/gpg.rc
set crypt_autosign = yes

# sidebar
set sidebar_visible=yes
set sidebar_width=30


Few words about the hooks. I have two mail accounts, work and personal, both hosted at gmail. As I don't want to check both accounts, I've configured my work account to forward all the mail to my personal account and configured personal account to place all such mail into a separate folder.

So, the first default '.' hook just says that we generally use personal account. The second hook for 'mywork' says that when in 'mywork' folder, we change 'from' address to work address and use smtp from work account for sending (could be configured on gmail side as well to add working address for account, but I don't like this way for various reasons).

The third hook is for an alias at example.org which I use for some maillists, so I change 'from' address to it so maillist software allows me to write there. In this case I've added this address to gmail account, so no special 'smtp_url' magic is required.

So, it doesn't seem to be complex to configure mutt this way and I'm totally happy with such a configuration.

Update @ 24th, May 2010: I've wrote a replacement for goobook and switched to it instead: http://empt1e.blogspot.com/2010/05/introducing-goocaa-or-on-way-to-my.html