Mutt is Slow

Recently it started bothering me that Mutt is very slow.

I use it with IMAP and I have a number of mailboxes there, usually 1 mailbox per a mailing list. Each of these mail boxes contains relatively large amount of messages, for example, in FreeBSD's svn-ports-all mailbox I currently have 170.000+ messages.

It takes quite some time to open such a mailbox. The first measurements showed approx. 80-90 seconds. And it's not only for the first pass, it happens every time when I open the mailbox.

I noticed that mutt is using bdb to store the headers cache and decided to try the other backend: tokyo cabinet. It helped to improve things a little, but not significantly, only down to 60 seconds.

The other thing I tried is to use per-mailbox header cache file instead of a common one. If you set header_cache in your ~/.muttrc to a directory, it'll use per-mailbox files, otherwise, if it points to a file, it'd be a single cache. I didn't notice any performance changes after this though.

I was wondering if using some sort of memory-based filesystem would improve performance, but, unfortunately, the header cache size is approx. 2.5G and my total RAM is 4G, so it doesn't look like it'd make any sense.

I've made a call graph to get a better picture of what's going on:

Looking at this picture, it appears that the most time consuming operations of the mailbox opening are:

• Iterating through IMAP server response (15.19% out of 74.64%)
• Checking cache (33.32% out of 74.64%)
• Sorting headers (16.6% out of 74.64%)

So it looks like cache handling contribution to the overall time is not that huge as I thought, i.e. even if we remove that part completely, it'd still take tens of seconds to open mailbox and that's no good.

So, actually, I have an impression that the current design with blunt fetching of all the messages simply cannot cope with large amount of data.

Probably it should do something like that:

• Keep cache only for amount of messages visible on the screen (maybe x3-5 of that to be on the safe side, but still that would be under 1000)
• Display stuff from cache right away, then pull the newer messages and rebuild the view
• Lazy-load older messages when user is scrolling (not sure how doable that with ncurses though).

As even doing a PoC appears to be a quite time-consuming task, I'll probably take a look if there are some other terminal-based MUAs that could handle large amounts of email better.