Quality IMAP, Part 1: Your Mailbox Is A Database
37 points by gasche
37 points by gasche
Author of the post here, happy to answer questions!
Have you ever had fantasies about putting the people who wrote RFCs 9051 and 822 into Jigsaw trap type situation?
RFC822 has been obsoleted by RFC5322 which is much better ;-).
Jokes aside, as cryptic as these specifications can sometimes be, I think it is quite remarkable that it is feasible to implement them today with few resources (we're not a big company or anything), with a modern design and implementation techniques, and this way be compatible with billions of devices, decades of emails, and an ecosystem used day-to-day all around the globe.
This is not something that we can say for a lot of the technology that we see around these days. (And that's why I think there is still value in sticking with these old protocols even when they are imperfect.)
No mention of imap-codec in the post. (also nlnet funded). Are you rolling your own IMAP encoding/decoding? Haven't taken a look at the code yet.
We do use imap-codec in Aerogramme, and it has been working perfectly for us! As far as I know it has always just worked and allowed us to focus on other things than IMAP parsing, which is great.
Makes sense! I expected reading about IMAP parsing in your blogpost, but if you were using imap-codec from the beginning you shouldn't have the same pain as others who roll their own do :)
btw, if you have a test instance somewhere, I'd love to verify my email client against your implementation. I have contact info on my lobste.rs profile
it would be great to test meli with aerogramme. I'm currently fixing issues with our test deployment, but will send you a message with details as soon as I have something working!
IMAP is complicated stuff. For my own implementation of an IMAP server, I've written down some gotchas a while ago which you might find helpful https://raphting.dev/imap-is-indeed-complicated/
The authors explain that the Dovecot testsuite helped them implement their own program. I wonder if their own efforts could in turn generate a cross-implementation testsuite that would be shared with other authors of email clients. This is tricky because the fuzzing effort concentrates on their internal AST structure for email (that it can be printed and reparsed correctly), and different implementations will use different internal ASTs for emails.
Ideally there would exist a sort of standard/conventional format for "structured email addresses" that is an easy-to-parse sort of AST for email, independent of a specific library's implementation. Then the authors (or really anyone) could use their fuzzing testsuite to generate streams of (parsed email, structured email) pairs, that could be used for testing by any author of an email library -- assuming they have a parser from structured emails to their own internal AST.
I was experimenting with generating fuzz input from the abnf for addresses, headers, email, last weekend.
Without a corpus to preseed your fuzzing, you get nonsense like this which is useful for edge cases but not reflective of real-world date:
"From:,,,<\"\\ \\6#W\\=\\ \".*-{*|'S$&.\"\".${*!&?.\"\".\"\\\r!*\\ ~!k\\%!n\\q\\8!\\ !\".&-~`~'{''.\"\\Q\\9!g!\"@[i]>,((%\\ \\Uq((\\ I{\u{b}\\A($T=b\\ v\\ ))\\f((H?\\\n)(!\\ |\\2\\\n&)\\ H)O\\ )(\\[&((\\ ((\\ \\b.b(((\\Ydv\u{c}i\\X3<\\=)((8\\\r)\\'\\ J)\\:'R\\')S(!\\ (%)(((y/e((\\O\\e(5( \\\t()($((\\ )(\u{7f}(\\'(%(\\{#!\\ ((\\\n)q(b(((8((\\ (\\v(\\ /)\\ (()\\ \\>Io \r\n )))\\ '\\ !(\\5\\ #T\"\\Uz(\\ \\R(\\\"(v\\+(\\ %&\\E(\\S)(\\ {O\\ J\\9n((($\\\r(%)\\E({)!]#)\\ (()\\ E+\\Zq)\\ (6,'()\\\t( &(:(\\]\\\t&\\ )t\\\r\\ \\\t\u{b}()(((((\\@)B(\\ \\|R:)(\\\n%)\\ (\\ \t\r\n n))\\ (Y\\ \\~\\P((\"(\\ \\D((\u{1d}Z$\\\n(P(((\u{7f}(z((\\\nyj\\,)\\ ))(d)&!((@L)(\\6\\ )\\ %!))\\ (*()((\\\t\\\t)\\1))Y&D(]\"\\(\\U\\:(]8y)P, \r\n (]\"\\M%(B((Y!\\D(\\ p\\E\\(7 (((((5)(($3(((u)\\a\\ \\F():((;\\J(|\\\n#(/\\ )(((\\O(((\\?\\G)\\ ))()\\ g(x\\%\\ \\R!:)\\[(i!'((j(h(\\ (\\M\\d/(\\ (\u{e}(R\\ \\D\\ (((W'A!))(\\\u{4}\\ (\\ )\\ \\'\\\r\\\0(%\\#}\\>(E\\ \\ \\ \"\\\")((()I.((\\U\\R$)(\\ \\ )()`)\\ (`(|\\q)('\\ \\ (\u{b}(($\\&()\\ \\{(()a\\ 2 \r\n (\\y\\\t\\\n\\\n\\F\\ \\G\\T)($(o*)\\ )\\ )\\9w\\ m\\T(((\\\r()j((q \\i4%\\\0%(\\\0\\U(}u\")\\\t \r\n \\ (((\\ \\\t^9\\ \\X\\ O*)\\C()\\|))(\\i#"
or
"Date:Sat, 09 Jun \r\n\t 4110 \r\n 47:54MDT\r\n"
I think it's a worthy endeavor. A public test data repository is valuable. The problem for sourcing corpuses is that most of public email datasets are very homogeneous. For example public mailing lists will only have plain text, not much variety in addresses, etc.
I had not heard of aerogramme. Fascinating how there seems to be a flurry of activity around new email (servers, clients, etc.). Great write up by the way!!!