[neomutt-devel] Labels/Keywords/Tags framework in mutt [Was: Add support to IMAP custom keywords (#399)

guyzmo z+mutt+neomutt at m0g.net
Fri Feb 17 11:41:56 CET 2017

Ok, I'm pulling the discussion out of the issue #399, because I believe
we need to take a step back, where we need to have some design thought.
I agree with David that we need to factorise how different labels
schemes work, including notmuch.

Now the question that lies is how do we unify/factorise those tools,
while making it easy to take advantage of what they're good for.

We agree that keywords/labels/tags API should implement the following
minimal feature set:

• add one or more tags
• remove one or more tags
• fetch/show tags for a message
• lookup messages based on tags
 • using a filter (within the limit command)
 • by creating a virtual folder

We have several backends, including

• X-Label
• IMAP keywords
• notmuch
• "two other keywording standards in documented use" (please fill the blank ☺)

AFAICT, X-Label is a backend that overlaps with all. The other backends
are context-dependent, and thus mutually exclusive.

that being said.

On Thu, Feb 16, 2017 at 02:22:12PM -0800, David Champion wrote:
> * On 16 Feb 2017, guyzmo wrote: 
> > I don't get why you think there should be no differences between X-Label
> > and IMAP keyword? Those two features are two independant backends for
> > storing tags, and they can run in parallel.
> Why does the backend matter? This is just a storage format/access
> mechanism. It's still the same email message. If I apply keywords to an
> IMAP message and then save it to a Maildir, should I lose the keywords?
> I don't lose any other properties of the message when I do that.  Why
> are keywords different?

The great thing about X-Labels is that they stay with the message, and
thus can be used as a reflection of what has been stored in other

The bad thing about X-Labels is that they stay with the message, and
they get leaked in case an user forwards or bounces a message.

All other backends keep the labels separate from the message itself, and
they also are context-bound, notmuch works only with Maildirs, IMAP
labels only work with IMAP. Also, we can consider the case of readonly
accesses, that makes separate indexing relevant.

> Should I have separate search mechanisms for each keyword type in my
> mailbox?

For the search mechanism, different backend have different properties.

All the backends offer the similar core feature of filtering a loaded
mailbox using the limit command.

Notmuch is the fastest way to search, and it offers a search over labels,
but also over the headers and full-text. So you can make very fine
grained searches over your mailbox (with tags, text words, the from and
between two dates).

IMAP also has some constraints, IIRC, mostly on the allowed syntax with
also some special reserved keywords. You can also run searches on some
IMAP servers over headers, text and labels.

So for both IMAP and Notmuch, there's the possibility to create a
"virtual folder" (as the notmuch feature currently implements), that is
the result of a search. And that's a feature I'm not aware exists with

> There are two other keywording standards in documented use even for flat
> on-disk (non-IMAP) mailboxes.  Do we implement distinct feature sets for
> those?

I never heard of those, are they implemented in mutt?

> > The X-Label is to store tags within the mail's headers, whereas the IMAP
> > keywords is accessible only through an IMAP connection.
> As you say, that's a backend.  It's still functionally a keyword.

Indeed, but with different features, you might want different content.
One might want to store a common set of labels in all storages, and some
more in one storage. Some storages are mutually exclusive (like notmuch
and imap), whereas X-Label intersects with all. X-Label cannot work on
read-only mailboxes, notmuch can.

Also, mutating the mails to add labels can cause side-effects like
re-indexing by the indexer. I vaguely remember a discussion on the topic
that was I think on the notmuch mailing list a while ago. I'll try to
dig it up again.

> > So with your approach, you'd be introducing a form of redundancy between
> > the two backends. Also, AFAICT, the X-Label is being designed to accept
> > free form text, whereas the IMAP keywords are having a strict grammar,
> Not sure what you mean by redundancy.  My view is that I'm reducing
> redundant functionality by combining all these into a single feature
> set.  It should have been done long ago.

factorising functionality and combining are two different things. I
think the current discussion is sane, because we need to figure how to
factorise the code and the features with an approach that can cover all
the use cases.

> I implemented X-Label in mutt originally.  Nothing else had
> labels/tags/keywords, so there was no standard to follow, but the idea
> was very much to allow arbitrary, abstract tagging.  Now there are
> several standards, and it makes sense for mutt to play along with the
> rest of the world.  But nothing prevents continuing to have freeform
> text - we are just going to let you have more than one of them now.

I'm not advocating to change X-Label when used alone!

> > free form text, whereas the IMAP keywords are having a strict grammar,
> > and backend implementations can vary in how they behave. The IMAP
> > keywords is also in use by other MUAs (like Mail.app) to colourise the
> > mails.
> You might know more about how keywords are supported in the IMAP
> protocol than I do, but I haven't seen any indication that this can't
> work.

What can do the most can do the less, for sure. There just needs to be a
mechanism to validate the update of the IMAP keywords, and give feedback
to the user when a keyword can't be stored.

> > if it broke badly, it might be a good time to actually gather forces and
> > make a proper design that fits all the label tagging situations!
> My point of view is that we did that a long time ago.  If there are
> actual problems, of course I want to address them.  But I do not want a
> mail application with a separate subsystem for each approach to message
> keyword tagging.  That's not useful.

well, notmuch is not integrated in that model, you've worked alone on
adding IMAP support to your backend, and someone else submitted a PR for
a similar feature. So a long time ago, you've done a great X-Label
feature, but since then new stuff emerged, and need to be factored in.

We can regret the forking history of karel doing his own fork, but at
the same time, it's been a good way for him to experiment with fresh
eyes and implement a unique feature set. Now all we need is to
factorise it within a global keyword and vfolder featureset.



More information about the neomutt-devel mailing list