[neomutt-users] Figuring out IMAP w/O365, xoauth2, and pizauth

[Andrew Sullivan]

Hi,

I've made a few desultory postings about this before, but I don't seem to make any progress debugging it, so I thought I'd try to summarize in one place.  Note that I am treating all these issues as potentially related, but they may not be.

I have neomutt configured to connect to more than one account.  To achieve this, I use various hooks to configure the variables appropriate to that account when changing mailboxes.  I use it on MacOS.

I access both mailboxes via IMAP and (SMTP) submission.  The first is a bog-standard dovecot+postfix configuration that currently still uses password-based login.  That mostly works flawlessly.  The other is an Office365 Exchange service using the O365 implementation of IMAP.  Authentication is via xoauth2.

I tried to use the mutt_oauth2.py mechanism recommended in the manual, but while I was able to get it to work on its own it seemed to present a lot of trouble once I tried to integrate it with all the hooks for account switching.  Fortunately, I found pizauth (https://github.com/ltratt/pizauth), which seems to be working for me.  But there are a few issues.

First, it seems that neomutt interacts badly with the system going to sleep while in a mailbox index.  It appears that the sleep mode disconnects the network connection, and when reawakened neomutt doesn't know what to do with the now-defunct connection it had.  If there were no mails marked deleted but where the mailbox is disconnected, then _usually_ one can get reconnected to the dovecot-hosted mailbox by cycling through accounts.  Neomutt tries to use the connection, fails, and then authenticates again.  This is about 50% reliable, also, on the O365 connection: the rest of the time, it claims to reauthenticate but there are no mails in the relevant inbox, and you have to restart neomutt to get back to a functioning mailbox.

If the mailbox had marked-deleted but unexpunged messages, however, I get stuck: trying to leave the affected mailbox results in an expunge confirmation, and then since the connection is dead the expunge fails and I get into a confirm-expunge, fail loop (it doesn't matter whether I say yes or no to the expunge question).

The IMAP connection to O365 also seems to be flaky.  It disconnects from time to time without any real explanation.  I have tried fiddling with the timeouts on both the IMAP and pizauth systems, and none of this seems to matter.

In both of the latter cases (the expunge-loop problem and IMAP confused on O365), I often find I have to eXit in a forced way.  Perhaps 30% of the time when I do that, neomutt reports a ssegfault.  I won't dump the entire system log report in here, but messages tend to look like this:

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000008
Exception Codes:       0x0000000000000001, 0x0000000000000008
Exception Note:        EXC_CORPSE_NOTIFY

VM Region Info: 0x8 is not in any region.  Bytes before following region: 105553518919672
       REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
       UNUSED SPACE AT START
--->  
       MALLOC_NANO (reserved)   600018000000-600020000000 [128.0M] rw-/rwx SM=NUL  ...(unallocated)

If there is more that I can do to try to help diagnose this issue (these issues?), I'm happy to try.  Previous efforts have not led me to anything conclusive, but I guess I'm vaguely hopeful that a segfault might provide enough info to suggest what's happening.

Thanks again to the developers for the work on this.

System details:

% neomutt -v
NeoMutt 20230407
Copyright (C) 1996-2022 Michael R. Elkins and others.
NeoMutt comes with ABSOLUTELY NO WARRANTY; for details type 'neomutt -vv'.
NeoMutt is free software, and you are welcome to redistribute it
under certain conditions; type 'neomutt -vv' for details.

System: Darwin 21.6.0 (arm64)
ncurses: ncurses 6.4.20221231 (compiled with 6.4.20221231)
libiconv: 1.11
libidn2: 2.3.4 (compiled with 2.3.4)
GPGME: 1.19.0
OpenSSL: OpenSSL 1.1.1t  7 Feb 2023
libnotmuch: 5.6.0
PCRE2: 10.42 2022-12-11
storage: tokyocabinet, lmdb
compression: zlib

Configure options: --prefix=/opt/homebrew/Cellar/neomutt/20230407 --sysconfdir=/opt/homebrew/etc --autocrypt --gss --disable-idn --idn2 --lmdb --nls --notmuch --pcre2 --sasl --sqlite --tokyocabinet --zlib --with-lua=/opt/homebrew/opt/lua --with-ncurses=/opt/homebrew/opt/ncurses --with-ssl=/opt/homebrew/opt/openssl at 1.1 --with-sqlite=/opt/homebrew/opt/sqlite

Compilation CFLAGS: -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D__EXTENSIONS__ -D_XOPEN_SOURCE_EXTENDED -I/opt/homebrew/opt/libgpg-error/include -I/opt/homebrew/Cellar/libassuan/2.5.5/include -I/opt/homebrew/Cellar/gpgme/1.19.0/include -I/opt/homebrew/Cellar/neomutt/20230407/include -I/opt/homebrew/include/lua -I/opt/homebrew/Cellar/neomutt/20230407/include -I/opt/homebrew/Cellar/neomutt/20230407/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/Cellar/neomutt/20230407/include -DNCURSES_WIDECHAR -I/opt/homebrew/Cellar/neomutt/20230407/include -I/opt/homebrew/Cellar/openssl at 1.1/1.1.1t/include -I/opt/homebrew/Cellar/libidn2/2.3.4_1/include -I/opt/homebrew/Cellar/pcre2/10.42/include -I/opt/homebrew/Cellar/neomutt/20230407/include -I/opt/homebrew/Cellar/tokyo-cabinet/1.4.48/include -I/opt/homebrew/Cellar/neomutt/20230407/include

Default options:
   +attach_headers_color +compose_to_sender +compress +cond_date +debug 
   +encrypt_to_self +forgotten_attachments +forwref +ifdef +imap +index_color 
   +initials +limit_current_thread +multiple_fcc +nested_if +new_mail +nntp +pop 
   +progress +quasi_delete +regcomp +reply_with_xorig +sensible_browser +sidebar 
   +skip_quoted +smtp +status_color +timeout +tls_sni +trash 

Compile options:
   +autocrypt +fcntl -flock -fmemopen +futimens +getaddrinfo -gnutls +gpgme 
   -gsasl +gss +hcache -homespool +idn -inotify -locales_hack +lua -mixmaster 
   +nls +notmuch +openssl +pcre2 +pgp +sasl +smime +sqlite +sun_attachment 

MAILPATH="/var/mail"
PKGDATADIR="/opt/homebrew/Cellar/neomutt/20230407/share/neomutt"
SENDMAIL="/usr/sbin/sendmail"
SYSCONFDIR="/opt/homebrew/etc"



-- 
Andrew Sullivan
ajs at anvilwalrusden.com