Where Man Wins over ... Gsuite

May 2018

Starting configuration: Alpine 2.00 patched and procmail bundled with openSUSE 11.3 (since ages)
Ending configuration: the same plus fetchmail
Works also: Alpine 2.21 patched etc. under openSUSE Leap 42.3 (Sep 2018)
Works also: Alpine 2.25 patched etc. under openSUSE Leap 42.3 (Jan 2024)

The case is something I would have gladly be dispensed with. Our institute has been happily using for e-mail a local sendmail MTA (Mail Transfer Agent) since the early '90s, and I have been using pine and its successor Alpine as MUA (Mail User Agent) since more or less the same time.
Actually my main and only "e-mail crisis" so far had been the transition from the IBM-mainframe based VM Rice mailer to pine in the moment we abandoned Bitnet (and the main alternatives were VAX VMS mail under Decnet, and mailx under Unix and TCP-IP Internet, with poorer user interfaces).

I always found pine and Alpine a close-to-ideal MUA for the following reasons:

• it is deeply and easily customizable editing a single .pinerc configuration file
• it is a fully compliant IMAP client, allowing to access different mail servers in a single instance
• it easily supports a lot of folders, including the good old native Unix folder format;
  in my current configuration I have 888 folders in 55 different folder collections
• inclusive of a very powerful aggregate operation which allows to select groups of messages according to various conditions and move or deal with them in a single go
• it has a nice addressbook handling, inclusive of the fact one can associate the folder where both ingoing and outgoing messages for one person go (Fcc):
  I always hated MUAs which store "sent mail" in a different place from "received mail"
• it nicely handles MIME digests in particular in conjuction with custom key configuration, for instance with a single keystroke
  • $ : i move a message to a "rejected spam" folder
  • S : I submit the spam folder to the local spamassassin Bayes learner
  • D : I split a MIME digest from a mailing list (I subscribe to mailing lists always in digest mode when possible, receiving one mail digest per day) into a temporary folder with individual messages
• it supports multiple roles so one can send as a specific address by choice, or according to the origin of the mail
• It has a nice colour handling for highlighting of screen parts and message classification
• it handles Usenet news in the same way as e-mail
• however it is a line-mode oriented application running in a terminal, which may have the following inconvenients:
  • it allows to deal with one message (received or being sent) at a time
  • it might not be ideal when one is travelling a lot (which I don't do any more), although this can be overcome in many ways.
  • One way is of course to ssh to your machine and use its local Alpine
  • Another one is to install Alpine on a laptop and point it to a local IMAP server on your machine (I used this combination in 2002 from ESO Paranal, and I was able to access and store all my correspondence at my home institute, as well as keep some local folders ... in that case I was able to figure out the ESO SMTP server to use, nowadays I'd need a SASL-authenticated session to my institute SMTP server).
  • Alternatively, once I've activated the local IMAP server (and my choice was for UW IMAP coming from the same developers as pine), I can access it with any webmail client ...
  • ... not by chance I choose Web Alpine which is a different story

• back in 1997 I wrote a procmail wrapper which received IAU Circulars (just starting to be distributed by e-mail) and distributed them to interested local staff as well as printing a lookalike of the old circulars sent by post.
• I route particular messages to particular folders, e.g. seminar announcements, or when I was dealing with INAF Internal Communication (this was in conjunction with an Alpine role).
• I route messages from system crontabs to particular "virtual" folders, and the same for mailing lists. Usually the virtual folder points to my own inbox but not necessarily. This has the advantage that when on holiday or away for a long period, I can repoint the virtual folder to /dev/null for unimportant messages, or to an offline store for those I want to read when back (so my inbox accessed remotely via IMAP/webmail is less cluttered)
• I can strip and munge messages with unwanted HTML duplicates or top posting
• I have my own second-level spam filtering beyond our greylisting and spamassassin. What survives can be unconditionally black- or whitelisted for specific addresses or stored in several levels of folders with different "spammosity"

• Finally I use formail (part of procmail suite) to split MIME digests in conjunction with the above mentioned Alpine key

Our institution decide to move to an institution-wide solution based on Gsuite (the "commercial" implementation of gmail), which implies that:

• all mail is received on Google mail server
• ... which of course can be accessed from anywhere with their web mailer (which means working their way
• ... which can also be accessed by any IMAP client, which however has to do with the more-or-less notorious non-standard implementations of IMAP by Google (for instance their concept of "labels" instead of "folders")
• but the main point is once mail is received on the server, it is already delivered and therefore all delivery-time filtering as the one done by procmail is no longer possible

• continue to use Alpine as my MUA
• keep mail stored on the "foreign" Google server for the minimum possibly time
• (except for spam, that can stay there)
• have mail processed by my set of procmail filters

The only snag is that IMAP deletion by Google is faulty. If you Delete and eXpunge a message from the Gsuite inbox, it disappears from there but it remains in the All Mail Gsuite pseudo-folder and cannot be deleted unless one uses the Gsuite web mail interface.

The snag is again that IMAP deletion by Google is faulty. If you Delete and eXpunge a message from the Gsuite All Mail pseudo-folder it is marked for deletion but cannot be expunged unless one uses the Gsuite web mail interface to delete it.
Conversely (at least with my Alpine configuration) if one moves a message from a Gsuite folder to another Gsuite folder, it is deleted and expunged from the origin folder and appears in the destination. Instead if one moves a message from a Gsuite folder to a local folder, it is not deleted.

• So far I used the third way. In the user mode, fetchmail reads a configuration file .fetchmailrc. Fetchmail has a rich man page and a FAQ on its site. Anyhow the basic configuration file can be as short as two lines (or even just the second one)
  set daemon 300
  poll server protocol xxx portxxx username xxx password xxx etc.

  fetchmail can be invoked in operation mode just typing

  fetchmail

  If it is not running, it will detach and remain there as a daemon and revive every 5 minutes.
  Otherwise re-invoking it will just awaken it ("background fetchmail at 9367 awakened" which will also tell you the pid, i.e. in this case 9367).

  Note that fetchmail -quit will not terminate fetchmail. It will kill the current background process, but restart a new one.
  The only way to terminate a fetchmail daemon is to kill it by pid with e.g.
  kill -9 9367 or
  killall fetchmail

• For testing purposes use fetchmail in non-daemon verbose mode running it once as
  fetchmail -vvv --nodetach --nosyslog

• The second way (user crontab) looks the preferred solution. It allows to run fetchmail in a single instance every five minutes, not leaving processes around.

  In this way .fetchmailrc shall contain only the "poll" line (no "set daemon" line).

  The crontab entry is

  */5 * * * * /usr/bin/fetchmail &> /dev/null

  Do NOT put --nodetach in the crontab invocation (the fetchmail process and its invoking shell will remain forever until killed, which is not what you want).
  Redirect output to /dev/null otherwise crontab will return fetchmail log messages in a mail every 5 minutes (this could be useful only if you want to see what's happening for debugging purposes)

So what remains is to find the correct way to poll the Gsuite servers

• fetchmail retrieves by default only the new mail in inbox, but not that which has been already read (by Gsuite webmailer or by Alpine IMAP or any other client).
  This can be corrected using fetchmail fetchall option.
• Gsuite does not actually delete moved messages but moves them to the All Mail pseudofolder.
• and we know already that All Mail content cannot be eXpunged other than using Gsuite webmailer.

Note however that IMAP is the only way to let fetchmail access Gsuite folders other than the inbox. For instance if you'd ever want to retrieve Gsuite's spam use (note syntax):

The solution is close to perfect. All messages (read and unread) are fetched from Gsuite inbox and processed via procmail. However they are not fully deleted from the Gsuite server: they are moved to the Bin folder, where however they can be Deleted and eXpunged manually.

The smtphost yourhostname option can be added only to prevent innocuous error messages in the fetchmail log of the form
fetchmail: connection to localhost:smtp [::1/25] failed: Connection refused.
which are due to the fact fetchmail tries by default an IPV6 connection before an IPV4. If, as usual here, sendmail is not configured for IPV6, fetchmail generates the error and passes on to IPV4. Putting your host name there suppresses the error message. Fetchmail works normally in both cases.

All my tests were done accessing the Gsuite web mailer using Pale Moon 25.6 which forces use of the "basic HTML mode" because it is not one of the Gsuite supported browsers (a very bad concept anyhow). Once I'll have upgraded to Pale Moon 27 I may repeat some of them or fudge it to mimic a supported one.
The information provided in this page is for mere information and does not provide any guarantee express or implied that will work for anybody's configuration or purposes. Do your own tests at your own risk!

In particular my configuration which transfers mail asap from Gsuite to my machine is not suited for travellers, unless they set up a private IMAP server on their machine.

Two more things ... one is another Google IMAP peculiarity. The other could be a "local" peculiarity.

The Google IMAP peculiarity is that also the "sent-mail" folder is somewhat special.
If I use Gmail's webmailer to send a mail for test to myself (or elsewhere), this is saved in the "sent-mail" folder (which cannot be disabled or changed apparently). Later, when fetchmail runs, it grabs also the content of the "sent-mail" and not just the inbox.
This can be an annoying "feature", generating duplicates, if I have sent test messages to myself.

The peculiarity occurs in our arrangement. Our machines are in a NIS networks, which includes an aliases map. Typically, all users had aliases of the form user: user@machine or user: user@imap-server. During the transition in which a temporary domain is used, aliases can be changed to user: user@g.institution (where g.institution is the Gsuite temporary domain). This routes messages addressed to old addresses here to the Gsuite inbox.
However when fetchmail attempts to download from Gsuite's inbox, the latter aliases feeds it back to Gsuite. So the message is moved from Gsuite's inbox to Gsuite's bin but never ingested !
It is necessary to maintain the alias user: user@machine to tell fetchmail that mail "terminates here" (on the machine where fetchmail runs).

Update January 2024

In December 2023 I have been informed that Gsuite would require 2-factor authentication (2FA) since end January 2024. A further annoyance ...

As expected enabling 2FA in Gsuite (which for me would imply as baseline receiving a code via SMS and entering it on the web, probably only for connections from home and just once from work, which uses a static IP) did cause problems to my configuration above. Remember that I want:

• Have my email stored on my work machine and not permanently on Gsuite's server
• Have my mail "delivered" automatically to my work machine, which is achieved using fetchmail, in order to have incoming mail processed by procmail.
• Note that procmail is used not just for my own sorting-in-folders, but also for service purposes (it is a back end to process Google forms for input in our database for students and associates).
• As auxiliary functions I may need sometime to check Gsuite's inbox in the 5-min interval when mail is transiting before being fetched. From Alpine.
• As further auxiliary function I may need typically once per day to access Gsuite's Spam and Bin (thrashcan) directories to check false positive spam, and to delete the leftovers of fetchmail (Gsuite does not delete fetched mail as instructed, but stores it in Bin). Also from Alpine.

• The first and simplest workaround is:
  • get an alternate (non-google) e-mail address, with proper IMAP handling
  • instruct Gsuite to forward all incoming mail to this alternate address and delete it
  • configure fetchmail to fetch from the alternate address
  • configure alpine to access the alternate address inbox, as well as the other alternate address folders
• This allows to use procmail as wished
• It also allows to use alpine to check mail "in transit before it's fetched"
• But this does not allow to clean Gsuite's Bin (unfortunately the forwarded-and-deleted messages are stored in Bin) or to check Gsuite's Spam, except using Gmail's web interface.

• A second workaround would be to allow fetchmail and alpine to access Gsuite's IMAP via OAUTH2. However this is not compatible with the versions I am using and would be rather awkward.
• A simpler variant is to instruct Gsuite to generate an "app password". This password can be used instead of the normal account password both with fetchmail and alpine.
• This effectively so far repristinates the wished behaviour.
  The forwarding to the alternate address would be no longer necessary, however I preferred to leave it active.

• At this point the only annoyance would be that alpine requires each time in a session the two passwords (the unobvious Gsuite's app password and the alternate address one).
• This could be overcome using an alpine version supporting password storage protected by a master password.
• So I retrieved the latest "all patches" alpine version (2.25), configured it --with-passfile ...
• ... defined a master password
• ... stored once forever the passwords of all IMAP and POP servers (and also SMTP for alternate roles) with the lighter encryption
• ... this way I enter once per session just the master password, and I can easily access all folders comfortably