Where Man Wins over ... Gsuite

May 2018

For a recent (2024) update see here

Starting configuration: Alpine 2.00 patched and procmail bundled with openSUSE 11.3 (since ages)
Ending configuration: the same plus fetchmail
Works also: Alpine 2.21 patched etc. under openSUSE Leap 42.3 (Sep 2018)
Works also: Alpine 2.25 patched etc. under openSUSE Leap 42.3 (Jan 2024)

The case is something I would have gladly be dispensed with. Our institute has been happily using for e-mail a local sendmail MTA (Mail Transfer Agent) since the early '90s, and I have been using pine and its successor Alpine as MUA (Mail User Agent) since more or less the same time.
Actually my main and only "e-mail crisis" so far had been the transition from the IBM-mainframe based VM Rice mailer to pine in the moment we abandoned Bitnet (and the main alternatives were VAX VMS mail under Decnet, and mailx under Unix and TCP-IP Internet, with poorer user interfaces).

I always found pine and Alpine a close-to-ideal MUA for the following reasons:

I also found procmail a very powerful Mail Delivery Agent (default in our sendmail arrangement), which allows to do very efficiently mail filtering at delivery time. With this I did (and mostly still do) things like:
So you may wonder, if all this is so nicely working why are you telling us here and what you had to fight with the machine ?

I had to fight exactly because I wanted to retain all the above !

Our institution decide to move to an institution-wide solution based on Gsuite (the "commercial" implementation of gmail), which implies that:


What I want to achieve is the following: The solution for that (or part of it) is fetchmail, also a good piece of public, standard-compliant software which should be available on any Linux system.
A first issue (in case only that is desired) is accessing Gsuite incoming mail by Alpine.

Alpine supports a feature called "incoming folder collections". I.e. in addition to the normal INBOX (i.e. your /var/spool/mail/user folder if you receive locally on your machine) you can define a number of folders which are checked as the inbox (i.e. "the bell rings" when mail arrives). They can be remote IMAP or POP folders, or even local folders fed by procmail (I had already a couple of the latter, for INAF Internal Communication and for the INAF "discussioni" mailing list management, and a POP server on the Civic Network of Milan).

To add Gsuite inbox one has to:

If you do this your folder list screen (Alpine command L) will appear sort-of like this

Folder-Collection <Incoming-Folders>
------------------------------------------------------------
INBOX (14/64) INAF disc (0/4) ComIntern Gsuite(1/1) RCM (?)

You can now open Gsuite inbox as any normal folder, read mail, and save it anywhere in your other folders.

The only snag is that IMAP deletion by Google is faulty. If you Delete and eXpunge a message from the Gsuite inbox, it disappears from there but it remains in the All Mail Gsuite pseudo-folder and cannot be deleted unless one uses the Gsuite web mail interface.


A second issue (in case one wants deeper access to Gsuite via Alpine) is accessing other Gsuite "folders" by Alpine.

For this one has to define a "folder collection" for the Gsuite server. Such collection will include all other folders but the inbox (the inbox has to be managed as described above). Proceed as follows

If you do this your folder list screen (Alpine command L) will appear sort-of like this

Folder-Collection <All Gsuite>
--------------------------------------------------------------------------
All Mail[/] Bin[/] Drafts[/] Important[/] Sent Mail[/] Spam[/] Starred[/]

You can now open any Gsuite inbox as any normal folder, read mail, and save it anywhere in your other folders (local or even Gsuite's).
You can Delete and eXpunge messages in any folder but the mysterious All Mail pseudo-folder. EXpunge works immediately and does not save in the Gsuite dustbin (the Bin folder).

The snag is again that IMAP deletion by Google is faulty. If you Delete and eXpunge a message from the Gsuite All Mail pseudo-folder it is marked for deletion but cannot be expunged unless one uses the Gsuite web mail interface to delete it.
Conversely (at least with my Alpine configuration) if one moves a message from a Gsuite folder to another Gsuite folder, it is deleted and expunged from the origin folder and appears in the destination. Instead if one moves a message from a Gsuite folder to a local folder, it is not deleted.


All the above does not cure the issue of applying procmail filters to Gsuite incoming mail.

The solution to that is using fetchmail.

Fetchmail retrieves the content of a remote IMAP or POP folder, and feeds it into the local MTA as if it were mail received locally. This allows implicitly all delivery time filtering to work normally !

Fetchmail is intended to work as a daemon, and to poll periodically the remote servers.

Rumours are that one should not poll Gsuite servers more frequently than every 5 minutes otherwise one might be banned.

The daemon can be started either at system level (at boot), or at user level via a crontab, or at user level on the command line.

So what remains is to find the correct way to poll the Gsuite servers


A first unsatisfactory attempt is to use fetchmail with Gsuite IMAP.

I started with this entry in .procmailrc:

poll imap.gmail.com protocol imap port 993 username 'Name.Surname@inaf.it' password mypassword
and got a certificate error.

The reason is that fetchmail tries to get the certificates Google has and verify them with the certificating authorities' certificates. One can find on the network complicated ways to use SSL to get such certificates and combine them in a local PEM file.
But all this is unnecessary. At least in my case all necessary certificates are already stored in the system in /etc/ssl/certs. So just write in .procmailrc:

poll imap.gmail.com protocol imap port 993 username 'Name.Surname@inaf.it' password mypassword ssl sslcertpath /etc/ssl/certs

This way fetchmail works, it retrieves the content of the Gsuite inbox and submits it to local procmail filters. However this is not optimal for several reasons:

Note however that IMAP is the only way to let fetchmail access Gsuite folders other than the inbox. For instance if you'd ever want to retrieve Gsuite's spam use (note syntax):

poll imap.gmail.com protocol imap port 993 username 'Name.Surname@inaf.it' password mypassword folder "[Gmail]/Spam" ssl sslcertpath /etc/ssl/certs

The satisfactory way is to use fetchmail with Gsuite POP.

Gsuite allows concurrent use of POP and IMAP, so this solution allows to use POP for fetchmail and IMAP for Alpine at the same time.

The solution is close to perfect. All messages (read and unread) are fetched from Gsuite inbox and processed via procmail. However they are not fully deleted from the Gsuite server: they are moved to the Bin folder, where however they can be Deleted and eXpunged manually.

The smtphost yourhostname option can be added only to prevent innocuous error messages in the fetchmail log of the form
fetchmail: connection to localhost:smtp [::1/25] failed: Connection refused.
which are due to the fact fetchmail tries by default an IPV6 connection before an IPV4. If, as usual here, sendmail is not configured for IPV6, fetchmail generates the error and passes on to IPV4. Putting your host name there suppresses the error message. Fetchmail works normally in both cases.


The cherry on top is how to get rid of the stuff in Gsuite Bin using Alpine.

Of course one can go to the Bin folder, select all messages, delete and expunge them manually, but one wants to do it with a single keystroke.

In Alpine configuration (M S C) locate Key Definition Rules (this requires Alpine has been compiled with a special patch allowing user key definitions) and add a rule like this, exactly with this syntax, all commas and double quotes as shown.

"_SCREEN_ == {folder} _PKEY_ == {B} => _COMMAND_{M,L,W,"All Gsuite",^M,G,"Bin",^M,;,A,A,D,X}"

and now automagically from anywhere in Alpine you type L B (capital B) and that will go to Gsuite's dustbin and clear it !


Final notes and disclaimer:

All my tests were done accessing the Gsuite web mailer using Pale Moon 25.6 which forces use of the "basic HTML mode" because it is not one of the Gsuite supported browsers (a very bad concept anyhow). Once I'll have upgraded to Pale Moon 27 I may repeat some of them or fudge it to mimic a supported one.
The information provided in this page is for mere information and does not provide any guarantee express or implied that will work for anybody's configuration or purposes. Do your own tests at your own risk!

In particular my configuration which transfers mail asap from Gsuite to my machine is not suited for travellers, unless they set up a private IMAP server on their machine.


Addendum June 2018

Two more things ... one is another Google IMAP peculiarity. The other could be a "local" peculiarity.

The Google IMAP peculiarity is that also the "sent-mail" folder is somewhat special.
If I use Gmail's webmailer to send a mail for test to myself (or elsewhere), this is saved in the "sent-mail" folder (which cannot be disabled or changed apparently). Later, when fetchmail runs, it grabs also the content of the "sent-mail" and not just the inbox.
This can be an annoying "feature", generating duplicates, if I have sent test messages to myself.

The peculiarity occurs in our arrangement. Our machines are in a NIS networks, which includes an aliases map. Typically, all users had aliases of the form user: user@machine or user: user@imap-server. During the transition in which a temporary domain is used, aliases can be changed to user: user@g.institution (where g.institution is the Gsuite temporary domain). This routes messages addressed to old addresses here to the Gsuite inbox.
However when fetchmail attempts to download from Gsuite's inbox, the latter aliases feeds it back to Gsuite. So the message is moved from Gsuite's inbox to Gsuite's bin but never ingested !
It is necessary to maintain the alias user: user@machine to tell fetchmail that mail "terminates here" (on the machine where fetchmail runs).

Update January 2024

In December 2023 I have been informed that Gsuite would require 2-factor authentication (2FA) since end January 2024. A further annoyance ...

As expected enabling 2FA in Gsuite (which for me would imply as baseline receiving a code via SMS and entering it on the web, probably only for connections from home and just once from work, which uses a static IP) did cause problems to my configuration above. Remember that I want:

This is "broken" by 2FA because after enabling it, both fetchmail and alpine cannot any longer manage authentication.
Therefore I found a number of (redundant) workarounds (some were tested and implemented before enabling 2FA, some afterwards).

sax.iasf-milano.inaf.it/~lucio/WWW/WhereManWins/gs.html :: original creation 2018 nov 29 14:14:57 CET :: last edit 2024 Jan 22 12:30:09 CET