imapsync/FAQ.d/FAQ.Gmail.txt

#!/bin/cat
$Id: FAQ.Gmail.txt,v 1.33 2017/09/05 15:11:20 gilles Exp gilles $

This documentation is also at http://imapsync.lamiral.info/#doc

=======================================================================
              Imapsync tips for Gmail accounts.
=======================================================================

Questions anwswered in this FAQ are:

Q. Can I use imapsync to transfer from or to Gmail accounts?

Q. How many days does it take to transfer X GB?

Q. How to synchronize from Gmail to Gmail?

Q. How to synchronize from XXX to Gmail?

Q. How to synchronize from Gmail to XXX?

Q. I can't authenticate with Gmail via IMAP
   and Gmail says "Please log in via your web browser"

Q. Can not open imap connection on [imap.gmail.com]
   Unable to connect to imap.gmail.com

Q. Can I safely use --useuid for Gmail transfers?

Q. Gmail does not really delete messages in folder [Gmail]/All Mail
   What happens? What can I do?

Q. Can I use the Extension of the SEARCH command: X-GM-RAW described at
   https://support.google.com/mail/answer/7190?hl=en
   https://developers.google.com/gmail/imap_extensions#extension_of_the_search_command_x-gm-raw

Q. How to avoid the [IMAP] prefix on Gmail side?

Q. Does imapsync have the capability to do 2 stage authentication?

Q. How to use XOAUTH2 to globally authenticate gmail users?

Q. How to use XOAUTH to globally authenticate gmail users?

Q. How to use a Gmail account to backup several different imap accounts?

Q. How to migrate email from gmail to google apps?

=======================================================================
Q. Can I use imapsync to transfer from or to Gmail accounts?

R. Yes. But IMAP access to a Gmail account is not allowed by default so 
it has to be allowed in the Gmail configuration part:
 -> Settings
 -> Forwarding and POP/IMAP
 -> IMAP Access
 -> Enable IMAP

=======================================================================
Q. How many days does it take to transfer X GB?

R. Basically it takes X days to transfer X GB per account.
   Gmail has usage limits per day and use throttlers when
   they are overtaken
   http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
   From the previous link:

   * it's 2X days to upload  X GB to Gmail, it's why I suggest to add
   --maxbytespersecond 10000
   for uploading messages to Gmail
   * it's X/2 days to download X BG from Gmail, it's why I suggest to add
   --maxbytespersecond 20000
   for downloading messages from Gmail
  
   That's theoretical values that always work in practice. Try
   upper values and see if they still work.

   How Gmail says limits are reached?
   This is either a disconnection with
   "BYE Session expired, please login again"
   or a very small rate, less than 1 Kib/s
=======================================================================
Q. How to synchronize from Gmail to Gmail?

R. Use the following example:

./imapsync \
           --host1 imap.gmail.com \
           --ssl1 \
           --user1 account1@gmail.com \
           --password1 gmailsecret1 \
           --host2 imap.gmail.com \
           --ssl2 \
           --user2 account2@gmail.com \
           --password2 gmailsecret2 \
           --maxbytespersecond 10000 \
           --automap \
           --exclude "\[Gmail\]$"   


Explanations:

--ssl1 --ssl2 are mandatory since Gmail only supports 
imap ssl connections.

--maxbytespersecond 10000 ( 10 kBytes/s ) option is here to avoid 
locking or errors when imap transfers exceed Gmail maximum limit. 
See http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
--maxbytespersecond 10000 is not mandatory in the sense Gmail may 
allow you to use an upper value than 1 GBytes per 24h without 
disconnections.


--automap is not mandatory but it's a feature to automatically
map folder names based on the Gmail user configuration itself, par account.
It will save manual folder names changes or the use of --regextrans2 
or --f1f2 to map folder names.
For example, imap folder "[Gmail]/Sent Mail" may be mapped as one of
E-mails enviados
Enviada
Enviado
Gesendet
G<EFBFBD>nderildi
Inviati
Sendt
Skickat
Verzonden
etc.
on both sides, host1 or host2, maybe differently, sometimes in 
incomprehensible alphabets, a headache for imap sysadmins.
See a listing here:
http://stackoverflow.com/questions/2185391/localized-gmail-imap-folders/2185548#2185548

--exclude "\[Gmail\]$"   is just there to avoid a warning error
when selecting this not used folder.

=======================================================================


=======================================================================
Q. How to synchronize from XXX to Gmail?

R. Use the following example:

imapsync --host1 mail.oldhost.com     \
         --user1 my_email@oldhost.com \
         --password1 password   \
         --host2 imap.gmail.com \
         --user2 my_email@gmail.com \
         --password2 password \
         --ssl2 \
         --maxbytespersecond 10000 \
         --maxsize 25000000 \
         --automap   \
         --expunge1  \
         --addheader \
         --exclude "\[Gmail\]$"      \
         --regextrans2 "s/[ ]+/_/g"  \
         --regextrans2 "s/['\^\"\\\\]/_/g"

Explanations:

--ssl2 is mandatory since Gmail only supports imap ssl connections.

--maxbytespersecond 10000 ( 10 kBytes/s ) option is here to avoid 
locking or errors when imap transfers exceed Gmail maximum limit. 
See http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
--maxbytespersecond 10000 is not mandatory in the sense Gmail may 
allow you to use an upper value than 1 GBytes per 24h without 
disconnections.

--maxsize 25000000 is mandatory since Gmail limits messages size
up to 25 MB. This value increases over time, it was 10 MB some 
years ago so you can try higher values. The Gmail page about
this limit is https://support.google.com/mail/answer/6584

--automap is optional but it will save manual folder names
changes or the use of --regextrans2 or --f1f2 to map folder names.

--expunge1 is optional. It deletes messages marked \Deleted on host1.
Imapsync syncs messages with all their flags, Gmail takes the messages
marked \Deleted but deletes or moves them just after.
Option --expunge1 really removes messages marked \Deleted on host1 
so they are not synced at all.


The --addheader option is there because "Sent" folder messages 
sometimes lack the "Message-Id:" and "Received:" headers needed 
by imapsync to identify messages (only when --useuid is not used). 
So option --addheader adds a "Message-Id" header consisting of
the imap UID of the message on the host1 folder, like
"Message-Id: 12345@imapsync".

--exclude "\[Gmail\]$" is there to avoid a small examine/select error:
"Could not examine: 43 NO [NONEXISTENT] Unknown Mailbox: [Gmail] 
(now in authenticated state) (Failure)".

--regextrans2 "s/[ ]+/_/g" is there to convert blank characters not 
accepted by gmail to character _ underscore. In fact only leading
and trailing blank characters are problems with gmail, and
also successive blanks ending with the IMAP error 
"NO [CANNOT] Folder contains excess whitespace (Failure)"
If you want to change only leading and trailing blank characters
then use the following instead
On Linux/Unix:
  --regextrans2 "s,(/|^) +,\$1,g" --regextrans2 "s, +(/|$),\$1,g"
On Windows:
  --regextrans2 "s,(/|^) +,$1,g" --regextrans2 "s, +(/|$),$1,g"


--regextrans2 "s/[\^]/_/g" is mandatory.  It converts, since
not accepted by gmail, character ^ to character _ underscore.

--regextrans2 "s/['\"\\\\]/_/g" is optional. It converts 
characters ' or " or \ to character _ underscore.

You can select folders exported to imap within the gmail preferences.
Select or unselect some "System labels", depending on your needs.


=======================================================================
Q. How to synchronize from Gmail to XXX?

R. Use this example:

./imapsync \
           --host1 imap.gmail.com \
           --user1 gilles.lamiral@gmail.com \
           --password1 gmailsecret \
           --host2 localhost \
           --user2 tata \
           --password2 tatasecret \
           --ssl1 \
           --maxbytespersecond 20000 \
           --useheader="X-Gmail-Received" \
           --useheader "Message-Id" \
           --automap \
           --regextrans2 "s,\[Gmail\].,," \
           --skipcrossduplicates \
           --folderfirst "Work" \
           --folderfirst "Friends" \
           --folderlast  "CanWait" \
           --folderlast  "[Gmail]/All Mail"


Explanations:

--ssl1 is mandatory since Gmail only supports imap ssl connections.

--maxbytespersecond 20000 ( 20 kBytes/s ) option is here to avoid 
locking or errors when imap transfers exceed Gmail maximum limit. 
See http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
--maxbytespersecond 20000 is not mandatory in the sense Gmail may 
allow you to use an upper value than 2 GBytes per 24h without 
disconnections.

--useheader="X-Gmail-Received" --useheader "Message-Id" are not
mandatory. I use them because I found (several years ago, it may
have changed) that Gmail always adds a different header 
"X-Gmail-Received:" to all messages it gets. So the identification
by imapsync can not fail using this header. "Message-Id" is there
for safety about this Gmail rule.

--automap is optional but it will save manual folder names
changes or the use of --regextrans2 to map folder names.

--regextrans2 "s,\[Gmail\].,,"
If your destination imap server doesn't like "[Gmail]" name, 
get rid of this "[Gmail]" part with that.

You can select folders exported to imap within the gmail preferences,
for example you may unselect all "System labels".

--skipcrossduplicates is optional but it can save Gigabytes of hard
disk memory. Within imap protocol, Gmail presents Gmail labels as 
folders, so a message labeled "Work" "ProjectX" "Urgent" ends up 
in three different imap folders "Work" "ProjectX" and "Urgent"
after an imap sync. --skipcrossduplicates prevent this behavior.

An issue with --skipcrossduplicates is that the first label synced
by imapsync goes to its corresponding folder and other labels are
ignored. This way, at least you can choose what labels have the 
priority by using the --folderfirst option. For example
--folderfirst "Work" will sync messages labeled "Work" before
messages labeled "CanWait" or "Urgent". By default imapsync 
syncs folders (Gmail labels) using the classical alphanumeric order.

--folderlast "CanWait" will sync only messages that have the
label CanWait and only it.

--folderlast "[Gmail]/All Mail", in conjunction with option 
--skipcrossduplicates, will only put in "[Gmail]/All Mail"
the messages that are not labeled at all.



=======================================================================
Q. I can't authenticate with Gmail via IMAP
   and Gmail says "Please log in via your web browser"

R0. My current settings on two Gmail account are made
with 0) 1) 2)

0) Login via a browser to the Gmail account concerned.

1) The 2-step verification is OFF
https://myaccount.google.com/security?hl=en&pli=1&nlr=1#signin

2) Access for less secure apps is turned ON
https://www.google.com/settings/security/lesssecureapps
https://support.google.com/accounts/answer/6010255?hl=en

R1. See Coert Grobbelaar solution:
https://web.archive.org/web/20150906230041/http://security.stackexchange.com/questions/86404/how-do-i-interact-with-google-to-import-email-via-imapsync

R2. I had the same issue one time (mars 2015) logging to Gmail 
with imapsync.
The Gmail imap message error said "Please log in via your web browser"
so I logged for this account via a web browser,
it asked me to receive a code via a mobile, I said yes,
I entered the code and everything went ok. 


=======================================================================
Q. Can not open imap connection on [imap.gmail.com]: 
   Unable to connect to imap.gmail.com

R0. It looks like this issue is related to ipv6. Both ipv4 and ipv6 
protocols should work with gmail and imapsync, I test that regularly, 
imapsync works fine for both ipv4 and ipv6. 
If you disable ipv6 then disable also ipv6 resolution! 

The default names resolution order is to present ipv6 name resolutions 
first. If you know how to make ipv4 answers be taken before ipv6 
then tell me.

R1. First solution, run imapsync with the option --inet4:

   imapsync ... --inet4

R2. A second solution is to use directly gmail ipv4 ip address:

  imapsync ... --host1 64.233.184.108 

In case it changes, get it with any command showing the imap.gmail.com 
name resolution, try one of those:

  nslookup imap.gmail.com
  host     imap.gmail.com
  ping     imap.gmail.com

Or go to http://ping.eu/nslookup/ to get the resolution.

Thanks to Chris Nolan to report, understand and fix this issue!

=======================================================================
Q. Can I safely use --useuid for Gmail transfers?

R. Yes, but I suggest to not use --useuid for Gmail transfers.

Using UIDs is useless with Gmail in the case of global duplicates
(duplicates across different folders). Gmail always accept a global
duplicate message as a new message, giving imapsync a new UID for this
message, and throw it away because it already has it. Gmail
will do this at each run so imapsync will always try to copy the
message, and Gmail will always accept and throw away the new copy. It
ends up with no duplicates on Gmail but a waste of bandwith and time,
which is the opposite goal of --usecache implied by --useuid.


=======================================================================
Q. Gmail does not really delete messages in folder [Gmail]/All Mail
   What happens? What can I do?

R. It's true and explained in Gmail documentation at
https://support.google.com/mail/answer/78755?hl=en

To really remove messages in folder "[Gmail]/All Mail", they have to
be moved to the "Trash" folder and be deleted from "Trash".

=======================================================================
Q. Can I use the Extension of the SEARCH command: X-GM-RAW described at
   https://support.google.com/mail/answer/7190?hl=en
   https://developers.google.com/gmail/imap_extensions#extension_of_the_search_command_x-gm-raw

R. Sure. Example, to search only emails with attachment and in unread state:

On Unix:

  imapsync ... --search 'X-GM-RAW "has:attachment in:unread"'

On Windows:

  imapsync.exe ... --search "X-GM-RAW ""has:attachment in:unread"""

=======================================================================
Q. How to avoid the [IMAP] prefix on Gmail side?
How to stop creating folder with this prefix?
Any switch we can use? e.g. [IMAP]/Archive

R. No switch in imapsync since [IMAP]/ prefix is done by Gmail,
   it might be configurable within Gmail parameters configuration.


=======================================================================
Q. Does imapsync have the capability to do 2 stage authentication?

R. No, imapsync doesn't support 2 stage authentication. 
Reading https://support.google.com/mail/answer/1173270?hl=en
it looks like it can't because imapsync uses imap protocol.

So you have to follow the Google recommendation and generate an 
application-specific password or normal authentication 
or use XOAUTH or XOAUTH2.

=======================================================================
Q. How to use XOAUTH2 to globally authenticate gmail users?

R. Yes, but really tested on Unix systems, not sure on Windows. See:
   http://imapsync.lamiral.info/FAQ.d/FAQ.XOAUTH2.txt

=======================================================================
Q. How to use XOAUTH to globally authenticate gmail users?

R0. XOAUTH is considered obsolete and superseded by XOAUTH2
    See http://imapsync.lamiral.info/FAQ.d/FAQ.XOAUTH2.txt

=======================================================================
Q. How to use a Gmail account to backup several different imap accounts?

R. For each account named xxx use:

  imapsync ... --subfolder2 xxx/xxx

It syncs the account xxx under a sub-subfolder xxx/xxx. This way there
is no supplementary label created on the multi-archive Gmail 
destination account. No labels all over the place and all original
xxx sub-folders show up nested within xxx/xxx.

=======================================================================
Q. How to migrate email from gmail to google apps?

R. Take a look at:
http://www.linux-france.org/prj/imapsync_list/msg00639.html

http://biasecurities.com/2009/02/migrate-email-from-gmail-to-google-apps/
http://www.thamtech.com/blog/2008/03/29/gmail-to-google-apps-email-migration/

=======================================================================
=======================================================================