imapsync/FAQ.d/FAQ.Office365.txt

$Id: FAQ.Office365.txt,v 1.33 2022/01/05 11:17:21 gilles Exp gilles $

This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Office365.txt


======================================================================
            Imapsync tips for Office365.
======================================================================

Questions answered in this FAQ are:

Q. Can I use imapsync to transfer from or to Office365 accounts?

Q. Can imapsync work for users that are administrators for 
an Office 365 domain? (quick answer: no)

Q. Does imapsync support OAUTH2 authentication for Office365 accounts?

Q. How to sync from Office365 to XXX?

Q. How to sync from XXX to Office365

Q. For Office365 I have double and triple-checked the username and
password spelling but I still get a "LOGIN failed". Any clue?

Q. How can I access an Office365 shared mailbox?

Q. How can I use a shared account as a backup account for several
mailboxes and so avoid spending too many dollars in backup accounts?

Q. Office365 fails with "User is authenticated but not connected".

Q. I see "NO Maximum size of appendable message has been exceeded"
   What can I do with that?

Q. The imap connection to Office365 is not working very efficiently,
   is there a solution to fix that?

Q. Office365 throttles the sync and says: 
   "Request is throttled. Suggested Backoff Time: 299961 milliseconds".
   What can I do with that?

Q. What are the receive and sending limits of Office365?

Q. The sync fails with many "Trying command when NOT connected!".
   What can I do?

Q. How to see or migrate public folders on Office365?
R. https://docs.microsoft.com/en-us/exchange/collaboration/public-folders/migrate-to-exchange-online?view=exchserver-2019
   https://www.exchangesavvy.com/moving-your-public-folders-to-office-365-what-you-need-to-know/

Q. Office365 refuses to create the folder named "Files" with the error
   "NO Folder name is reserved". What happens?

Q. Office365 users complain that a folder named "Files" contains 
   messages with no sender.

Q. From XXX to Office365, read receipts are all resent again after sync.
   Even for old messages. How can I fix that?

Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
   DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error   

Q. From XXX Office365 I get this error message sometimes: 
   "BAD Command Argument Error 11". What does it mean?

Q. From XXX to Office365 the flag Flagged does not seem to be well synced. 
   What can I do?

Q. How to migrate from or to Office 365 with an admin/authuser account?

Q. Couldn't create folder [trash] "Mailbox already exists".


Now the questions again with their answers.

======================================================================
Q. Can I use imapsync to transfer from or to Office365 accounts?

R0. Yes. But IMAP access to Office365 account is not always allowed 
by default so it has to be allowed in the server configuration part.
   
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365

R1. Imapsync doesn't support OAUTH2 authentication for Office365 
accounts yet. That's not totally true, if you have an access token
to access your mailbox then you can authenticate with imapsync this
way:

  imapsync ... --oauthaccesstoken1 tokenfile

where "tokenfile" is a file containing the access token.

R2. Enable double-step authentication and configure it but after use 
an "app password" with imapsync. Details:

a) Go to 
https://account.microsoft.com/security

b) Click on "Advanced Security Options"
Turn "Two-step verification" on. Follow the steps and finish".

c) Then now "App passwords" is available.
Click on "Create a new app password". 
Use this password to authenticate with imapsync.

d) Delete this app password when the job with imapsync is finished.

R3. Also, check a license is assigned to that account in Office365.

R4. Microsoft introduced something called "security defaults" which is enabled
by default for new tenants. One of the rules blocks IMAP access as of imapsync.

The funny thing is that you can't disable a single rule of this 
security package without buying additional licenses.
Switching the whole thing off allows the IMAP login.

Also, disable double-step authentication on the Azure/Active Directory portal.
See here:
https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults
Thanks to Stephan Buhre for this R4 answer.

R5. Are there special characters in the password?
    https://imapsync.lamiral.info/FAQ.d/FAQ.Authentication_failure.txt
    https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Windows.txt
    https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Unix.txt
    https://imapsync.lamiral.info/FAQ.d/FAQ.Passwords_on_Mac.txt

R6. Triple check the hostname then. Try all of these:
 * outlook.office365.com
 * imap-mail.outlook.com
 * imap.outlook.com

======================================================================
Q. Can imapsync work for users that are administrators for 
an Office 365 domain? (quick answer: no)

R. I doesn't seem possible to use imap for administrators, so imapsync
won't be able to work for an administrator's mailbox.
See:
https://github.com/imapsync/imapsync/issues/310#issuecomment-1002396218
https://exhaust.lewiscollard.com/post/146866104/office365-to-migadu-migration/

Solution: as explained in the article above, use Davmail as a proxy to 
access this mailbox. http://davmail.sourceforge.net/

Thanks to Lewis Collard for this report and solution.

======================================================================
Q. Does imapsync support OAUTH2 authentication for Office365 accounts?

R. Yes but partially. Imapsync won't help you getting an access token
but if you have one then you can use it with imapsync this way:

Office365 as source:

  imapsync ... --oauthaccesstoken1 tokenfile1

Office365 as destination:

  imapsync ... --oauthaccesstoken2 tokenfile2

======================================================================
Q. How to sync from Office365 to XXX?

R0. IMAP is not enabled by default on Office365, see how to enable it:
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365

If IMAP cannot be enabled or if it doesn't work well, you can try DavMail 
(http://davmail.sourceforge.net/) which acts as an IMAP gateway 
through Outlook Web Access.

R. On Windows, use:

  imapsync.exe ... --office1

On Unix, use:

  imapsync ... --office1

Option --office1 is like (release 1.970 or higher):
  
  imapsync ... --host1 outlook.office365.com \
               --ssl1 \
               --exclude "^Files$" 
  
======================================================================
Q. How to sync from XXX to Office365

R0. IMAP is not enabled by default on Office365, see how to enable it:
https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/enable-or-disable-pop3-or-imap4-access
https://docs.microsoft.com/en-us/exchange/troubleshoot/configure-mailboxes/pop3-imap-owa-activesync-office-365

If IMAP cannot be enabled or if it doesn't work well, you can try DavMail 
(http://davmail.sourceforge.net/) which acts as an IMAP gateway 
through Outlook Web Access.

R. Here is a command-line resume that solves most encountered issues when 
   migrating to Office365.

    imapsync ... --office2

which is equivalent to (in imapsync release 1.870 or higher):

  imapsync ... \
    --host2 outlook.office365.com \
    --ssl2 \
    --maxsize 45000000 \
    --maxmessagespersecond 4 \
    --disarmreadreceipts \
    --regexmess "s,(.{10239}),$1\r\n,g" \
    --f1f2 "Files=Files_renamed_by_imapsync"


On Linux, you can also try the "reformime" command 
that can be used like:

  imapsync ... --maxlinelengthcmd "reformime -r7"

To get reformime, install the "maildrop" package.

On Linux again, there is a good Python script in the tarball that can 
fix several things that Exchange or O365 have issues with.

Use it like this:

  ./imapsync ... --pipemess W/tools/fix_email_for_exchange.py

It often does some miracles on messages.

======================================================================
Q. For Office365 I have double and triple-checked the username and 
password spelling but I still get a "LOGIN failed". Any clue?

R. Go to "Q. Can I use imapsync to transfer from or to Office365 accounts?"

======================================================================
Q. How can I access an Office365 shared mailbox?

R. First, create a shared mailbox, for example shared@example.com. 
Then give full permissions to a licensed account user@example.com.
Now with imapsync, use the licensed user login with the syntax: 

  user@example.com\shared@example.com

and the password for user@example.com

Caveat: Character \ is a special character, so
use double-quotes around the user login, like:

  imapsync ... --user1 "user@example.com\shared@example.com"

Sources:
https://adam-hand.com/2017/07/25/connect-a-shared-mailbox-from-o365-to-outlook-via-imap/
https://www.arclab.com/en/kb/email/imap-settings-shared-mailbox-office-exchange-online.html
https://social.technet.microsoft.com/Forums/en-US/336e02ee-6767-4810-90a0-1352bd7cc9e9/office-365-how-to-access-a-shared-mailbox-using-imap-client?forum=onlineservicesexchange

======================================================================
Q. How can I use a shared account as a backup account for several
mailboxes and so avoid spending too many dollars in backup accounts?

R. Use the option --subfolder2 backup_foo

  imapsync ... --user2 sharedloginsyntax --subfolder2 backup_foo

To restore, do the reverse with:

  imapsync ... --user1 sharedloginsyntax --subfolder1 backup_foo


======================================================================
Q. Office365 fails with "User is authenticated but not connected".

R1. "The message User is authenticated but not connected is due to a 
   bug in the Office365 server's IMAP implementation. If the client 
   presents a valid user name but an invalid password, the server 
   accepts the login, but subsequent commands fail with the 
   aforementioned error message." Source:
https://unix.stackexchange.com/questions/164823/user-is-authenticated-but-not-connected-after-changing-my-exchange-password
Thanks to James Abbottsmith for this link and explanation at
https://github.com/imapsync/imapsync/issues/32#issuecomment-153561647

R2. Miguel Alameda reported understanding and solving this issue 
    like this, the context was admin/authuser:
    "The admin user had not permission in the target mailbox."

======================================================================
Q. I see "NO Maximum size of appendable message has been exceeded"
   What can I do with that?

R. Office365 supports send/receive max message sizes of up to 150MB 
   but you need to make changes in your tenant(s) to support it.

The following PowerShell command will increase the message sizes that 
can be sent/received.  The trick in getting IMAPSync to work is to 
apply these settings to the accounts performing the migration,
NOT the accounts associated with the target mailbox (assuming you're
using service accounts to perform transfers on behalf of users).

  Set-mailbox -Identity $UPN -MaxReceiveSize 150mb -MaxSendSize 150mb

e.g. 

  Set-mailbox -Identity "migrationaccount@testtenant.onmicrosoft.com" -MaxReceiveSize 150mb -MaxSendSize 150mb

We're transferring data between Office 365 tenants so we set these 
values on the migration accounts in the source and target tenants.

Thanks to Sean McDougall, Ian Thomas & Matt Wilks from Toronto 
for this FAQ item.

======================================================================
Q. The imap connection to Office365 is not working very efficiently,
   is there a solution to fix that?

R. Yes. Try DavMail
   http://davmail.sourceforge.net/
   I don't use it myself but a user, Yannick Palanque, reported
   great results using it.

======================================================================
Q. The sync fails with many "Trying command when NOT connected!".
   What can I do?

R. The --debugimap option can show you more details, especially 
   messages like this one:
   "BAD Request is throttled. Suggested Backoff Time: 178755 milliseconds"
   In that case, see the next faq item and its fixes.

======================================================================
Q. Office365 throttles the sync and says: 
   "Request is throttled. Suggested Backoff Time: 299961 milliseconds".
   What can I do with that?

R. Office365 has throttle mechanisms to limit any huge usage.
   Sometimes imapsync transfers are too stressful for servers.
   The following message 
   "Request is throttled. Suggested Backoff Time: 299961 milliseconds"
   comes from the imap Office365 server, imapsync just reports it
   before being disconnected from it.

   To solve the throttles issues from 0365, there are two solutions
   at least:

R1. Call Microsoft Office365 and ask them to remove the limits on your 
  mailboxes. That's not a joke, they do it for 90 days usually, 
  sometimes only after you reach the second technician you call, 
  the first one usually be not enough competent to understand 
  what you're talking about (I would be glad to remove this bad
  fact).

R2. Play with options --maxbytespersecond or --maxmessagespersecond
    or --exitwhenover

  imapsync ... --maxbytespersecond 100_000

  imapsync ... --maxmessagespersecond 2

  imapsync ... --exitwhenover 1_000_000_000

I don't know the upper value that avoids the default throttling from 0365
and I guess it changes over time.

======================================================================
Q. What are the receive and sending limits of Office365?

R1. 3600 messages per hour (22 July 2021). Maybe 1200 as imapsync can 
   be viewed as a single sender. Tell me what you experienced, 
   I'll update this point.

https://docs.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#receiving-and-sending-limits

In that case, imapsync can adapt to this with (1 message/second = 3600 messages per hour)

    imapsync ... --maxmessagespersecond 1
or
    imapsync ... --maxmessagespersecond 0.33

R2. I also found "Microsoft theoretically allows for about 300MB of
throughput per user per hour."  at
https://www.systools.in/blog/microsoft-office-365-throttling-policy/

In that case, imapsync can adapt to this with (83333 bytes/second = 300 MBytes/hour)

    imapsync ... --maxbytespersecond 83333

======================================================================
Q. Office365 refuses to create the folder named "Files" with the error
   "NO Folder name is reserved". What happens?

R. The folder Files is a standard folder in Office365.
   It should not be synced in IMAP. See the next question.

======================================================================
Q. Office365 users complain that a folder named "Files" contains 
   messages with no sender.

R0. To fix this, add --exclude Files

  imapsync ... --exclude Files

  If you use --office1 then imapsync will add this exclusion 
  automatically like using  the option:
  
  imapsync ... --exclude "^Files$" 

  If you use --office2 then imapsync will add a renaming of any
  "Files" folder on host1, like using the option:
  
  imapsync ... --f1f2 "Files=Files_renamed_by_imapsync"

  The host2 account ends up with a folder named 
  "Files_renamed_by_imapsync", but no complaining.

R1. This folder "Files" seems to be a standard folder in Exchange Online,
    but it is not. This folder contains all attachments in every email
    that is in the mailbox but without any headers.
    
    This causes some confusion for users as these appear in their search
    results as duplicate lines but without the sender details or even
    the message body.

   This folder seems to be usually hidden so IMAP clients can’t see it, 
   but for some reason sometimes it becomes visible.

R2. It looks to be a common problem with Exchange Online. 
    I’m not sure what causes the folder to appear.

   More info here:
https://answers.microsoft.com/en-us/msoffice/forum/msoffice_o365admin-mso_exchon-mso_o365b/exclude-the-exchange-online-system-folder-called/2adbdf84-db4a-4c7f-ac29-738757980a0d
https://answers.microsoft.com/en-us/msoffice/forum/msoffice_o365admin-mso_dep365-mso_o365b/no-sender-emails-in-files-folder/534bae8f-a7d7-4f5f-8ed7-5bad0d5fa23f

  (This question/answer is taken quasi verbatim from Perttu Aaltonen)

======================================================================
Q. From XXX to Office365, read receipts are all resent again after sync.
   Even for old messages. How can I fix that?

R. Imapsync can remove the header containing this read-receipt request.

On Unix or Windows use:

  imapsync ... --disarmreadreceipts

Since read receipts should be sent for unseen messages that will go 
to a seen state after the migration, you could be strict and apply 
the regex only to seen messages. 
Selecting seen message can be done with:

  imapsync ... --search1 "SEEN" --disarmreadreceipts

A second run has to be run without the --disarmreadreceipts
for unseen messages:

  imapsync ... --search1 "UNSEEN" 

If fact --disarmreadreceipts is just an option equivalent to:

  --regexmess 's{\A(.*?(?! ^$))^Disposition-Notification-To:(.*?)$}{$1X-Disposition-Notification-To:$2}igxms' 

That regex changes the header Disposition-Notification-To. It prefixes
it with an X- so that it becomes inactive.
Disposition-Notification-To: blabla
becomes
X-Disposition-Notification-To: blabla

Thanks to David Karnowski for pointing and solving this issue.

======================================================================
Q. DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error
   DEBUG: IO/Socket/SSL.pm:1043: local error: SSL read error   

R1. "SSL read or write error" happens sometimes, it isn't related to 
    imapsync directly but to the ssl underlying library when communicating
    with Exchange in TLS/SSL encrypted mode. 
    Next runs should put the sync further, so rerun the syncs 
    until it is well completed.

R2. Another solution is to remove --tls or --ssl options for Exchange
    and accept clear text syncs.

R3. See also the FAQ FAQ.SSL_errors.txt
    https://imapsync.lamiral.info/FAQ.d/FAQ.SSL_errors.txt

======================================================================
Q. From XXX Office365 I get this error message sometimes: 
   "BAD Command Argument Error 11". What does it mean?

R. This error message comes from the Office365 IMAP server when it
encounters any problem. Most of the time it is one of the following:

  * Some messages are bigger than the size limit. 
    45 MB by default on Office365.
    I don't know if it can be upped by configuration for Office365.
    If you can't fix this limit on Office365 then use the option
    --maxsize 45000000 for 45 MB to tell imapsync to skip those messages.

  imapsync ... --maxsize 45000000 # 45 MB for Office365
  

  * Quota reached. The whole account is full.
    It can be upped by configuration.

  * You use --synclabels --resynclabels from a previous command line
    related to syncing from Gmail to Gmail. Remove them.

  * Some messages have some lines too long. Use option --maxlinelength
    to skip messages whose max line length is over some bytes.
    --maxlinelength 1000 is an RFC2822 must but most servers support
    higher values. Office365 supports 10500 characters line length:

  imapsync ... --maxlinelength 10500

  In case you prefer fixing messages with long lines the hard way,
  instead of skipping them with --maxlinelength 10500, just use:

  On Windows
  imapsync ... --regexmess "s,(.{10500}),$1\r\n,g"

  On Unix
  imapsync ... --regexmess 's,(.{10500}),$1\r\n,g'
  
Have also in mind that Office365 closes the connection after 10 errors
encountered so you might also see "BYE Connection closed" errors from
Office365, which means Office365 leaves the session and says goodbye,
come back later. Redo some sync then.

On Linux, there is a good Python script in the tarball that can 
fix several things that Office365 has issues with.

Use it like this:

  ./imapsync ... --pipemess W/tools/fix_email_for_exchange.py

It often does some miracles on messages.
It's called fix_email_for_exchange because Office365 is Exchange,
or at least started to be an Exchange server with the same issues.

======================================================================
Q. From XXX to Office365 the flag Flagged does not seem to be well synced. 
   What can I do?

R. Use the following trick. Run imapsync twice, one with --regexflag
   and one without, like this:

1) imapsync ... --regexflag "s/\\Flagged//g"
2) imapsync ...

You can add --debugflags if you want to see what imapsync gets and 
does in detail with flags.

The magic of this trick is on ignoring the \Flagged flag on the first 
sync and setting it on the second sync, with STORE instead of APPEND.
This Office365 bug seems that Office365 gets and sets well the Flagged 
flag with APPEND in IMAP but then it forgets it with other protocols;
With STORE it sets and gets the "\Flagged" flag everywhere.

Thanks to Dave Murray and Simon Savva for reporting and solving 
this issue.

======================================================================
Q. How to migrate from or to Office 365 with an admin/authuser account?

Note from Yago Torres Fernandez:
(a working command using admin/authuser on host2 Office 365)

  imapsync ... --authuser2 user_admin@domain.com --user2 user_to_be_migrated@domain.com  ^
               --password2 XXXX --ssl2  ^

but previously in Office365, you must do something like the following, using Powershell:

  Add-MailboxPermission -identity user_to_be_migrated@domain.com -user user_admin@domain.com -accessrights fullaccess -inheritancetype all


Note from Betsy Lawlor: You can use global modern authentication with two factor 
on Exchange Online (M365) but you must have "AllowBasicAuthImap" on
the admin account you are using to migrate the mail.

Remark: PLAIN authentication is the only way to go with --authuser1 for now.
So don't use --authmech1 SOMETHING with --authuser1 admin_user,
it will not work.
Same behavior with the --authuser2 option.

Note from Rafael Alvarez Ballesteros:
When you get an Office365 license you will receive an admin user to handle
your licenses and products like admin@yourcompanyname.onmicrosoft.com.
This account is the administrator account; some weeks or months ago Microsoft 
has decided you need to use two-factor authentication by default, 
so if two factors authentication is enabled you will no be able to sync 
the mail (it will not connect to host2).
OWA (I think this is the two-factor authentication) needs to be disabled 
globally and enabled individually on the users you want to.
One account can have the right to access other mailboxes no matter if admin or 
any other account but needs to have OWA disabled to be able to connect 
to the office365 server.

Note from Martin Paulucci:
I had to remove the domain part for the user
but not for the admin.  Example:

  imapsync ... --authuser2 user_admin@domain.com --user2 user_to_be_migrated

See also:
http://www.linux-france.org/prj/imapsync_list/msg02203.html
Subject: RE: [imapsync] Office 365 - 'Master User'?
Date:    Mon, 1 Jun 2015 17:53:54 +0000


======================================================================
Q. Couldn't create folder [trash] "Mailbox already exists".

R. Some servers take care of character cases in folder names, 
   some servers do not, like Exchange. Since not respecting the case
   can merge two different folders into one then imapsync respects the case. 

   For example, if a host1 server has a folder name called "trash"
   and the host2 server already has a folder "Trash" or "TRASH"
   then imapsync will try to create the folder "trash" on host2
   because trash and Trash are different strings. But if host2
   does not respect character case it will consider folder "trash"
   already exists and will say it. That's the error message reported
   by imapsync: "Mailbox already exists". This message comes from the server.

   The folder creation fails but messages are well transferred in
   so take a look at this warning, understand why it happens
   and it should be fine most of the time.

   To avoid this warning use --regextrans2 to map the folder names

   imapsync ...  --regextrans2 "s/^trash$/Trash/"

   If there are two folders Trash and trash on host1 then both
   will be merged into only one Trash folder on host2. 
   In case of the option --delete2 is used the regextans2 above becomes
   mandatory, otherwise imapsync will sync messages from the
   first Trash and then delete them when syncing trash.

   If you want to avoid merging folders that are considered different 
   on host1 but are considered the same on destination host2 because of
   case sensitivities and insensitivities, use  --nomixfolders

======================================================================
======================================================================