imapsync/FAQ.d/FAQ.APPEND_errors.txt

#!/bin/cat
$Id: FAQ.APPEND_errors.txt,v 1.13 2021/05/29 07:58:44 gilles Exp gilles $

This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.APPEND_errors.txt

======================================================================
              Dealing with Imapsync APPEND errors.
======================================================================

Questions answered in this FAQ are:


Q. I have an "could not append" error with 
   "Message contains invalid header" at the end, like:
   Err 1/20: - msg INBOX/6 {40666} could not append ... NO Message contains invalid header
   What can I do to transfer these emails?

Q. For some messages, the imapsync log says 
   "could not append", sometimes followed by an explicit message 
   describing what went wrong, or sometimes followed by a not very
   useful message "socket closed while reading data from server"
   What can I do?

Q. The append error message is "NO Message contains NUL characters"
   What can I do?


Now the questions again with their answers.


======================================================================
Q. I have an "could not append" error with 
   "Message contains invalid header" at the end, like:
   Err 1/20: - msg INBOX/6 {40666} could not append ... NO Message contains invalid header
   What can I do to transfer these emails?


R0. Append in the IMAP protocol is the command to add a message in the folder.
    The error string "NO. Message contains invalid header" comes from the 
    destination IMAP server at host2, it doesn't like the message and rejects it.
   
R1. Header lines must be in 7bit encoding.
   If they aren't in 7bit the IMAP server can refuse them as they break 
   the IMAP RFC specification.

   A solution is to transform all 8bit characters to the Z character with:
   
   imapsync ... --regexmess "tr [\x80-\xff] [Z]"

   Caveat: this transformation transforms both the email header and its body.
   It's not perfect but I guess it's better than no message transferred.
   So consider using it in a second pass, not the first, as 8-bit 
   characters in the body message are allowed and changing them when
   not needed is not a good thing.


R2. Some crappy email systems, like virus checkers or any other crappy 
    software tool dealing with your messages, can add headers to tell 
    the world that they've done a great job. But sometimes they didn't,
    and they fucked up your messages by not respecting the standard
    and added ugly non-rfc compliant headers.
    
    To remove an ugly header, let's call it "X-Spam-Report", that spreads over 
    several lines beginning or, that's the ugly part, not beginning with a space:

    imapsync ... --regexmess 's{X-Spam-Report:.*?\n(^[a-zA-Z0-9\-]+:|^\r?\n)}{$1}xms'

    Thanks to Damien SAUTEREAU for reporting and solving this issue.
    

======================================================================
Q. For some messages, the imapsync log says 
   "could not append", sometimes followed by an explicit message 
   describing what went wrong, or sometimes followed by a not very
   useful message "socket closed while reading data from server"
   What can I do?

R0. Append errors are host2 problems, append is the imap term to copy 
    a message to the destination account. 
    A problem is that the assosiated "socket closed ..." error message happens
    for several different issues. So I list here several potential issues
    and their solutions if they exist.
    R1 deals about too long lines in messages on Windows.
    R2 deals about too long lines in messages on Unix.
    R3 deals with quota reached.

R1. On Windows, add --regexmess "s,(.{9900}),$1\r\n,g"

Some messages have too long lines; for example, 
Exchange supports only 9900 characters line length.
Use this option to add "new line" characters (also called CRLF)
to wrap lines longer than 9900 characters. 
The regex means "add one CRLF every 9900".

  imapsync.exe ... --regexmess "s,(.{9900}),$1\r\n,g"

R2. On Unix, add --pipemess "reformime -r7". The command reformime
    usually belongs to the package called "maildrop".

  imapsync ... --pipemess "reformime -r7"

I reproduce here the "reformime" manual part explaining what does 
the option "-r7" 

$ man reformime |more
REFORMIME(1)                Double Precision, Inc.                REFORMIME(1)

NAME
       reformime - MIME E-mail reformatting tool

SYNOPSIS
       reformime [options...]

DESCRIPTION
       reformime is a utility for reformatting MIME messages.

       Generally, reformime expects to see an RFC 2045[1] compliant message on
       standard input
...
OPTIONS
...
       -r
           Rewrite message, adding or standardizing RFC 2045[1] MIME headers.

       -r7
           Like -r but also convert 8bit-encoded MIME sections to
           quoted-printable.
...
   Adding RFC 2045 MIME headers
       The -r option performs the following actions:

       If there is no Mime-Version:, Content-Type:, or
       Content-Transfer-Encoding: header, reformime adds one.

       If the Content-Transfer-Encoding: header contains 8bit or raw, but only
       seven-bit data is found, reformime changes the
       Content-Transfer-Encoding header to 7bit.

       -r7 does the same thing, but also converts 8bit-encoded content that
       contains eight-bit characters to quoted-printable encoding.

       
R3. Look for the quota limit on host2. If it is the current mailbox size, don't
    search elsewhere. Increase the quota on the destination account.

======================================================================
Q. The append error message is "NO Message contains NUL characters"
   What can I do?

R. Use:

  imapsync ...  --skipmess "m/(\x00)+\Z/"

The option --skipmess skips messages matching a pattern, the 
messages are not synced, you just avoid the error messages concerning
those messages.

If you want to sync them, use instead:

  imapsync ...  --regexmess "s/(\x00)+\Z//g"


======================================================================
======================================================================