imapsync/FAQ.d/FAQ.Folders_Mapping.txt

#!/bin/cat
$Id: FAQ.Folders_Mapping.txt,v 1.23 2021/04/28 23:42:04 gilles Exp gilles $

This documentation is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Folders_Mapping.txt



=======================================================================
             Imapsync tips about changing folders names.
=======================================================================

Questions answered in this FAQ are:

Q. What is the imapsync default behaviour with folders?

Q. What encoding imapsync uses to deal with folder names?

Q. How can I encode folder names for imapsync without getting a headache?

Q. What are the easy ways to change folder names?

Q. What should I know before playing with the --regextrans2 option?

Q. How to remove the string "INBOX." in the name of destination folders?

Q. How to change only INBOX name to Inbox_Migrated?

Q. How to sync all folders to a single INBOX folder?

Q. How to sync all folders to a single WHOLE folder?

Q. How to sync a complete account in a subfolder called FOO?

Q. How to substitute all characters dot "." by underscores "_" ?

Q. How to substitute all doublequotes " by underscores _ ?

Q. How to substitute all characters * % . and # by underscores _ ?

Q. How to change folder names like this:
[mail/Sent Items] -> [Sent]
[mail/Test]       -> [INBOX/Test]
[mail/Test2]      -> [INBOX/Test2]

Q. Is it possible to synchronize all messages from one server to 
   another without recreating the folder structure and the target server.

Q. I would like to move emails from InBox to a sub-folder called,
   say "2010-INBOX" based on the date. Like all emails received in the
   year 2010 should be moved to the folder called "2010-INBOX".

Q. How to sync a complete account in a subfolder called FOO,
   the hard way?

Q. How to transform all folder names to lowercase?

Q. How to transform all folder names to uppercase?

Q. How to transform all folder names to lowercase, except the first 
   character in uppercase?


Now the questions again with their answers.


=======================================================================
Q. What is the imapsync default behaviour with folders?

R. Imapsync syncs all folders, one by one, in alphanumeric order,
   reproducing their name identical from the source account at host1 to 
   the destination account at host2, except for the prefix and the 
   separator which are automatically adapted for host2.

=======================================================================
Q. What encoding imapsync uses to deal with folder names?


R. The IMAP protocol has a specific way to code folders names,
especially when these names use non-ascii 7bit characters.
This encoding is called utf7imap. 

Imapsync uses the same encoding as IMAP, it uses utf7imap.

=======================================================================
Q. How can I encode folder names for imapsync without getting a headache?


R. In order to well specify folders names on the command line, imapsync
   prints the complete folder list of both sides at the beginning 
   of each run.

The left column is the encoding you have to use, without the first enclosing 
square brackets [], the right column is the human utf8 view.

=======================================================================
Q. What are the easy ways to change folder names?

R. This document gives most examples with the powerful but complex
option --regextrans2. Before using --regextrans2 you should 
consider using --automap and --f1f2 because they are simpler 
to understand and use.

 --automap           : guesses folders mapping, for folders like
                       "Sent", "Junk", "Drafts", "All", "Archive", "Flagged".

 --f1f2    str1=str2 : Force folder str1 to be synced to str2,
                       --f1f2 overrides --automap and --regextrans2.

 --subfolder2   str  : Syncs the whole host1 folders hierarchy under the
                       host2 folder named str.
                       (It does it internally by adding two 
                       --regextrans2 options before all others.
                       Add --debug to see what's really going on.)


=======================================================================
Q. What should I know before playing with the --regextrans2 option?

The things to know and understand before playing with --regextrans2 are:

*) --regextrans2 is used to transform folders names.

*) --regextrans2 applies after the default 
   inversion prefix1 <-> prefix2 and sep1 <-> sep2.
   So, when elaborating the regex you should focus on
   the right part of the default mapping printed by
   imapsync, the part showing the host2 folder name.
   The section to look at is either the folder size section
   or within the folder loop located "Here" between the lines
++++ Looping on each folder
Here
++++ End looping on each folder
   
*) Several --regextrans2 are possible, they will be applied in the order
   of the command line, each one on the result of the previous one.

*) --regextrans2 uses Perl regex mechanism so it may be hard to master
   this part. It is powerful but not very simple.

*) Windows vs Unix quotes. 

   On windows don't use single quotes ' around the regex string, 
   use double quotes instead, like --regextrans2 "myregex"

   On Linux/Unix use single quotes ' around the regex string, it is
   easier to get what we want with single quotes since the shell 
   won't change the inner string. Use --regextrans2 'myregex'

*) Good method to elaborate any --regextrans2 string

   First, elaborate the --regextrans2 string with --dry --justfolders options. 

     imapsync ... --dry --justfolders

   With --dry imapsync shows the transformations it will do without 
   really doing them, --dry is the "do nothing" mode.
   With --justfolders imapsync will work only with folders, 
   messages won't be taken into account, so it will be fast
   and focused on the folders names.

   When the output shows what you expect imapsync to do with folders
   names, you can remove the --dry option. Keep the --justfolders 
   option in order to see if the destination server host2 accepts
   to create the folders.

   When everything is ok with folders you might remove --justfolders,
   imapsync will also transfer messages.
   Showing folders sizes is good then transferring messages, it allows
   ETA calculation and it's a supplementary check on folders.

=======================================================================
Q. How to remove the string "INBOX." in the name of destination folders?


R. To remove "INBOX." in the name of destination folders:

On Linux:

  imapsync ... --regextrans2 's/^INBOX\.(.+)/$1/'

On Windows:

  imapsync ... --regextrans2 "s/^INBOX\.(.+)/$1/"

=======================================================================
Q. How to change only INBOX name to Inbox_Migrated?

R. 
   imapsync ... --regextrans2 's{^INBOX$}{Inbox_Migrated}'


=======================================================================
Q. How to sync all folders to a single INBOX folder?

R. 

  imapsync ... --regextrans2 "s/.*/INBOX/"

=======================================================================
Q. How to sync all folders to a single WHOLE folder?

R. 

  imapsync ... --regextrans2 "s,.*,WHOLE,"


=======================================================================
Q. How to sync a complete account in a subfolder called FOO?

R. With imapsync release 1.641 and former, simply use:

  imapsync ... --subfolder2 FOO

Examples to this subfolder problem for previous releases
can be seen below, in another Q/R section.


=======================================================================
Q. How to substitute all characters dot "." by underscores "_" ?

R.
    --regextrans2 "s,\.,_,g"

=======================================================================
Q. How to substitute all doublequotes " by underscores _ ?

On Linux/Unix:

  --regextrans2 's,\",_,g' 

On Windows:

  --regextrans2 s,\^",_,g

=======================================================================
Q. How to substitute all characters * % . and # by underscores _ ?

On Linux/Unix:

  --regextrans2 'tr,*%.#,_,'

On Windows:

  --regextrans2 "tr,*%.#,_,"

You can increase the *%.# list by any unwanted character,
plus, the order doesn't count. See below for &

It is a bad idea to substitute & characters since &
is a character to encode non-ascii characters in IMAP folder names.


=======================================================================
Q. How to change folder names like this:
[mail/Sent Items] -> [Sent]
[mail/Test]       -> [INBOX/Test]
[mail/Test2]      -> [INBOX/Test2]

On Linux/Unix:

    --regextrans2 's,^mail/Sent Items$,Sent,' \
    --regextrans2 's,^mail/,INBOX/,'

a) I want folders 'Sent', 'Sent Messages', 'Gesendet' from host1 
    to be merged to folder 'Sent' on host2.

  --regextrans2 "s{Sent|Sent Messages|Gesendet}{Sent}"

b) I do not want this rule a) to apply to any subfolders,
    e.g. 'folder/Sent' or 'folder/Sent Messages' etc. but only
    basefolders.

So you have to add an anchor character ^ meaning "beginning of the string",
and an anchor character $ meaning "end of the string" like this:

  --regextrans2 's{^Sent$|^Sent Messages$|^Gesendet$}{Sent}'

c) I want just a) to be combined with a subfolder, for example:

  INBOX                -> Subfolder/INBOX
  Sent                 -> Subfolder/Sent
  Sent Messages        -> Subfolder/Sent
  Gesendet             -> Subfolder/Sent
  folder/Sent          -> Subfolder/folder/Sent
  folder/Sent Messages -> Subfolder/folder/Sent
  folder/Gesendet      -> Subfolder/folder/Sent

To achieve this, use:

  --subfolder2 Subfolder --regextrans2 's{Sent|Sent Messages|Gesendet}{Sent}'

d) I want a) and b) to be combined with a subfolder, for example:

  INBOX                -> Subfolder/INBOX
  Sent                 -> Subfolder/Sent
  Sent Messages        -> Subfolder/Sent
  Gesendet             -> Subfolder/Sent
  folder/Sent          -> Subfolder/folder/Sent
  folder/Sent Messages -> Subfolder/folder/Send Messages
  folder/Gesendet      -> Subfolder/folder/Gesendet
  ...

This one is trickier to achieve:

  --subfolder2 Subfolder --regextrans2 's{^Subfolder/Sent$|^Subfolder/Sent Messages$|^Subfolder/Gesendet$}{Subfolder/Sent}'


======================================================================
Q. Is it possible to synchronize all messages from one server to 
   another without recreating the folder structure and the target server.

R. Yes. This question has already been answered but a little of 
   redundancy can't hurt, can he?

For example, to synchronize all messages in all folders on host1
to folder INBOX only on host2:

1) First try (safe mode):

  --regextrans2 "s/.*/INBOX/" --dry --justfolders

2) See if the output says everything you want imapsync to do, 
   --dry option is safe and does nothing real.
   
3) Remove --dry 
   Check the imap folder tree on the target side, you should
   only have one: the classical INBOX.

4) Remove --justfolders


=======================================================================
Q. I would like to move emails from InBox to a sub-folder called,
   say "2010-INBOX" based on the date. Like all emails received in the
   year 2010 should be moved to the folder called "2010-INBOX".

R. 2 ways :

a) With imapsync:
-----------------

  imapsync ... \
    --search "SENTSINCE 1-Jan-2010 SENTBEFORE 31-Dec-2010"
    --f1f2 "INBOX=2010-INBOX" \
    --folder INBOX

See also the FAQ FAQ.Messages_Selection.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt
to get more examples about how to select messages.

b) Manually:
------------

1) You create a folder INBOX.2010-INBOX

2) Mostly every email software allow sorting by date. In INBOX, you
   select from 1 January to 31 December 2010 messages with the shift key.
   (in mutt, use ~d)

3) Cut/paste in INBOX.2010-INBOX



=======================================================================
Q. How to sync a complete account in a subfolder called FOO,
   the hard way?

R0. See the easy above using --subfolder2

R1. Several ways to do it.

  a) Separator is dot character "." and "INBOX" prefixes every folder

On Linux/Unix:

    --regextrans2 's,^INBOX(.*),INBOX.FOO$1,'

On Windows:

    --regextrans2 "s,^INBOX(.*),INBOX.FOO$1,"

  or:
  
  b) Separator is the slash character "/" and there is no prefix 

On Linux/Unix:

   --regextrans2 's,(.*),FOO/$1,'
   
   or with doublequotes
   
   --regextrans2 "s,(.*),FOO/\$1,"

On Windows:

   --regextrans2 "s,(.*),FOO/$1,"

=======================================================================
Q. How to transform all folder names to lowercase?

R. 

First, some documentation to understand how the regexes work.
perldoc perlre says:

  \l          lowercase next char (think vi)
  \u          uppercase next char (think vi)
  \L          lowercase till \E (think vi)
  \U          uppercase till \E (think vi)
  \E          end case modification (think vi)
  \Q          quote (disable) pattern metacharacters till \E


On Linux:

If the separator on host2 is the character slash "/", use:

  --regextrans2 's,([^/]+),\L$1\E,g'

If the separator on host2 is the character dot ".", use:

  --regextrans2 's,([^.]+),\L$1\E,g'

On Windows:

If the separator on host2 is the character slash "/", use:

  --regextrans2 "s,([^/]+),\L$1\E,g"

If the separator on host2 is the character dot ".", use:

  --regextrans2 "s,([^.]+),\L$1\E,g"


=======================================================================
Q. How to transform all folder names to uppercase?

R. 

On Linux:

If the separator on host2 is the character slash "/", use:

  --regextrans2 's,([^/]+),\U$1\E,g'

If the separator on host2 is the character dot ".", use:

  --regextrans2 's,([^.]+),\U$1\E,g'

On Windows:

If the separator on host2 is the character slash "/", use:

  --regextrans2 "s,([^/]+),\U$1\E,g"

If the separator on host2 is the character dot ".", use:

  --regextrans2 "s,([^.]+),\U$1\E,g"

=======================================================================
Q. How to transform all folder names to lowercase, except the first 
   character in uppercase?

R. 

On Linux:

If the separator on host2 is the character slash "/", use:

  --regextrans2 's,([^/]+),\u\L$1\E,g'

If the separator on host2 is the character dot ".", use:

  --regextrans2 's,([^.]+),\u\L$1\E,g'

On Windows:

If the separator on host2 is the character slash "/", use:

  --regextrans2 "s,([^/]+),\u\L$1\E,g"

If the separator on host2 is the character dot ".", use:

  --regextrans2 "s,([^.]+),\u\L$1\E,g"


=======================================================================
=======================================================================