imapsync/TODO

#!/bin/cat
# $Id: TODO,v 1.230 2020/01/03 22:39:22 gilles Exp gilles $

This documentation is also at http://imapsync.lamiral.info/#doc

===========================================================================
TODO file for imapsync
----------------------


SUGGESTED 2020_01_03 by Gilles
Bugfix. When running ./imapsync --tests --testslive
Host Nb folders are 0, which is false.


SUGGESTED 2020_01_03 by Gilles
https://lamiral.info/~gilles/imapsync/cover_db/imapsync.html
is result of ./imapsync --tests --testslive
Increase the subroutine coverage to 100% by
removing unused ones or add a test call for the remaining. 

SUGGESTED 2019_12_26 by Gilles
For /X Look at 
https://en.wikipedia.org/wiki/ModSecurity


SUGGESTED 2019_12_26 by Gilles
Look at 
https://metacpan.org/pod/Test::Warnings


SUGGESTED 2019_12_26 by https://github.com/masbaehr
Add the "throttle" word in the README about 
--maxmessagespersecond --maxbytespersecond
See https://github.com/imapsync/imapsync/issues/204


SUGGESTED 2016_07_12 by Fronik
With --automap apply the mapping to subfolders of mapped folders.
If
Sent => Envoyes
then
Sent.Foo     =>  Envoyes.Foo
Sent.Foo.Bar =>  Envoyes.Foo.bar


SUGGESTED 2019_10_22 by Gilles
Add the download link in imapsync itself, with --releasecheck

SUGGESTED 2019_10_20 by Olav Seyfarth
One suggestion/request regarding INSTALL.OnlineUI.txt: Please add a
typical NGINX setup since NGINX is used very often now and requires a
quite different setup: one has to set up a fcgi service.

SUGGESTED 2019_10_02 by Joel from Blue Net Box
Fix this issue:
<EFBFBD>imapsync_bin_Darwin<EFBFBD> can<61>t be opened because its integrity cannot be verified.
This software needs to be updated. Contact the developer for more information.
MacOS 10.15 is the new macOS Catalina (in the last beta phase)
This error message happens in a shell window.
Same imapsync_bin_Darwin in macOS 10.14 works fine.

SUGGESTED 2019_10_02 by Gilles
Look at Traefik to automatic ssl services (webserver).


RE_SUGGESTED 2019_09_26 by Andre Adrian (rotlaus)
Allow @ in subfolder names #195
https://github.com/imapsync/imapsync/issues/195
Allow also & (imap_utf7 character)
Or add a --nosanetizesubfolder option.

SUGGESTED 2019_07_01 by Jeffrey Thatplayer
https://github.com/imapsync/imapsync/issues/181
Make sanitizing an option for --subfolder1 --subfolder2
Two options? --nosanitize1 --nosanitize2 
Reflecting more I wonder if sanitizing  for --subfolder1 is just stupid.
It's not, it's good from the usage point of view, same string to point out
the destination backup/restore folder


SUGGESTED 2019_09_23 by Celko Michal.
My proposal is to create a status file.
it would be a command line option, ex. 
  imapsync ... --status-file /var/www/hosting/tmp/xyz.json
When I start imapsync process with this option, it would generate
and update this status file every few seconds, so any application
can access it and see what's going on with this process without
parsing logs or keeping up with the log format when it changes.
Example of such status file would be:
In progress
{
"status": "In progress",
"progress": "245/5000",
"Message": "",
"Debug": "",
"Process ID": "pid123456"
}
On error example
{
"status": "Fail",
"progress": "245/5000",
"Message": "ERRCODE:123",
"Debug": "LOG_imapsync/2019_09_22_09_01_42_058_abc@example.com_JDoe@example.com.txt",
"Process ID": "pid123456"
}
On complete example*
{
"status": "Completed",
"progress": "5000/5000",
"Message": "",
"Debug": "LOG_imapsync/2019_09_22_09_01_42_058_abc@example.com_JDoe@example.com.txt",
"Process ID": ""
}


SUGGESTED 2019_07_30 by Gilles
Only since imapsync 1.774 the release check is done with 
\r\n to end lines, instead of the not RFC compliant \n.
If I upgrade Apache on imapsync.lamiral.info
ii  apache2-mpm-worker     2.2.22-13+deb7u7 amd64            Apache HTTP Server - high speed threaded model
then any imapsync prior to 1.774 will fail to GET the VERSION file.
Release check life with names:

PeerAddr "linux-france.org"      in release 1.350
"Host: www.linux-france.org\n\n" in release 1.350 to 1.466

PeerAddr "imapsync.lamiral.info" in release 1.374 and upper
"Host: www.linux-france.org\n\n" in release 1.464 but not after
"Host: ks.lamiral.info\n\n"      in release 1.467 and upper

"Host: ks.lamiral.info\r\n\r\n"  in release  1.836 and upper

Solution to test: Use the directive 
HttpProtocolOptions Unsafe # available from 2.2.32 or 2.4.24 
http://httpd.apache.org/docs/2.4/mod/core.html#httpprotocoloptions
ks5 should be ok: Server Version: Apache/2.4.33 (FreeBSD)



SUGGESTED on 31 Jan 2017 by tibbsbrookside
Try this code XOAUTH2
https://github.com/imapsync/imapsync/issues/87#issuecomment-276298841


SUGGESTED 2019_06_18 by Konrad Wawryn
Add NTLMv2 support
It shouldn't be that hard to support NTLMv2 because
the underlying NTLM module can do it already.
Delegate all the stuff to underlying Mail-IMAPClient.

SUGGESTED 2019_06_05 by Gilles
Code equivalent W/tools/fix_email_for_exchange.py
inside imapsync

SUGGESTED 2019_04_10 by Gilles
With signals show also their numbers
https://perldoc.perl.org/Config.html

SUGGESTED 2019_04_03 by Katja Wolf and Gilles
Put a summary like the one in the log file 
in the source and the destination mailbox.

SUGGESTED 2019_03_19 by Gilles
In the final errors listing, add lines
* Host1 banner
* Host2 banner
* Command line parameters
I'm a little fade up to ask them 
in order to understand the context


SUGGESTED 2019_03_04 by Gilles
Write a FAQ about deciphering with imapsync.
Test it!
If the encryption is made using gpg
then the decryption via --pipemess could be just the command:
  imapsync ... --pipemess 'gpg --decrypt --no-batch --no-tty --override-session-key 9:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
where you previously ran:
  gpg --decrypt --no-batch --show-session-key some_rfc822_filecrypted.txt
in order to get the session-key "9:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

SUGGESTED 2019_02_28 by Gilles
Default tmpdir could become ./TMP_imapsync 
for --usecache if there is no cache in the 
historical tmpdir File::Spec->tmpdir(  )
Why? Some systems like Windows do change
their tmpdir upon reboot.
Some Unixes do remove their /tmp upon reboot.


SUGGESTED 2018_09_11 by Gilles
Use Smart::Comments in long loop 
Use Smart::Comments in debug/verbose mode

SUGGESTED 2018_09_11 by Gilles
Take a look at
Test::NoWarnings
Test::Warn

SUGGESTED 2018_05_08 by James B. Byrne
Sync also the quota with SETQUOTA
https://tools.ietf.org/html/rfc2087

SUGGESTED 2017_12_19 by Gilles
In foldersizes, add info about the oldest message found: INTERNALDATE EPOCH DAYS_FROM_NOW 
See draft in sub info_date_from_uid


SUGGESTED 2017_08_31 by Gilles
Makefile. Add a "make docker" goal that build, test and publish the 
docker package on ks3.

SUGGESTED 2017_08_22 by Ismael Baena
when hmailserver hierarchy delimiter is <20>\<5C>
When imapsync asks hmailserver about namespace, hmailserver returns <20>\\<5C> instead of <20>\<5C>. 
After that, imapsync fails when accessing any folder. Comparing hmailserver logs between 
imapsync and outlook accessing the account, I see outlook quoting every path 
while imapsync don<6F>t, and outlook doesn<73>t fail.
I wrote a message to the hmailserver forum, 
https://www.hmailserver.com/forum/viewtopic.php?f=7&t=31503


SUGGESTED 2017_05_04 by Gilles
Verif ipv6 connection, seems to fail now
./imapsync --host1 2603:1026:200:52::2 --host2 2603:1026:200:52::2 --justconnect
...
Host1: connecting on host1 [2603:1026:200:52::2] port [143]
Host1: Can not open imap connection on [2603:1026:200:52::2]: Unable to connect to 2603:1026:200:52::2: Invalid argument

SUGGESTED 2017_05_04 via github
Review the "make install" in Makefile

SUGGESTED 2017_04_26 by Franco Fassio
--usecache with trailing dots foo... in folder name breaks on NTFS.


SUGGESTED 2016_11_16 by Fl<46>vio Zarur Lucarelli LucaNet Sistemas
Getting subjects of messages for duplicates and print them.
Dates and sizes are easier printable because imapsync gets them each time.

SUGGESTED 2016_10_24 by Gilles
Verify regex --delete2foldersbutnot and --delete2foldersonly are ok
before anything.

SUGGESTED 2016_10_07 by Gilles
Write a FAQ file for packagers,
mostly how to get the dependency Perl modules complete.

SUGGESTED 2016_10_07 by Gilles
Write a FAQ file about all errors and explain what they mean.
Maybe propose an action to solve them in the code,
or even solve them directly in the execution.

SUGGESTED 2016_10_06 by Gilles
Document or fix behavior (do a AND),
curently "--search" makes "--maxage" and "--minage" ignored.

SUGGESTED 2016_09_16 by Gilles
Add a check of all --regextrans2 before going further,
like what is done with --regexmess

SUGGESTED 2016_08_25 by David Carter
It would be helpful if imapsync reported Message-Id:/From:/Subject
as well as folder name and message UID when it skips a message.

SUGGESTED 2016_09_02 by Gilles & David Carter
Exchange Online errors that need a relogin:
 "BAD User is authenticated but not connected."
 "error while reading data from server: Connection reset by peer"
 "timeout waiting 1200s for data from server"


SUGGESTED 2016_08_31 by Gilles
Rename $expungeaftereach $expungeaftereach1 or $expunge1aftereach



SUGGESTED 2016_08_19 by Gilles
Go back to SSL_VERIFY_PEER but include
SSL_ca_file inside imapsync or near.


SUGGESTED 2016_07_29 by Gilles from CSS3 Coursera course
Take a look at
https://modernizr.com/

SUGGESTED 2016_07_29 by Gilles
Move website to HTML5
https://about.validator.nu/

SUGGESTED 2016_07_07 by Jean-Dominique Delyon.
Add a way to know easily which account transfers went wrong.
G<EFBFBD>n<EFBFBD>rer un fichier des comptes qui ont rencontr<74> des probl<62>mes
et afficher le contenu <20> la fin de la boucle sur les comptes.

SUGGESTED 2016_06_29 by Gilles
Clarify system flags in RFC
Example to add:
If \Forwarded is not in PERMANENTFLAGS but \* is, then transform
\Forwarded to Forwarded. And by default.

SUGGESTED 2016_06_13 by David Carter
--pipemess could also treat stderr
` $command < $input_tmpfile > $output_tmpfile 2> $error_tmpfile `


SUGGESTED 2016_06_22 by Gilles
Add  --quiet mode
Make --quiet mode being switched by a signal

SUGGESTED 2016_06_12 by Gilles
Fix many perlcritic, all?
DONE *  60 perlcritic Double-sigil dereference (Severity 2)
DONE * 627 perlcritic violations of CodeLayout::ProhibitTrailingWhitespace. (Severity 1).
DONE * 458 perlcritic violations of ValuesAndExpressions::ProhibitInterpolationOfLiterals. "Useless interpolation of literal string"
DONE * 420 perlcritic violations of CodeLayout::ProhibitParensWithBuiltins. "Builtin function called with parentheses". (Severity: 1).



SUGGESTED 2016_06_09 by Gilles and David Carter
In order to avoid useless headaches from --regexmess, add
--regexheader
--regexbody

SUGGESTED 2016_06_09 by David Carter
Add --pipemess-on-fail
If, and only if, the target server responses with NO or BAD then have another go
with a sanitized version of the message.

SUGGESTED 2016_06_01 by M. Beaubien
Add flag --noheaderwarnings or better name
no header by parse_headers so taking whole header with BODY.PEEK[HEADER]
no header found so adding our own [Message-Id: <151648@imapsync>]

SUGGESTED 2016_04_17 by Gilles
Add a --passfile to allow user=>password style file.

WANTED 2016_02_10
Add stats about "Messages found crossduplicate on host1"

WANTED 2016_01_28 Stephen Sookdeo
Able to send email without attachment (in case size is the problem) and maybe note in main body that attachment was removed...
See http://unix.stackexchange.com/questions/174707/remove-delete-attachments-from-email-server-imap
http://unix.stackexchange.com/a/174726/89270

imapsync ... --pipemess 'remove_attachments --sizemin 10MB --addnote "Attachment was removed" '

WANTED 2016_03_06
Add a real test to authenticate with a ssl certificate.

WANTED 2016_03_08
Check if --ssl1 and --tls1 are used together and warn abort and abort in that case.
Same for --ssl2 and --tls2

Start a wiki for imapsync.

Write a Windows tutorial, TUTORIAL_Windows.html.

Write a good practices migration tips document, GOOD_PRACTICES.html

Add uidexpunge with --delete if possible (like with --delete2).

Add the possibility to specify a unity with --maxsize and --minsize
Example --maxsize 10MB --minsize 10KB
B
KB
MB
GB
TB


WANTED 2015_12_16 Gilles Lamiral
Add "df -i" with usecache and abort if the number of messages
to transfer will exhaust empty inodes used by the cache.
Looks like module Filesys::DfPortable will help for Unix and Win32.

==== Similar skip/error messages ====
WANTED 2016_01_28 Stephen Sookdeo
List all emails that gives errors so you know exactly which. List with detailed info.
Then ability to perform action on these specific emails like delete or ignore per email.

WANTED 2015_06_02 Karen F Bath.
Add skipped messages in the final dump.
Print the list of messages not copied and why (duplicates or void header).
I would like to request if you could add additional errors to the bottom,
as we find that things like MaxLineLength and maxsize limit are classed
as skipped messages and in my opinion are errors; as the email message
is not transferred but this is not logged at the bottom.
We have our own scan script which we run on all log files at the end
and copy the users logs into subfolders that have issues.
I've attached a list of things we search for.

"Error", "Output"
NO Mailbox already exists, "Folder"
NO LOGIN failed, "NoLogin"
Could not create folder, "Linux"
Error sending, "Linux"
Write failed 'no error caught', "Linux"
line length exceeds maxlinelength, "Linux"
BAD Command Argument Error. 11, "Linux"
Failure: can not open, "Linux"
No not connected, "Linux"
Write failed 'Connection reset by peer', "Linux"
could not be fetched:, "Linux"
NO Mailbox does not exist,"Error"
exceeds maxsize limit 20971520 bytes,"Error"
error while reading data from server: Connection reset by peer (4x),"Error"
Folder [Inbox] already exists,"Error"

WANTED Add a well described problem for each problem detected
and counted in error counter statistics.
==== End Similar skip/error messages ====


WANTED 2015_05_18
I'd like to be able to print the messages subjects, on some logs:
For instance:
    msg INBOX.Archived/95551 marked \Deleted on host2 [GyDD6WpsFEtyBzNFnv]
    msg [Gmail]/All Mail/281079 copying to INBOX.Archived
It would be great to be able to print the email subject, in order to debug or to get more useful information.


WANTED 2015_04_25
Add duplicates test option.




WANTED 2015_03_06
Dealing with Content-Type Message/Partial
Extract the components of the partial messages and construct them
with reformime as one message which can then be transferred. (Larry Moore)
See also uudeview http://www.fpx.de/fp/Software/UUDeview/
(Larry said uudeview is weird on Partial issue, too old maybe)

Apply --disarmreadreceipts only to UNSEEN messages.

Add an exit value when exiting because of --exitwhenover
The goal is to know easily why to restart later.

Write a Mail::imapsync package and use it.

One day, when I have really nothing better to do, evaluate:
http://www.rackspace.com/apps/email_hosting/migrations
http://www.yippiemove.com/
http://www.migrationwiz.com/
http://www.microsoft.com/download/en/details.aspx?id=1329 "Microsoft Transporter Suite"


Add --mark-as-deleted1 --mark-as-deleted2 as
aliases for --noexpunge1 --delete1 and --noexpunge2 --delete2

Fix bug found by Pavel Stano on 01/06/2012 (june) imapsync never stop login
when login fails with a "* BYE Temp error" from server.

Consider /var/tmp/ instead of /tmp (/tmp is destoyed
on some Unix at reboot)

Fix long path over than 260 character on Win32 with --usecache
Fix inode crunching with --usecache
Think about Digest::SHA or Digest::SHA::PurePerl.
Think about a file database like DBM instead.
Look at https://en.wikipedia.org/wiki/Radix_tree (David M. advice)
Look at leveldb (Simon Th actux advice)
Look at http://fallabs.com/kyotocabinet/ (JYB advice)
Also Redis (Ludovic Danigo advice)

I am writing now to suggest that imapsync provide info containing the parameters
with which a message is determined to be a duplicate, so that if I wanted to confirm
they are in fact duplicates by looking at the messages themselves, enough info would
be available to easily locate them.  One or two duplicates per folder are easy to
find, but I had one large folder with quite a few, and in this case it would have
been a big help.


Explain that users can win time/bandwidth by using --expunge

Fix "\Forwarded" flag bug in courier.
Does \lalala can be forbidden (courier does a
"16 NO Error in IMAP command received by server"
with
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited

Add sync imap keywords. Sync Gmail labels to imap keyword
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=503159
http://www.linux-france.org/prj/imapsync_list/msg00022.html
http://mail.google.com/support/bin/answer.py?hl=en&answer=77657
http://mail.google.com/support/bin/answer.py?answer=78892

Add and option to sync to & from files.

Add an --aclregextrans2 flag.
"Today we discovered, that Cyrus and Dovecot use different characters for
their ACLs. Syncing ACLs vom Cyrus to Dovecot (at least 1.2) doesn't
work. Cyrus uses c and d, Dovecot uses k and x instead."
Peer Heinlein.

Add more information about skipped messages.

Add Rick Romero patch with
--quiet   No output at all
--showstats  intended for use with --quiet

Take a look at Mail::IMAPTalk
Simon Bertrang said "way better performance, less problems, easier to use and no
issues so far". Sounds good!

imapsync doesn't report well. It should says "I had
to sync 123 messages but I could transfer only 99 messages"
Maybe count messages not transferred because they're dupplicate.

Fix bug "not possible to use space in the imap password"

Add kerberos authentification

Fix bug with folder names starting with an asterisk: *Archiv
Sending: 220 SELECT *Archiv/Support/foobar
Read: 220 BAD Protokollfehler: "Invalid folder specified in SELECT command"

Add LITERAL+ [RFC 2088] support to imapsync.

Make a m4 file for lsm.imapsync and generate it at each public release.

Add --verbose from Kjetil jumbo patch.

Read the IMAP RFC http://www.faqs.org/rfcs/rfc3501.html

Read:
http://www.washington.edu/imap/documentation/commndmt.txt.html

Add cyrus link about INBOX. namespace
http://asg.web.cmu.edu/cyrus/download/imapd/altnamespace.html

===========================================================================
Now the TODO done! (or not)
===========================================================================

DONE      2019/12/16 revision 1.974 by Gilles
Use Debug_fh to set where go the --debugimap outputs.
SUGGESTED 2017_09_04 by Gilles
STDOUT + logfile instead of STDERR with Mail::IMAPClient output.
Should be easy using $imap->Debug_fh( $fileHandle ) ;

DONE      2019_12_11 by Gilles
SUGGESTED 2019_11_28 by Gilles
Find a place to add this advice:
https://github.com/imapsync/imapsync/issues/201#issuecomment-559500077
Place: https://imapsync.lamiral.info/FAQ.d/FAQ.Docker.txt


DONE      2019_11_27 by Gilles 
SUGGESTED 2019_09_30 by Gilles
Update the docker build 
Mail::IMAPClient 3.38 to latest
Ok Mail::IMAPClient 3.42 in docker gilleslamiral/imapsync 1.967 a82b5d5ff2f0

DONE      2019_11_19 by Gilles revision 1.961 
SUGGESTED 2019_11_13 by Flavio
Are --delete2foldersonly or --delete2folders or --delete2foldersbutnot
compatible with --subfolder2 ? 
Yes but fix this
--delete2foldersonly SUB --subfolder SUB    deletes SUB
while 
--delete2foldersonly SUB. --subfolder SUB   does not delete SUB
(seen the dot?)


DONE      2019_07_30 by Gilles release 1.955
SUGGESTED 2018_05_15 by Massimo Maggi
https://github.com/imapsync/imapsync/issues/141
Document --skipcrossduplicates and --debugcrossduplicates in imapsync --help

DONE 2019_04_28 by Gilles
SUGGESTED 2016_10_03 by Gilles
Do like tail -F on logfile when 
"another imapsync may be curently running",
instead of quit.
Add --tail --notail and --tail on by default in cgi context.

DONE 2019_04_27 by Gilles
SUGGESTED 2019_04_25 by Gilles
Add length of $cache_dir to
Local cache directory: $cache_dir


DONE      2019/04/11  by Gilles revision 1.930
SUGGESTED 2016_08_07 by Gilles
Add a meaningful exit value to all die:
  * fatal software dependency, Perl modules
  * fatal parameter issue
  * fatal connect issue
  * fatal login issue
  * fatal permission file issue (open)
  * fatal IMAP issue
  * fatal IMAP disconnection
Maybe replace all die by exit
Add a meaningful exit value to all exit
  * exit at end but with errors
  * exit at middle because of errormax
  * exit by signal


DONE      2017_07_13
SUGGESTED 2016_03_11
Add a FAQ about Authentication failures and quoting.
FAQ.d/FAQ.Authentication_failure.txt
FAQ.d/FAQ.Passwords_on_Unix.txt  
FAQ.d/FAQ.Passwords_on_Windows.txt

DONE      2017/05/24 by Gilles revision 1.813
SUGGESTED 2017_04_26 by Franco Fassio
Check that --maxbyteperseconds does not interfer with --timeout 
by sleeping too long.


DONE by Gilles revision 1.795 date: 2017/04/22
SUGGESTED 2016_12_01 by Gilles
When working under with webmin and virtualmin imapsync
acts as a cgi, it should not. Find a way to avoid cgi mode
even under a web server. Solution: edit the sub under_cgi_context()

DONE 2019_04_08 by Gilles
SUGGESTED 2019_03_06 by Gilles
when exit by signal, exit by sending a signal to itself:
How to be a proper program
https://www.cons.org/cracauer/sigint.html


DONE 2018_11_20 by Gilles
Count the messages that are on host2 but not on host1.
(hum... see DONE 2018_09_01)

DONE 2018_09_01 by Gilles
Count messages on host2 that are not in host1

DONE 2018/04/18 revision 1.875 by Gilles
SUGGESTED 2018_06_29 by Michael Hesse
Remove the warning on Windows
<EFBFBD>No such signal: SIGUSR1 at script/imapsync line 11744.<2E>


DONE      2017_11_29 by Gilles
SUGGESTED 2017_04_11 by Stefano Lo Cascio
Add a --resyncflags on by default but allowing --noresyncflags

DONE      2017_10_27 by Gilles in revision 1.841
SUGGESTED 2017_09_02 by Gilles
Check that @ARGV is empty after readding it by GetOptions since
remaining arguments are not used and might signal something wrong 
in the command line arguments.


DONE      2017_08_31 by Gilles
SUGGESTED 2016_12_14 by Gilles
Look at Pod::Usage and convert the actual doc to it

DONE      2017_04_24 by Gilles
SUGGESTED 2016_11_29 by Gilles
Fix bug passfile no exist => login
Should be fatal sooner.

DONE      2017/03/13 by gilles in revision 1.782
SUGGESTED 2016_06_29 by Gilles
Add --gmail1 --gmail2 --gmail12 to set automatically advices from
https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt
Add --exchange --office365 to set automatically advices from
https://imapsync.lamiral.info/FAQ.d/FAQ.Exchange.txt
Add --domino1 --domino2

DONE by Gilles revision 1.755 date: 2017/01/12
SUGGESTED by Gilles
Try to act as --tls if CAPABILITY allows it and if there is no --notls.
Fallback without --tls if --tls fails.

DONE      2016_09_29 by Gilles
SUGGESTED 2016_04_13 by Gilles
Split --noabletosearch in --noabletosearch1 --noabletosearch2


SUGGESTED 2016_09_29 by Gilles
DONE with revision 1.764 2017/01/19
Try to use TLS or SSL by default unless not asked to do so.

DONE      2016_07_27 by Gilles
SUGGESTED 2016_06_15 by Gilles
Fix sub check_lib_version and its call.
It was ok because of the Perl line "use Mail::IMAPClient 3.30".



SUGGESTED 2016_06_22 by Gilles
DONE      2016_07_29 by Gilles
Make reconnections launched by a signal Ctrl-c, aka INT signal.
2 consecutive Ctrl-c within 1 second then exit the program.
Inline help when Ctrl-c is hit

SUGGESTED 2016_07_15 by Alan Williams
DONE      2016_07_20 for IO::Socket::SSL
Add --inet4 --inet6 to force AF_INET or AF_INET6 ip connexion,
aka ipv4 or ipv6.

SUGGESTED 2016_06_24 by Jean-Dominique Delyon
DONE      2016_06_30 by Gilles
Add --delete1emptyfolders
Destroy host1 empty folders.
Be carefull of non-empty subfolders!


SUGGESTED 2016_07_01 by Gilles
DONE      2016_07_01 by Gilles
Add exit explanation after errorsdump

DONE 2016_01_06. Write a Unix tutorial, TUTORIAL_Unix.html.

DONE by Gilles revision 1.677 date: 2016/01/19
Move --help documentation into the man page so that description is easier to find.

DONE 2015_12_26. WANTED 2015_03_24
Add --sslargs with usage like:
imapsync ...  --sslargs 'SSL_version=SSLv3' --sslargs 'SSL_use_cert=1' \
              --sslargs 'SSL_verify_mode=SSL_VERIFY_PEER'
See perldoc IO::Socket::SSL for all possibilities.


DONE 2015_12_25 Gilles Lamiral
Add --ssldebug 0-4

DONE 2015_10_06. Find a way to avoid passwords in --debugimap unless needed.
Proposed in Mail-IMAPClient via $imap->Showcredentials()
in patch at
See https://rt.cpan.org/Public/Bug/Display.html?id=107592

DONE 2015_12_03 WANTED 2015_11_24 Jens Herrmann
Add --logdir to allow imapsync choosing the filename while allowing
user to choose the dirname part.

DONE. 2015_08_10.
Guess separators and prefixes when NAMESPACE is not available,
instead of forcing the user to guess and set them.

DONE 2015_08_03. WANTED 2015_07_21
Fix W/learn/imap_utf7_encode
--regextrans2 "s,El&AOk-ments,&AMk-l&AOk-ments,"

DONE      2014/10/06 by Gilles revision 1.597
SUGGESTED 2016_08_31 by Gilles & David Carter
(I don't know why the DONE date older than the SUGGESTED date)
Detect equivalent folders by upper/lower case on host1
and add a way to not merge them when host2 is case insensitive.

DONE. Add a FAQ entry about long path over than 260 character on Win32.

DONE. Build and distribute a standalone Darwin Mac OS X binary.
It's called imapsync_bin_Darwin


DONE. Add a NOP for host2 for each fake copy in --dry mode.
Goal is to avoid timeouts happening only because of --dry

DONE 2015_05_09 WANTED 2015_04_25
Add an option "--subfolder2 FOO" to move all folders to a subfolder FOO:
On Windows:
    --regextrans2 "s,${h2_prefix}(.*),${h2_prefix}FOO${h2_sep}$1," \
    --regextrans2 "s,^INBOX$,${h2_prefix}FOO${h2_sep}INBOX,"


DONE. Quota are read if available and warning is printed if 90% quota reached.
Can you setup an option to make it stop if the destination mailbox reports
that it is over quota?


DONE. Add stdin/stdout filter before transfer:
"Now i asked me, how to modify your perl program to work with
that - in example, to write each mail to stdout, pipe that to the
conversion program, and read the result from stdin - and this all before
the mail will transfer to the target imap-server"
http://www.courier-mta.org/maildrop/reformime.html
Look at IPC::Open2, assuming a good and safe pipe,
or write more code handling all the bad stranger behaviours possible.
Option name: --pipemess "command arg1 arg2 ..."


DONE. Transform messages with too long linelength.
Office365 and Exchange abort connexion
when they encounter a message with more than 10000 characters on a single line.
Could be done with the previous --pipemess with reformime.
Thanks to Larry Moore for suggesting reformime tool!
Usage: --maxlinelength 9900 maxlinelengthcmd 'reformime -r7'


DONE. Add DavMail in Similar Softwares section. http://davmail.sourceforge.net/

DONE. Convert folder names to utf-8 and print them next to the utf-7 ones.
Look at imapsync/W/learn/
./imap_utf7 data_utf7

DONE. Separate the online site in a folder site/ or www/  and make the upper directory simple.

DONE. --skipemptyfolders
Is it possible to run imapsync in a way that empty folders on host1
aren't created at all on host2? Yes with --skipemptyfolders

NO. Add an option to remove empty folders on host1, except INBOX.
Use --skipemptyfolders instead

DONE. Subject is output when APPEND fails. Done since a long time.
Suggestion: it's very difficult to track down messages which are behaving
funny during the sync. It would be great - and presumably easy to code -
to have an option to have imapsync display e.g. the subject of an
e-mail when it gets synced, rather than just the message ID and the date/time.


DONE. Add W/learn/imap_utf7  in imapsync, the folders lists output looks a good place,
if transcription is needed.

DONE. Add options --log and --logfile to log both on screen and to a file.
DONE. Document the tee command solution (Unix and Windows) in the FAQ.

NEVER. Add an option to implement the faq entry about copying a contact folder.
(write a dedicated software instead!)

NEVER. Use examine() instead of select() in --dry mode.
Add a method doing the switch automagicaly.

DONE. Write --disarmreadreceipts that does:
  --regexmess 's{\A(.*?(?! ^$))^Disposition-Notification-To:(.*?)$}{$1X-Disposition-Notification-To:$2}igxms'

DONE. With --foldersizes make the listing on host2 independent of case
(avoid "not yet created" if the folder exists in another case string)

DONE. Write --maxmessagespersecond and  --maxbytespersecond for staying under Microsoft's throttling policies.

DONE. Inform about --addheader when the problem occurs "(no wanted headers so we ignore this message)"

DONE. Print the timeout value.
DONE. Inform --timeout is is seconds.

DONE. Add an entry to http://lsm.qqx.org/lsm/
See lsm.imapsync

DONE. Fix the mailing-list archive bug with From at
the beginning of a line
http://www.linux-france.org/prj/imapsync_list/msg00307.html

DONE. Remove 2.2.9 Mail::IMAPClient support.

DONE. Add OAUTH autentication support
https://developers.google.com/google-apps/gmail/oauth_overview

DONE. Add --delete1 as an alias for --delete

DONE. Add current date at the beginning of the run, useful when imapsync
doesn't finish with statistics.

DONE. Add automatic convertion or detection  when
separator inversion produces an invalid character.
From uw to cyrus, for example :
FoldA/FoldB.ext ->  FoldA.FoldB/ext

DONE. Not done since useless now (--useuid)
Add a --skipheaderinfolder option

DONE. Not fixed since only reported once a long time ago.
Fix this:
> - Erreur avec la traditionnelle diff<66>rence entre Windows
> et Linux sur les retour-chariots : le calcul de la
> longueur du message ou des ent<6E>tes <20> envoyer au serveur
> cible n'est pas bon sur une machine Windows.
> Ci-dessous la modif :
>
>         # No NL Count on Windows my $length = ( -s $file ) + $bare_nl_count;
>         my $length = ( -s $file );
I wonder if it is Windows or the imap server used.


DONE. No Debian package anymore.
Add debian packaging in the Makefile.
Write to the debian maintener about that.

DONE with --addheader
Add an option to solve syncing Sent folder when no good header
are available:
http://www.linux-france.org/prj/imapsync_list/msg01151.html
http://www.linux-france.org/prj/imapsync_list/msg01158.html

DONE. Add a note about
"One other thing: You might want to warn idiots like me, that if your
cache resides on a filesystem with a limited number of inodes such as
ext4, your inodes will be exhausted really fast (check with df -i). I
now use a dedicated partition with reiserfs."

DONE. Add an option --exitwhenover 2500000000 to avoid to be banned or locked
http://support.google.com/a/bin/answer.py?hl=en&answer=1071518

DONE. Check if a message exists before fetching it (could have been deleted recently)
      (Check avoided when UID mode is not available)

DONE. Change default --useheader 'Message-ID' --useheader Date to
--useheader 'Message-ID' --useheader 'Received'
because Exchange changes Date header! (but not Received:)

DONE. Add "output to reflect everything that imapsync was doing".
Not everything but flag synchronization will be nice"
--debugflags does this.

DONE. Add a script to install automaticaly all needed Perl modules
if needed. Not a script but command lines in INSTALL file.

DONE. Post on imapsync mailing-list when a new release comes.
No since I no longer distribute it gratis, imapsync detects
itself a new release and buyers are informed by a specific
list. I know some packagers (FreeBSD) use the Changelog file
to detect an update.

DONE. Subscribe users to imapsync_update mailing-list automatically after the payment.
(done in paypal_send)

DONE. NO became porn! Post on newsgroup comp.mail.imap when a new release comes.
http://groups.google.fr/group/comp.mail.imap

DONE. The --search option allows this (and many other things)
Add option --exclude_messages_with_flag

DONE. (yahoo login is standard now)
Add yahoo imap support:
http://elearningcentral.blogspot.com/2010/07/how-to-use-imapsync-with-yahoo-imap.html
http://www.bwebcentral.com/utils/imapsync-yahoo
See patches/imapsync-yahoo


DONE. Add --subscribe by default.
Subscribe only when needed (not already subscribed).

DONE. Ask Nick Czeczulin why he wrote patent "Method for mailbox migration"
http://www.google.com/patents/about?id=qI_IAAAAEBAJ

DONE. Add --search option allowing to select messages with any IMAP SEARCH command.

DONE. Make --usecache works with --maxage --maxsize etc.

DONE. Fix Exchange 2010 SP1 issue with --foldersizes when
host2 folders don't exist. $imap->exists calls STATUS.
Is it RFC compliant or an Exchange bug?
Exchange quit after 10 errors.

DONE. Make --delete2 works with --useuid

DONE. write a comment to http://blog.migrationwiz.com/2010/12/09/imapsync-vs-migrationwiz/

DONE. Read http://bugs.gentoo.org/show_bug.cgi?id=354831
      Nice conversation.

DONE. Look https://fedorahosted.org/released/imapsync/

DONE. Take a look at https://bitbucket.org/imapsync/imapsync

DONE. Explain expunge behavior in help message.

DONE. Add --authmd51 --authmd52 to permit authmd5 by host.

DONE. Write option --delete2foldersonly regex.
Example: to permit a sync in a subfolder with --delete2folder
--regextrans2 's#(.*)#NEW/$1#'  --delete2foldersonly /^NEW/

DONE. Write option --delete2foldersbutnot regex.
Example: to permit a sync but not deleting folder OLD
--delete2foldersbutnot /^OLD/

DONE. Add cache to speed up transfer. Option --usecache

DONE. There was a time imapsync took the whole message when the
header was bad (empty). But it was bad for speed with big
messages (nowadays there are always big messages in
mailboxes). May be the best is to take a part of the
body. Have to code this.

DONE. Add a --delete2folders option
"When syncing mailboxes with imapsync, is there a way to delete folders in the
target account? The --delete2 option only seems to delete individual
messages, not folders."

DONE. Add NTLM authentification support. Mail-IMAPClient-3.25
support it.
http://cpansearch.perl.org/src/BUZZ/NTLM-1.05/NTLM.pm
http://curl.haxx.se/rfc/ntlm.html

DONE. Evaluate memory consumption with (or better):
print qx{ ps o pid,pcpu,comm,vsz,rss,size $$ }, "\n"
Search memory leaks with
Test-Weaken Test-Memory-Cycle Devel-Cycle Devel-Leak Test-Weaken
sh -x tests.sh ll_bigmail
sh -x tests.sh ll_memory_consumption
are good candidate to stress memory.
No memory leak detected just up to 8 memory copies of the same data
in Mail::IMAPClient.
Wrote ./learn/memory_consumption to show that it is a Mail::IMAPClient issue.

DONE.Be more effiscient with large mailboxes
Write a Mail::IMAPClient::fetch_hash allowing selecting messages to fetch
4 hours estimated time coding. Time spent 4h30 (with public release and emails)

DONE. Bugfix. Duplicate messages on host2 are not deleted with --delete2
Reason: "Skipping msg #120:508 in host2 folder INBOX.2005-INBOX (duplicate so we ignore this message)"

DONE. Ask namespace capability after login (dovecot 2).
It was a bug from library caching CAPABILITY.

DONE. Write a clean_exit() replacing each die() or exit() call.
Wrote exit_clean() and die_clean() to remove pid file.

DONE. Add a --pidfile option.

DONE. Add NOOP commands to avoid timeouts.
Useless since reconnect mode permits timeouts.

DONE. Add --subscribeall option.
Is it possible to have a option that subscribes all folders regardless of
subscription on the source server? Perhaps --subscribeall?

DONE. Add an option to make imapsync automatically
reconnect when the connection drops

DONE. Add Google adsence to imapsync website to see if it can help.

DONE. Be "FLAGS.SILENT" the normal behavior instead of "+FLAGS.SILENT".

DONE. Add TLS support with patches/imapsync-1.217_tls_support.patch

USELESS. Make --skipheader be multiple. Useless since we can
use the or perl regex '^X-|^Date|^From'


USELESS. Add a --tmpfile option.

DONE. Add --justlogin options
to check usernames and passwords.

DONE. Add a --tmpdir option.

DONE. Fix bug "Found that if folders have a space at the end of
the name, it will not create the folder name on the new
server, nor will it copy that folder's email."
Added regression test checking this.
Could not reproduce this bug.
Seems to be a old imapsync bug with 2.2.9


DONE. (contacted Date::Manip maintener) Fix bug with Date::Manip 6.02

DONE. Explain howto change only the header with --regexmess

DONE.
Change IsUnconnected behavior. If IsUnconnected then print
stats and die. Avoid logout.

DONE. Remove --password "secret"  in outpout.

DONE. Make "--delete 2" be a fatal error.

DONE.
Write to zimbra coders about the date presentation:
Sorting by default by internal date is bad with migrations
and delayed arrivals. An user option would be good.

DONE.
Add failure return code in case of
last FOLDER if $from->IsUnconnected();
last FOLDER if   $to->IsUnconnected();
See Phil Lobbes messages (18 Sep 2008)
Subject: BUG: imapsync may silently disconnect leading to unmigrated data without warnings

DONE. Fix bad english language.

DONE. Add a better test to check non existing folders on destination
server. (status and select are bad).

DONE. Fix the buggy --include behavior with no --folder --folderrec or --subscribed.

DONE. Update the distribution list including imapsync in the
INSTALL chapter. Also in README.

DONE. Start FAQ entries about good options with such imap server.

DONE. Talk about mbox and maildir format when "Invalid header"
problem is encountered. (See FAQ "Invalid header")

DONE. Look at http://barnson.org/node/81
An old blog about imap migration.

DONE. Normally, "no header used or found" means imapsync found a message
with no header. It may be a bad message with really no header
or it may be a imap server problem, the server gives no header
for this message. In that case imapsync gets the whole message
to see if there is the same on the other side. It slows the transfer
of course. I think I'll change imapsync behavior and let it
give up those bad messages missing an header.

DONE. Make --syncinternaldates turn on by default

DONE. Check imapsync with gmail (dates problem?).

DONE. Start an imap internet scan project.

DONE. Fix --ssl --justconnect bug

DONE. Fix --ssl needs --authmech login with gmail. Why?

DONE. Try a 50Mo message transfer (slow ?)
fast!

DONE. Add my amazon wishlist link.

DONE. Remove file rfc2342.txt "IMAP4 Namespace"

DONE. Add an --exactsync option to remove target messages
that are not on source. Maybe --delete2 is a better name.

DONE. Fix the "word too large" or "Command line too long"
      bug with some imap servers.

DONE. Add features from Herman (patch against 1.139 and
again with 1.156 by Kjetil Torgrim Homme)
 - Support for authenticating as different (admin) user
 - Support for different auth mechanisms (we needed PLAIN)

DONE. Make --include --exclude options possibly be a list.

DONE. Look at http://freshmeat.net/projects/freshmeat-submit/

DONE in FAQ.Talk about :
"I think notes is right but MS Eudora is bad with dates.
(netscape or thunderbird is ok with dates)
use the option --syncinternaldates "

DONE. Create a imapsync mailing list.
May be I won't since we can use comp.mail.imap
http://groups.google.fr/group/comp.mail.imap
to talk about imapsync

DONE. Add a --recurse option when --folder option is used.
No I won't since this feature can be done by doing a
--include '^INBOX.MyFolder'
Yes done with --folderrec

Done. Look at comp.mail.imap http://groups.google.fr/groups?q=imapsync
In fact I set up an google alert on imapsync so I read
any posted question about imapsync posted  to any group
followed by google.

DONE. Add usage about --regextrans2
To remove INBOX. in the name of destination folders:
--regextrans2 's/^INBOX\.(.+)/$1/'
To sync a complete account in a subfolder called FOO:
--regextrans2 's/^INBOX(.*)/INBOX.FOO$1/'

DONE. Make the --justconnect a real "just connect" connection,
not a auth connection like it is now.

DONE. Add --prefix1 option.
The need is to remove this prefix when building target folder names.

DONE. Read the IMAP RFC http://www.rfc-editor.org/rfc/rfc2342.txt
Respect the NAMESPACE behavior like the RFC.

DONE. Add in doc: Does imapsync support IMAP over TLS (IMAPS)?

DONE. Add the licence of each IMAP software (the free ones).

DONE. Add  MailEnable version 1.54 if
Javier Gomez succeed. He failed.
MailEnable is an early stage developpment IMAP server.

DONE. Update doc with ALL options.

DONE. Think about speed.

DONE. Pb if "to separator" is in "from folder" name.
Have to choose a caracter != to separator and
not in from folders. The solution can be to
just exchange the two separator caracters.

DONE. Look at the separator() function in Mail::IMAPClient

DONE. Add SASL support MD5 : DIGEST-MD5 and CRAM-MD5
see authenticate in IMAPClient.pm
Test the new Mail::IMAPClient (2.1.4 -> 2.2.6 or sup)
userdbpw -hmac-md5 | userdb userdb set hmac-md5pw
http://www.inter7.com/courierimap/INSTALL.html

DONE. Add a --subscribe option to subscribe folders on the
      destination server.

DONE. Add a --subscribed option to transfert only subscribed
      folder on the source server.

DONE. Add a CREDITS file.

DONE. Pb with namespace INBOX. Some IMAP servers
don't have this CAPABILITY
http://www.inter7.com/courierimap/FAQ.html#namespace
http://www.rfc-editor.org/rfc/rfc2342.txt

DONE. Pb with date

DONE. Pb with \Recent flag

DONE. Add the list of tested IMAP servers (successes and failures).