News about next imapsync, currently distributed , next and previous releases (back to menu)

imapsync was written on

See ChangeLog to read what has been done in details since 2001.

New features or bugfixes since previous releases:

• 1.836 More secure by default, ssl or tls activation!
• Enhancement: An Imapsync Docker image available!
• Usability: Now goes to SSL by default if port 993 is open. Use --nosslcheck to avoid that.
• Usability: Now goes to TLS by default if possible, ie, only if STARTTLS is in CAPABILITY. If you want only TLS and nothing else, use --tls1 --nossl1
• Usability: Now if you want a basic imap connection on port 143 with no default encryption behavior, ie, no ssl nor tls, then use --nossl1 --notls1 for host1 and --nossl2 --notls2 for host2.
• Enhancement: Added --gmail1 and --gmail2 to simplify Gmail options setting. It sets parameters suggested in the Gmail FAQ --ssl, --host, etc.
• Enhancement: Added --office1 and --office2 to simplify Office 365 options setting. It sets parameters suggested in the Exchange/Office365 FAQ.
• Enhancement: Added --domino1 and --domino2 to simplify Domino options setting. It sets parameters suggested in the Domino FAQ.
• Enhancement: Added --maxsleep in order to avoid timeouts with --maxbytespersecond and --maxmessagespersecond options. By default imapsync will sleep 2 seconds maximum, like if the command line contained --maxsleep 2
• Enhancement: Added --maxbytesafter in order to start --maxbytespersecond limitation only after --maxbytesafter amount of data transferred. Usefull for Gmail limits, for example, in order to active a 50K/s limit rate only after 500 MB of data transfer, use --maxbytesafter 500_000_000 --maxbytespersecond 50_000
• Enhancement: Added --testsunit in order to run any unit test individualy from the command line. Several --testsunit are allowed. Example: imapsync --testsunit tests_true --testsunit tests_always_fail
• Enhancement: Added password setting via environment variables IMAPSYNC_PASSWORD1 and IMAPSYNC_PASSWORD2
• Usability: No more useless and false warning "says it has NO CAPABILITY for AUTHENTICATE LOGIN"
• Usability: Options --delete1 and --delete are now aliases. Option --delete1 is preferable over --delete (--delete is still supported).
• Usability: Now prints always permanentflags info. It helps to understand most flag issues at first run, without --debugflags
• Usability: Now prints "could not append ( Subject:[$subject], Date:[$h1_date], Size:[$h1_size] )" when append fails.
• Usability: Option --showpasswords now shows also passwords with --debugimap. Useful to debug quoting issues.
• Usability: --ipv4 is now synonym of --inet4 and --ipv6 is now synonim of --inet6
• Usability: Added --testslive6 to check pure ipv6 connectivity.
• Enhancement: Added --noabletosearch1 and --noabletosearch2; Still support --noabletosearch, which turn on both --noabletosearch1 and --noabletosearch2
• Enhancement: Added --abort option to terminate a previous call still running. In command line context --abort uses the pidfile to know what to abort. In cgi context, ie online, exact same credentials are needed in order to really abort the other sync.
• Enhancement: Added milliseconds in the default logfile name since several runs is possible within one second in cgi context or on a powerful machine.
• Docker context: Added docker context in order to be run under the nobody user without permission issues.
• Docker context: Can run imapsync --tests under nobody user on Unix (or at least Linux).
• Bugfix: Fixed issue "SSL routines:ssl3_check_cert_and_algorithm:dh key too small" with openssl-dh-key-too-small-error SSL_cipher_list => 'DEFAULT:!DH'
• CGI context: Allow parameters passed by POST.
• CGI context: Abort, before doing anything, if the server load is already too heavy, and invite to come later depending on the current load (1, 5, or 15 minutes later).
• Bug fix: Guessed prefix is the empty string even when there is no folders (which is a bad sign anyway, since INBOX should be listed).
• Bug fix: Option --skipmess could not work most of the time. I guess it was a mistake arrived by badly converting an "unless" to an "if". Perl critics with no tests added => caveat emptor!
• Refactoring: Removed Mail::IMAPClient overload definitions
• Dependency added: IO::Socket::SSL
• Dependency added: Sys::MemInfo
• Dependency added: Pod::Usage

• 1.727 https website and CGI on the way!
• Security: Thanks to free and easy letsencrypt, Imapsync website is available in https: https://imapsync.lamiral.info/
• Enhancement: Added option --delete1emptyfolders. It deletes empty folders on host1, INBOX excepted. Useful with --delete since what remains on host1 is only what failed to be synced.
• Enhancement: Imapsync can be used as a CGI program. Still experimental. The web UI interface has to be written.
• Enhancement: XOAUTH2 authentication works with json files without extra tool openssl. Old pk12 files are still supported (still with openssl).
• Enhancement: Memory usage passed from 5x to 2x of the biggest message size. Some users do send 700MB messages, imapsync needed 3.5 GB of RAM, now it crunches only 1.4 GB, a more common RAM value available on small hosts.
• Enhancement: Added option --inet4 to force ipv4 in ssl and tls modes. Useful with Gmail and Office365 when ipv6 is disable on the imapsync host but not ipv6 resolution.
• Enhancement: Reconnect to imap servers with a Ctrl-c. Exit with two Ctrl-c within one second or one Ctrl-\
• Enhancement: Copy is not done when one --pipemess command ends with an empty message generated.
• Enhancement: STDERR of --pipemess commands are reported on STDOUT and in the logfile (they were ignored before).
• Enhancement:
• Dependency: Added Perl module dependency of Readonly to transform hard coded magic numbers into names. Mandatory module.
• Dependency: Added Perl module dependency of Test::MockObject for some tests. Only a "require" for now.
• Security: In CGI mode, remove all content in the unsafe "eval"uated options (--regextrans2 --regexflag --regexmess --skipmess --pipemess --delete2foldersonly --delete2foldersbutnot). Consider these options not available in CGI mode.
• Bug fix: Fixed -1 wrong output in statistics for big accounts (%d => %s) in printf.
• Bug fix: Uncommented call to $test_builder->reset( ) Don't know why it was commented but it was bad for knowing tests go wrong.
• Bug fix: Check connected state in sub create_folder(). Avoid multiple useless failures.
• Refactoring: Wrote Perl package Imapsync::Getopt::Long to handle CGI requests like command line options. Started as a copy of Luke Ross Getopt::Long::CGI
• Refactoring: Perlcritic cleanup 627 violations of CodeLayout::ProhibitTrailingWhitespace.
• Refactoring: Perlcritic cleanup 60 Double-sigil dereferences.
• Refactoring: Perlcritic cleanup 458 ValuesAndExpressions::ProhibitInterpolationOfLiterals. "Useless interpolation of literal string"
• Refactoring: Perlcritic cleanup 420 CodeLayout::ProhibitParensWithBuiltins. "Builtin function called with parentheses". (Severity: 1).
• Refactoring: Perlcritic cleanup, various small advices.
• Refactoring: Replaced print by myprint, printf by myprintf, sprinf by mysprintf. Will ease --quiet mode and general output handling.

• 1.684 Small changes, bugfixes
• Bug fix: Imapsync.exe updated to Mail-IMAPClient-3.38 (3.37 had a bug with credentials containing special characters)
• Usability: Added inline help about "use --noexpungeaftereach to speed up" if --delete is used.
• Usability: Added --sslargs1 in documentation, usage option list.
• Bug fix: No more warning about "Use of uninitialized value $sockargs[7] in join or string at" in ssl or tls mode. Was due to "'SSL_version' => undef".
• Bug fix: No more "Can't use an undefined value as a HASH reference at /usr/bin/imapsync line 1247."
• Bug fix: Added --expungeaftereach check in sub delete_message_on_host1().

• 1.678 Improved website! (I hope...)
• Enhancement: Added --sslargs1 and --sslargs2 to pass any ssl parameter for host1 or host2 connection. Example: --sslargs1 SSL_verify_mode=1 --sslargs1 SSL_version=SSLv3
• Enhancement: Added --timeout1 int and --timeout2 int in seconds (--timeout int still available to set both with the same value)
• Enhancement: Added --debugssl int. Default is like --debugssl 1 (Only print out errors).
• Enhancement: Added several polls to know where I shall focus improvements.
• Usability: Added env_proxy call in sub xoauth2() to read proxy settings from environment variable http_proxy without using PERL_LWP_ENV_PROXY=1
• Usability: Wrote TUTORIAL_Unix.
• Usability: Check f1 folder exists when specified by --f1f2 and warns when it doesn't
• Usability: Added --automap --justfolders --dry in imapsync_example.sh and imapsync_example.bat
• Bug fix: Added require Encode::Byte to solve "The locale codeset (cp1252) isn't one that perl can decode" on Win32.
• Refactoring: Removed --allow3xx option.
• Refactoring: Continue to move to one global $sync-> in order to reduce number of parameters in routines.

• 1.670 Folders mapping made easy with --automap
• Enhancement: Added option --automap that guesses folders mapping, for folders like "Sent", "Junk", "Drafts", "All", "Archive", "Flagged".
  • IMAP Special-Use folders described in rfc6154 are supported and also well known names for Exchange and common clients.
  • Automap is turned off by default, to try it a safe method is to use --automap --justautomap, it will exit early
• Enhancement: Added --f1f2 str1=str2 : Force folder str1 to be synced to str2.
  • Option --f1f2 overrides any --automap mapping and any --regextrans2 transformation.
  • Example --f1f2 Spam=Junk to map Spam folder to Junk.
• Enhancement: Added --justautomap to exit after seeing what will happen with --automap and --f1f2 options.
• Enhancement: Added IMAP4 QUOTA values (RFC 2087) when they are available. A warning goes up when it's time to find space.
• Enhancement: Added IMAP4 ID extension.
  • ID feature is on by default if supported by the IMAP servers, use --noid to turn it off.
  • ID Infos returned by the imap servers are shown.
  • Infos sent to the servers are name=imapsync, version=`imapsync --version`, os=$OSNAME`, vendor='Gilles LAMIRAL' and date=`rcs date of release`.
  • For github or other distributors you can hack this part by searching $imapsync_id_github
• Enhancement: Added --logdir path to change the default log directory. Default is LOG_imapsync/
• Usability: Added folders counting in outputs.
• Usability: Better output in the login part
• Usability: Guess prefixes and separators instead of forcing the user to find them.
• Usability: Output, added Host1: or Host2: at the beginning of lines.
• Bug fix: SSL_VERIFY_NONE in --ssl and --tls modes
• Bug fix: Fixed xoauth2 calls. Now XOAUTH2 Works on Windows --authmech1 XOAUTH2 with openssl installed. See FAQ.d/FAQ.XOAUTH2.txt.
• Bug fix: Fixed IO::Socket::SSL constants issue with latest releases
• Bug fix: Changed "Host1: checking all wanted folders exist" not efficient algorithm in order to allow efficiently a 2.4 million folders account. Yes, some accounts have this.
• Refactoring: Started to use global $sync-> in order to reduce number of parameters in routines.
• Refactoring: Some perlcritic fixes.

• 1.644 Mac OS X standalone binary provided!
• Enhancement: Mac OS X standalone binary called imapsync_bin_Darwin
• Enhancement: Added option --subfolder2 SUB Move whole host1 folders hierarchy under folder SUB.
• Enhancement: Added --fetch_hash_set "1:*" to permit Mail2World success. Need a patched Mail::IMAPClient 3.35 in sub fetch_hash()
• Usability: No folders sizes if --justfolders, unless really wanted.
• Usability: No warning about messages when --dry --justfolders are used together.
• Bug fix: Added NOOP in --dry mode during fake APPEND in order to avoid timeouts

• 1.636
• Enhancement: Added errors dump listing at the end, before last folder sizes or the statistics. Turned on by default. Use --noerrorsdump to avoid it.
• Enhancement: Added --maxlinelengthcmd that will be called upon when a line over --maxlinelength value is detected.
• Enhancement: Exchange maxlinelength issue fixed with --maxlinelengthcmd 'reformime -r7' on Linux. Install maildrop package to get reformime command
• Enhancement: Added --testslive. Running imapsync --testslive performs a live test on real accounts (I own).
• Enhancement: Added --pipemess in order to pass all message to an external filter tool like reformime -r7.
• Enhancement: Added xoauth2 support, available on Unix with underlying openssl. Thanks to Joaquin Lopez.
• Enhancement: Added --skipmess regex to skip messages matching a regex. Example --skipmess 'm/[\x80-ff]/' to slip messages with non-7bit characters ( ie a byte somewhere begins with 1 )
• Usability: Better output of folders excluded by --exclude and folders included by --include
• Usability: Added inline help in many places. What you can do with switches to change imapsync behaviour, contextually. Learn imapsync the easy way.
• Bug fix: imapsync --tests should work under any circumstance now, any Unix, Windows, exe or bin or script. 561 non-regression tests
• Bug fix: Options --folderfirst and --folderlast generated an error when their value folder does not exist. Existence is checked.
• Bug fix: Option --disarmreadreceipts used to change Disposition-Notification-To in the body when not available in the header. Now never changes the body in all cases.
• Bug fix: Change default useheader values. Now it is really like --useheader "Message-Id" --useheader "Received". HMailServer replies two lines with --useheader "Message-Id" --useheader "Message-ID" in older imapsync releases.
• Bug fix: in imap_utf7_decode() + character must not be escaped. Was a bug with Cyrillic characters
• Refactoring: Started to split the FAQ file in several parts in FAQ.d/*
• Refactoring: Split the INSTALL file in several parts in INSTALL.d/*

• 1.607 Folders shown also in utf8.
• Caveat to upgrade on Unix!: New Unicode::String Perl module dependency, for utf8 output. See the INSTALL file.
• Enhancement: Added --skipemptyfolders to skip syncing empty host1 folders. They are not created on host2.
• Enhancement: Windows exe now uses IO::SSL 2.002 instead of 1.98 (is it really an enhancement?).
• Usability: Remove /x from regexes applied with --include and --exclude. Blanks no longer have to be explicit with \ or [ ].
• Usability: Added utf8 output for folder names, protocol native utf7imap special characters are not user friendly readable
• Usability: Moved foldersizes output after folders lists output.
• Usability: Added total size transferred after each message copied.
• Usability: After ETA, added number of total to be synced, as a denominator dddd after each message copied, like nnn/dddd where nnn is the number of messages copied. Before there was only nnn without /dddd.
• Bug fix: Cache fix on Windows. File paths with brackets [] are special, they have to be escaped with glob. For example, [Gmail] must be written [[]Gmail[]].
• Bug fix: Added several checks if IsUnconnected. Goal avoid imap commands while disconnected. Reconnexion would be better anyway.

• 1.597 Small things
• Enhancement: Added --nomixfolders to avoid merging folders that are considered different on host1 but the same on destination host2 because of case sensitivities and insensitivities.
• Bug fix: Fixed "imapsync doesn't see created folders in the listing of folder sizes at the end". (second time this bug shows up)

• 1.592 Logging by default! (an internal feature now)
• Caveat to upgrade: New IO::Tee Perl module dependency, for logging. See the INSTALL file.
• Windows upgrade: Win32 binary is now distributed in a zip file with two batches scripts ready to use it.
• Enhancement: Added logging by default in subdirectory ./LOG_imapsync/
• Enhancement: Added option --log turned on by default. Use --nolog to disable logging.
• Enhancement: Added option --logfile. Default is like ./LOG_imapsync/YYYY_MM_DD_HH_MM_SS_user2.txt where YYYY_MM_DD_HH_MM_SS is the launching date and time like 2014_05_20_23_45_10
• Enhancement: Added option --disarmreadreceipts to fix resending read receipts on host2. A must with Exchange as host2.
• Enhancement: Added FTGate support (no EXAMINE imap command used anymore).
• Usability: Updated the code to easily switch on --noreleasecheck by default but allowing --releasecheck (some users want this feature).
• Usability: Warn syncing messges will not be simulated when --dry mode is on and folder on host2 does not exist yet.
• Bug fix: Make TLS works with last IO::Socket::SSL default value (Added SSL_verify_mode => 'SSL_VERIFY_NONE') in --tls mode.
• Bug fix: Fixed bug when creating folders on host2 with --sep2=\\ (Lotus Notes).
• Bug fix: Try to get folder size when a folder on host2 has the same name but with a different case like OLD_MAIL -> old_mail. This avoids "does not exist" output.

• 1.584
• Enhancement: Added --minmaxlinelength to select messages with long lines only. It helps to diagnostic Echange error on messages with lines longer than 9000 characters
• Enhancement: Added --debugmaxlinelength
• Bug fix: --ssl1 --tls2 was buggy because of default SSL_VERIFY_PEER. "Can not go to tls encryption on [localhost]:Unable to start TLS: Cannot determine peer hostname for verification..."
• Bug fix: Check if going to tls is ok, exit otherwise with explicit error message. Thanks to Dennis Schridde for reporting this ugly bug that deserves this CVE-2014-2014.
• Debug: Print separator given by NAMESPACE even when --sep1 oe --sep2 is used.
• Debug: Prints prefix given by NAMESPACE even when --prefix1 or --prefix2 is used. It is a preparation for advising not to use --prefix unless suggested.

Good changes made before, listed because they can help:

• Enhancement: Added --skipcrossduplicates to avoid copying messages that are already copied in another folder, good from Gmail to X.
• Enhancement: Added --debugcrossduplicates to print which messages (UIDs) are skipped with --skipcrossduplicates (and in what other folders they are).
• Enhancement: Added --folderfirst string: sync this folder first. --folderfirst "Work" --folderfirst "Friends"
• Enhancement: Added --folderlast string: sync this folder last. --folderlast "[Gmail]/All Mail" etc.
• Enhancement: Added --maxmessagespersecond to limit messages tranfer rate and fix Office365 throttle limitation (use --maxmessagespersecond 4).
• Enhancement: Added --maxbytespersecond to limit byte transfer rate.
• Enhancement: Added --debugmemory option that prints memory consumption after each message is copied.
• Enhancement: Added --authmech EXTERNAL.
• Enhancement: Adapted behavior for allowing --maxdate --mindate with --noabletosearch. Use internat date instead of Date: header.
• Enhancement: QQMail IMAP4Server success (with --noabletosearch option)
• Usability: Apply same treatment to --delete2duplicates as --delete2. If --uidexpunge2 can be done, do it, else do --expunge2 (unless --nouidexpunge2 or --noexpunge2).
• Usability: Added --timeout in the --help output. Default timeout is 120 seconds now (was nothing by imapsync, 600 with underlying modules).
• Enhancement/Bug fix: Added --ssl1_SSL_version and --ssl2_SSL_version to force the SSL_version in case the default auto-negociation does not work. Example: --ssl1 --ssl1_SSL_version SSLv3
• Enhancement: Added option --messageidnodomain to fix Mailenable bug changing the domain part of Message-Id header, and avoid duplicates
• Enhancement: Added option --syncflagsaftercopy to fix Mailenable bug not getting the flags with the APPEND
• Enhancement: Added option --maxlinelength to skip messages whose max line length is over a number of bytes. Exchange 2013 and Office365 need --maxlinelength 1000 (which is a RFC2822 must) to avoid disconnections.
• Usability: Added --fixInboxINBOX, turned on by default, to map automatically Inbox INBOX folder names.
• Usability: IMAP server removing consecutive spaces on the header part doesn't generate duplicates (MailEnable does that)
• Usability: SSL_verify_mode 0 to avoid warning about Man-In-The-Middle.
• Enhancement: Added XOAUTH authentication. Thanks to Eduardo Bortoluzzi Junior.
• Enhancement: Added --search1 and --search2 to allow different searches on each host.
• Usability: Added --delete1 as an alias for --delete
• Speed: Option --nocheckmessageexists is activated by default since --checkmessageexists often slow down transfers too much.
• Usability: Option --foldersizesatend is on if --foldersizes is on. Off if --nofoldersizesatend
• Usability: Added host2 minus host1 statistic: number of messages and bytes. One difference at the start and one at the end. Useful to detect quickly a difference between host2 and host1. Need both --foldersizes and --foldersizesatend options (which are on by default).
• Usability: Added --foldersizesatend. It gets the folders sizes at the end of the whole transfer. Turned on by default (use --nofoldersizesatend to turn it off).
• Enhancement: Added option --delete2duplicates; it deletes messages in host2 that are duplicates. --delete2duplicates is on when --delete2 is set unless --nodelete2duplicates is set too.
• Enhancement: Added option --pidfilelocking; it aborts imapsync, when just launched, in case another imapsync may be running. Turned off by default
• Usability: Added option --showpasswords. it shows passwords on output instead of "MASKED". Turned off by default. Useful to restart a run by reading the log or to see what happen with special characters.
• Bug fix: Added option --fixslash2 to avoid the 'Invalid mailbox name' error. It acts when --sep1 is character / and --sep2 is not character / and host1 folder name contains --sep2 characters. Turned on by default (--nofixslash2 to unable it).
• Enhancement: Added option --noabletosearch to allow the listing of messages without the imap "SEARCH ALL" command. It's useful for playing with poor imap servers like Softalk 7.6.4 (8.6 is fine with SEARCH ALL).
• Enhancement: Added --noexpungeaftereach to speedup --delete --expunge from Gmail.
• Enhancement: Added option --nocheckmessageexists to speed up with Tobit imap server as host1.
• Enhancement: Added option --exitwhenover option to avoid locking when transfers exceed maximum limit. See for example this Gmail constraint
  • imapsync ... --exitwhenover 2500000000 # 2.5GB if host1 is Gmail
  • imapsync ... --exitwhenover 500000000 # 500MB if host2 is Gmail
• Usability: Default headers used to identify a message are like --useheader "Message-Id" --useheader "Received" (Exchange compatibility bu default).
• Enhancement: Added option --addheader to fix the issue "ignored (no header so we ignore this message)" (on Sent folder usually). Thanks to Ameir Abdeldayem
• Enhancement: Added --nofilterflags to check any STORE flags by ignoring PERMANENTFLAGS (Exchange tests)

This document last modified on ($Id: news.shtml,v 1.50 2017/09/11 03:04:46 gilles Exp gilles $)
Top of the page