]
[--noreleasecheck]
@@ -317,7 +317,7 @@ IMAP SERVERS
- Hotmail since hotmail.com does not provide IMAP access
- Outlook.com since outlook.com does not provide IMAP access
- Success stories reported with the following 53 imap servers (software
+ Success stories reported with the following 55 imap servers (software
names are in alphabetic order):
- 1und1 H mimap1 84498 [host1] H mibap4 95231 [host1]
@@ -325,6 +325,7 @@ IMAP SERVERS
- Apple Server 10.6 Snow Leopard [host1]
- Archiveopteryx 2.03, 2.04, 2.09, 2.10 [host2], 3.0.0 [host2]
(OSL 3.0) http://www.archiveopteryx.org/
+ - Atmail 6.x [host1]
- Axigen Mail Server Version 8.0.0
- BincImap 1.2.3 (GPL) (http://www.bincimap.org/)
- CommuniGatePro server (Redhat 8.0) (Solaris), CommuniGate Pro 5.2.17[host2] (CentOS 5.4)
@@ -373,6 +374,7 @@ IMAP SERVERS
- OpenWave
- Oracle Beehive [host1]
- Qualcomm Worldmail (NT)
+ - QQMail IMAP4Server [host1] [host2] https://en.mail.qq.com/
- Rockliffe Mailsite 5.3.11, 4.5.6
- Samsung Contact IMAP server 8.5.0
- Scalix v10.1, 10.0.1.3, 11.0.0.431, 11.4.6
@@ -464,5 +466,5 @@ SIMILAR SOFTWARES
Feedback (good or bad) will often be welcome.
- $Id: imapsync,v 1.555 2013/07/23 11:41:10 gilles Exp gilles $
+ $Id: imapsync,v 1.564 2013/08/18 19:28:47 gilles Exp gilles $
diff --git a/TODO b/TODO
index 8691162..9475bd2 100644
--- a/TODO
+++ b/TODO
@@ -1,5 +1,5 @@
#!/bin/cat
-# $Id: TODO,v 1.122 2013/08/03 02:08:33 gilles Exp gilles $
+# $Id: TODO,v 1.123 2013/08/16 00:49:52 gilles Exp gilles $
TODO file for imapsync
----------------------
@@ -17,8 +17,14 @@ http://www.yippiemove.com/
http://www.migrationwiz.com/
http://www.microsoft.com/download/en/details.aspx?id=1329 "Microsoft Transporter Suite"
-Print the timeout value.
-Inform --timeout is is seconds.
+Convert folder names to utf-8 and print them next to the uft-7 ones.
+Look at imapsync/W/learn/
+./imap_utf7 data_utf7
+
+DONE. Inform about --addheader when the problem occurs "(no wanted headers so we ignore this message)"
+
+DONE. Print the timeout value.
+DONE. Inform --timeout is is seconds.
Add an exit value when exiting because of --exitwhenover
The goal is to know easily why to restart later.
@@ -50,6 +56,13 @@ Look at leveldb (Simon Th actux advice)
Look at http://fallabs.com/kyotocabinet/ (JYB advice)
Also Redis (Ludovic Danigo advice)
+I am writing now to suggest that imapsync provide info containing the parameters
+with which a message is determined to be a duplicate, so that if I wanted to confirm
+they are in fact duplicates by looking at the messages themselves, enough info would
+be available to easily locate them. One or two duplicates per folder are easy to
+find, but I had one large folder with quite a few, and in this case it would have
+been a big help.
+
Find a way to avoid passwords in --debugimap unless needed.
diff --git a/TUTORIAL.html b/TUTORIAL.html
new file mode 100644
index 0000000..1d73641
--- /dev/null
+++ b/TUTORIAL.html
@@ -0,0 +1,129 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tutorial for imapsync
+
+Introduction
+
+Three Internet protocols share the big pie
+access of almost all email accounts : POP, IMAP, HTTP.
+
+
+The oldest one is POP, Post Office Protocol, it allows only
+one main box, also called INBOX.
+The second protocol is IMAP, Internet Message Access Protocol, which allows
+a hierarchy of mailboxes also called folders, allows also concurrent accesses,
+tagging with flags, search by many criterium like date, subject, size etc.
+The third protocol is HTTP, HyperText Transfer Protocol, via webmails.
+Webmails often offer the same features than imap servers and,
+since webmails background is often an imap server,
+a parallel access via IMAP.
+
+
+Software imapsync is a command line tool to
+copy, migrate, backup or synchronize IMAP mailboxes.
+
+
+Command line means imapsync is not graphical, it is textual,
+you have to type characters on your keyboard.
+Your fingers will not suffer anyway because
+I wrote examples nearly ready to go.
+Most of the time you only have to change values
+and adapt them to your context.
+
+
+Do not be afraid, the mouse will not be forsaken.
+You can still use the mouse to launch an editor,
+select/copy/paste complete examples
+and run the little script you edit with a doubleclick.
+
+
+Imapsync runs on Unix or Windows.
+It is because imapsync is written in the Perl language
+and thanks to the Perl creators Perl runs everywhere.
+Outside imapsync life is different;
+Historically Windows came after Unix and the marvelous designers
+of this old time decided it would be very cool
+to not share the same syntax for doing the same things.
+Thanks guys, great thinking!
+
+
+To avoid you to learn by headaches a system you do not master
+I will give all examples in both worlds, Unix and Windows.
+Macintosh users are in the Unix world now but do not tell them,
+it can hurt the olders.
+
+
+Conventions
+
+Imapsync has many options but you can ignore most of them
+and still make great transfers.
+
+
+In this documentation I write all the examples as a complete example.
+In order to simplify the reading or the printing,
+the command is written on several lines.
+It could be written on one single line;
+if you prefer the whole command on one line then
+just remove the last character of each line,
+it is the "\" character on Unix examples
+or the "^" character on Windows examples.
+
+
+For example, on Unix
+
+
+ imapsync \
+ --host1 imap.truc.org \
+ --user1 foo \
+ --password1 secret1 \
+ ...
+
+
+is equivalent to
+
+
+ imapsync --host1 imap.truc.org --user1 foo --password1 secret1 ...
+
+
+
+and on Windows
+
+
+ imapsync ^
+ --host1 imap.truc.org ^
+ --user1 foo ^
+ --password1 secret1 ^
+ ...
+
+
+is equivalent to
+
+
+ imapsync --host1 imap.truc.org --user1 foo --password1 secret1 ...
+
+
+
+
+
+
diff --git a/TUTORIAL.t2t b/TUTORIAL.t2t
new file mode 100644
index 0000000..996a2c1
--- /dev/null
+++ b/TUTORIAL.t2t
@@ -0,0 +1,97 @@
+
+
+
+% $Id: TUTORIAL.t2t,v 1.2 2013/08/16 00:48:57 gilles Exp gilles $
+
+= Tutorial for imapsync =
+
+== Introduction ==
+
+Three Internet protocols share the big pie
+access of almost all email accounts : POP, IMAP, HTTP.
+
+The oldest one is POP, Post Office Protocol, it allows only
+one main box, also called INBOX.
+The second protocol is IMAP, Internet Message Access Protocol, which allows
+a hierarchy of mailboxes also called folders, allows also concurrent accesses,
+tagging with flags, search by many criterium like date, subject, size etc.
+The third protocol is HTTP, HyperText Transfer Protocol, via webmails.
+Webmails often offer the same features than imap servers and,
+since webmails background is often an imap server,
+a parallel access via IMAP.
+
+
+Software imapsync is a command line tool to
+copy, migrate, backup or synchronize IMAP mailboxes.
+
+
+Command line means imapsync is not graphical, it is textual,
+you have to type characters on your keyboard.
+Your fingers will not suffer anyway because
+I wrote examples nearly ready to go.
+Most of the time you only have to change values
+and adapt them to your context.
+
+Do not be afraid, the mouse will not be forsaken.
+You can still use the mouse to launch an editor,
+select/copy/paste complete examples
+and run the little script you edit with a doubleclick.
+
+Imapsync runs on Unix or Windows.
+It is because imapsync is written in the Perl language
+and thanks to the Perl creators Perl runs everywhere.
+Outside imapsync life is different;
+Historically Windows came after Unix and the marvelous designers
+of this old time decided it would be very cool
+to not share the same syntax for doing the same things.
+Thanks guys, great thinking!
+
+To avoid you to learn by headaches a system you do not master
+I will give all examples in both worlds, Unix and Windows.
+Macintosh users are in the Unix world now but do not tell them,
+it can hurt the olders.
+
+
+== Conventions ==
+
+Imapsync has many options but you can ignore most of them
+and still make great transfers.
+
+In this documentation I write all the examples as a complete example.
+In order to simplify the reading or the printing,
+the command is written on several lines.
+It could be written on one single line;
+if you prefer the whole command on one line then
+just remove the last character of each line,
+it is the "\" character on Unix examples
+or the "^" character on Windows examples.
+
+For example, on Unix
+
+```
+ imapsync \
+ --host1 imap.truc.org \
+ --user1 foo \
+ --password1 secret1 \
+ ...
+```
+is equivalent to
+```
+ imapsync --host1 imap.truc.org --user1 foo --password1 secret1 ...
+```
+
+and on Windows
+```
+ imapsync ^
+ --host1 imap.truc.org ^
+ --user1 foo ^
+ --password1 secret1 ^
+ ...
+```
+is equivalent to
+```
+ imapsync --host1 imap.truc.org --user1 foo --password1 secret1 ...
+```
+
+
+
diff --git a/VERSION b/VERSION
index 0a92d41..4b5d34e 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-1.555
+1.564
diff --git a/VERSION_EXE b/VERSION_EXE
index 35f5e47..4b5d34e 100644
--- a/VERSION_EXE
+++ b/VERSION_EXE
@@ -1 +1 @@
-1.558
+1.564
diff --git a/W/.BUILD_EXE_TIME b/W/.BUILD_EXE_TIME
index c35dff0..bc03f25 100644
--- a/W/.BUILD_EXE_TIME
+++ b/W/.BUILD_EXE_TIME
@@ -224,3 +224,7 @@
1375497013 END 1.558 : samedi 3 août 2013, 04:30:13 (UTC+0200)
1375534760 BEGIN 1.558 : samedi 3 août 2013, 14:59:20 (UTC+0200)
1375536286 END 1.558 : samedi 3 août 2013, 15:24:46 (UTC+0200)
+1376617407 BEGIN 1.563 : vendredi 16 août 2013, 03:43:27 (UTC+0200)
+1376618294 END 1.563 : vendredi 16 août 2013, 03:58:14 (UTC+0200)
+1376856084 BEGIN 1.564 : dimanche 18 août 2013, 22:01:24 (UTC+0200)
+1376856924 END 1.564 : dimanche 18 août 2013, 22:15:24 (UTC+0200)
diff --git a/W/learn/imapbyhand b/W/learn/imapbyhand
index fc8e169..ce04934 100644
--- a/W/learn/imapbyhand
+++ b/W/learn/imapbyhand
@@ -1,4 +1,5 @@
+# $Id: $
# How to remove a message by hand.
telnet imap.truc.com 143
@@ -10,3 +11,21 @@ a64 STORE 1:1 +FLAGS (\DELETED)
a75 EXPUNGE
a86 LOGOUT
+# When no SEARCH ALL available
+1 LOGIN test@coook.cn 1234@com
+3 SELECT INBOX
+4 UID SEARCH 1:11638
+5 UID FETCH 11620:11637 (RFC822.SIZE)
+6 UID FETCH 11000:11637 (RFC822.SIZE)
+7 UID FETCH 2:11637 (RFC822.SIZE)
+8 UID FETCH 1:11637 (RFC822.SIZE)
+9 UID FETCH 1:11638 (RFC822.SIZE)
+10 UID FETCH 1:11639 (RFC822.SIZE)
+10 UID FETCH 11000:11640 (RFC822.SIZE) BAD
+
+10 UID FETCH 1:11639 (RFC822.SIZE) OK
+
+11 STATUS INBOX MESSAGES
+11 STATUS INBOX (MESSAGES)
+11 STATUS INBOX (MESSAGES UIDNEXT)
+
diff --git a/W/paypal_reply/paypal_build_invoices b/W/paypal_reply/paypal_build_invoices
index 19111f3..9cf0810 100755
--- a/W/paypal_reply/paypal_build_invoices
+++ b/W/paypal_reply/paypal_build_invoices
@@ -1,6 +1,6 @@
#!/bin/sh
-# $Id: paypal_build_invoices,v 1.54 2013/07/28 04:19:32 gilles Exp gilles $
+# $Id: paypal_build_invoices,v 1.55 2013/08/17 23:43:38 gilles Exp gilles $
# usage: sh paypal_build_invoices /g/var/paypal_invoices/????
@@ -39,7 +39,8 @@ cp /home/gilles/public_html/AGIL/factures/000/facture_imapsync-000.tex /g/var/pa
#/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 2519 /g/paypal/paypal_2013_04_complet.csv
#/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 2572 /g/paypal/paypal_2013_05_complet.csv
#/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 2627 /g/paypal/paypal_2013_06_complet.csv
-/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 2682 /g/paypal/paypal_2013_07_complet.csv
+#/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 2682 /g/paypal/paypal_2013_07_complet.csv
+/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 2741 /g/paypal/paypal_2013_08_complet.csv
: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 147 /g/paypal/paypal_2010_11_complet.csv
: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 214 /g/paypal/paypal_2010_12_complet.csv
@@ -73,9 +74,10 @@ cp /home/gilles/public_html/AGIL/factures/000/facture_imapsync-000.tex /g/var/pa
: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 2519 /g/paypal/paypal_2013_04_complet.csv
: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 2572 /g/paypal/paypal_2013_05_complet.csv
: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 2627 /g/paypal/paypal_2013_06_complet.csv
+: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 2682 /g/paypal/paypal_2013_07_complet.csv
set -x
-: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 2682 /g/paypal/paypal_2013_07_complet.csv
+: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 2741 /g/paypal/paypal_2013_08_complet.csv
set +x
# La totale
diff --git a/imapsync b/imapsync
index 4c8a44b..7bc80e1 100755
--- a/imapsync
+++ b/imapsync
@@ -22,7 +22,7 @@ Synchronises mailboxes between two imap servers.
Good at IMAP migration. More than 52 different IMAP server softwares
supported with success, few failures.
-$Revision: 1.558 $
+$Revision: 1.564 $
=head1 SYNOPSIS
@@ -114,7 +114,7 @@ The option list:
[--nofoldersizes] [--nofoldersizesatend]
[--dry]
[--debug] [--debugimap][--debugimap1][--debugimap2]
- [--timeout ] [--fast]
+ [--timeout ]
[--split1] [--split2]
[--reconnectretry1 ] [--reconnectretry2 ]
[--noreleasecheck]
@@ -367,7 +367,7 @@ Failure stories reported with the following 3 imap servers:
- Hotmail since hotmail.com does not provide IMAP access
- Outlook.com since outlook.com does not provide IMAP access
-Success stories reported with the following 53 imap servers
+Success stories reported with the following 55 imap servers
(software names are in alphabetic order):
- 1und1 H mimap1 84498 [host1] H mibap4 95231 [host1]
@@ -375,6 +375,7 @@ Success stories reported with the following 53 imap servers
- Apple Server 10.6 Snow Leopard [host1]
- Archiveopteryx 2.03, 2.04, 2.09, 2.10 [host2], 3.0.0 [host2]
(OSL 3.0) http://www.archiveopteryx.org/
+ - Atmail 6.x [host1]
- Axigen Mail Server Version 8.0.0
- BincImap 1.2.3 (GPL) (http://www.bincimap.org/)
- CommuniGatePro server (Redhat 8.0) (Solaris), CommuniGate Pro 5.2.17[host2] (CentOS 5.4)
@@ -423,6 +424,7 @@ Success stories reported with the following 53 imap servers
- OpenWave
- Oracle Beehive [host1]
- Qualcomm Worldmail (NT)
+ - QQMail IMAP4Server [host1] [host2] https://en.mail.qq.com/
- Rockliffe Mailsite 5.3.11, 4.5.6
- Samsung Contact IMAP server 8.5.0
- Scalix v10.1, 10.0.1.3, 11.0.0.431, 11.4.6
@@ -534,7 +536,7 @@ Entries for imapsync:
Feedback (good or bad) will often be welcome.
-$Id: imapsync,v 1.558 2013/08/03 01:13:37 gilles Exp gilles $
+$Id: imapsync,v 1.564 2013/08/18 19:28:47 gilles Exp gilles $
=cut
@@ -655,13 +657,14 @@ my(
$messageidnodomain,
$fixInboxINBOX,
$maxlinelength,
+ $uidnext_default,
);
# main program
# global variables initialisation
-$rcs = '$Id: imapsync,v 1.558 2013/08/03 01:13:37 gilles Exp gilles $ ';
+$rcs = '$Id: imapsync,v 1.564 2013/08/18 19:28:47 gilles Exp gilles $ ';
$total_bytes_transferred = 0;
$total_bytes_skipped = 0;
@@ -844,7 +847,7 @@ if ( $uidexpunge2 and not Mail::IMAPClient->can( 'uidexpunge' ) ) {
exit_clean( 3 ) ;
}
-if ( $delete2 and not defined( $uidexpunge2 ) ) {
+if ( ( $delete2 or $delete2duplicates ) and not defined( $uidexpunge2 ) ) {
if ( Mail::IMAPClient->can( 'uidexpunge' ) ) {
print "Info: will act as --uidexpunge2\n" ;
$uidexpunge2 = 1 ;
@@ -872,6 +875,8 @@ if ($syncinternaldates) {
print "Info: turned OFF syncinternaldates\n";
}
+
+
if (defined($authmd5) and ($authmd5)) {
$authmd51 = 1 ;
$authmd52 = 1 ;
@@ -908,6 +913,10 @@ $authuser2 ||= $user2;
print "Info: will try to use $authmech1 authentication on host1\n";
print "Info: will try to use $authmech2 authentication on host2\n";
+$timeout = defined( $timeout ) ? $timeout : 120 ;
+print "Info: imap connexions timeout is $timeout seconds\n";
+
+
$syncacls = (defined($syncacls)) ? $syncacls : 0 ;
$foldersizes = (defined($foldersizes)) ? $foldersizes : 1 ;
$foldersizesatend = (defined($foldersizesatend)) ? $foldersizesatend : $foldersizes ;
@@ -920,6 +929,10 @@ $fastio2 = (defined($fastio2)) ? $fastio2 : 0;
$reconnectretry1 = (defined($reconnectretry1)) ? $reconnectretry1 : 3;
$reconnectretry2 = (defined($reconnectretry2)) ? $reconnectretry2 : 3;
+# Since select_msgs() returns no messages when uidnext does not return something
+# then $uidnext_default is never used. So I have to remove it.
+$uidnext_default = 999999 ;
+
@useheader = ( "Message-Id", "Message-ID", "Received" ) unless ( @useheader ) ;
my %useheader ;
@@ -1221,7 +1234,7 @@ FOLDER: foreach my $h1_fold ( @h1_folders_wanted ) {
last FOLDER if $imap2->IsUnconnected();
my $h1_msgs_all_hash_ref = { } ;
- my @h1_msgs = select_msgs( $imap1, $h1_msgs_all_hash_ref, $search1 );
+ my @h1_msgs = select_msgs( $imap1, $h1_msgs_all_hash_ref, $search1, $h1_fold );
last FOLDER if $imap1->IsUnconnected();
my $h1_msgs_nb = scalar( @h1_msgs ) ;
@@ -1231,7 +1244,7 @@ FOLDER: foreach my $h1_fold ( @h1_folders_wanted ) {
$debug and print "Host1 selecting messages of folder [$h1_fold] took ", timenext(), " s\n";
my $h2_msgs_all_hash_ref = { } ;
- my @h2_msgs = select_msgs( $imap2, $h2_msgs_all_hash_ref, $search2 ) ;
+ my @h2_msgs = select_msgs( $imap2, $h2_msgs_all_hash_ref, $search2, $h2_fold ) ;
last FOLDER if $imap2->IsUnconnected();
my $h2_msgs_nb = scalar( @h2_msgs ) ;
@@ -1306,7 +1319,8 @@ FOLDER: foreach my $h1_fold ( @h1_folders_wanted ) {
$h1_fir_ref = $imap1->fetch_hash( \@h1_msgs, "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h1_fir_ref )
if ( @h1_msgs ) ;
}else{
- $h1_fir_ref = $imap1->fetch_hash( '1:999999', "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h1_fir_ref )
+ my $uidnext = $imap1->uidnext( $h1_fold ) || $uidnext_default ;
+ $h1_fir_ref = $imap1->fetch_hash( "1:$uidnext", "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h1_fir_ref )
if ( @h1_msgs ) ;
}
$debug and print "Host1 getting flags idate and sizes of folder [$h1_fold] took ", timenext(), " s\n";
@@ -1323,7 +1337,7 @@ FOLDER: foreach my $h1_fold ( @h1_folders_wanted ) {
my $rc = parse_header_msg($imap1, $m, $h1_heads_ref, $h1_fir_ref, 'Host1', \%h1_hash);
if (! defined($rc)) {
my $h1_size = $h1_fir_ref->{$m}->{"RFC822.SIZE"} || 0;
- print "Host1 $h1_fold/$m size $h1_size ignored (no wanted headers so we ignore this message)\n" ;
+ print "Host1 $h1_fold/$m size $h1_size ignored (no wanted headers so we ignore this message. To solve this: use --addheader)\n" ;
$total_bytes_skipped += $h1_size;
$nb_msg_skipped += 1;
$h1_nb_msg_noheader +=1;
@@ -1353,8 +1367,17 @@ FOLDER: foreach my $h1_fold ( @h1_folders_wanted ) {
$debug and print "Host2 getting flags idate and sizes of folder [$h2_fold]\n" ;
@$h2_fir_ref{@h2_msgs} = ( ); # fetch_hash can select by uid with last arg as ref
- $h2_fir_ref = $imap2->fetch_hash( \@h2_msgs, "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h2_fir_ref)
- if (@h2_msgs);
+
+
+ if ( $abletosearch ) {
+ $h2_fir_ref = $imap2->fetch_hash( \@h2_msgs, "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h2_fir_ref)
+ if (@h2_msgs) ;
+ }else{
+ my $uidnext = $imap2->uidnext( $h2_fold ) || $uidnext_default ;
+ $h2_fir_ref = $imap2->fetch_hash( "1:$uidnext", "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h2_fir_ref )
+ if ( @h2_msgs ) ;
+ }
+
$debug and print "Host2 getting flags idate and sizes of folder [$h2_fold] took ", timenext(), " s\n" ;
my @h2_msgs_duplicate;
@@ -1687,6 +1710,8 @@ sub sync_flags {
$h1_flags = flags_for_host2( $h1_flags, $permanentflags2 ) ;
+ $h2_flags = flagsCase( $h2_flags ) ;
+
( $debug or $debugflags ) and
print "Host1 flags filt msg $h1_fold/$h1_msg flags( $h1_flags ) Host2 $h2_fold/$h2_msg flags( $h2_flags )\n" ;
@@ -2238,8 +2263,8 @@ sub banner_imapsync {
my @argv = @_ ;
my $banner_imapsync = join("",
'$RCSfile: imapsync,v $ ',
- '$Revision: 1.558 $ ',
- '$Date: 2013/08/03 01:13:37 $ ',
+ '$Revision: 1.564 $ ',
+ '$Date: 2013/08/18 19:28:47 $ ',
"\n",localhost_info(), "\n",
"Command line used:\n",
"$0 ", command_line_nopassword( @argv ), "\n",
@@ -2794,7 +2819,7 @@ sub foldersizes {
}
my $hash_ref = { } ;
- my @msgs = select_msgs( $imap, undef, $search_cmd ) ;
+ my @msgs = select_msgs( $imap, undef, $search_cmd, $folder ) ;
$nb_msgs = scalar( @msgs ) ;
my $biggest_in_folder = 0 ;
@$hash_ref{ @msgs } = ( undef ) if @msgs ;
@@ -2802,7 +2827,8 @@ sub foldersizes {
if ( $abletosearch ) {
$imap->fetch_hash( \@msgs, "RFC822.SIZE", $hash_ref) or die_clean("$@" ) ;
}else{
- $imap->fetch_hash( '1:999999', "RFC822.SIZE", $hash_ref ) or die_clean( "$@" ) ;
+ my $uidnext = $imap->uidnext( $folder ) || $uidnext_default ;
+ $imap->fetch_hash( "1:$uidnext", "RFC822.SIZE", $hash_ref ) or die_clean( "$@" ) ;
}
for ( keys %$hash_ref ) {
my $size = $hash_ref->{ $_ }->{ "RFC822.SIZE" } ;
@@ -3095,26 +3121,35 @@ sub tests_ucsecond {
return( ) ;
}
+
sub select_msgs {
- my ( $imap, $msgs_all_hash_ref, $search_cmd ) = @_ ;
+ my ( $imap, $msgs_all_hash_ref, $search_cmd, $folder ) = @_ ;
+ my ( @msgs ) ;
+
+ if ( $abletosearch ) {
+ @msgs = select_msgs_by_search( $imap, $msgs_all_hash_ref, $search_cmd, $folder ) ;
+ }else{
+ @msgs = select_msgs_by_fetch( $imap, $msgs_all_hash_ref, $search_cmd, $folder ) ;
+ }
+
+}
+
+sub select_msgs_by_search {
+ my ( $imap, $msgs_all_hash_ref, $search_cmd, $folder ) = @_ ;
my ( @msgs, @msgs_all ) ;
# Need to have the whole list in msgs_all_hash_ref
# without calling messages() several times.
# Need all messages list to avoid deleting useful cache part
# in case of --search or --minage or --maxage
+
if ( ( defined( $msgs_all_hash_ref ) and $usecache )
or ( not defined( $maxage ) and not defined( $minage ) and not defined( $search_cmd ) )
) {
- if ( $abletosearch ) {
- $debugdev and print "Calling messages()\n" ;
- @msgs_all = $imap->messages() ;
- }else{
- $debugdev and print "Calling fetch_hash()\n" ;
- @msgs_all = sort { $a <=> $b } keys( %{$imap->fetch_hash( '1:999999', "RFC822.SIZE")} ) ;
- $debugdev and print "Done fetch_hash()\n" ;
- }
-
+
+ $debugdev and print "Calling messages()\n" ;
+ @msgs_all = $imap->messages() ;
+
return if ( $#msgs_all == 0 && !defined( $msgs_all[0] ) ) ;
if ( defined( $msgs_all_hash_ref ) ) {
@@ -3136,6 +3171,57 @@ sub select_msgs {
return( @msgs );
}
+
+sub select_msgs_by_fetch {
+ my ( $imap, $msgs_all_hash_ref, $search_cmd, $folder ) = @_ ;
+ my ( @msgs, @msgs_all, %fetch ) ;
+
+ # Need to have the whole list in msgs_all_hash_ref
+ # without calling messages() several times.
+ # Need all messages list to avoid deleting useful cache part
+ # in case of --search or --minage or --maxage
+
+
+ $debugdev and print "Calling fetch_hash()\n" ;
+ my $uidnext = $imap->uidnext( $folder ) or return( ) ;
+ %fetch = %{$imap->fetch_hash( "1:$uidnext", "INTERNALDATE") } ;
+ @msgs_all = sort { $a <=> $b } keys( %fetch ) ;
+ $debugdev and print "Done fetch_hash()\n" ;
+
+ return( ) if ( $#msgs_all == 0 && !defined( $msgs_all[0] ) ) ;
+
+ if ( defined( $msgs_all_hash_ref ) ) {
+ @{ $msgs_all_hash_ref }{ @msgs_all } = () ;
+ }
+ # return all messages
+ if ( not defined( $maxage ) and not defined( $minage ) and not defined( $search_cmd ) ) {
+ return( @msgs_all ) ;
+ }
+
+ if ( defined( $search_cmd ) ) {
+ print "Warning: strange to see --search with --noabletosearch, an error can happen\n" ;
+ @msgs = $imap->search( $search_cmd ) ;
+ return( @msgs ) ;
+ }
+
+ # we are here only if $maxage or $minage is defined
+ my( @max, @min, $maxage_epoch, $minage_epoch ) ;
+ if ( defined( $maxage ) ) { $maxage_epoch = $timestart_int - 86400 * $maxage ; }
+ if ( defined( $minage ) ) { $minage_epoch = $timestart_int - 86400 * $minage ; }
+ foreach my $msg ( @msgs_all ) {
+ my $idate = $fetch{ $msg }->{'INTERNALDATE'} ;
+ #print "$idate\n" ;
+ if ( defined( $maxage ) and ( epoch( $idate ) >= $maxage_epoch ) ) {
+ push( @max, $msg ) ;
+ }
+ if ( defined( $minage ) and ( epoch( $idate ) <= $minage_epoch ) ) {
+ push( @min, $msg ) ;
+ }
+ }
+ @msgs = msgs_from_maxmin( \@max, \@min ) ;
+ return( @msgs ) ;
+}
+
sub select_msgs_by_age {
my( $imap ) = @_ ;
@@ -3144,26 +3230,53 @@ sub select_msgs_by_age {
if ( defined( $maxage ) ) {
@max = $imap->sentsince( $timestart_int - 86400 * $maxage ) ;
}
- if ( defined($minage ) ) {
+ if ( defined( $minage ) ) {
@min = $imap->sentbefore( $timestart_int - 86400 * $minage ) ;
}
+
+ @msgs = msgs_from_maxmin( \@max, \@min ) ;
+ return( @msgs ) ;
+}
+
+sub msgs_from_maxmin {
+ my( $max_ref, $min_ref ) = @_ ;
+ my( @max, @min, @msgs, @inter, @union ) ;
+
+ @max = @$max_ref ;
+ @min = @$min_ref ;
+
SWITCH: {
unless( defined( $minage ) ) { @msgs = @max ; last SWITCH } ;
unless( defined( $maxage ) ) { @msgs = @min ; last SWITCH } ;
my ( %union, %inter ) ;
foreach my $m ( @min, @max ) { $union{ $m }++ && $inter{ $m }++ }
- @inter = keys( %inter ) ;
- @union = keys( %union ) ;
+ @inter = sort { $a <=> $b } keys( %inter ) ;
+ @union = sort { $a <=> $b } keys( %union ) ;
# normal case
if ( $minage <= $maxage ) { @msgs = @inter ; last SWITCH } ;
# just exclude messages between
if ( $minage > $maxage ) { @msgs = @union ; last SWITCH } ;
}
-
return( @msgs );
}
+sub tests_msgs_from_maxmin {
+ my @msgs ;
+ $maxage = 200 ;
+ @msgs = msgs_from_maxmin( [ '1', '2' ], [ '2', '3' ] ) ;
+ ok( 0 == compare_lists( [ '1', '2' ], \@msgs ), 'msgs_from_maxmin: maxage++' ) ;
+ $minage = 100 ;
+ @msgs = msgs_from_maxmin( [ '1', '2' ], [ '2', '3' ] ) ;
+ ok( 0 == compare_lists( [ '2' ], \@msgs ), 'msgs_from_maxmin: -maxage++minage-' ) ;
+ $minage = 300 ;
+ @msgs = msgs_from_maxmin( [ '1', '2' ], [ '2', '3' ] ) ;
+ ok( 0 == compare_lists( [ '1', '2', '3' ], \@msgs ), 'msgs_from_maxmin: ++maxage-minage++' ) ;
+ $maxage = undef ;
+ @msgs = msgs_from_maxmin( [ '1', '2' ], [ '2', '3' ] ) ;
+ ok( 0 == compare_lists( [ '2', '3' ], \@msgs ), 'msgs_from_maxmin: ++minage-' ) ;
+}
+
sub lastuid {
my $imap = shift ;
@@ -4544,7 +4657,7 @@ sub check_last_release {
}
sub imapsync_version {
- my $rcs_imapsync = '$Id: imapsync,v 1.558 2013/08/03 01:13:37 gilles Exp gilles $ ' ;
+ my $rcs_imapsync = '$Id: imapsync,v 1.564 2013/08/18 19:28:47 gilles Exp gilles $ ' ;
my $imapsync_version ;
if ( $rcs_imapsync =~ m{,v\s+(\d+\.\d+)}xo ) {
@@ -4980,7 +5093,7 @@ sub list_folders_in_2_not_in_1 {
map { $h2_folders_not_in_h1{$_} = 1} @h2_folders_not_in_h1 ;
@h2_folders_not_in_h1 = list_keys_in_2_not_in_1( \%h2_folders_from_1_all, \%h2_folders_not_in_h1) ;
- return( reverse @h2_folders_not_in_1 );
+ return( reverse @h2_folders_not_in_h1 );
}
sub delete_folders_in_2_not_in_1 {
@@ -5195,7 +5308,7 @@ sub epoch {
my ( $mday, $month, $year, $hour, $min, $sec, $sign, $zone_h, $zone_m ) ;
my $time ;
- if ( $d =~ m{(\d{2})-([A-Z][a-z]{2})-(\d{4})\s(\d{2}):(\d{2}):(\d{2})\s((?:\+|-))(\d{2})(\d{2})}xo ) {
+ if ( $d =~ m{(\d{1,2})-([A-Z][a-z]{2})-(\d{4})\s(\d{2}):(\d{2}):(\d{2})\s((?:\+|-))(\d{2})(\d{2})}xo ) {
#print "internal: [$1][$2][$3][$4][$5][$6][$7][$8][$9]\n" ;
( $mday, $month, $year, $hour, $min, $sec, $sign, $zone_h, $zone_m )
= ( $1, $2, $3, $4, $5, $6, $7, $8, $9 ) ;
@@ -5218,6 +5331,12 @@ sub tests_epoch {
ok( '1282658400' eq epoch( '24-Aug-2010 12:00:00 -0200' ), 'epoch 24-Aug-2010 12:00:00 -0200 -> 1282658400' ) ;
ok( '1282658400' eq epoch( '24-Aug-2010 16:01:00 +0201' ), 'epoch 24-Aug-2010 16:01:00 +0201 -> 1282658400' ) ;
ok( '1282658400' eq epoch( '24-Aug-2010 14:01:00 +0001' ), 'epoch 24-Aug-2010 14:01:00 +0001 -> 1282658400' ) ;
+
+ ok( '1280671200' eq epoch( '1-Aug-2010 16:00:00 +0200' ), 'epoch 1-Aug-2010 16:00:00 +0200 -> 1280671200' ) ;
+ ok( '1280671200' eq epoch( '1-Aug-2010 14:00:00 +0000' ), 'epoch 1-Aug-2010 14:00:00 +0000 -> 1280671200' ) ;
+ ok( '1280671200' eq epoch( '1-Aug-2010 12:00:00 -0200' ), 'epoch 1-Aug-2010 12:00:00 -0200 -> 1280671200' ) ;
+ ok( '1280671200' eq epoch( '1-Aug-2010 16:01:00 +0201' ), 'epoch 1-Aug-2010 16:01:00 +0201 -> 1280671200' ) ;
+ ok( '1280671200' eq epoch( '1-Aug-2010 14:01:00 +0001' ), 'epoch 1-Aug-2010 14:01:00 +0001 -> 1280671200' ) ;
return( ) ;
}
@@ -5327,12 +5446,15 @@ Several options are mandatory.
--ssl2 : Use an SSL connection on host2.
--tls1 : Use an TLS connection on host1.
--tls2 : Use an TLS connection on host2.
+--timeout : Connections timeout in seconds. Default is 120.
+ 0 means no timeout.
--folder : Sync this folder.
--folder : and this one, etc.
--folderrec : Sync this folder recursively.
--folderrec : and this one, etc.
--include : Sync folders matching this regular expression
+ Blancs like in "foo bar" have to be written "foo\\ bar"
--include : or this one, etc.
in case both --include --exclude options are
use, include is done before.
@@ -5513,7 +5635,6 @@ sub usage_complete {
is the number of messages handled per request.
default is like --split1 500.
--split2 : same thing on host2.
---timeout : imap connect timeout.
EOF
return( ) ;
}
@@ -5690,7 +5811,7 @@ sub tests_debug {
SKIP: {
skip "No test in normal run" if ( not $tests_debug );
- tests_subject( ) ;
+ tests_msgs_from_maxmin( ) ;
}
return( ) ;
}
@@ -5741,6 +5862,7 @@ sub tests {
tests_fix_Inbox_INBOX_mapping( ) ;
tests_max_line_length( ) ;
tests_subject( ) ;
+ tests_msgs_from_maxmin( ) ;
}
return( ) ;
}
diff --git a/index.shtml b/index.shtml
index 3ab01a4..6282c3b 100644
--- a/index.shtml
+++ b/index.shtml
@@ -5,7 +5,7 @@
Official imapsync migration tool ( release <!--#exec cmd="cat ./VERSION"--> )
-
+
@@ -117,7 +117,7 @@ total is 93 millions for 2011, 91 millions for 2012, 66 millions for half 2013
-1.558
+1.564
Enhancement:
Enhancement:
Enhancement:
@@ -127,7 +127,7 @@ total is 93 millions for 2011, 91 millions for 2012, 66 millions for half 2013Usability:
Usability:
Bug fix:
-Bug fix:
+:
Bug fix:
Bug fix:
Refactoring:
@@ -138,14 +138,30 @@ total is 93 millions for 2011, 91 millions for 2012, 66 millions for half 2013
-- 1.558 Better Office365 and Mailenable handling
+- 1.564 QQMail support
+- Enhancement: Adapted behavior for allowing --maxdate --mindate with --noabletosearch.
+Use internat date instead of Date: header.
+- Enhancement: QQMail IMAP4Server success (with --noabletosearch option)
+- Usability: Apply same treatment to --delete2duplicates as --delete2.
+If --uidexpunge2 can be done, do it, else do --expunge2 (unless --nouidexpunge2 or --noexpunge2).
+- Usability: Added --addheader suggestion in outpout in case messages with no header are found.
+- Usability: Added --timeout in the --help output. Default timeout is 120 seconds now
+(was nothing by imapsync, 600 with underlying modules).
+
+
+
+- 1.558 SSL fix.
+- Enhancement/Bug fix: Added --ssl1_SSL_version and --ssl2_SSL_version to force the SSL_version
+in case the default auto-negociation does not work. Example: --ssl1 --ssl1_SSL_version SSLv3
+
+
+
+- 1.555 Better Office365 and Mailenable handling
- Enhancement: Added option --messageidnodomain to fix Mailenable bug changing the domain part of Message-Id header, and avoid duplicates
- Enhancement: Added option --syncflagsaftercopy to fix Mailenable bug not getting the flags with the APPEND
- Enhancement: Added option --maxlinelength to skip messages whose max line length is over a number of bytes.
Exchange 2013 and Office365 need --maxlinelength 1000 (which is a RFC2822 must) to avoid disconnections.
-- Enhancement/Bug fix: Added --ssl1_SSL_version and --ssl2_SSL_version to force the SSL_version
-in case the default auto-negociation does not work. Example: --ssl1 --ssl1_SSL_version SSLv3
-
+
- Usability: Added back the banner of IMAP servers, ie, the first line given after the connection established.
- Usability: Added --fixInboxINBOX, turned on by default, to map automatically Inbox INBOX folder names.
- Usability: IMAP server removing consecutive spaces on the header part doesn't generate duplicates (MailEnable does that)
@@ -621,7 +637,7 @@ I like it.
Now the long reported success stories list:
-53 different imap server softwares supported!
+55 different imap server softwares supported!
[host1] means "source server" and [host2] means "destination server":
@@ -635,6 +651,7 @@ I like it.
Apple Server 10.6 Snow Leopard [host1]
Archiveopteryx 2.03, 2.04, 2.09, 2.10 [host2], 3.0.0 [host2]
(OSL 3.0) (http://www.archiveopteryx.org/)
+ Atmail 6.x [host1] http://atmail.com/
Axigen Mail Server Version 8.0.0 (http://www.axigen.com/)
BincImap 1.2.3 (GPL) (http://www.bincimap.org/)
CommuniGatePro server (Redhat 8.0) (Solaris), CommuniGate Pro 5.2.17[host2] (CentOS 5.4)
@@ -695,6 +712,7 @@ I like it.
(http://www.oracle.com/technetwork/middleware/beehive/)
Qualcomm Worldmail (NT) (http://www.eudora.com/worldmail/)
Rockliffe Mailsite 5.3.11, 4.5.6 (http://www.mailsite.com/)
+ QQMail IMAP4Server [host1] [host2] (See FAQ) https://en.mail.qq.com/
Samsung Contact IMAP server 8.5.0
Scalix v10.1, 10.0.1.3, 11.0.0.431, 11.4.6 (http://www.scalix.com/)
SmarterMail, Smarter Mail 5.0 Enterprise, Smarter Mail 5.5 [host1], SmarterMail Professional 10.2 [host1]. (http://www.smartertools.com/)
@@ -757,19 +775,19 @@ Prices are given par mailbox and may be outdated (december 2011).
![Viewable With Any Browser]()
@@ -779,7 +797,7 @@ alt="Viewable With Any Browser" />
This document last modified on
-($Id: index.shtml,v 1.175 2013/08/03 16:24:06 gilles Exp gilles $)
+($Id: index.shtml,v 1.180 2013/08/19 21:30:37 gilles Exp gilles $)