diff --git a/ChangeLog b/ChangeLog index c22609c..6b2e318 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,17 +1,34 @@ RCS file: RCS/imapsync,v Working file: imapsync -head: 1.476 +head: 1.480 branch: locks: strict - gilles: 1.476 + gilles: 1.480 access list: symbolic names: keyword substitution: kv -total revisions: 476; selected revisions: 476 +total revisions: 480; selected revisions: 480 description: ---------------------------- -revision 1.476 locked by: gilles; +revision 1.480 locked by: gilles; +date: 2012/01/08 05:11:03; author: gilles; state: Exp; lines: +8 -7 +a1.net fix. +---------------------------- +revision 1.479 +date: 2011/12/16 05:30:27; author: gilles; state: Exp; lines: +33 -26 +Unified outout in --debug mode. Lines start with Host1 or Host2. +---------------------------- +revision 1.478 +date: 2011/12/16 04:10:32; author: gilles; state: Exp; lines: +11 -9 +Changed output about messages ignored to better reflect what is going on. +---------------------------- +revision 1.477 +date: 2011/12/11 10:07:09; author: gilles; state: Exp; lines: +8 -8 +--split down to 100 to follow the recommendation of RFC2683 section 3.2.1.5 +Thanks to Phil Pearl for this advice. +---------------------------- +revision 1.476 date: 2011/12/10 01:33:50; author: gilles; state: Exp; lines: +34 -8 Applied Ameir Abdeldayem patch to deal with errors like "Host1 Sent/15 size 1428 ignored (no header so we ignore this message)" diff --git a/FAQ b/FAQ index 4265cc9..a345516 100644 --- a/FAQ +++ b/FAQ @@ -1,5 +1,5 @@ #!/bin/cat -# $Id: FAQ,v 1.100 2011/12/10 01:38:36 gilles Exp gilles $ +# $Id: FAQ,v 1.102 2012/01/08 06:40:50 gilles Exp gilles $ +------------------+ | FAQ for imapsync | @@ -139,6 +139,22 @@ Q. Is is legal? R. Yes, the license permits it http://imapsync.lamiral.info/COPYING +======================================================================= +Q. I use --useuid which uses a cache in /tmp or --tmpdir, the hostnames + host1 or host2 has changed but mailboxes are the same. Will imapsync + generate duplicate messages on next runs? + +R. Yes + +Q. How can I fix this? + +R. The cache path reflects hostnames or ip adresses, just change the + directory names of host1 or host2. Use --dry to see if next runs + will generate duplucates. + By default the cache is like + + /tmp/imapsync_cache/host1/user1/host2/user2/... + ======================================================================= Q. How can I speed up transfers? @@ -463,29 +479,29 @@ Solution: run imapsync a second time. imapsync synchronizes flags on each run. ======================================================================= -Q. Some passwords contain * and " characters. Login fails. +Q. On Unix, some passwords contain * and " characters. Login fails. R. Use a backslash to escape the characters: - imapsync --password1 \"password\" + imapsync ... --password1 \"password\" It works for the star * character, I don't know if it works for the " character. ======================================================================= -Q. Out of memory on FreeBSD +Q. On Windows, some passwords contain $ characters. Login fails. +R1. Enclose passwords between "" -R. http://groups.google.com/group/lucky.freebsd.questions/browse_thread/thread/f4218e4252863328 + imapsync ... --password1 "zzz$zz$$z" -See the user limit with the command - ulimit -a -To change it, try - ulimit -d 1000000000 -Also -http://www.unixadmintalk.com/f41/perl-out-memory-sbrk-9112/ -The default hard datasize limit on FreeBSD is 512MB. To raise it, put this -(or more) in /boot/loader.conf and reboot: +R2. Prefix each $ character with a ^ since ^ is the escape character + on Windows -kern.maxdsiz="1024M" + imapsync ... --password1 zzz^$zz^$^$z + +For a password that is exactly the 8 characters string $%&<>|^" +you have to enter + + imapsync ... --password1 "$%%&<>|^"^" ======================================================================= Q. With huge account (many messages) when it comes to reading the diff --git a/Makefile b/Makefile index 2564221..bc9e1a8 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ -# $Id: Makefile,v 1.89 2011/11/13 08:53:12 gilles Exp gilles $ +# $Id: Makefile,v 1.90 2012/01/08 06:39:38 gilles Exp gilles $ .PHONY: help usage all @@ -40,7 +40,7 @@ hello: echo "$(IMAPClient)" -all: ChangeLog README VERSION imapsync_elf_x86.bin +all: ChangeLog README VERSION imapsync_elf_x86.bin imapsync.exe testp : @@ -281,11 +281,11 @@ upload_lfo: /home/gilles/public_html/www.linux-france.org/html/prj/imapsync/.htaccess sh ~/memo/lfo-rsync -upload_index: index.shtml +upload_index: index.shtml FAQ validate --verbose index.shtml rcsdiff index.shtml rsync -avH index.shtml root@ks.lamiral.info:/var/www/imapsync/ - rsync -avH index.shtml \ + rsync -avH index.shtml FAQ \ ../../public_html/www.linux-france.org/html/prj/imapsync/ sh $(HOME)/memo/lfo-rsync diff --git a/README b/README index 34e2e29..73236dc 100644 --- a/README +++ b/README @@ -3,7 +3,7 @@ NAME Synchronise mailboxes between two imap servers. Good at IMAP migration. More than 44 different IMAP server softwares supported with success. - $Revision: 1.476 $ + $Revision: 1.480 $ SYNOPSIS To synchronise imap account "foo" on "imap.truc.org" to imap account @@ -305,9 +305,10 @@ IMAP SERVERS names are in alphabetic order): - 1und1 H mimap1 84498 [host1] - - a1.net imap.a1.net IMAP4 Ready WARSBL614 00029c23 [host1] + - a1.net imap.a1.net IMAP4 Ready [host1] - Archiveopteryx 2.03, 2.04, 2.09, 2.10 [host2], 3.0.0 [host2] (OSL 3.0) http://www.archiveopteryx.org/ + - Axigen Mail Server Version 8.0.0 - BincImap 1.2.3 (GPL) (http://www.bincimap.org/) - CommuniGatePro server (Redhat 8.0) (Solaris), CommuniGate Pro 5.2.17[host2] (CentOS 5.4) - Courier IMAP 1.5.1, 2.2.0, 2.1.1, 2.2.1, 3.0.8, 3.0.3, 4.1.1 (GPL) @@ -443,5 +444,5 @@ SIMILAR SOFTWARES Feedback (good or bad) will often be welcome. - $Id: imapsync,v 1.476 2011/12/10 01:33:50 gilles Exp gilles $ + $Id: imapsync,v 1.480 2012/01/08 05:11:03 gilles Exp gilles $ diff --git a/VERSION b/VERSION index aaad981..7a29603 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.476 +1.480 diff --git a/VERSION_EXE b/VERSION_EXE index aaad981..7a29603 100644 --- a/VERSION_EXE +++ b/VERSION_EXE @@ -1 +1 @@ -1.476 +1.480 diff --git a/W/paypal_reply/paypal_build_invoices b/W/paypal_reply/paypal_build_invoices index 1db2d8c..1cfd4ac 100755 --- a/W/paypal_reply/paypal_build_invoices +++ b/W/paypal_reply/paypal_build_invoices @@ -1,6 +1,6 @@ #!/bin/sh -# $Id: paypal_build_invoices,v 1.22 2011/12/02 04:08:46 gilles Exp gilles $ +# $Id: paypal_build_invoices,v 1.23 2012/01/08 00:33:04 gilles Exp gilles $ # usage: sh paypal_build_invoices /g/var/paypal_invoices/???? @@ -20,9 +20,11 @@ set -x #/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 917 /g/paypal/paypal_2011_08_complet.csv #/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 999 /g/paypal/paypal_2011_09_complet.csv #/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 1094 /g/paypal/paypal_2011_10_complet.csv -/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 1185 /g/paypal/paypal_2011_11_complet.csv +#/g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 1185 /g/paypal/paypal_2011_11_complet.csv /g/public_html/imapsync/W/paypal_reply/paypal_bilan --write_invoices --first_in 1263 /g/paypal/paypal_2011_12_complet.csv +: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 147 /g/paypal/paypal_2010_11_complet.csv +: /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 214 /g/paypal/paypal_2010_12_complet.csv : /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 294 --usdeur 1.3385 /g/paypal/paypal_2011_01_complet.csv : /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 382 /g/paypal/paypal_2011_02_complet.csv : /g/public_html/imapsync/W/paypal_reply/paypal_bilan --first_in 473 /g/paypal/paypal_2011_03_complet.csv diff --git a/imapsync b/imapsync index b0eb113..3a44010 100755 --- a/imapsync +++ b/imapsync @@ -20,7 +20,7 @@ Synchronise mailboxes between two imap servers. Good at IMAP migration. More than 44 different IMAP server softwares supported with success. -$Revision: 1.476 $ +$Revision: 1.480 $ =head1 SYNOPSIS @@ -354,9 +354,10 @@ Success stories reported with the following 44 imap servers (software names are in alphabetic order): - 1und1 H mimap1 84498 [host1] - - a1.net imap.a1.net IMAP4 Ready WARSBL614 00029c23 [host1] + - a1.net imap.a1.net IMAP4 Ready [host1] - Archiveopteryx 2.03, 2.04, 2.09, 2.10 [host2], 3.0.0 [host2] (OSL 3.0) http://www.archiveopteryx.org/ + - Axigen Mail Server Version 8.0.0 - BincImap 1.2.3 (GPL) (http://www.bincimap.org/) - CommuniGatePro server (Redhat 8.0) (Solaris), CommuniGate Pro 5.2.17[host2] (CentOS 5.4) - Courier IMAP 1.5.1, 2.2.0, 2.1.1, 2.2.1, 3.0.8, 3.0.3, 4.1.1 (GPL) @@ -518,7 +519,7 @@ Entries for imapsync: Feedback (good or bad) will often be welcome. -$Id: imapsync,v 1.476 2011/12/10 01:33:50 gilles Exp gilles $ +$Id: imapsync,v 1.480 2012/01/08 05:11:03 gilles Exp gilles $ =cut @@ -627,13 +628,14 @@ my( $usecache, $debugcache, $cacheaftercopy, $wholeheaderifneeded, %h1_msgs_copy_by_uid, $useuid, $h2_uidguess, $addheader, + %h1, %h2, ); # main program # global variables initialisation -$rcs = '$Id: imapsync,v 1.476 2011/12/10 01:33:50 gilles Exp gilles $ '; +$rcs = '$Id: imapsync,v 1.480 2012/01/08 05:11:03 gilles Exp gilles $ '; $total_bytes_transferred = 0; $total_bytes_skipped = 0; @@ -728,8 +730,8 @@ check_lib_version() or exit_clean(0) if ($justbanner); # By default, 1000 at a time, not more. -$split1 ||= 500; -$split2 ||= 500; +$split1 ||= 100; +$split2 ||= 100; $host1 || missing_option("--host1") ; $port1 ||= (defined $ssl1 and !defined $tls1) ? 993 : 143; @@ -1120,17 +1122,18 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { my $h1_msgs_all_hash_ref = { } ; my @h1_msgs = select_msgs( $imap1, $h1_msgs_all_hash_ref ); - ( $debug or $debugLIST ) and print "Host1 LIST: ", scalar( @h1_msgs ), " messages [@h1_msgs]\n" ; - #( $debug or $debugLIST ) and print "Host1 LIST: ", scalar( @$h1_msgs_all_ref ), " messages [@$h1_msgs_all_ref]\n" ; - # internal dates on host2 are after the ones on host1 - # normally... + my $h1_msgs_nb = scalar( @h1_msgs ) ; + $h1{ $h1_fold }{ 'messages_nb' } = $h1_msgs_nb ; + + ( $debug or $debugLIST ) and print "Host1 LIST: $h1_msgs_nb messages [@h1_msgs]\n" ; my $h2_msgs_all_hash_ref = { } ; my @h2_msgs = select_msgs( $imap2, $h2_msgs_all_hash_ref ) ; - ( $debug or $debugLIST ) and print "Host2 LIST: ", scalar(@h2_msgs), " messages [@h2_msgs]\n"; - #( $debug or $debugLIST ) and print "Host2 LIST ALL: ", scalar( keys %$h2_msgs_all_hash_ref ), " messages [", - # join( ' ', keys( %$h2_msgs_all_hash_ref ) ), "]\n" ; + my $h2_msgs_nb = scalar( @h2_msgs ) ; + $h2{ $h2_fold }{ 'messages_nb' } = $h2_msgs_nb ; + + ( $debug or $debugLIST ) and print "Host2 LIST: $h2_msgs_nb messages [@h2_msgs]\n"; my $cache_base = "$tmpdir/imapsync_cache/$host1/$user1/$host2/$user2"; my $cache_dir = cache_folder( $cache_base, $h1_fold, $h2_fold ); @@ -1204,7 +1207,7 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { my $rc = parse_header_msg($imap1, $m, $h1_heads_ref, $h1_fir_ref, 'Host1', \%h1_hash); if (! defined($rc)) { my $h1_size = $h1_fir_ref->{$m}->{"RFC822.SIZE"} || 0; - print "Host1 $h1_fold/$m size $h1_size ignored (no header so we ignore this message)\n"; + print "Host1 $h1_fold/$m size $h1_size ignored (no wanted headers so we ignore this message)\n" ; $total_bytes_skipped += $h1_size; $nb_msg_skipped += 1; $h1_nb_msg_noheader +=1; @@ -1218,8 +1221,11 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { $h1_nb_msg_duplicate += 1; } } + my $h1_msgs_duplicate_nb = scalar( @h1_msgs_duplicate ) ; + $h1{ $h1_fold }{ 'duplicates_nb' } = $h1_msgs_duplicate_nb ; + + $debug and print "Host1 selected: $h1_msgs_nb duplicates: $h1_msgs_duplicate_nb\n" ; $debug and print "Host1 whole time parsing headers took ", timenext(), " s\n"; - $debug and print "\n"; $debug and print "Host2 parsing headers of folder [$h2_fold]\n"; @@ -1238,7 +1244,7 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { my $rc = parse_header_msg($imap2, $m, $h2_heads_ref, $h2_fir_ref, 'Host2', \%h2_hash); my $h2_size = $h2_fir_ref->{$m}->{"RFC822.SIZE"} || 0; if (! defined($rc)) { - print "Host2 $h2_fold/$m size $h2_size ignored (no header so we ignore this message)\n"; + print "Host2 $h2_fold/$m size $h2_size ignored (no wanted headers so we ignore this message)\n" ; $h2_nb_msg_noheader += 1 ; } elsif(0 == $rc) { # duplicate @@ -1247,6 +1253,10 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { push(@h2_msgs_duplicate, $m); } } + my $h2_msgs_duplicate_nb = scalar( @h2_msgs_duplicate ) ; + $h2{ $h2_fold }{ 'duplicates_nb' } = $h2_msgs_duplicate_nb ; + + $debug and print "Host2 selected: $h2_msgs_nb duplicates: $h2_msgs_duplicate_nb\n" ; $debug and print "Host2 whole time parsing headers took ", timenext(), " s\n"; $debug and print "++++ Verifying [$h1_fold] -> [$h2_fold]\n"; @@ -1318,7 +1328,7 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { } my $h2_uidnext = $imap2->uidnext( $h2_fold ) ; - $debug and print "uidnext h2: $h2_uidnext\n" ; + $debug and print "Host2 uidnext: $h2_uidnext\n" ; $h2_uidguess = $h2_uidnext ; MESS: foreach my $m_id (@h1_hash_keys_sorted_by_uid) { my $h1_size = $h1_hash{$m_id}{'s'}; @@ -1333,7 +1343,7 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { else{ # already on host2 my $h2_msg = $h2_hash{$m_id}{'m'} ; - $debug and print "msg $h1_fold/$h1_msg equals $h2_fold/$h2_msg\n" ; + $debug and print "Host1 found msg $h1_fold/$h1_msg equals Host2 $h2_fold/$h2_msg\n" ; $total_bytes_skipped += $h1_size ; $nb_msg_skipped += 1 ; $debugcache and print "touch $cache_dir/${h1_msg}_$h2_msg\n" if ( $usecache ) ; @@ -1348,9 +1358,9 @@ FOLDER: foreach my $h1_fold (@h1_folders_wanted) { # Good my $h2_size = $h2_hash{$m_id}{'s'}; $debug and print - "msg $h1_fold/$h1_msg sizes $h1_size <> $h2_size $h2_fold/$h2_msg\n"; + "Host1 size msg $h1_fold/$h1_msg = $h1_size <> $h2_size = Host2 $h2_fold/$h2_msg\n"; if( $delete ) { - print "msg $h1_fold/$h1_msg deleted on host1\n"; + print "Host1 msg $h1_fold/$h1_msg deleted\n"; unless( $dry ) { $imap1->delete_message( $h1_msg ); $h1_nb_msg_deleted += 1; @@ -1403,7 +1413,7 @@ sub size_filtered_flag { sub sync_flags { my ( $h1_fold, $h1_msg, $h2_fold, $h2_msg, $permanentflags2, $h1_fir_ref, $h2_fir_ref ) = @_ ; - $debug and print "flags $h1_fold/$h1_msg -> $h2_fold/$h2_msg\n"; + $debug and print "Host1 flags $h1_fold/$h1_msg -> Host2 $h2_fold/$h2_msg\n"; my $h1_size = $h1_fir_ref->{$h1_msg}->{"RFC822.SIZE"} ; return() if size_filtered_flag( $h1_size ) ; @@ -1413,7 +1423,7 @@ sub sync_flags { my $h2_flags = $h2_fir_ref->{ $h2_msg }->{ "FLAGS" } || '' ; ( $debug or $debugflags ) and - print "init msg h1 $h1_fold/$h1_msg flags( $h1_flags ) h2 $h2_fold/$h2_msg flags( $h2_flags )\n" ; + print "Host1 flags init msg $h1_fold/$h1_msg flags( $h1_flags ) Host2 $h2_fold/$h2_msg flags( $h2_flags )\n" ; # RFC 2060: This flag can not be altered by any client $h1_flags =~ s@\\Recent\s?@@gi ; @@ -1427,16 +1437,16 @@ sub sync_flags { my $diff = compare_lists( \@h1_flags, \@h2_flags ); #$diff = 1 ; - ( $debug or $debugflags ) and print "filt msg h1 $h1_fold/$h1_msg flags( $h1_flags ) h2 $h2_fold/$h2_msg flags( $h2_flags )\n" ; + ( $debug or $debugflags ) and print "Host1 flags filt msg $h1_fold/$h1_msg flags( $h1_flags ) Host2 $h2_fold/$h2_msg flags( $h2_flags )\n" ; $diff and ( $debug or $debugflags ) - and print "msg $h2_fold/$h2_msg replacing h2 flags( $h2_flags ) with h1 flags( $h1_flags )\n"; + and print "Host2 flags msg $h2_fold/$h2_msg replacing h2 flags( $h2_flags ) with h1 flags( $h1_flags )\n"; # This sets flags so flags can be removed with this # When you remove a \Seen flag on host1 you want to it # to be removed on host2. Just add flags is not what # we need most of the time. if ( ! $dry and $diff and ! $imap2->store( $h2_msg, "FLAGS.SILENT (@h1_flags)" ) ) { - print "- msg $h2_fold/$h2_msg could not add flags [@h1_flags]: ", + print "Host2 flags msg $h2_fold/$h2_msg could not add flags [@h1_flags]: ", $imap2->LastError, "\n"; #$nb_errors++; } @@ -1845,8 +1855,8 @@ sub banner_imapsync { my @argv_copy = @_; my $banner_imapsync = join("", '$RCSfile: imapsync,v $ ', - '$Revision: 1.476 $ ', - '$Date: 2011/12/10 01:33:50 $ ', + '$Revision: 1.480 $ ', + '$Date: 2012/01/08 05:11:03 $ ', "\n",localhost_info(), "\n", "Command line used:\n", "$0 ", command_line_nopassword(@argv_copy), "\n", @@ -3498,7 +3508,7 @@ sub parse_header_msg { my $headstr; if ( ( ! $headnum ) and ( $wholeheaderifneeded ) ){ - print "$s uid $m_uid no header by classic way so taking whole header\n"; + print "$s uid $m_uid no header by parse_headers so taking whole header with BODY.PEEK[HEADER]\n" ; $imap->fetch($m_uid, "BODY.PEEK[HEADER]"); my $whole_header = $imap->_transaction_literals; @@ -3633,7 +3643,7 @@ sub check_last_release { } sub imapsync_version { - my $rcs = '$Id: imapsync,v 1.476 2011/12/10 01:33:50 gilles Exp gilles $ '; + my $rcs = '$Id: imapsync,v 1.480 2012/01/08 05:11:03 gilles Exp gilles $ '; $rcs =~ m/,v (\d+\.\d+)/; my $VERSION = ($1) ? $1: "UNKNOWN"; return($VERSION); diff --git a/index.shtml b/index.shtml index c1a0071..447e1ac 100644 --- a/index.shtml +++ b/index.shtml @@ -5,7 +5,7 @@ Imapsync: an IMAP migration tool ( release <!--#exec cmd="cat VERSION"--> ) - + @@ -45,8 +45,8 @@

imapsync software is a command line tool allowing incremental and recursive IMAP transfers from one mailbox to another, both anywhere on the internet -or in your local network. Incremental means you can stop the transfer at any time -and restart it later efficiently. Recursive means all folders hierarchy can be copied. +or in your local network. "Incremental" means you can stop the transfer at any time +and restart it later efficiently. "Recursive" means all folders hierarchy can be copied.

imapsync is useful for imap account migration or imap account backup. @@ -60,11 +60,14 @@ or mbsync (written by Michael for 2 ways synchronizations.

+

Alternatives to imapsync are listed in the Similar softwares section. +

+

Some numbers for 2011

@@ -90,11 +93,24 @@ for 2 ways synchronizations.

New features or bugfixes since previous releases:

+ + + @@ -354,14 +370,16 @@ Don't hesitate to have a try, I will help you and make efforts to switch them to
  • (2011) MDaemon 12.0.3 as host2 but MDaemon is supported as host1. MDaemon is simply buggy with the APPEND IMAP command with any IMAP email client.
    • -

    Now the long reported success stories list: 44 different imap server softwares supported!
    +

    Now the long reported success stories list: 46 different imap server softwares supported!
    [host1] means "source server" and [host2] means "destination server":

    -

    Similar softwares

    +

    Similar softwares