#!/bin/cat
# $Id: INSTALL.OnlineUI.txt,v 1.49 2021/03/19 09:39:01 gilles Exp gilles $
This documentation is also located online at
https://imapsync.lamiral.info/INSTALL.d/
https://imapsync.lamiral.info/INSTALL.d/INSTALL.OnlineUI.txt
=======================================================================
Installing imapsync online
=======================================================================
I'm now confident with /X since the /X service is up and running quite
well since January 2017. Anyway, if you run this service on your own
online, you take responsibility for it.
=======================================================================
Hardware consideration
RAM used per imapsync process, mean value: 230 MB.
Average_bandwidth_rate: 345 KiB/s ~ 2.8 Mbps.
Load mean: 0.8 on a CPU 4 cores "Intel(R) i5-2320 3.00GHz K8-class"
=======================================================================
Installation
You have to be a little familiar with what a CGI script is
and how to activate a CGI script on the Apache
HTTP server, or any other HTTP server. I have received demands
to run it on the Ngnix HTTP server but I haven't played with it yet.
Linux is also a preferred platform (I run /X service on Linux and FreeBSD).
I have tested this visual interface on Mac. It works.
For now, it demands some skills few Mac users have.
Drop me a note in case you want to do that.
I have tested this visual interface on Windows,
it fails on Windows because of some hardcoded Unix paths.
I'm working on it to be Windows ok but it's not done yet (May 2020).
Some users have successfully installed a /X visual interface
on Windows using a Linux VM machine.
The web visual user interface frontend /X is compounded in four
files: a html5 file, a CSS file, a javascript file, and
a logo image:
* https://imapsync.lamiral.info/X/imapsync_form_extra.html
* https://imapsync.lamiral.info/X/imapsync_form.css
* https://imapsync.lamiral.info/X/imapsync_form.js
* https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
You can do a "view source" to see the HTML file as
it is written, and a "save" to get it locally.
The three other files can be saved the same way or
with a command named "wget". I suggest using wget,
see below the ready-to-use command lines.
Those four files can be put anywhere on a web server,
as long as they stand in the same directory. If you
want to put them in different directories, just change
the content of imapsync_form_extra.html to reflect the change,
ie, change the two lines referencing imapsync_form.css and imapsync_form.js
href="imapsync_form.css" (near the beginning of imapsync_form_extra.html)
src="imapsync_form.js" (near the end of imapsync_form_extra.html)
I let you change the image logo as an exercise, it's safe if you fail.
The actual imap syncing work is done by imapsync acting as a CGI,
the visual interface is only there to give imapsync the parameters
needed for the sync.
Use at least Perl module CGI.pm release 4.08 (2014-10-18)
to avoid the bug "Undefined subroutine CGI::multi_param".
You can use the command named cpanm to upgrade CGI.pm to
its last version, it's the easiest way.
Print the CGI.pm release with:
perl -MCGI -e 'print "$CGI::VERSION\n"'
If it is under release 4.08 (2014-10-18) then upgrade it with
cpanm CGI
It is a good thing to remove the old one if it was installed
by a distribution package, I let you this part as an exercise too.
To check and fix the Perl modules dependencies, run:
cd
wget -N https://imapsync.lamiral.info/prerequisites_imapsync
sh prerequisites_imapsync
To make imapsync work as a CGI script, there are two conditions.
First, imapsync has to work by itself on the web host.
If imapsync doesn't work by itself, as a command line,
then it won't work as a CGI script.
Second, imapsync has to work by itself on the web host using
the Unix user running the webserver. Detailed examples to
verify that will be provided further in this document.
The imapsync_form_extra.html file in action calls the CGI location
/cgi-bin/imapsync
which has to be imapsync itself, the file script (not the directory).
The very latest and relatively stable imapsync is at
https://imapsync.lamiral.info/imapsync
This file is the program file used verbatim for the service given at
https://imapsync.lamiral.info/X/
Copy the three files imapsync_form.* on a directory that is exported
by your HTTP server.
Copy the imapsync script on the cgi-bin/ directory
allowing CGIs and you'll have your imapsync visual interface
and service. The cgi-bin/ directory is usually outside the
hierarchy exported to anybody by the HTTP server.
The default Apache 2.4 timeout is 60 seconds, one minute, 300 for older Apache.
https://httpd.apache.org/docs/2.4/mod/core.html#timeout
Mine is now "Timeout 3600", 3600 seconds, an hour. I chose this huge
timeout value because imapsync can spend a long time without talking while
getting the headers of huge folders of 100k messages.
If you intend to offer this service for huge mailboxes or for a long
time, I strongly recommand you to set this "Timeout 3600" in
the Apache configuration right now because you will sure end up with this
timeout issue in a few months.
Search for timeouts in the Apache error log to see if you have timeout issues.
Now that I have explained the general context for any system,
I'll describe concrete examples on several systems,
Debian/Ubuntu and Centos. Feedbacks show that the Centos
process is easier in case you don't know very much any of the
Linux distributions.
=============================================================================
A) Concrete example on a Debian server with Apache:
Imapsync script place on the server disk is
/usr/lib/cgi-bin/imapsync
This classical /cgi-bin directory is usually already configured
in the Apache configuration file
/etc/apache2/sites-available/default-ssl
or
/etc/apache2/sites-available/default
This configuration file contains the following section
somewhere, maybe in comments for now, ie, with
some # characters at the beginning to make them ignored:
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
AllowOverride None
# Next line "no-gzip 1" is to avoid output buffering,
# clients can then see the log during the sync
SetEnv no-gzip 1
Options +ExecCGI -MultiViews
# Choose either one or the other, depending on your Apache version
# Lines beginning with # are ignored
# For Apache 2.2
#Order allow, deny
#Allow from all
# Apache 2.4
Require all granted
That's all for the Apache configuration side.
Now get, test, and install the latest imapsync:
cd
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
# some basic tests
./imapsync
./imapsync --testslive
cp imapsync /usr/lib/cgi-bin/
Assuming that the Unix account running Apache is www-data,
check that it will work under Apache with this command:
su -s /bin/sh -c 'SERVER_SOFTWARE=foo /usr/lib/cgi-bin/imapsync' www-data
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
The last command should print something like:
Status: 200 OK to sync IMAP boxes. Load on bar is ...
...
You can also verify that the webserver is not buffering its output with the
command:
wget -nv -S -O- 'http://localhost/cgi-bin/imapsync?testslive=1&simulong=10'
You should get the output as time goes on. If you don't get the
output as time goes on, ie you see no output then all output
at once, it means the webserver is buffering. Fix it with
the "SetEnv no-gzip 1" described above.
The UI front-end file place on the server disk in
this example is
/var/www/html/X/imapsync_form_extra.html
but it can be placed anywhere on the disk, the important
thing is that it has to be served by the webserver.
mkdir /var/www/html/X/
cd /var/www/html/X/
wget -N \
https://imapsync.lamiral.info/X/imapsync_form_extra.html \
https://imapsync.lamiral.info/X/imapsync_form.css \
https://imapsync.lamiral.info/X/imapsync_form.js \
https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
ln -s imapsync_form_extra.html index.html
The imapsync process working directory in cgi mode is
/var/tmp/imapsync_cgi/
it is not configurable unless changing it in
imapsync directly, it is hard-coded in imapsync.
In this directory will go the log files and
the pid files.
Check
http://yourhost/X/imapsync_form_extra.html
or the safer
https://yourhost/X/imapsync_form_extra.html
That's all for installing a /X service on Debian.
=============================================================================
B) Here is a concrete example on a Centos 7 server with the Apache
webserver httpd:
First, follow and apply the section "Centos 7 and latest imapsync"
at https://imapsync.lamiral.info/INSTALL.d/INSTALL.Centos.txt
Then:
yum install httpd
systemctl restart httpd
cpanm CGI
mkdir /var/www/html/X/
cd /var/www/html/X/
wget -N \
https://imapsync.lamiral.info/X/imapsync_form_extra.html \
https://imapsync.lamiral.info/X/imapsync_form.css \
https://imapsync.lamiral.info/X/imapsync_form.js \
https://imapsync.lamiral.info/X/logo_imapsync_Xn.png
ln -s imapsync_form_extra.html index.html
cd
wget -N https://imapsync.lamiral.info/imapsync
chmod +x imapsync
# some basic tests
./imapsync
./imapsync --testslive
cp imapsync /var/www/cgi-bin/
Assuming that the Unix account running Apache is "apache",
which is the default Apache user on Centos system,
check that it will work under Apache with this command:
# a real synchronization but not in cgi context
cd /tmp
su -s /bin/sh -c '/var/www/cgi-bin/imapsync --testslive' apache
# in cgi context but just the imapsync command with no parameter
cd
su -s /bin/sh -c 'SERVER_SOFTWARE=foo /var/www/cgi-bin/imapsync' apache
# a real synchronization in cgi context
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
The last command should print something like:
Status: 200 OK to sync IMAP boxes. Load on bar is ...
...
You can also verify that the webserver is not buffering its output with the
command:
wget -nv -S -O- 'http://localhost/cgi-bin/imapsync?testslive=1&simulong=10'
You should get the output as time goes on. If you don't, no output then all
output at once, it means the webserver is buffering. Fix it with
the "SetEnv no-gzip 1" described above.
Now check
http://yourhost/X/imapsync_form_extra.html
or the safer
https://yourhost/X/imapsync_form_extra.html
That's all for installing a /X service on Centos 7.
B bis) How about Centos 8?
Follow the procedure for Centos 7. While imapsync is ok on the command line,
you will encounter some permission denied in the CGI context. Something like:
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
...
Host1 failure: can not open imap connection on host1 [test1.lamiral.info]
with user [test1]: Unable to connect to test1.lamiral.info: Permission denied
The issue might come from SELinux. I haven't dig into SELinux enough to give
you the commands that will allow imapsync online and only it while maintaining
SELinux in enforcing mode.
Quick solution:
sestatus
setenforce 0
sestatus
To go back to the previous state:
setenforce 1
sestatus
Nota bene
=========
You may also want to avoid being placed by systemd in
a directory like (where xxx are crypto hash characters):
/var/tmp/systemd-private-xxx-httpd.service-xxx/tmp/
In that case, edit the file
/usr/lib/systemd/system/httpd.service
and in it, replace the line
PrivateTmp=true
by the line
PrivateTmp=false
then reload systemd and restart httpd with the commands:
systemctl daemon-reload
systemctl restart httpd
systemctl status httpd
=======================================================================
=================== Bandwidth statistics ==========================
=======================================================================
If you want the bandwidth statistics like the ones at the bottom
of the page and following the image link, more detailed at
https://imapsync.lamiral.info/vnstat/vnstati.html
Those stats are generated by vnstat
https://humdi.net/vnstat/
Vnstat is already available as a package in most Linux distros.
The images are generated by the following commands, every minute:
vnstati -s -o /var/www/vnstat/vnstat_s.png
vnstati -h -o /var/www/vnstat/vnstat_h.png
vnstati -hg -o /var/www/vnstat/vnstat_hg.png
vnstati -hs -o /var/www/vnstat/vnstat_hs.png
vnstati -d -o /var/www/vnstat/vnstat_d.png
vnstati -m -o /var/www/vnstat/vnstat_m.png
vnstati -y -o /var/www/vnstat/vnstat_y.png
vnstati -t -o /var/www/vnstat/vnstat_t.png
vnstati -vs -o /var/www/vnstat/vnstat_vs.png
vnstati -5 -o /var/www/vnstat/vnstat_5.png
=======================================================================
====================== Troubleshooting ============================
=======================================================================
The log says the temporary directory is
/var/tmp/imapsync_cgi/
but it is not in the system.
It may be that the apache or httpd service is run by systemd with a jailed
temporary directory.
Solution:
find /etc/systemd/ /usr/lib/systemd/ |xargs grep -s PrivateTmp
If systemd jails Apache then you'll find a line like:
/etc/systemd/system/multi-user.target.wants/apache2.service:PrivateTmp=true
(Debian/Ubuntu)
or
/usr/lib/systemd/system/httpd.service:PrivateTmp=true
(Centos)
Replace this line "PrivateTmp=true"
in /etc/systemd/system/multi-user.target.wants/apache2.service
or
/usr/lib/systemd/system/httpd.service
with:
PrivateTmp=false
Then reload the systemd daemon and restart the apache2 service
Debian:
systemctl daemon-reload
systemctl restart apache2
systemctl status apache2
Centos:
systemctl daemon-reload
systemctl restart httpd
systemctl status httpd
Then retry
wget -nv -S -O- http://localhost/cgi-bin/imapsync?testslive=1
Look now if /var/tmp/imapsync_cgi/ is there.
=======================================================================
If you encounter this issue:
Failed to find a valid digest in the 'integrity' attribute for resource
'https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js'
with computed SHA-256 integrity 'kZMXypKF3if9/5v2tP9UHBvS/535tSyH7vjszruyCso='.
The resource has been blocked.
It may be because of AdBlock.
Verification:
wget https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js
cat jquery.min.js | openssl dgst -sha384 -binary | openssl base64 -A
gives exactly what is in imapsync_form_extra.html
more imapsync_form_extra.html
...
src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"
integrity="sha384-xBuQ/xzmlsLoJpyjoggmTEz8OWUFM0/RC5BsqQBDX2v5cMvDHcMakNTNrHIW2I5f"
So if your https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js
is not what it should be, your access looks compromised.
Thanks to Dominik Ulrich for this insight!
=======================================================================
=======================================================================
This part is for hackers only.
If you want to use the UI but make it more complicated things than
just run imapsync then use the following files:
imapsync_shell_wrapper instead of imapsync itself
imapsync_form_wrapper.js instead of imapsync_form.js
imapsync_form_wrapper.html instead of imapsync_form.html
How to get those files:
wget -N https://imapsync.lamiral.info/X/imapsync_shell_wrapper \
https://imapsync.lamiral.info/X/imapsync_form_wrapper.js \
https://imapsync.lamiral.info/X/imapsync_form_wrapper.html
Centos:
chmod +x imapsync_shell_wrapper
cp imapsync_shell_wrapper /var/www/cgi-bin/
Debian:
chmod +x imapsync_shell_wrapper
cp imapsync_shell_wrapper /usr/lib/cgi-bin/
Normally, you only have to change the script imapsync_shell_wrapper
to suit your needs.
Have in mind that the abort button will kill only one imapsync so
it is not a working button in case of successive imapsync runs.
=======================================================================
=======================================================================
====== mod_perl failure ======
This part is for mod_perl experts only.
The script imapsync doesn't work under Modperl::Registry
nor under ModPerl::PerlRun. So read on if you think you
are better than me.
I tried the standard way, telling how any cgi Perl script
can be run under mod_perl perlrun, but it fails with imapsync.
Any hint welcome!
# This is a Debian example
# install mod-perl with
apt-get install libapache2-mod-perl2
# edit the file /etc/apache2/mods-available/perl.conf
# with the following lines
more /etc/apache2/mods-available/perl.conf
PerlModule ModPerl::PerlRun
Alias /perl-run/ /usr/lib/cgi-bin/
SetHandler perl-script
PerlResponseHandler ModPerl::PerlRun
PerlOptions +ParseHeaders
Options +ExecCGI
# Enable the Apache perl module
a2enmod perl
# Verify perl.conf and perl.load are in directory mods-enabled/
ls mods-enabled/perl.*
# Reload Apache
apachectl graceful
# Verify imapsync works under perl-run
curl http://localhost/perl-run/imapsync
=======================================================================
=======================================================================