Mike Fährmann
6174a5c4ef
[download] adjust filename extension on filetype mismatch
...
(closes #63 )
2018-01-17 18:37:06 +01:00
Mike Fährmann
ebe9b0a04c
another attempt at downloader retry behavior
...
This commit changes the general behavior from
'Retry on every exception and abort on DownloadError' to
'Only retry on DownloadRetry exceptions and abort on every other one'
The previous version would have retried on several states which
would have no chance of ever succeeding (invalid URLs, etc.)
2017-12-07 15:31:14 +01:00
Mike Fährmann
79bcaa8726
improve downloader retry behavior
...
- only retry download on 5xx and 429 status codes
- immediately fail on 4xx status codes
2017-11-10 21:46:18 +01:00
Mike Fährmann
42e948584d
fix downloader error handling
...
RequestException being a subclass of OSError caused all exceptions
during file downloads to be ignored/re-raised.
2017-11-07 15:23:07 +01:00
Mike Fährmann
707b15b586
create missing directories for 'part-directory'
...
also some code improvements regarding downloader config values
2017-10-27 12:22:45 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files ( #29 )
...
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
2017-10-26 00:16:48 +02:00
Mike Fährmann
9a41002b77
fix partial downloads for 'text:' URLs
...
Using a filesize in bytes as offset into a Python string is not
a good idea if said file contains non-ASCII characters.
2017-10-25 15:04:45 +02:00
Mike Fährmann
963670d73b
add options to control usage of .part files ( #29 )
...
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options
Disabling .part files restores the behaviour of the old downloader
implementation.
2017-10-24 23:33:44 +02:00
Mike Fährmann
b0353aa02d
rewrite download modules ( #29 )
...
- use '.part' files during file-download
- implement continuation of incomplete downloads
- check if file size matches the one reported by server
2017-10-24 12:53:03 +02:00
Mike Fährmann
e2b5cd9918
change config-path for 'retries' and 'timeout'
2017-03-26 18:24:46 +02:00
Mike Fährmann
0b5076815d
always delete incompletely downloaded files
2017-03-21 15:53:43 +01:00
Mike Fährmann
22910f9562
improve error handling of http file downloads
...
(#10 )
2017-03-16 04:17:35 +01:00
Mike Fährmann
4f123b8513
code adjustments according to pep8
2017-01-30 19:40:15 +01:00
Mike Fährmann
3c1daef839
don't delete downloaded files in certain edge cases
2016-11-27 23:43:25 +01:00
Mike Fährmann
29692c5784
get extension from Content-Type header if not provided
2016-09-30 12:32:48 +02:00
Mike Fährmann
ecc6542fc8
change required parameter type to file-like objects
2015-12-21 22:46:49 +01:00
Mike Fährmann
a8c0b4531d
fix issue with Ctrl+c on windows
2015-12-02 01:01:33 +01:00
Mike Fährmann
4b377ccc09
use output-module during downloads
2015-12-01 21:22:58 +01:00
Mike Fährmann
28fa7c53b4
docstrings and other small fixes for downloaders
2015-04-10 21:45:41 +02:00
Mike Fährmann
deef91eddc
initial commit
2014-10-12 21:56:44 +02:00
