1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-25 20:22:36 +01:00
Commit Graph

1400 Commits

Author SHA1 Message Date
Mike Fährmann
b17a5d6f3b
give downloader classes proper names 2018-11-16 14:40:05 +01:00
Mike Fährmann
3c25fa2dad
update build_testresult_db.py script 2018-11-15 22:58:14 +01:00
Mike Fährmann
7f6a0be982
adjust some tests 2018-11-15 22:50:04 +01:00
Mike Fährmann
baad7b0fa5
[twitter] unpack API responses when logged in (closes #123) 2018-11-14 11:49:35 +01:00
Mike Fährmann
3bdfc15be1
[pinterest] don't crash on pins without image info 2018-11-14 11:46:14 +01:00
Mike Fährmann
8ef84a6823
add option to enable/disable specific downloader modules
... and write URLs with no (active) downloader to unsupported-file
2018-11-13 18:06:36 +01:00
Mike Fährmann
14ee6bf611
[behance] handle external URLs with youtube-dl 2018-11-13 15:10:23 +01:00
Mike Fährmann
36425122ff
[artstation] handle external URLs with youtube-dl 2018-11-13 14:27:02 +01:00
Mike Fährmann
bd8670d925
[gfycat] extend URL pattern 2018-11-11 21:19:11 +01:00
Mike Fährmann
2fa28a2609
update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
7e2d6bcd62
[deviantart] fix original image downloads 2018-11-10 19:16:10 +01:00
Mike Fährmann
9e12e073ab
[2chan] fix extraction 2018-11-10 19:15:21 +01:00
Mike Fährmann
966a9ca3a0
update test results 2018-11-10 19:14:54 +01:00
Mike Fährmann
e26ba682a2
enforce utf-8 encoding for input files (#120) 2018-11-10 18:27:01 +01:00
Mike Fährmann
a36259d8f1
update setup.py
- add Python version check
- add classifiers
- simplify sys.exit() usage
2018-10-24 14:43:37 +02:00
Mike Fährmann
fd8ed35591
[turboimagehost] fix extraction 2018-10-23 21:08:24 +02:00
Mike Fährmann
c69150f715
[imagefap] fix extraction
also adds tags to gallery-metadata and converts suitable values to int
2018-10-20 18:32:25 +02:00
Mike Fährmann
d1f3d32eec
[fallenangels] unescape chapter titles 2018-10-20 18:31:26 +02:00
Mike Fährmann
655549df7c
[downloader:ytdl] add several options
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.

downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.

downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
2018-10-20 18:26:49 +02:00
Mike Fährmann
d3d7f01543
add 'prepare()' step for post-processors
This allows post-processors to modify the destination path before
checking if a file already exists.
2018-10-18 22:32:03 +02:00
Mike Fährmann
c9861ca812
adjust message for status_code based exceptions
from: 5xx HTTP Error: Reason
to  : 5xx: Reason

The "HTTP Error" part was in there to emulate Request's error messages
from response.raise_for_status(), but it reads a lot better without.
2018-10-18 15:09:49 +02:00
Mike Fährmann
eb1c24b286
[imagebam] detect nonexistent galleries 2018-10-17 15:21:47 +02:00
Mike Fährmann
6ed629f2b6
allow specifying number of skips before abort/exit (closes #115)
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
2018-10-13 17:21:55 +02:00
Mike Fährmann
e1d306cc48
update unit test results 2018-10-13 16:54:30 +02:00
Mike Fährmann
8faf03ed84
[pixiv] use refresh_token based authentication
The first login will still use username and password, but everything
afterwards will use the refresh_token obtained from that.

This will prevent pixiv from sending a "New login to pixiv" email every
time a new access_token is requested.
2018-10-12 22:26:27 +02:00
Mike Fährmann
2221cf97ff
implement 'update()' for caches 2018-10-12 22:18:29 +02:00
Mike Fährmann
d8492df51b
[deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
c00dce2adc
[behance] enable 'categorytransfer' 2018-10-09 23:40:49 +02:00
Mike Fährmann
1532d1b690
fix 'range' tests and update a few test results 2018-10-08 23:53:58 +02:00
Mike Fährmann
392a081657
update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
48a8717a7c
add 'output.num-to-str' option
... to convert any numeric values to string when outputting them as JSON
(during '--dump-json' or otherwise)
2018-10-08 20:28:54 +02:00
Mike Fährmann
af3f81c7d9
add '--no-check-certificate' command-line option 2018-10-07 21:46:00 +02:00
Mike Fährmann
0514d6a0ae
make --filter and --range config-file options
The functionality of --(chapter-)filter and --(chapter-)range are now
also exposed as the following config-file options:

- extractor.*.image-filter
- extractor.*.image-range
- extractor.*.chapter-filter
- extractor.*.chapter-range

TODO: update configuration.rst
2018-10-07 21:39:56 +02:00
Mike Fährmann
4a348990f4
adjust value resolution for retries/timeout/verify options
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.

'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.

Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
2018-10-07 21:13:39 +02:00
Mike Fährmann
f647f5d9c3
use 'verify' option for regular HTTP requests 2018-10-06 16:38:43 +02:00
Mike Fährmann
ca6ac4db6a
fix 'content' tests 2018-10-05 21:10:33 +02:00
Mike Fährmann
188876d814
implement youtube-dl downloader module
URLs starting with 'ytdl:' will now be handled by youtube-dl.
There is probably a lot to fix and improve, but the basic use case
works.

TODO:
- format selection and ytdl options in general
- better filename/path handling
- ytdl support for "unsupported URLs"
- ...
2018-10-05 18:05:11 +02:00
Mike Fährmann
f4df6c2396
[pixiv] remove 'type' and 'page' query parameter handling
The "new and improved" /member_illust.php and /bookmark.php listings
don't quite work with how things were.
2018-10-03 16:08:36 +02:00
Mike Fährmann
d70db2d555
Revert "[komikcast] fix extraction"
This reverts commit 5507f5ce2e.
2018-10-02 20:38:42 +02:00
Mike Fährmann
d69db60e2a
update unit test results 2018-10-02 20:37:46 +02:00
Mike Fährmann
f8b3b00249
[twitter] add experimental 'videos' option (#99)
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
2018-09-30 21:52:23 +02:00
Mike Fährmann
5507f5ce2e
[komikcast] fix extraction 2018-09-29 16:37:30 +02:00
Mike Fährmann
8080071174
[flickr] improve album metadata (closes #109) 2018-09-29 16:21:55 +02:00
Mike Fährmann
537448ba6e
[yuki] fix extraction of older threads (closes #112) 2018-09-29 11:38:55 +02:00
Mike Fährmann
1acaed73e0
[warosu] improve extraction and metadata
- convert values to int
- unquote original filenames
- don't parse posts twice
2018-09-28 13:03:12 +02:00
Mike Fährmann
2cf3f53839
[yuki] add thread extractor (closes #111) 2018-09-28 12:47:32 +02:00
Mike Fährmann
17611bfec0
update build_supportedsites.py script 2018-09-28 12:43:19 +02:00
Mike Fährmann
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection 2018-09-24 13:12:54 +02:00
Mike Fährmann
c402cc4047
[hentaifoundry] add 'popular' and 'recent' extractors
for "Popular Pictures" and "Recent Pictures" listings
2018-09-24 13:11:18 +02:00
Mike Fährmann
a5fc311dfa
[hentaifoundry] add 'favorite' extractor 2018-09-22 21:23:29 +02:00