1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-25 12:12:34 +01:00
Commit Graph

1371 Commits

Author SHA1 Message Date
Mike Fährmann
392a081657
update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
48a8717a7c
add 'output.num-to-str' option
... to convert any numeric values to string when outputting them as JSON
(during '--dump-json' or otherwise)
2018-10-08 20:28:54 +02:00
Mike Fährmann
af3f81c7d9
add '--no-check-certificate' command-line option 2018-10-07 21:46:00 +02:00
Mike Fährmann
0514d6a0ae
make --filter and --range config-file options
The functionality of --(chapter-)filter and --(chapter-)range are now
also exposed as the following config-file options:

- extractor.*.image-filter
- extractor.*.image-range
- extractor.*.chapter-filter
- extractor.*.chapter-range

TODO: update configuration.rst
2018-10-07 21:39:56 +02:00
Mike Fährmann
4a348990f4
adjust value resolution for retries/timeout/verify options
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.

'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.

Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
2018-10-07 21:13:39 +02:00
Mike Fährmann
f647f5d9c3
use 'verify' option for regular HTTP requests 2018-10-06 16:38:43 +02:00
Mike Fährmann
ca6ac4db6a
fix 'content' tests 2018-10-05 21:10:33 +02:00
Mike Fährmann
188876d814
implement youtube-dl downloader module
URLs starting with 'ytdl:' will now be handled by youtube-dl.
There is probably a lot to fix and improve, but the basic use case
works.

TODO:
- format selection and ytdl options in general
- better filename/path handling
- ytdl support for "unsupported URLs"
- ...
2018-10-05 18:05:11 +02:00
Mike Fährmann
f4df6c2396
[pixiv] remove 'type' and 'page' query parameter handling
The "new and improved" /member_illust.php and /bookmark.php listings
don't quite work with how things were.
2018-10-03 16:08:36 +02:00
Mike Fährmann
d70db2d555
Revert "[komikcast] fix extraction"
This reverts commit 5507f5ce2e.
2018-10-02 20:38:42 +02:00
Mike Fährmann
d69db60e2a
update unit test results 2018-10-02 20:37:46 +02:00
Mike Fährmann
f8b3b00249
[twitter] add experimental 'videos' option (#99)
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
2018-09-30 21:52:23 +02:00
Mike Fährmann
5507f5ce2e
[komikcast] fix extraction 2018-09-29 16:37:30 +02:00
Mike Fährmann
8080071174
[flickr] improve album metadata (closes #109) 2018-09-29 16:21:55 +02:00
Mike Fährmann
537448ba6e
[yuki] fix extraction of older threads (closes #112) 2018-09-29 11:38:55 +02:00
Mike Fährmann
1acaed73e0
[warosu] improve extraction and metadata
- convert values to int
- unquote original filenames
- don't parse posts twice
2018-09-28 13:03:12 +02:00
Mike Fährmann
2cf3f53839
[yuki] add thread extractor (closes #111) 2018-09-28 12:47:32 +02:00
Mike Fährmann
17611bfec0
update build_supportedsites.py script 2018-09-28 12:43:19 +02:00
Mike Fährmann
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection 2018-09-24 13:12:54 +02:00
Mike Fährmann
c402cc4047
[hentaifoundry] add 'popular' and 'recent' extractors
for "Popular Pictures" and "Recent Pictures" listings
2018-09-24 13:11:18 +02:00
Mike Fährmann
a5fc311dfa
[hentaifoundry] add 'favorite' extractor 2018-09-22 21:23:29 +02:00
Mike Fährmann
1c95a0173f
[hentaifoundry] split 'artist' into 'user'+'artist'
and some smaller changes ...

'user' is the name of the account an image is listed at and
'artist' is now the name of the account who created the image.

For example "https://www.hentai-foundry.com/user/Tenpura/faves/pictures"
- 'user': Tenpura
- 'artist' of the only image: LewdBrush
2018-09-22 21:21:07 +02:00
Mike Fährmann
55f5c87160
[postprocessor:ugoira] add 'libx264-prevent-odd' option
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
2018-09-21 20:01:14 +02:00
Mike Fährmann
8c8da11bb8
do not create directory structures when using '-s' 2018-09-21 17:55:04 +02:00
Mike Fährmann
e066f35118
update extractor tests 2018-09-21 11:25:56 +02:00
Mike Fährmann
006f75b538
[hentaifoundry] rewrite + more metadata
- extract width, height, artist per image
- improve pattern regex
- better extensibility for other listings
2018-09-21 11:23:51 +02:00
Mike Fährmann
eeb7424783
[hentaifoundry] add support for "scraps" (#110) 2018-09-20 13:41:23 +02:00
Mike Fährmann
6ea9a78588
[wallhaven] add login capabilities
Being logged in is required to access NSFW wallpapers.
2018-09-19 21:04:01 +02:00
Mike Fährmann
c9290d8212
[wallhaven] add wallpaper and search extractors
todo:
- login support to gain access to NSFW wallpapers
- extractors for tag-, similar-, latest-listings
- skip() support
2018-09-17 21:26:13 +02:00
Mike Fährmann
26cbcb3a72
[flickr] improve error handling (#109) 2018-09-17 10:12:14 +02:00
Mike Fährmann
2be4c9ffe3
[sankaku] small code improvements 2018-09-16 21:01:28 +02:00
Mike Fährmann
529aa21dd9
move FileAdapter definition into recursive.py 2018-09-16 20:59:22 +02:00
Mike Fährmann
31a5c7c2c0
release version 1.5.3 2018-09-14 18:18:43 +02:00
Mike Fährmann
22ab509a70
[bobx] rename "model" to "idol" extractor 2018-09-14 18:11:36 +02:00
Mike Fährmann
99137f1bee
[sankaku] send login info as formdata
Previously they were erroneously send as URL parameters.
2018-09-14 17:54:15 +02:00
Mike Fährmann
fa64c38d5b
[sankaku] fix pagination for user favorites (#106) 2018-09-14 17:51:46 +02:00
Mike Fährmann
69fd61ea86
[bobx] add gallery and model extractors 2018-09-13 20:13:12 +02:00
Mike Fährmann
0232d80cec
[deviantart] convert 'published_time' to int (fixes #108)
The 'published_time' field (a timestamp) changed from integer to string
and caused journal creation to fail.
2018-09-13 19:52:01 +02:00
Mike Fährmann
7742cf8601
[tumblr] change 'reblogs' option (#103)
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
2018-09-10 15:40:25 +02:00
Mike Fährmann
f1695567e8
adjust values in template config file
(#104)
2018-09-09 14:10:55 +02:00
Mike Fährmann
d4d95d3154
[tumblr] improve rewrite rules for video URLs 2018-09-09 14:09:47 +02:00
Mike Fährmann
542a25c389
[ngomik] fix extraction 2018-09-09 13:45:40 +02:00
Mike Fährmann
a666ddd16b
[tumblr] extend 'reblogs' functionality (#103)
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
2018-09-07 19:13:52 +02:00
Mike Fährmann
c9b8e6aefc
[reddit] fix submission-ID parsing (#104)
Uppercase characters caused a ValueError exception
2018-09-07 18:27:54 +02:00
Mike Fährmann
488abeca0b
[hentaicafe] adjust default directory format
A separate folder for each chapter is rather pointless if almost all
manga have only one chapter each.
2018-09-07 18:25:58 +02:00
Mike Fährmann
b4eca2633e
[tumblr] support /archive URLs 2018-09-06 11:09:13 +02:00
Mike Fährmann
aa1de70da0
[tumblr] recognize inline videos (#102) 2018-09-06 10:37:40 +02:00
Mike Fährmann
3ecea4cf36
[hentaicafe] add chapter and manga extractors (#101) 2018-09-05 21:08:40 +02:00
Mike Fährmann
41249f3ead
improve extractor.get_downloader() 2018-09-05 18:17:16 +02:00
Mike Fährmann
eb3185d6a3
update exception hierarchy 2018-09-05 18:15:33 +02:00