1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Commit Graph

96 Commits

Author SHA1 Message Date
Mike Fährmann
7b562907c3
[nijie] add favorites extractor
adds support for 'https://nijie.info/user_like_illust_view.php?id=...'
2018-03-31 18:54:25 +02:00
Mike Fährmann
f39153b6e9
[nhentai] add extractor for search results 2018-03-28 17:21:44 +02:00
Mike Fährmann
52d41c41e7
[exhentai] add extractor for favorited galleries 2018-03-27 18:58:42 +02:00
Mike Fährmann
a993d0ea90
release version 1.3.2 2018-03-23 15:27:56 +01:00
Mike Fährmann
e7525b1b0e
[artstation] add challenge extractor (#80) 2018-03-23 15:06:09 +01:00
Mike Fährmann
b25ae30a95
reorder options in gallery-dl.conf 2018-03-22 10:05:53 +01:00
Mike Fährmann
40ca562d7b
[artstation] add album extractor (#80) 2018-03-17 17:36:31 +01:00
HRXN
241eda8c3d Update supportedsites.rst (#81)
Add ArtStation to Supported Sites.
2018-03-16 13:40:59 +01:00
Mike Fährmann
f0e3861ec7
update configuration.rst
- move `base-directory` entry to extractor options
  (it should've been there since v1.0.0 / commit e3bfb83)
- rename "General" to "Miscellaneous" and move it down a bit, since it
  has only two options left
- mention the ability to use a list for tumblr.posts
- replace any appearance of bato.to
2018-03-16 12:00:55 +01:00
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
d6ef52897c
[imgchili] remove module
All previously hosted images yield a 404
and the main page is just a logo.
2018-03-13 11:02:58 +01:00
Mike Fährmann
7847ab1d5a
[imagehosts] remove even more dead sites
All removed sites either
- reject all incoming connections or
- display a message from their domain registrar
2018-03-12 21:25:13 +01:00
Mike Fährmann
74c8f83056
update gallery-dl.conf and add gallery-dl-example.conf
- gallery-dl.conf contains all config options set to their default
  values, with the exception of 'filename', 'directory', and
  'cache.file'

- gallery-dl-example.conf contains a potential real-world configuration
  file and demonstrates some more involved options
2018-03-12 18:15:30 +01:00
Mike Fährmann
1400868f53
[mangadex] general improvements
- support >100 chapter entries per manga
- custom archive ID format
- detect non-existing chapters
2018-03-06 14:15:15 +01:00
Mike Fährmann
749fbbfa6c
[mangadex] add chapter- and manga-extractor 2018-03-05 18:37:21 +01:00
Mike Fährmann
6e38cf5aab
[mangareader] use 'https://'
The site now redirects from http://mangareader.net/
to https://mangareader.net/
2018-03-02 17:19:17 +01:00
Mike Fährmann
8cdce21dcb
make archive keys user-configurable 2018-02-25 21:57:01 +01:00
Mike Fährmann
8f338347b6
[imagehosts] cleanup
removed
- chronos.to      - unable to resolve hostname
- coreimg.net     - same
- imgmaid.net     - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt    - cloudflare error 522, host down
- img4ever.net    - read timeout
2018-02-23 01:05:42 +01:00
Mike Fährmann
e1e0668ca8
add option to set default replacement field value
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
2018-02-23 00:59:20 +01:00
Mike Fährmann
8704d850bf
add explicit proxy support (#76)
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
2018-02-19 18:45:06 +01:00
Mike Fährmann
3cec533c28
Merge branch 'archive' 2018-02-12 18:07:58 +01:00
Mike Fährmann
7f7c16ae37
add option to specify additional key-value pairs 2018-02-08 23:10:58 +01:00
Mike Fährmann
54c36a8a34
[subapics] add chapter- and manga-extractor (#70) 2018-02-04 22:02:10 +01:00
Mike Fährmann
2dd3aeeeae
[komikcast] add chapter- and manga-extractor (#70) 2018-02-04 22:02:10 +01:00
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option
… as well as a config file equivalent
2018-02-01 22:00:44 +01:00
Mike Fährmann
d951f13e37
add config option for unsupported-URL file
for consistency's sake
2018-01-28 18:42:10 +01:00
Mike Fährmann
97f4f15ec0
add option to write logging output to a file
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
2018-01-26 18:51:51 +01:00
Mike Fährmann
1e93955170
[batoto] remove module
Site officially shut down on 2018.01.18
2018-01-23 21:37:32 +01:00
Mike Fährmann
40d35c87bc
[paheal] add tag- and post-extractors (closes #69) 2018-01-15 16:39:05 +01:00
Mike Fährmann
cc0c2cca57
[reddit] add extractor for reddit-hosted images (closes #68) 2018-01-14 18:55:42 +01:00
Mike Fährmann
35e09869d1
[mangapark] fix image URLs and use HTTPS 2018-01-12 14:59:49 +01:00
Mike Fährmann
29d75fc3fa
[tumblr] add support for OAuth authentication (#65) 2018-01-11 14:11:37 +01:00
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com 2018-01-09 17:54:37 +01:00
Mike Fährmann
5b094328b5
[puremashiro] add chapter- and manga-extractor (closes #66)
Also adds support for region subtags in language codes (e.g. en-us)
2018-01-07 21:50:43 +01:00
Mike Fährmann
03b8a548cb
[tumblr] change reblogs default value to true (#61) 2018-01-06 15:52:08 +01:00
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts (#61)
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use (closes #56)
Gelbooru's API allows access to all images and is not restricted
to the first 20000.

This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
2017-12-18 01:16:46 +01:00
Mike Fährmann
0a9a07a6e1
[slideshare] improve metadata; flake8
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
2017-12-13 21:16:49 +01:00
Mike Fährmann
19a6ae57b2
[sankaku] add pool extractor 2017-12-12 19:45:10 +01:00
Mike Fährmann
4fb6803fa6
add option to sleep before each download 2017-12-04 17:33:10 +01:00
Mike Fährmann
300346ecdf
[mangazuki] remove extractors
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
2017-12-04 13:36:04 +01:00
Mike Fährmann
8f518e03f8
add options to set maximum download rate
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option

This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.

[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
2017-12-02 01:47:26 +01:00
Mike Fährmann
214972bc9a
[gelbooru] use manual extraction
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875)

This also adds an extractor for image-pools.
2017-11-29 20:48:17 +01:00
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior (#48)
- posts   : list of post-types to inspect
- inline  : scan post bodies for inline images
- external: follow external links
2017-11-23 15:32:54 +01:00
Mike Fährmann
e6814aebe2
add 'extractor.*.user-agent' config option 2017-11-15 14:01:33 +01:00
Mike Fährmann
2d0cfb33e1
[xvideos] add user profile extractor (#45) 2017-11-02 17:28:35 +01:00
Mike Fährmann
707b15b586
create missing directories for 'part-directory'
also some code improvements regarding downloader config values
2017-10-27 12:22:45 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files (#29)
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
2017-10-26 00:16:48 +02:00
Mike Fährmann
963670d73b
add options to control usage of .part files (#29)
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options

Disabling .part files restores the behaviour of the old downloader
implementation.
2017-10-24 23:33:44 +02:00