1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Commit Graph

282 Commits

Author SHA1 Message Date
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts (#61)
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use (closes #56)
Gelbooru's API allows access to all images and is not restricted
to the first 20000.

This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
2017-12-18 01:16:46 +01:00
Mike Fährmann
4fb6803fa6
add option to sleep before each download 2017-12-04 17:33:10 +01:00
Mike Fährmann
8f518e03f8
add options to set maximum download rate
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option

This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.

[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
2017-12-02 01:47:26 +01:00
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior (#48)
- posts   : list of post-types to inspect
- inline  : scan post bodies for inline images
- external: follow external links
2017-11-23 15:32:54 +01:00
Mike Fährmann
e6814aebe2
add 'extractor.*.user-agent' config option 2017-11-15 14:01:33 +01:00
Mike Fährmann
707b15b586
create missing directories for 'part-directory'
also some code improvements regarding downloader config values
2017-10-27 12:22:45 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files (#29)
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
2017-10-26 00:16:48 +02:00
Mike Fährmann
963670d73b
add options to control usage of .part files (#29)
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options

Disabling .part files restores the behaviour of the old downloader
implementation.
2017-10-24 23:33:44 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve (fixes #44)
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
2017-10-14 23:01:33 +02:00
Mike Fährmann
abfd2924a9
update configuration.rst
- add instructions on how to register your own applications
- combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
Mike Fährmann
f3fbaa5c3e
[reddit] allow users to override the API User-Agent
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].

[1] https://github.com/reddit/reddit/wiki/API#rules
2017-10-10 17:29:46 +02:00
Mike Fährmann
2ef3c35c98
smaller textual changes
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
2017-10-09 23:23:19 +02:00
Mike Fährmann
75d3a1f72f
[deviantart] always download original images
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.

[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
2017-10-07 13:07:34 +02:00
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords' 2017-09-09 18:48:28 +02:00
Mike Fährmann
54c0715135
allow users to set their own API access_tokens/client_ids 2017-09-09 17:50:19 +02:00
Mike Fährmann
b8862ff15e
add 'downloader.http.verify' option
(also: change the default 'timeout' from None to 30)
2017-08-31 15:21:08 +02:00
Mike Fährmann
3c9f190757
extend output of --list-keywords 2017-08-10 17:36:21 +02:00
Mike Fährmann
f08af03845
Merge branch 'cookies' 2017-07-25 14:04:53 +02:00
Mike Fährmann
00a23fe208
update configuration.rst 2017-07-21 18:32:56 +02:00
Mike Fährmann
9be8f7e106
[deviantart] add "extractor.deviantart.flat" option
Setting this to 'false' downloads images into individual subdirectories
for each gallery-folder or favourite-collection, otherwise it is just
creating a flat list of images.
2017-07-12 17:05:31 +02:00
Mike Fährmann
9edbd6ff01
update configuration.rst 2017-07-04 19:34:34 +02:00
Mike Fährmann
06c4cae05b
extend the output of '--list-extractors'
It now includes category and subcategory values for
each extractor class.
2017-06-28 18:51:47 +02:00
H R X N
b757687606 Update configuration.rst (#24)
Fixed minor typo.
2017-06-28 13:55:39 +02:00
Mike Fährmann
598e3a8d88
update configuration.rst 2017-06-27 18:56:24 +02:00
Mike Fährmann
86561d7c8b
update configuration.rst 2017-06-27 17:44:02 +02:00
Mike Fährmann
d3b04076f7
add .netrc support (#22)
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.

The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
2017-06-24 12:17:26 +02:00
Mike Fährmann
fbe8c519e2
update README
- mention and link to docs/configuration.rst
- add OAuth section
2017-06-23 16:14:51 +02:00
Mike Fährmann
ccb7973b57
update configuration.rst 2017-06-20 16:20:28 +02:00
H R X N
b206b831ee Update configuration.rst
Fix heading.
2017-06-19 14:12:43 +02:00
Mike Fährmann
48b444e276
add a list of all configuration options
(the format might be subject to change; putting everything  in
 one single table is even worse)
2017-06-18 22:16:26 +02:00