1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Commit Graph

49 Commits

Author SHA1 Message Date
Mike Fährmann
d33f5a7423
[wallhaven] rewrite
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
  IDs that can't be translated to the new ID system
- support direct links to wallpapers
2019-05-31 14:53:02 +02:00
Mike Fährmann
f1893b2b5b
[deviantart] add 'folders' option (#276) 2019-05-30 17:28:12 +02:00
Mike Fährmann
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option (#279)
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
2019-05-27 22:24:48 +02:00
Mike Fährmann
9890bfdf23
[flickr] improve code and metadata
- simplify pagination
- add more metadata and slightly change its structure
  - convert suitable values to int or list
  - move keys from ["photo"] to the base level
- proper video support (#246)
- rename method and variable names to better fit with other extractors
2019-05-14 22:10:50 +02:00
Mike Fährmann
1e4d351ad3
[danbooru] add authentication support (closes #151)
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
2019-01-09 14:19:07 +01:00
Mike Fährmann
28f9539551
[tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba
[tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
2fa28a2609
update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
f8b3b00249
[twitter] add experimental 'videos' option (#99)
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
2018-09-30 21:52:23 +02:00
Mike Fährmann
f1695567e8
adjust values in template config file
(#104)
2018-09-09 14:10:55 +02:00
Mike Fährmann
c9b8e6aefc
[reddit] fix submission-ID parsing (#104)
Uppercase characters caused a ValueError exception
2018-09-07 18:27:54 +02:00
Mike Fährmann
10365394d7
[twitter] add support for user-timelines (closes #96)
also adds a 'retweets' option to filter retweeted content
2018-08-17 20:04:11 +02:00
Mike Fährmann
b164231bca
[sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
5f27cfeff6
[deviantart] remove prefer-public option
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048
[deviantart] improve public-private token switching
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
Mike Fährmann
ff436692bf
["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
87853538b4
[yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
6ac403c5d3
add postprocessor config example 2018-06-08 18:31:59 +02:00
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
This reverts commit 4a26ae32df.
2018-04-09 16:08:08 +02:00
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
b25ae30a95
reorder options in gallery-dl.conf 2018-03-22 10:05:53 +01:00
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
74c8f83056
update gallery-dl.conf and add gallery-dl-example.conf
- gallery-dl.conf contains all config options set to their default
  values, with the exception of 'filename', 'directory', and
  'cache.file'

- gallery-dl-example.conf contains a potential real-world configuration
  file and demonstrates some more involved options
2018-03-12 18:15:30 +01:00
Mike Fährmann
8704d850bf
add explicit proxy support (#76)
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
2018-02-19 18:45:06 +01:00
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option
… as well as a config file equivalent
2018-02-01 22:00:44 +01:00
Mike Fährmann
d951f13e37
add config option for unsupported-URL file
for consistency's sake
2018-01-28 18:42:10 +01:00
Mike Fährmann
97f4f15ec0
add option to write logging output to a file
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
2018-01-26 18:51:51 +01:00
Mike Fährmann
03b8a548cb
[tumblr] change reblogs default value to true (#61) 2018-01-06 15:52:08 +01:00
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts (#61)
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use (closes #56)
Gelbooru's API allows access to all images and is not restricted
to the first 20000.

This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
2017-12-18 01:16:46 +01:00
Mike Fährmann
4fb6803fa6
add option to sleep before each download 2017-12-04 17:33:10 +01:00
Mike Fährmann
8f518e03f8
add options to set maximum download rate
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option

This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.

[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
2017-12-02 01:47:26 +01:00
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior (#48)
- posts   : list of post-types to inspect
- inline  : scan post bodies for inline images
- external: follow external links
2017-11-23 15:32:54 +01:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files (#29)
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
2017-10-26 00:16:48 +02:00
Mike Fährmann
963670d73b
add options to control usage of .part files (#29)
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options

Disabling .part files restores the behaviour of the old downloader
implementation.
2017-10-24 23:33:44 +02:00
Mike Fährmann
5fa42336a2
[sankaku] add warning for unauthenticated users
also improve URL pattern and add missing options to default config file
2017-10-16 21:21:08 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve (fixes #44)
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
2017-10-14 23:01:33 +02:00
Mike Fährmann
2ef3c35c98
smaller textual changes
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
2017-10-09 23:23:19 +02:00
Mike Fährmann
470bbe9d8c
fix smaller stuff
- change filename option in example config file
- adapt default filename format for mangafox
- remove unnecessary newline

[skip ci]
2017-09-11 17:07:29 +02:00
Mike Fährmann
b8862ff15e
add 'downloader.http.verify' option
(also: change the default 'timeout' from None to 30)
2017-08-31 15:21:08 +02:00
Mike Fährmann
74a2b5af06
update supportedsites.rst 2017-07-15 15:01:30 +02:00
Mike Fährmann
3460dc8950
update gallery-dl.conf 2017-07-14 08:23:11 +02:00
Mike Fährmann
d3b04076f7
add .netrc support (#22)
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.

The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
2017-06-24 12:17:26 +02:00
Mike Fährmann
fbe8c519e2
update README
- mention and link to docs/configuration.rst
- add OAuth section
2017-06-23 16:14:51 +02:00
Mike Fährmann
5f55c854b9
[flickr] replace getPublic... API call with regular ones 2017-06-12 16:37:06 +02:00
Mike Fährmann
d5a70f2580
add simple progress indicator for multiple URLs (#19)
The output can be configured via the 'output.progress'
config value.

Possible values:
    - true:     Show the default progress indicator
                "[{current}/{total}] {url}" (default)
    - false:    Never show the progress indicator
    - <string>: Show the progress indicator using this
                as a custom format string(1).
                Possible replacement keys are:
                - current: current URL index
                - total  : total number of URLs
                - url    : current URL

(1) https://docs.python.org/3/library/string.html#formatstrings
2017-06-09 20:12:15 +02:00
Mike Fährmann
090e11b35d
[reddit] enable user authentication with OAuth2 (#15)
Call '$ gallery-dl oauth:reddit' to get a refresh_token
for your account.
2017-06-08 16:17:13 +02:00
Mike Fährmann
1e416e5227
update README 2017-05-09 15:43:55 +02:00