1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 18:53:21 +01:00
Commit Graph

65 Commits

Author SHA1 Message Date
Mike Fährmann
f02a768b5c
[danbooru] add 'ugoira' option (#406)
to choose between ZIP archives or converted video files
for Ugoira posts
2019-08-31 21:51:52 +02:00
Mike Fährmann
7c09545f70
[downloader:ytdl] add 'outtmpl' option (#395) 2019-08-24 22:47:59 +02:00
Mike Fährmann
5a210991b6
Remove control characters from filesystem paths
- add 'path-remove' option to specify the set of characters that
 should be removed
- rename 'restrict-filenames' to 'path-restrict'
- #348, #380
2019-08-16 23:12:16 +02:00
leezhi0513
42bb0cdb9b Update gallery-dl.conf (#378) 2019-08-11 14:44:50 +02:00
Mike Fährmann
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option 2019-08-08 16:54:20 +02:00
Mike Fährmann
3957d27d79
[deviantart] add 'quality' option (#369) 2019-08-03 11:40:35 +02:00
Mike Fährmann
547ea71463
[downloader.ytdl] add 'forward-cookies' option (#352)
The "long" name is necessary because just calling it 'cookies' would
clash with how the lookup for '--cookies' is implemented.
2019-07-24 21:19:11 +02:00
Mike Fährmann
b1bea8aaeb
add 'restrict-filenames' option (#348) 2019-07-23 17:41:24 +02:00
Mike Fährmann
21991acc49
add 'ciphers' option; update default User-Agent 2019-07-19 17:14:40 +02:00
Mike Fährmann
feb98cf196
[twitter] improve 'content' formatting; add option (#338)
- include emoticons
- leave newlines intact
- remove pic.twitter.com/ links at the end
2019-07-17 16:02:51 +02:00
Mike Fährmann
f7b5c4c3e7
use values of 'retries' options correctly
The RE-tries option now specifies exactly that: the maximum number a
failed HTTP request is re-tried. For example a value of 2 will now
correctly stop after 3 attempts: the initial one + 2 re-tries.

The maximum wait-time now also caps at 30min and increases exponentially
for both extractor.request() and downloader.http.download().
2019-06-30 23:10:18 +02:00
Mike Fährmann
15e4ddf46d
implement custom logging formatter
supports custom log message formats for each loglevel and, by
extension, custom ANSI codes and colors for errors and warnings

(#304)
2019-06-21 20:17:58 +02:00
Mike Fährmann
db3f52881a
add 'mtime' option 2019-06-20 17:19:44 +02:00
Mike Fährmann
e05a96db5e
[deviantart] rename 'stash' to 'extra' (#302)
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
2019-06-10 21:05:25 +02:00
Mike Fährmann
c23bf263fe
[deviantart] rename 'external' to 'stash' (#302)
restrict extracted URLs to ones from https://sta.sh/...
2019-06-09 11:16:02 +02:00
Mike Fährmann
2fb85178da
[deviantart] add 'external' option (#302)
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
2019-06-06 18:53:50 +02:00
Mike Fährmann
d33f5a7423
[wallhaven] rewrite
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
  IDs that can't be translated to the new ID system
- support direct links to wallpapers
2019-05-31 14:53:02 +02:00
Mike Fährmann
f1893b2b5b
[deviantart] add 'folders' option (#276) 2019-05-30 17:28:12 +02:00
Mike Fährmann
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option (#279)
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
2019-05-27 22:24:48 +02:00
Mike Fährmann
9890bfdf23
[flickr] improve code and metadata
- simplify pagination
- add more metadata and slightly change its structure
  - convert suitable values to int or list
  - move keys from ["photo"] to the base level
- proper video support (#246)
- rename method and variable names to better fit with other extractors
2019-05-14 22:10:50 +02:00
Mike Fährmann
1e4d351ad3
[danbooru] add authentication support (closes #151)
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
2019-01-09 14:19:07 +01:00
Mike Fährmann
28f9539551
[tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba
[tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
2fa28a2609
update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
f8b3b00249
[twitter] add experimental 'videos' option (#99)
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
2018-09-30 21:52:23 +02:00
Mike Fährmann
f1695567e8
adjust values in template config file
(#104)
2018-09-09 14:10:55 +02:00
Mike Fährmann
c9b8e6aefc
[reddit] fix submission-ID parsing (#104)
Uppercase characters caused a ValueError exception
2018-09-07 18:27:54 +02:00
Mike Fährmann
10365394d7
[twitter] add support for user-timelines (closes #96)
also adds a 'retweets' option to filter retweeted content
2018-08-17 20:04:11 +02:00
Mike Fährmann
b164231bca
[sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
5f27cfeff6
[deviantart] remove prefer-public option
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048
[deviantart] improve public-private token switching
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
Mike Fährmann
ff436692bf
["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
87853538b4
[yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
6ac403c5d3
add postprocessor config example 2018-06-08 18:31:59 +02:00
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
This reverts commit 4a26ae32df.
2018-04-09 16:08:08 +02:00
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
b25ae30a95
reorder options in gallery-dl.conf 2018-03-22 10:05:53 +01:00
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
74c8f83056
update gallery-dl.conf and add gallery-dl-example.conf
- gallery-dl.conf contains all config options set to their default
  values, with the exception of 'filename', 'directory', and
  'cache.file'

- gallery-dl-example.conf contains a potential real-world configuration
  file and demonstrates some more involved options
2018-03-12 18:15:30 +01:00
Mike Fährmann
8704d850bf
add explicit proxy support (#76)
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
2018-02-19 18:45:06 +01:00
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option
… as well as a config file equivalent
2018-02-01 22:00:44 +01:00
Mike Fährmann
d951f13e37
add config option for unsupported-URL file
for consistency's sake
2018-01-28 18:42:10 +01:00
Mike Fährmann
97f4f15ec0
add option to write logging output to a file
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
2018-01-26 18:51:51 +01:00
Mike Fährmann
03b8a548cb
[tumblr] change reblogs default value to true (#61) 2018-01-06 15:52:08 +01:00
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts (#61)
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use (closes #56)
Gelbooru's API allows access to all images and is not restricted
to the first 20000.

This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
2017-12-18 01:16:46 +01:00
Mike Fährmann
4fb6803fa6
add option to sleep before each download 2017-12-04 17:33:10 +01:00
Mike Fährmann
8f518e03f8
add options to set maximum download rate
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option

This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.

[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
2017-12-02 01:47:26 +01:00
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior (#48)
- posts   : list of post-types to inspect
- inline  : scan post bodies for inline images
- external: follow external links
2017-11-23 15:32:54 +01:00