Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 18:53:21 +01:00
Code Issues Releases Wiki Activity
3,303 Commits 6 Branches 164 Tags 15 MiB
c83668c2ff
Commit Graph

313 Commits

Author SHA1 Message Date
Mike Fährmann
9f638c2e01
[twitter] add 'replies' option (closes #705) 2020-04-29 23:20:06 +02:00
Mike Fährmann
5b676ea59d
[e621] document username & password support (#640) 2020-03-14 01:15:22 +01:00
Mike Fährmann
59edcdc822
[hitomi] restore metadata fields from before f33b13a 
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
 2020-03-12 23:43:41 +01:00
Mike Fährmann
a45fbc38ea
[pixiv] implement 'avatar' option (#595, #623) 2020-03-09 21:18:16 +01:00
Mike Fährmann
6e759fe2f5
reword note for 'extractor.deviantart.refresh-token' (#612) 2020-02-19 23:39:27 +01:00
Mike Fährmann
67ea80222c
link to post processor example in 'pixiv.ugoira' docs 2020-02-14 01:22:16 +01:00
Mike Fährmann
74e684e828
[twitter] change default value for 'videos' to 'true' 
Every other 'videos' option defaulted to 'true', except Twitter.
 2020-02-14 01:03:42 +01:00
Mike Fährmann
ab13f81fd7
add example to 'cookies' docs 2020-02-13 23:44:38 +01:00
Mike Fährmann
c7cf9dd111
[furaffinity] support classic layout (#284) 2020-02-12 21:39:43 +01:00
Mike Fährmann
da0d5f6092
[oauth] add 'port' option (#604) 2020-02-09 13:45:44 +01:00
Mike Fährmann
091f9a107d
[docs] add examples of how to filter manga chapters by language 
(closes #599)
 2020-02-07 23:21:44 +01:00
Mike Fährmann
dff33b260c
[reddit] add 'videos' option 2020-01-31 23:45:02 +01:00
Mike Fährmann
56f1c96168
implement 'parent-directory' option (#551) 2020-01-29 18:32:37 +01:00
Mike Fährmann
6703b8a86b
[blogger] implement video extraction (closes #587) 2020-01-24 23:37:23 +01:00
Mike Fährmann
37247dbaff
miscellaneous fixes 2020-01-19 22:53:06 +01:00
Mike Fährmann
25d5ec4ff3
[twitter] add option to extract TwitPic embeds (#579) 2020-01-18 21:31:29 +01:00
Mike Fährmann
254f7c3999
implement a post-processor module to compare file versions 
(#530)
 2020-01-18 21:08:25 +01:00
Mike Fährmann
87c8b89ddd
[postprocessor:metadata] add 'directory' option (#520) 2020-01-02 20:58:10 +01:00
Mike Fährmann
48e42e73fb
[reddit] change default value for 'comments' to '0' 2019-12-20 16:54:59 +01:00
Mike Fährmann
bf658fd84b
[vsco] implement 'videos' option 2019-12-19 18:34:19 +01:00
Mike Fährmann
95c90722ee
[instagram] implement 'videos' option (closes #521) 2019-12-19 18:34:06 +01:00
Mike Fährmann
15f9bb3d14
add option to disable pyOpenSSL usage (#508) 
(pyOpenSSL is now disabled by default)
 2019-12-08 21:21:00 +01:00
Mike Fährmann
93b43f74f4
update docs/configuration.rst 
- fix description of "skip": "enumerate"
- use line breaks for lists of multiple examples
- improve formatting of auto-generated man pages
 2019-12-06 01:19:23 +01:00
Mike Fährmann
e17907ee2a
change default value of 'cookies-update' to 'true' 2019-12-05 23:43:49 +01:00
Mike Fährmann
26d2334550
[postprocessor:metadata] rename 'format' to 'content-format' 
Just to be consistent with the other 'extension-format' option name,
and only 'format' is also still accepted.
 2019-11-30 17:27:49 +01:00
Mike Fährmann
a412531451
[postprocessor:metadata] implement 'extension-format' option 
closes #477
 2019-11-30 17:26:17 +01:00
Mike Fährmann
d45fabb79d
match user profile handling on deviantart and newgrounds 2019-11-22 23:20:21 +01:00
Mike Fährmann
dd5d2b2eac
[deviantart] add user profile extractor (#377, #419) 2019-11-07 18:29:49 +01:00
Mike Fährmann
9e88e7a344
[postprocessor:exec] improve (#421, #413) 
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
 2019-11-03 21:45:45 +01:00
Mike Fährmann
c01ff78467
[twitter] extend 'videos' option to force extraction with ytdl 
(closes #459)
 2019-11-01 22:06:07 +01:00
Mike Fährmann
a5e6252137
update deviantart client-id/-secret instructions (#451, #424) 2019-10-21 16:57:45 +02:00
Mike Fährmann
389d2d7e38
implement 'cookies-update' option (#445) 2019-10-19 15:23:55 +02:00
Mike Fährmann
1adafdd3d0
document cache file requirement for DeviantArt refresh tokens 2019-10-13 23:01:57 +02:00
Mike Fährmann
8af59a4bba
fix & update docs 
- update Requests links
- add example for --exec
- set '-dev' version
 2019-10-11 18:36:25 +02:00
Mike Fährmann
46ba173ded
[reddit] fix documentation inconsistencies (closes #429) 
- Require 'reddit.comments' to be a number and convert it to an
  integer to be extra sure
- Link to the README's OAuth section were appropriate
 2019-09-27 17:34:10 +02:00
Mike Fährmann
ee8b654464
[instagram] implement 'highlights' option (closes #329) 2019-09-21 23:38:20 +02:00
Mike Fährmann
f02a768b5c
[danbooru] add 'ugoira' option (#406) 
to choose between ZIP archives or converted video files
for Ugoira posts
 2019-08-31 21:51:52 +02:00
Mike Fährmann
c6c5cb1898
improve 'deviantart.quality' description 2019-08-30 18:41:18 +02:00
Mike Fährmann
7c09545f70
[downloader:ytdl] add 'outtmpl' option (#395) 2019-08-24 22:47:59 +02:00
Mike Fährmann
5a210991b6
Remove control characters from filesystem paths 
- add 'path-remove' option to specify the set of characters that
 should be removed
- rename 'restrict-filenames' to 'path-restrict'
- #348, #380
 2019-08-16 23:12:16 +02:00
Mike Fährmann
e661a1ce19
update links to Request's documentation 2019-08-15 13:25:06 +02:00
Mike Fährmann
8dc42bb178
implement 'enumerate' for 'extractor.skip' (#306) 
[ci skip]
 2019-08-08 18:37:54 +02:00
Mike Fährmann
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option 2019-08-08 16:54:20 +02:00
Mike Fährmann
3957d27d79
[deviantart] add 'quality' option (#369) 2019-08-03 11:40:35 +02:00
Mike Fährmann
0609afd1e4
update default cache directory ... again 
Use a 'gallery-dl' subdirectory in ~/.cache to adhere to how other
programs store their cached data, and call os.makedirs() so it also
works without an existing ~/.cache directory.
 2019-08-01 22:11:00 +02:00
Mike Fährmann
a8b60b2bd9
change default cache directory for unix systems 
Use either $XDG_CACHE_HOME or ~/.cache (if the former isn't set)
and store potentially sensitive cookies and tokens in a user's
home directory and not in the world-readable /tmp.
 2019-07-31 22:56:14 +02:00
Mark Henrick
923e1bb714 [docs] Fix inconsistency about which sites have optional authentication (#359) 
* [docs] Fix inconsistency about which sites have optional authentication

* update authentication docs
 2019-07-29 18:22:31 +02:00
Mike Fährmann
a90280f4e7
[postprocessor:zip] add 'mode' option (#355) 2019-07-29 16:51:26 +02:00
Mike Fährmann
547ea71463
[downloader.ytdl] add 'forward-cookies' option (#352) 
The "long" name is necessary because just calling it 'cookies' would
clash with how the lookup for '--cookies' is implemented.
 2019-07-24 21:19:11 +02:00
Mike Fährmann
b1bea8aaeb
add 'restrict-filenames' option (#348) 2019-07-23 17:41:24 +02:00
Mike Fährmann
21991acc49
add 'ciphers' option; update default User-Agent 2019-07-19 17:14:40 +02:00
Mike Fährmann
feb98cf196
[twitter] improve 'content' formatting; add option (#338) 
- include emoticons
- leave newlines intact
- remove pic.twitter.com/ links at the end
 2019-07-17 16:02:51 +02:00
Mike Fährmann
8d1ae9b715
[tumblr] enable date-min/-max/-format options (#337) 2019-07-17 14:36:41 +02:00
Mike Fährmann
16c582aaf9
implement 'mtime' post-processor (#332) 
This can set a file's modification time according to a UNIX timestamp
or a datetime object from its metadata.
 2019-07-14 22:39:17 +02:00
Mike Fährmann
62097284fe
add 'download' option (#220) 2019-07-14 18:48:18 +02:00
Mike Fährmann
69205df68d
allow '-1' for infinite retries (#300) 2019-06-30 23:10:47 +02:00
Mike Fährmann
f7b5c4c3e7
use values of 'retries' options correctly 
The RE-tries option now specifies exactly that: the maximum number a
failed HTTP request is re-tried. For example a value of 2 will now
correctly stop after 3 attempts: the initial one + 2 re-tries.

The maximum wait-time now also caps at 30min and increases exponentially
for both extractor.request() and downloader.http.download().
 2019-06-30 23:10:18 +02:00
Mike Fährmann
f2000a69aa
implement 'image-unique' and 'chapter-unique' options (#303) 
The default value for both is 'false', i.e. duplicate URLs are NOT
ignored.

The previous behavior was to always ignore duplicate URLs to make
'--abort-on-skip' work properly when new images where added to the
beginning of a collection while gallery-dl is running.
 2019-06-29 22:50:17 +02:00
Mike Fährmann
15e4ddf46d
implement custom logging formatter 
supports custom log message formats for each loglevel and, by
extension, custom ANSI codes and colors for errors and warnings

(#304)
 2019-06-21 20:17:58 +02:00
Mike Fährmann
db3f52881a
add 'mtime' option 2019-06-20 17:19:44 +02:00
Mike Fährmann
e05a96db5e
[deviantart] rename 'stash' to 'extra' (#302) 
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
 2019-06-10 21:05:25 +02:00
Mike Fährmann
c23bf263fe
[deviantart] rename 'external' to 'stash' (#302) 
restrict extracted URLs to ones from https://sta.sh/...
 2019-06-09 11:16:02 +02:00
Mike Fährmann
2fb85178da
[deviantart] add 'external' option (#302) 
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
 2019-06-06 18:53:50 +02:00
Mike Fährmann
d33f5a7423
[wallhaven] rewrite 
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
  IDs that can't be translated to the new ID system
- support direct links to wallpapers
 2019-05-31 14:53:02 +02:00
Mike Fährmann
f1893b2b5b
[deviantart] add 'folders' option (#276) 2019-05-30 17:28:12 +02:00
Mike Fährmann
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option (#279) 
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
 2019-05-27 22:24:48 +02:00
Mike Fährmann
9890bfdf23
[flickr] improve code and metadata 
- simplify pagination
- add more metadata and slightly change its structure
  - convert suitable values to int or list
  - move keys from ["photo"] to the base level
- proper video support (#246)
- rename method and variable names to better fit with other extractors
 2019-05-14 22:10:50 +02:00
Mike Fährmann
70be494161
[plurk] add a 'comments' options (#212) 2019-04-14 22:12:46 +02:00
Mike Fährmann
369eb66125
consistently use '*' for rst lists 2019-03-28 16:21:41 +01:00
Mike Fährmann
c7a6b0ed90
[deviantart] add 'metadata' option (#189) 2019-03-21 14:49:42 +01:00
Mike Fährmann
12482553bd
update links to youtube-dl 2019-03-13 22:03:02 +01:00
Mike Fährmann
25aaf55514
[smugmug] improve format selection (closes #183) 
- use original image if available
- support video formats
- remove user info for ImageExtractor (it is no longer possible to get
  image owner information for a single image)
 2019-03-10 15:20:35 +01:00
Mike Fährmann
ae353ed3b0
provide "extractor" and "job" keys for logging output 
This allows for stuff like "{extractor.url}" and "{extractor.category}"
in logging format strings.
Accessing 'extractor' and 'job' in any way will return "None" if those
fields aren't defined, i.e. in general logging messages.
 2019-02-14 11:09:58 +01:00
Mike Fährmann
134487ffb0
[exhentai] stop extraction if image limit is exceeded (#141) 
can be turned off with the `exhentai.limits' option
 2019-01-26 18:40:39 +01:00
Mike Fährmann
b50b30f1c9
[photobucket] download subalbums (#117) 2019-01-22 14:05:18 +01:00
Mike Fährmann
277b52101a
add 'category-transfer' option 
[ci skip]
 2019-01-19 20:28:19 +01:00
Mike Fährmann
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes (#135) 2019-01-17 21:18:12 +01:00
Mike Fährmann
1e4d351ad3
[danbooru] add authentication support (closes #151) 
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
 2019-01-09 14:19:07 +01:00
Mike Fährmann
06cbf5f9c4
implement 'chapter-reverse' option (#149) 
Setting it to `true` will start with the latest chapter instead of the
first one.
 2019-01-07 18:22:33 +01:00
Mike Fährmann
e95b24f056
[reactor] add wait-min & -max options (#148) 2019-01-07 18:04:47 +01:00
Mike Fährmann
28f9539551
[tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba
[tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
c47482b110
smaller changes, missing docs, etc. 
- make 'netrc' extractor-specific
- rename 'downloader.enable' to 'enabled'
- document 'downloader.ytdl.format'
- consistent newlines in configuration.rst
 2018-11-16 18:18:07 +01:00
Mike Fährmann
8ef84a6823
add option to enable/disable specific downloader modules 
... and write URLs with no (active) downloader to unsupported-file
 2018-11-13 18:06:36 +01:00
Mike Fährmann
2fa28a2609
update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
655549df7c
[downloader:ytdl] add several options 
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.

downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.

downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
 2018-10-20 18:26:49 +02:00
Mike Fährmann
6ed629f2b6
allow specifying number of skips before abort/exit (closes #115) 
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
 2018-10-13 17:21:55 +02:00
Mike Fährmann
d8492df51b
[deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
392a081657
update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
f8b3b00249
[twitter] add experimental 'videos' option (#99) 
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
 2018-09-30 21:52:23 +02:00
Mike Fährmann
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection 2018-09-24 13:12:54 +02:00
Mike Fährmann
55f5c87160
[postprocessor:ugoira] add 'libx264-prevent-odd' option 
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
 2018-09-21 20:01:14 +02:00
Mike Fährmann
7742cf8601
[tumblr] change 'reblogs' option (#103) 
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
 2018-09-10 15:40:25 +02:00
Mike Fährmann
a666ddd16b
[tumblr] extend 'reblogs' functionality (#103) 
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
 2018-09-07 19:13:52 +02:00
Mike Fährmann
6ecb36d88c
[postprocessor:ugoira] add 'ffmpeg-output' option 2018-08-31 17:37:35 +02:00
Mike Fährmann
10365394d7
[twitter] add support for user-timelines (closes #96) 
also adds a 'retweets' option to filter retweeted content
 2018-08-17 20:04:11 +02:00
Mike Fährmann
b164231bca
[sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
5f27cfeff6
[deviantart] remove prefer-public option 
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
 2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048
[deviantart] improve public-private token switching 
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
 2018-07-25 12:52:36 +02:00
Mike Fährmann
886d662582
[deviantart] add option to minimize refresh-token usage 
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
 2018-07-24 21:44:57 +02:00
Mike Fährmann
54a0d72dc8
[postprocessor:ugoira] improve frame rate handling 
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.

This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
 2018-07-23 21:33:20 +02:00
Mike Fährmann
ff436692bf
["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
00032b828c
[deviantart] add 'wait-min' option 2018-07-14 11:52:21 +02:00
Mike Fährmann
269dc2bbd5
[sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
1d43cbbf52
[gelbooru] tag-splitting for non-api mode 2018-07-06 15:24:19 +02:00
Mike Fährmann
4a57509392
generalize tag-splitting option (#92) 
- extend functionality to other booru sites:
  - http://behoimi.org/
  - https://konachan.com/
  - https://e621.net/
  - https://rule34.xxx/
  - https://safebooru.org/
  - https://yande.re/
 2018-07-04 12:21:16 +02:00
Mike Fährmann
87853538b4
[yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
0c1c4557dd
[postprocessor:ugoira] add option for two-pass encoding 2018-06-20 18:48:10 +02:00
Mike Fährmann
eb7a1f3b98
[pixiv] rework ugoira handling 
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
 2018-06-18 17:57:57 +02:00
Mike Fährmann
b84e71da91
add postprocessor documentation to configuration.rst 2018-06-16 15:46:41 +02:00
Mike Fährmann
12797e3b1f
update configuration.rst 
... again

- some more 'Path' references
- fixed some inconsistencies and errors
- added note about logging config for files
 2018-05-28 22:14:38 +02:00
Mike Fährmann
c43f02245f
update configuration.rst 
- fix default values for 'log' and 'unsupportedfile'

[ci skip]
 2018-05-27 17:12:57 +02:00
Mike Fährmann
dacda69c9e
update configuration.rst 
- document logging options
- add a section for "custom types"

[ci skip]
 2018-05-27 16:50:35 +02:00
Mike Fährmann
53f36176fd
update configuration.rst 
- update the API Tokens & IDs section
  - mention redirect URIs for deviantart
  - include api-secret for tumblr
  - add instructions for smugmug
- [ci skip]
 2018-05-26 11:26:50 +02:00
Mike Fährmann
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements 
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
 2018-04-25 16:04:30 +02:00
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions 
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
 2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest (#83) 
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
 2018-04-16 20:03:28 +02:00
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order" 
This reverts commit 4a26ae32df.
 2018-04-09 16:08:08 +02:00
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
f0e3861ec7
update configuration.rst 
- move `base-directory` entry to extractor options
  (it should've been there since v1.0.0 / commit e3bfb83)
- rename "General" to "Miscellaneous" and move it down a bit, since it
  has only two options left
- mention the ability to use a list for tumblr.posts
- replace any appearance of bato.to
 2018-03-16 12:00:55 +01:00
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
8cdce21dcb
make archive keys user-configurable 2018-02-25 21:57:01 +01:00
Mike Fährmann
e1e0668ca8
add option to set default replacement field value 
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
 2018-02-23 00:59:20 +01:00
Mike Fährmann
8704d850bf
add explicit proxy support (#76) 
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
 2018-02-19 18:45:06 +01:00
Mike Fährmann
3cec533c28
Merge branch 'archive' 2018-02-12 18:07:58 +01:00
Mike Fährmann
7f7c16ae37
add option to specify additional key-value pairs 2018-02-08 23:10:58 +01:00
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option 
… as well as a config file equivalent
 2018-02-01 22:00:44 +01:00
Mike Fährmann
d951f13e37
add config option for unsupported-URL file 
for consistency's sake
 2018-01-28 18:42:10 +01:00
Mike Fährmann
97f4f15ec0
add option to write logging output to a file 
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
 2018-01-26 18:51:51 +01:00
Mike Fährmann
03b8a548cb
[tumblr] change reblogs default value to true (#61) 2018-01-06 15:52:08 +01:00
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts (#61) 
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
 2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use (closes #56) 
Gelbooru's API allows access to all images and is not restricted
to the first 20000.

This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
 2017-12-21 21:42:40 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication 
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
 2017-12-18 01:16:46 +01:00
Mike Fährmann
4fb6803fa6
add option to sleep before each download 2017-12-04 17:33:10 +01:00
Mike Fährmann
8f518e03f8
add options to set maximum download rate 
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option

This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.

[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
 2017-12-02 01:47:26 +01:00
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior (#48) 
- posts   : list of post-types to inspect
- inline  : scan post bodies for inline images
- external: follow external links
 2017-11-23 15:32:54 +01:00
Mike Fährmann
e6814aebe2
add 'extractor.*.user-agent' config option 2017-11-15 14:01:33 +01:00
Mike Fährmann
707b15b586
create missing directories for 'part-directory' 
also some code improvements regarding downloader config values
 2017-10-27 12:22:45 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files (#29) 
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
 2017-10-26 00:16:48 +02:00
Mike Fährmann
963670d73b
add options to control usage of .part files (#29) 
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options

Disabling .part files restores the behaviour of the old downloader
implementation.
 2017-10-24 23:33:44 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve (fixes #44) 
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
 2017-10-14 23:01:33 +02:00
Mike Fährmann
abfd2924a9
update configuration.rst 
- add instructions on how to register your own applications
- combine descriptions of a few config keys
 2017-10-12 23:37:28 +02:00
Mike Fährmann
f3fbaa5c3e
[reddit] allow users to override the API User-Agent 
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].

[1] https://github.com/reddit/reddit/wiki/API#rules
 2017-10-10 17:29:46 +02:00
Mike Fährmann
2ef3c35c98
smaller textual changes 
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
 2017-10-09 23:23:19 +02:00
Mike Fährmann
75d3a1f72f
[deviantart] always download original images 
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.

[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
 2017-10-07 13:07:34 +02:00
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords' 2017-09-09 18:48:28 +02:00
Mike Fährmann
54c0715135
allow users to set their own API access_tokens/client_ids 2017-09-09 17:50:19 +02:00
Mike Fährmann
b8862ff15e
add 'downloader.http.verify' option 
(also: change the default 'timeout' from None to 30)
 2017-08-31 15:21:08 +02:00
Mike Fährmann
3c9f190757
extend output of --list-keywords 2017-08-10 17:36:21 +02:00
Mike Fährmann
f08af03845
Merge branch 'cookies' 2017-07-25 14:04:53 +02:00
Mike Fährmann
00a23fe208
update configuration.rst 2017-07-21 18:32:56 +02:00
Mike Fährmann
9be8f7e106
[deviantart] add "extractor.deviantart.flat" option 
Setting this to 'false' downloads images into individual subdirectories
for each gallery-folder or favourite-collection, otherwise it is just
creating a flat list of images.
 2017-07-12 17:05:31 +02:00
Mike Fährmann
9edbd6ff01
update configuration.rst 2017-07-04 19:34:34 +02:00
Mike Fährmann
06c4cae05b
extend the output of '--list-extractors' 
It now includes category and subcategory values for
each extractor class.
 2017-06-28 18:51:47 +02:00
H R X N
b757687606 Update configuration.rst (#24) 
Fixed minor typo.
 2017-06-28 13:55:39 +02:00
Mike Fährmann
598e3a8d88
update configuration.rst 2017-06-27 18:56:24 +02:00
Mike Fährmann
86561d7c8b
update configuration.rst 2017-06-27 17:44:02 +02:00
Mike Fährmann
d3b04076f7
add .netrc support (#22) 
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.

The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
 2017-06-24 12:17:26 +02:00
Mike Fährmann
fbe8c519e2
update README 
- mention and link to docs/configuration.rst
- add OAuth section
 2017-06-23 16:14:51 +02:00
Mike Fährmann
ccb7973b57
update configuration.rst 2017-06-20 16:20:28 +02:00
H R X N
b206b831ee Update configuration.rst 
Fix heading.
 2017-06-19 14:12:43 +02:00
Mike Fährmann
48b444e276
add a list of all configuration options 
(the format might be subject to change; putting everything  in
 one single table is even worse)
 2017-06-18 22:16:26 +02:00