Mike Fährmann
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements
...
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
2018-04-25 16:04:30 +02:00
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions
...
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
2018-04-16 20:03:28 +02:00
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
...
This reverts commit 4a26ae32df
2018-04-09 16:08:08 +02:00
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order
2018-04-07 15:57:55 +02:00
Mike Fährmann
f0e3861ec7
update configuration.rst
...
- move `base-directory` entry to extractor options
(it should've been there since v1.0.0 / commit e3bfb83
2018-03-16 12:00:55 +01:00
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors
2018-03-14 14:05:14 +01:00
Mike Fährmann
8cdce21dcb
make archive keys user-configurable
2018-02-25 21:57:01 +01:00
Mike Fährmann
e1e0668ca8
add option to set default replacement field value
...
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
2018-02-23 00:59:20 +01:00
Mike Fährmann
8704d850bf
add explicit proxy support ( #76 )
...
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
2018-02-19 18:45:06 +01:00
Mike Fährmann
3cec533c28
Merge branch 'archive'
2018-02-12 18:07:58 +01:00
Mike Fährmann
7f7c16ae37
add option to specify additional key-value pairs
2018-02-08 23:10:58 +01:00
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option
...
… as well as a config file equivalent
2018-02-01 22:00:44 +01:00
Mike Fährmann
d951f13e37
add config option for unsupported-URL file
...
for consistency's sake
2018-01-28 18:42:10 +01:00
Mike Fährmann
97f4f15ec0
add option to write logging output to a file
...
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
2018-01-26 18:51:51 +01:00
Mike Fährmann
03b8a548cb
[tumblr] change reblogs default value to true ( #61 )
2018-01-06 15:52:08 +01:00
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts ( #61 )
...
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use ( closes #56 )
...
Gelbooru's API allows access to all images and is not restricted
to the first 20000.
This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication
...
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.
[*] e.g. https://polinaegorussia.deviantart.com/gallery/
--------------
known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
2017-12-18 01:16:46 +01:00
Mike Fährmann
4fb6803fa6
add option to sleep before each download
2017-12-04 17:33:10 +01:00
Mike Fährmann
8f518e03f8
add options to set maximum download rate
...
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option
This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.
[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
2017-12-02 01:47:26 +01:00
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior ( #48 )
...
- posts : list of post-types to inspect
- inline : scan post bodies for inline images
- external: follow external links
2017-11-23 15:32:54 +01:00
Mike Fährmann
e6814aebe2
add 'extractor.*.user-agent' config option
2017-11-15 14:01:33 +01:00
Mike Fährmann
707b15b586
create missing directories for 'part-directory'
...
also some code improvements regarding downloader config values
2017-10-27 12:22:45 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files ( #29 )
...
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
2017-10-26 00:16:48 +02:00
Mike Fährmann
963670d73b
add options to control usage of .part files ( #29 )
...
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options
Disabling .part files restores the behaviour of the old downloader
implementation.
2017-10-24 23:33:44 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve ( fixes #44 )
...
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
2017-10-14 23:01:33 +02:00
Mike Fährmann
abfd2924a9
update configuration.rst
...
- add instructions on how to register your own applications
- combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
Mike Fährmann
f3fbaa5c3e
[reddit] allow users to override the API User-Agent
...
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].
[1] https://github.com/reddit/reddit/wiki/API#rules
2017-10-10 17:29:46 +02:00
Mike Fährmann
2ef3c35c98
smaller textual changes
...
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
2017-10-09 23:23:19 +02:00
Mike Fährmann
75d3a1f72f
[deviantart] always download original images
...
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.
[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
2017-10-07 13:07:34 +02:00
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords'
2017-09-09 18:48:28 +02:00
Mike Fährmann
54c0715135
allow users to set their own API access_tokens/client_ids
2017-09-09 17:50:19 +02:00
Mike Fährmann
b8862ff15e
add 'downloader.http.verify' option
...
(also: change the default 'timeout' from None to 30)
2017-08-31 15:21:08 +02:00
Mike Fährmann
3c9f190757
extend output of --list-keywords
2017-08-10 17:36:21 +02:00
Mike Fährmann
f08af03845
Merge branch 'cookies'
2017-07-25 14:04:53 +02:00
Mike Fährmann
00a23fe208
update configuration.rst
2017-07-21 18:32:56 +02:00
Mike Fährmann
9be8f7e106
[deviantart] add "extractor.deviantart.flat" option
...
Setting this to 'false' downloads images into individual subdirectories
for each gallery-folder or favourite-collection, otherwise it is just
creating a flat list of images.
2017-07-12 17:05:31 +02:00
Mike Fährmann
9edbd6ff01
update configuration.rst
2017-07-04 19:34:34 +02:00
Mike Fährmann
06c4cae05b
extend the output of '--list-extractors'
...
It now includes category and subcategory values for
each extractor class.
2017-06-28 18:51:47 +02:00
H R X N
b757687606
Update configuration.rst ( #24 )
...
Fixed minor typo.
2017-06-28 13:55:39 +02:00
Mike Fährmann
598e3a8d88
update configuration.rst
2017-06-27 18:56:24 +02:00
Mike Fährmann
86561d7c8b
update configuration.rst
2017-06-27 17:44:02 +02:00
Mike Fährmann
d3b04076f7
add .netrc support ( #22 )
...
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.
The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
2017-06-24 12:17:26 +02:00
Mike Fährmann
fbe8c519e2
update README
...
- mention and link to docs/configuration.rst
- add OAuth section
2017-06-23 16:14:51 +02:00
Mike Fährmann
ccb7973b57
update configuration.rst
2017-06-20 16:20:28 +02:00
H R X N
b206b831ee
Update configuration.rst
...
Fix heading.
2017-06-19 14:12:43 +02:00
Mike Fährmann
48b444e276
add a list of all configuration options
...
(the format might be subject to change; putting everything in
one single table is even worse)
2017-06-18 22:16:26 +02:00
