Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Code Issues Releases Wiki Activity
1,908 Commits 6 Branches 164 Tags 16 MiB
c6c5cb1898
Commit Graph

286 Commits

Author SHA1 Message Date
Mike Fährmann
e4171d6baf
[luscious] add login capabilities (closes #159) 2019-01-28 17:14:15 +01:00
Mike Fährmann
134487ffb0
[exhentai] stop extraction if image limit is exceeded (#141) 
can be turned off with the `exhentai.limits' option
 2019-01-26 18:40:39 +01:00
Mike Fährmann
b50b30f1c9
[photobucket] download subalbums (#117) 2019-01-22 14:05:18 +01:00
Mike Fährmann
277b52101a
add 'category-transfer' option 
[ci skip]
 2019-01-19 20:28:19 +01:00
Mike Fährmann
4b441c162e
release version 1.6.3 2019-01-18 12:31:04 +01:00
Mike Fährmann
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes (#135) 2019-01-17 21:18:12 +01:00
Mike Fährmann
e53cdfd6a8
update build_supportedsites.py 2019-01-09 14:58:35 +01:00
Mike Fährmann
1e4d351ad3
[danbooru] add authentication support (closes #151) 
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
 2019-01-09 14:19:07 +01:00
Mike Fährmann
06cbf5f9c4
implement 'chapter-reverse' option (#149) 
Setting it to `true` will start with the latest chapter instead of the
first one.
 2019-01-07 18:22:33 +01:00
Mike Fährmann
e95b24f056
[reactor] add wait-min & -max options (#148) 2019-01-07 18:04:47 +01:00
Mike Fährmann
a36f52a730
[joyreactor] add extractor for search results (#148) 2019-01-03 16:25:56 +01:00
Mike Fährmann
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor (#113) 
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
 2018-12-26 18:56:57 +01:00
Mike Fährmann
28f9539551
[tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba
[tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
be52069cbc
update CHANGELOG and docs/supportedsites 2018-11-28 14:53:27 +01:00
Mike Fährmann
c47482b110
smaller changes, missing docs, etc. 
- make 'netrc' extractor-specific
- rename 'downloader.enable' to 'enabled'
- document 'downloader.ytdl.format'
- consistent newlines in configuration.rst
 2018-11-16 18:18:07 +01:00
Mike Fährmann
8ef84a6823
add option to enable/disable specific downloader modules 
... and write URLs with no (active) downloader to unsupported-file
 2018-11-13 18:06:36 +01:00
Mike Fährmann
2fa28a2609
update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
655549df7c
[downloader:ytdl] add several options 
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.

downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.

downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
 2018-10-20 18:26:49 +02:00
Mike Fährmann
6ed629f2b6
allow specifying number of skips before abort/exit (closes #115) 
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
 2018-10-13 17:21:55 +02:00
Mike Fährmann
d8492df51b
[deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
392a081657
update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
f8b3b00249
[twitter] add experimental 'videos' option (#99) 
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
 2018-09-30 21:52:23 +02:00
Mike Fährmann
2cf3f53839
[yuki] add thread extractor (closes #111) 2018-09-28 12:47:32 +02:00
Mike Fährmann
17611bfec0
update build_supportedsites.py script 2018-09-28 12:43:19 +02:00
Mike Fährmann
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection 2018-09-24 13:12:54 +02:00
Mike Fährmann
c402cc4047
[hentaifoundry] add 'popular' and 'recent' extractors 
for "Popular Pictures" and "Recent Pictures" listings
 2018-09-24 13:11:18 +02:00
Mike Fährmann
a5fc311dfa
[hentaifoundry] add 'favorite' extractor 2018-09-22 21:23:29 +02:00
Mike Fährmann
55f5c87160
[postprocessor:ugoira] add 'libx264-prevent-odd' option 
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
 2018-09-21 20:01:14 +02:00
Mike Fährmann
006f75b538
[hentaifoundry] rewrite + more metadata 
- extract width, height, artist per image
- improve pattern regex
- better extensibility for other listings
 2018-09-21 11:23:51 +02:00
Mike Fährmann
6ea9a78588
[wallhaven] add login capabilities 
Being logged in is required to access NSFW wallpapers.
 2018-09-19 21:04:01 +02:00
Mike Fährmann
c9290d8212
[wallhaven] add wallpaper and search extractors 
todo:
- login support to gain access to NSFW wallpapers
- extractors for tag-, similar-, latest-listings
- skip() support
 2018-09-17 21:26:13 +02:00
Mike Fährmann
22ab509a70
[bobx] rename "model" to "idol" extractor 2018-09-14 18:11:36 +02:00
Mike Fährmann
69fd61ea86
[bobx] add gallery and model extractors 2018-09-13 20:13:12 +02:00
Mike Fährmann
7742cf8601
[tumblr] change 'reblogs' option (#103) 
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
 2018-09-10 15:40:25 +02:00
Mike Fährmann
f1695567e8
adjust values in template config file 
(#104)
 2018-09-09 14:10:55 +02:00
Mike Fährmann
a666ddd16b
[tumblr] extend 'reblogs' functionality (#103) 
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
 2018-09-07 19:13:52 +02:00
Mike Fährmann
c9b8e6aefc
[reddit] fix submission-ID parsing (#104) 
Uppercase characters caused a ValueError exception
 2018-09-07 18:27:54 +02:00
Mike Fährmann
3ecea4cf36
[hentaicafe] add chapter and manga extractors (#101) 2018-09-05 21:08:40 +02:00
Mike Fährmann
75862715ac
[behance] add user extractor 2018-08-31 17:42:09 +02:00
Mike Fährmann
6ecb36d88c
[postprocessor:ugoira] add 'ffmpeg-output' option 2018-08-31 17:37:35 +02:00
Mike Fährmann
ab2bfaeb46
[ngomik] add replacement for 'subapics' 
http://subapics.com/ got discontinued and replaced by http://ngomik.in/.

ngomik.in is still displaying a link to the "old site" showing a big
"Account Suspended" sign.
 2018-08-23 15:29:53 +02:00
Mike Fährmann
e9dd2eff1d
[twitter] add extractor for media-tweet timelines (#96) 
For example "https://twitter.com/PicturesEarth/media".
They are different from normal timelines in that they do not contain
any (re)tweets from other users and feature all media the user ever
posted, including responses to other tweets.
 2018-08-19 20:46:12 +02:00
Mike Fährmann
9b1c39032c
[twitter] changes and improvements 
- rename User- to TimelineExtractor
- rename 'userid' to 'user_id' to conform to the other ..._id values
- adjust archive_fmt to deal with retweets
- emulate browser behavior for API calls
 2018-08-18 23:04:45 +02:00
Mike Fährmann
10365394d7
[twitter] add support for user-timelines (closes #96) 
also adds a 'retweets' option to filter retweeted content
 2018-08-17 20:04:11 +02:00
Mike Fährmann
63fa0b2006
[pinterest] add extractors for related pins 
Related pins can not be accessed by adding a "#related" fragment
to the end of a Pinterest URL, for example:
- https://www.pinterest.com/pin/858146903966145189/#related
- https://www.pinterest.com/g1952849/test-/#related

There are no explicit real URLs for related pins,
using an option to enable them results in "clunky" code,
and a custom "related:<URL>" scheme doesn't feel right either.
 2018-08-15 21:49:45 +02:00
Mike Fährmann
3c90df6635
[piczel] add user, folder and image extractors 2018-08-08 10:53:01 +02:00
Mike Fährmann
b164231bca
[sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
df082e923c
[behance] add gallery extractor (#95) 2018-08-01 21:46:55 +02:00
Mike Fährmann
5f27cfeff6
[deviantart] remove prefer-public option 
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
 2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048
[deviantart] improve public-private token switching 
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
 2018-07-25 12:52:36 +02:00
Mike Fährmann
886d662582
[deviantart] add option to minimize refresh-token usage 
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
 2018-07-24 21:44:57 +02:00
Mike Fährmann
54a0d72dc8
[postprocessor:ugoira] improve frame rate handling 
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.

This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
 2018-07-23 21:33:20 +02:00
Mike Fährmann
84854fcad7
[myportfolio] add user and gallery extractors (#95) 2018-07-19 18:56:45 +02:00
Mike Fährmann
c9f70e0a19
[paheal] use HTTPS 2018-07-17 21:25:03 +02:00
Mike Fährmann
ff436692bf
["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
00032b828c
[deviantart] add 'wait-min' option 2018-07-14 11:52:21 +02:00
Mike Fährmann
269dc2bbd5
[sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
1d43cbbf52
[gelbooru] tag-splitting for non-api mode 2018-07-06 15:24:19 +02:00
Mike Fährmann
c20c0a4820
[safebooru] add pool extractor 2018-07-04 12:24:57 +02:00
Mike Fährmann
4a57509392
generalize tag-splitting option (#92) 
- extend functionality to other booru sites:
  - http://behoimi.org/
  - https://konachan.com/
  - https://e621.net/
  - https://rule34.xxx/
  - https://safebooru.org/
  - https://yande.re/
 2018-07-04 12:21:16 +02:00
Mike Fährmann
188e956c4e
[imagefap] use HTTPS + update test results 2018-06-30 19:40:46 +02:00
Mike Fährmann
87853538b4
[yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
0c1c4557dd
[postprocessor:ugoira] add option for two-pass encoding 2018-06-20 18:48:10 +02:00
Mike Fährmann
eb7a1f3b98
[pixiv] rework ugoira handling 
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
 2018-06-18 17:57:57 +02:00
Mike Fährmann
b84e71da91
add postprocessor documentation to configuration.rst 2018-06-16 15:46:41 +02:00
Mike Fährmann
6ac403c5d3
add postprocessor config example 2018-06-08 18:31:59 +02:00
Mike Fährmann
cdcc3427a0
[simplyhentai] add video extractor (#89) 
All videos hosted on their own servers seem be to dead,
but myhentai.tv embeds, which are most of the videos, work fine.
 2018-05-30 11:25:23 +02:00
Mike Fährmann
12797e3b1f
update configuration.rst 
... again

- some more 'Path' references
- fixed some inconsistencies and errors
- added note about logging config for files
 2018-05-28 22:14:38 +02:00
Mike Fährmann
c43f02245f
update configuration.rst 
- fix default values for 'log' and 'unsupportedfile'

[ci skip]
 2018-05-27 17:12:57 +02:00
Mike Fährmann
dacda69c9e
update configuration.rst 
- document logging options
- add a section for "custom types"

[ci skip]
 2018-05-27 16:50:35 +02:00
Mike Fährmann
55b0913412
[simplyhentai] add gallery extractor (#89) 2018-05-27 15:25:04 +02:00
Mike Fährmann
53f36176fd
update configuration.rst 
- update the API Tokens & IDs section
  - mention redirect URIs for deviantart
  - include api-secret for tumblr
  - add instructions for smugmug
- [ci skip]
 2018-05-26 11:26:50 +02:00
Mike Fährmann
ecdc3475b8
[pixhost] support .to TLDs 2018-05-23 18:32:34 +02:00
Mike Fährmann
909d105ae6
[pixiv] add extractor for illusts from followed users 2018-05-15 13:05:15 +02:00
Mike Fährmann
f3483a2b7c
[smugmug] add OAuth support 2018-05-10 18:58:05 +02:00
Mike Fährmann
69a5e6ddb3
Merge branch 'master' into 1.4-dev 2018-05-04 10:19:02 +02:00
Mike Fährmann
82c50fa609
release version 1.3.5 2018-05-04 10:03:20 +02:00
Mike Fährmann
8bf3cdd82b
implement logging options 
Standard logging to stderr, logfiles, and unsupported URL files (which
are now handled through the logging module) can now be configured by
setting their respective option keys (log, logfile, unsupportedfile)
to a dict and specifying the following options;

- format:
    format string for logging messages
    available keys: see [1]
    default: "[{name}][{levelname}] {message}"
- format-date:
    format string for {asctime} fields in logging messages
    available keys: see [2]
    default: "%Y-%m-%d %H:%M:%S"
- level:
    the lowercase levelname until which the logger should activate;
    available levels are debug, info, warning, error, exception
    default: "info"
- path:
    path of the file to be written to
- mode:
    'mode' argument when opening the specified file
    can be either "w" to truncate the file or "a" to append to it (see [3])

If 'output.log', '.logfile', or '.unsupportedfile' is a string, it will
be interpreted, as it has been, as the filepath
(or as format string for .log)

[1] https://docs.python.org/3/library/logging.html#logrecord-attributes
[2] https://docs.python.org/3/library/time.html#time.strftime
[3] https://docs.python.org/3/library/functions.html#open
 2018-05-01 17:54:52 +02:00
Mike Fährmann
16e014baaa
[smugmug] added image and album extractor 
just some initial code that still requires a lot of work ...

TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth

It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
 2018-04-29 21:27:25 +02:00
Mike Fährmann
d96b3474e5
[puremashiro] remove module 
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
 2018-04-28 14:24:20 +02:00
Mike Fährmann
b44a296404
[gomanga] remove module 
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
 2018-04-28 14:24:21 +02:00
Mike Fährmann
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements 
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
 2018-04-25 16:04:30 +02:00
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions 
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
 2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest (#83) 
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
 2018-04-16 20:03:28 +02:00
Mike Fährmann
dd49127408
[spectrumnexus] remove module 
Site stopped hosting manga scans (http://view.thespectrum.net/)
 2018-04-16 09:45:07 +02:00
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order" 
This reverts commit 4a26ae32df.
 2018-04-09 16:08:08 +02:00
Mike Fährmann
564e12ca8f
replace 'imgyt' with 'imxto' 
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
 2018-04-09 15:53:20 +02:00
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
7b562907c3
[nijie] add favorites extractor 
adds support for 'https://nijie.info/user_like_illust_view.php?id=...'
 2018-03-31 18:54:25 +02:00
Mike Fährmann
f39153b6e9
[nhentai] add extractor for search results 2018-03-28 17:21:44 +02:00
Mike Fährmann
52d41c41e7
[exhentai] add extractor for favorited galleries 2018-03-27 18:58:42 +02:00
Mike Fährmann
a993d0ea90
release version 1.3.2 2018-03-23 15:27:56 +01:00
Mike Fährmann
e7525b1b0e
[artstation] add challenge extractor (#80) 2018-03-23 15:06:09 +01:00
Mike Fährmann
b25ae30a95
reorder options in gallery-dl.conf 2018-03-22 10:05:53 +01:00
Mike Fährmann
40ca562d7b
[artstation] add album extractor (#80) 2018-03-17 17:36:31 +01:00
HRXN
241eda8c3d Update supportedsites.rst (#81) 
Add ArtStation to Supported Sites.
 2018-03-16 13:40:59 +01:00
Mike Fährmann
f0e3861ec7
update configuration.rst 
- move `base-directory` entry to extractor options
  (it should've been there since v1.0.0 / commit e3bfb83)
- rename "General" to "Miscellaneous" and move it down a bit, since it
  has only two options left
- mention the ability to use a list for tumblr.posts
- replace any appearance of bato.to
 2018-03-16 12:00:55 +01:00
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
d6ef52897c
[imgchili] remove module 
All previously hosted images yield a 404
and the main page is just a logo.
 2018-03-13 11:02:58 +01:00
Mike Fährmann
7847ab1d5a
[imagehosts] remove even more dead sites 
All removed sites either
- reject all incoming connections or
- display a message from their domain registrar
 2018-03-12 21:25:13 +01:00
Mike Fährmann
74c8f83056
update gallery-dl.conf and add gallery-dl-example.conf 
- gallery-dl.conf contains all config options set to their default
  values, with the exception of 'filename', 'directory', and
  'cache.file'

- gallery-dl-example.conf contains a potential real-world configuration
  file and demonstrates some more involved options
 2018-03-12 18:15:30 +01:00
Mike Fährmann
1400868f53
[mangadex] general improvements 
- support >100 chapter entries per manga
- custom archive ID format
- detect non-existing chapters
 2018-03-06 14:15:15 +01:00
Mike Fährmann
749fbbfa6c
[mangadex] add chapter- and manga-extractor 2018-03-05 18:37:21 +01:00
Mike Fährmann
6e38cf5aab
[mangareader] use 'https://' 
The site now redirects from http://mangareader.net/
to https://mangareader.net/
 2018-03-02 17:19:17 +01:00
Mike Fährmann
8cdce21dcb
make archive keys user-configurable 2018-02-25 21:57:01 +01:00
Mike Fährmann
8f338347b6
[imagehosts] cleanup 
removed
- chronos.to      - unable to resolve hostname
- coreimg.net     - same
- imgmaid.net     - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt    - cloudflare error 522, host down
- img4ever.net    - read timeout
 2018-02-23 01:05:42 +01:00
Mike Fährmann
e1e0668ca8
add option to set default replacement field value 
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
 2018-02-23 00:59:20 +01:00
Mike Fährmann
8704d850bf
add explicit proxy support (#76) 
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
 2018-02-19 18:45:06 +01:00
Mike Fährmann
3cec533c28
Merge branch 'archive' 2018-02-12 18:07:58 +01:00
Mike Fährmann
7f7c16ae37
add option to specify additional key-value pairs 2018-02-08 23:10:58 +01:00
Mike Fährmann
54c36a8a34
[subapics] add chapter- and manga-extractor (#70) 2018-02-04 22:02:10 +01:00
Mike Fährmann
2dd3aeeeae
[komikcast] add chapter- and manga-extractor (#70) 2018-02-04 22:02:10 +01:00
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option 
… as well as a config file equivalent
 2018-02-01 22:00:44 +01:00
Mike Fährmann
d951f13e37
add config option for unsupported-URL file 
for consistency's sake
 2018-01-28 18:42:10 +01:00
Mike Fährmann
97f4f15ec0
add option to write logging output to a file 
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
 2018-01-26 18:51:51 +01:00
Mike Fährmann
1e93955170
[batoto] remove module 
Site officially shut down on 2018.01.18
 2018-01-23 21:37:32 +01:00
Mike Fährmann
40d35c87bc
[paheal] add tag- and post-extractors (closes #69) 2018-01-15 16:39:05 +01:00
Mike Fährmann
cc0c2cca57
[reddit] add extractor for reddit-hosted images (closes #68) 2018-01-14 18:55:42 +01:00
Mike Fährmann
35e09869d1
[mangapark] fix image URLs and use HTTPS 2018-01-12 14:59:49 +01:00
Mike Fährmann
29d75fc3fa
[tumblr] add support for OAuth authentication (#65) 2018-01-11 14:11:37 +01:00
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com 2018-01-09 17:54:37 +01:00
Mike Fährmann
5b094328b5
[puremashiro] add chapter- and manga-extractor (closes #66) 
Also adds support for region subtags in language codes (e.g. en-us)
 2018-01-07 21:50:43 +01:00
Mike Fährmann
03b8a548cb
[tumblr] change reblogs default value to true (#61) 2018-01-06 15:52:08 +01:00
Mike Fährmann
d235f68f59
[tumblr] add option to filter reblogged posts (#61) 
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
 2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e
[gelbooru] re-enable API use (closes #56) 
Gelbooru's API allows access to all images and is not restricted
to the first 20000.

This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
 2017-12-21 21:42:40 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication 
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
 2017-12-18 01:16:46 +01:00
Mike Fährmann
0a9a07a6e1
[slideshare] improve metadata; flake8 
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
 2017-12-13 21:16:49 +01:00
Mike Fährmann
19a6ae57b2
[sankaku] add pool extractor 2017-12-12 19:45:10 +01:00
Mike Fährmann
4fb6803fa6
add option to sleep before each download 2017-12-04 17:33:10 +01:00
Mike Fährmann
300346ecdf
[mangazuki] remove extractors 
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
 2017-12-04 13:36:04 +01:00
Mike Fährmann
8f518e03f8
add options to set maximum download rate 
- -r/--limit-rate as cmdline option
- downloader.http.rate as config option

This implementation very roughly uses the idea of the token bucket
algorithm [1] and mostly uses Wget's approach [2] as inspiration.

[1] https://en.wikipedia.org/wiki/Token_bucket
[2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111
 2017-12-02 01:47:26 +01:00
Mike Fährmann
214972bc9a
[gelbooru] use manual extraction 
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875)

This also adds an extractor for image-pools.
 2017-11-29 20:48:17 +01:00
Mike Fährmann
12de658937
[tumblr] add options to control extraction behavior (#48) 
- posts   : list of post-types to inspect
- inline  : scan post bodies for inline images
- external: follow external links
 2017-11-23 15:32:54 +01:00
Mike Fährmann
e6814aebe2
add 'extractor.*.user-agent' config option 2017-11-15 14:01:33 +01:00
Mike Fährmann
2d0cfb33e1
[xvideos] add user profile extractor (#45) 2017-11-02 17:28:35 +01:00
Mike Fährmann
707b15b586
create missing directories for 'part-directory' 
also some code improvements regarding downloader config values
 2017-10-27 12:22:45 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files (#29) 
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
 2017-10-26 00:16:48 +02:00
Mike Fährmann
963670d73b
add options to control usage of .part files (#29) 
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options

Disabling .part files restores the behaviour of the old downloader
implementation.
 2017-10-24 23:33:44 +02:00
Mike Fährmann
5fa42336a2
[sankaku] add warning for unauthenticated users 
also improve URL pattern and add missing options to default config file
 2017-10-16 21:21:08 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve (fixes #44) 
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
 2017-10-14 23:01:33 +02:00
Mike Fährmann
abfd2924a9
update configuration.rst 
- add instructions on how to register your own applications
- combine descriptions of a few config keys
 2017-10-12 23:37:28 +02:00
Mike Fährmann
f3fbaa5c3e
[reddit] allow users to override the API User-Agent 
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].

[1] https://github.com/reddit/reddit/wiki/API#rules
 2017-10-10 17:29:46 +02:00
Mike Fährmann
2ef3c35c98
smaller textual changes 
- swapped doc for deviantart.mature and .original
- updated gallery-dl.conf
- "transferred" -> "delegated"
 2017-10-09 23:23:19 +02:00
Mike Fährmann
75d3a1f72f
[deviantart] always download original images 
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.

[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
 2017-10-07 13:07:34 +02:00
Mike Fährmann
cebf800a7f
[foolfuuka] add support for more sites (#18) 
- https://arch.b4k.co
- https://archive.whatisthisimnotgoodwithcomputers.com
- https://archive.yeet.net

Notes:
- The name "whatisthisimnotgoodwithcomputers" is way too long ...
- archive.yeet.net is out of date and also blocked by 4chan servers
  - newest threads are 2 weeks old
  - using "https://archive.yeet.net" as Referer header results in
    "403 Forbidden" when accessing 4chan
 2017-09-16 21:36:16 +02:00
Mike Fährmann
470bbe9d8c
fix smaller stuff 
- change filename option in example config file
- adapt default filename format for mangafox
- remove unnecessary newline

[skip ci]
 2017-09-11 17:07:29 +02:00
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords' 2017-09-09 18:48:28 +02:00
Mike Fährmann
54c0715135
allow users to set their own API access_tokens/client_ids 2017-09-09 17:50:19 +02:00