1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Commit Graph

1469 Commits

Author SHA1 Message Date
Mike Fährmann
1734a6c879
[reactor] detect "circular" redirects (#148) 2019-01-09 14:59:15 +01:00
Mike Fährmann
e53cdfd6a8
update build_supportedsites.py 2019-01-09 14:58:35 +01:00
Mike Fährmann
1e4d351ad3
[danbooru] add authentication support (closes #151)
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
2019-01-09 14:19:07 +01:00
Mike Fährmann
06cbf5f9c4
implement 'chapter-reverse' option (#149)
Setting it to `true` will start with the latest chapter instead of the
first one.
2019-01-07 18:22:33 +01:00
Mike Fährmann
e95b24f056
[reactor] add wait-min & -max options (#148) 2019-01-07 18:04:47 +01:00
Mike Fährmann
8e01cf0ef8
[reactor] generalize extractors (#148)
- support *.reactor.cc domains
- combine joyreactor and pornreactor modules
2019-01-07 17:06:47 +01:00
Mike Fährmann
38500ad697
[postprocessor:metadata] first implementation (#135) 2019-01-05 16:39:05 +01:00
Mike Fährmann
1737d7f576
[joyreactor] fix and improve pagination (#148) 2019-01-03 22:13:38 +01:00
Mike Fährmann
8753627ef4
[joyreactor] improve error handling for faulty JSON (#148)
- remove all ASCII escape codes, not just \n and \r
- ignore faulty posts instead of letting the exception propagate
2019-01-03 16:31:25 +01:00
Mike Fährmann
a36f52a730
[joyreactor] add extractor for search results (#148) 2019-01-03 16:25:56 +01:00
Mike Fährmann
a303efb597
[mangadex] handle manga pages without chapters 2019-01-03 16:22:12 +01:00
Mike Fährmann
0afa913de4
[tumblr] add tests for hidden and private blogs (#145)
Hidden / dashboard-only blogs are pretty straightforward and "only"
require a valid 'access-token' and 'access-token-secret' for the given
'api-key' and 'api-secret', so that signed OAuth1.0 requests are possible.

Private / password protected blogs on the other hand are a bit
cumbersome. In addition to a valid 'access-token' and
'access-token-secret', they also require the account belonging to those
tokens to be a member of the blog itself. Knowing the password and
entering it in the website isn't enough to access a blog through the
API. Following a private blog is also impossible, so that option can't
work either.
2019-01-03 16:12:24 +01:00
Mike Fährmann
67cc0ac873
release version 1.6.2 2019-01-01 16:26:58 +01:00
Mike Fährmann
fa7fa2f8ff
[deviantart1 update tests] 2019-01-01 15:39:34 +01:00
Mike Fährmann
b7b5456a32
[kissmanga] use HTTPS 2018-12-30 14:04:46 +01:00
Mike Fährmann
259123732f
[readcomiconline] improve comic-page parsing 2018-12-30 13:19:23 +01:00
Mike Fährmann
0328a04a65
[cloudflare] don't output the whole challenge page
thanks to the embedded animated gifs this is just a bit too much
2018-12-30 13:09:23 +01:00
Mike Fährmann
4ab0960083
[reddit] add metadata to extracted URLs 2018-12-29 17:52:43 +01:00
Mike Fährmann
2f4f60de33
[tumblr] add tests for each post type 2018-12-27 22:41:42 +01:00
Mike Fährmann
98314aa04c
[mangapark] detect non-existent chapters 2018-12-27 21:41:50 +01:00
Mike Fährmann
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor (#113)
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
2018-12-26 18:56:57 +01:00
Mike Fährmann
f9ace0f4a3
[mangapark] fix manga extraction ... again 2018-12-26 18:56:57 +01:00
Mike Fährmann
28f9539551
[tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba
[tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
7471933d5f
use extractor.request for all other API calls
- deviantart
- pawoo
- pixiv
- reddit
2018-12-22 14:42:23 +01:00
Mike Fährmann
995844c915
[instagram] relax test pattern even more 2018-12-22 14:25:55 +01:00
Mike Fährmann
2e5f82e59e
[tumblr] don't follow 'external' Tumblr URLs (#139) 2018-12-22 14:05:43 +01:00
Mike Fährmann
c5d4f558c9
allow missing field access keys in format strings (#136) 2018-12-22 13:54:14 +01:00
Mike Fährmann
0c9762f00e
[mangapark] fix extraction 2018-12-22 13:52:48 +01:00
Mike Fährmann
c9ef5ed364
[luscious] ensure URLs have a scheme 2018-12-21 17:56:51 +01:00
Mike Fährmann
851ee9f89f
[sensescans] replace tests
the old ones got removed
2018-12-21 16:05:07 +01:00
Mike Fährmann
c14d44e1bc
[downloader:common] retry downloads on SSL errors (#130) 2018-12-14 16:33:04 +01:00
Mike Fährmann
0be7ee3106
[hitomi] fix image subdomains (closes #142)
galleries with an ID ending in 1 need some special treatment
2018-12-14 16:15:06 +01:00
Mike Fährmann
fe96835d25
[kissmanga] add fallback for chapter-string parsing (#20) 2018-12-14 16:08:36 +01:00
Mike Fährmann
4d73cc785d
update test results 2018-12-14 16:07:32 +01:00
Mike Fährmann
049a9575c4
[tumblr] fix inline extraction #2
Using only the "comment" field isn't enough ...

[ci skip]
2018-12-11 21:57:20 +01:00
Mike Fährmann
f6bf66f72c
[pixiv] create directory for each "work" item (#136) 2018-12-11 20:37:47 +01:00
Mike Fährmann
79f6755c60
[postprocessor:classify] handle missing "extension" (#138) 2018-12-11 20:10:02 +01:00
Mike Fährmann
b7a9f6cc49
[tumblr] improve inline extraction (#137) 2018-12-11 20:02:48 +01:00
Mike Fährmann
010da8372a
[instagram] relax test pattern 2018-12-11 19:59:28 +01:00
Mike Fährmann
1c6b9ba322
[readcomiconline] use HTTPS 2018-12-09 14:54:55 +01:00
Leonardo Taccari
2655a2ea02 Add support for instagram.com user profiles and pages (#134)
* [instagram] Add extractor for instagram.com user profiles and pages

The extractor scrapes `instagram.com/<user>' timelines and
`instagram.com/p/<shortcode>' by mimicking the behaviour of a web
browser and extracting the sharedData JSON of the single pages.

Please note that this mean that for user timelines we also do an
extra request to the `instagram.com/p/<shortcode>' page but this
permit to have consistent (and all) information about the media
fetched.

The MD5 logic used for X-Instagram-GIS was documented in

 <https://stackoverflow.com/questions/49786980/>

* [instagram] Test for keywords, not url for GraphImage and GraphSidecar

URLs returned by instagram seems not stable so avoid testing for
them and instead test for keyword returned.

* [instagram] Improve test of InstagramProfilepageExtractor

Also check the count of media returned.

* [instagram] Several cleanup and improvements

- Change description, subcategories to generate a better description in
  docs/supportedsite.rst
- Remove not needed InstagramExtractor.__init__()
- Use text.parse_int() instead of directly using int() (the former is more
  robust)
- Use self.request().json() instead of using json.loads() the
  self.request().text()
- Add `pattern:' to check the URLs where we do not have a stable URLs.
  It seems that only the subdomain is not stable.

Thanks to @mikf!
2018-12-09 12:52:14 +01:00
HRXN
e80ee77d71 tumblr.py: update regex for video (#133)
There seems to be another sub-domain for videos, apparently..
Not just
`vt(.media).tumblr`
`vtt(media).tumblr`
But also
`ve(.media).tumblr`
2018-12-09 09:07:46 +01:00
Mike Fährmann
9a98b6769d
use extractor.request for API calls (#130)
... at least for OAuth1.0 based APIs (flickr, smugmug, tumblr)
2018-12-04 21:29:06 +01:00
Mike Fährmann
0225d90078
add exception name and traceback for OSErrors 2018-12-04 19:24:50 +01:00
Mike Fährmann
ad2cefda6b
[tumblr] in case of exception use filename as 'hash' (#129)
While a filename might not be a real 'hash', or comparable to what
tumbler usually provides, it is still better than an empty string.
At least as long as "alternatives" in format strings aren't implemented.
2018-12-04 19:15:23 +01:00
Mike Fährmann
95636418ad
[tumblr] catch exception for 'hash' extraction (fixes #129) 2018-12-02 19:48:09 +01:00
Mike Fährmann
40e30694f3
[pinterest] fix pin.it redirects 2018-12-02 19:38:50 +01:00
Mike Fährmann
770200888e
[gfycat] use public API endpoint 2018-12-02 18:56:53 +01:00
Mike Fährmann
b1e22e8354
release version 1.6.1 2018-11-28 15:34:01 +01:00