1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Commit Graph

168 Commits

Author SHA1 Message Date
Mike Fährmann
6ba67b0537
[hypnohub] add extractors (closes #196) 2019-03-23 09:50:39 +01:00
Mike Fährmann
8d96a8ce4c
[500px] add user-, gallery-, and image-extractors (#185) 2019-03-20 17:32:36 +01:00
Mike Fährmann
6277a739e4
[35photo] add user-, genre-, and image-extractors (#162) 2019-03-18 01:11:30 +01:00
Mike Fährmann
9c0e2f294b
[shopify] add generic collection and product extractors (#175)
with fashionnova.com  as a default domain
2019-03-05 22:33:37 +01:00
Mike Fährmann
280531c8ff
[pururin] add gallery extractor (closes #174) 2019-02-25 14:54:57 +01:00
Mike Fährmann
19860655a3
[weibo] add 'user' and 'status' extractors 2019-02-17 18:18:31 +01:00
Mike Fährmann
abbd45d0f4
update handling of extractor URL patterns
When loading extractor classes during 'extractor.find(…)', their
'pattern' attribute will be replaced with a compiled version of itself.
2019-02-08 20:08:16 +01:00
Mike Fährmann
6284731107
simplify extractor constants
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
2019-02-08 13:45:40 +01:00
Mike Fährmann
2f3a021d72
[hentaicafe] restore functionality 2019-02-05 10:22:52 +01:00
Mike Fährmann
e1bf3b225e
[foolslide] dynamically generate extractor classes 2019-02-04 12:54:07 +01:00
Mike Fährmann
58a9eede38
[foolfuuka] dynamically generate extractor classes 2019-02-03 17:09:45 +01:00
Mike Fährmann
80a75a1ecf
[tsumino] add gallery extractor (#161) 2019-01-29 17:28:48 +01:00
Mike Fährmann
0156189468
[hentaifox] add chapter extractor (#160) 2019-01-28 18:00:32 +01:00
Mike Fährmann
d19bac71be
[photobucket] add 'album' extractor (#117) 2019-01-20 16:19:13 +01:00
Mike Fährmann
b8fed34548
add generalized extractors for Mastodon instances (#144)
Extractors for Mastodon instances can now be dynamically generated,
based on the instance names in the 'extractor.mastodon.*' config path.

Example:
{
    "extractor": {
        "mastodon": {
            "pawoo.net": { ... },
            "mastodon.xyz": { ... },
            "tabletop.social": { ... },
            ...
        }
    }
}

Each entry requires an 'access-token' value, which can be generated with
'gallery-dl oauth:mastodon:<instance URL>'.
An 'access-token' (as well as a 'client-id' and 'client-secret') for
pawoo.net is always available, but can be overwritten as necessary.
2019-01-19 14:28:59 +01:00
Mike Fährmann
8e01cf0ef8
[reactor] generalize extractors (#148)
- support *.reactor.cc domains
- combine joyreactor and pornreactor modules
2019-01-07 17:06:47 +01:00
Leonardo Taccari
2655a2ea02 Add support for instagram.com user profiles and pages (#134)
* [instagram] Add extractor for instagram.com user profiles and pages

The extractor scrapes `instagram.com/<user>' timelines and
`instagram.com/p/<shortcode>' by mimicking the behaviour of a web
browser and extracting the sharedData JSON of the single pages.

Please note that this mean that for user timelines we also do an
extra request to the `instagram.com/p/<shortcode>' page but this
permit to have consistent (and all) information about the media
fetched.

The MD5 logic used for X-Instagram-GIS was documented in

 <https://stackoverflow.com/questions/49786980/>

* [instagram] Test for keywords, not url for GraphImage and GraphSidecar

URLs returned by instagram seems not stable so avoid testing for
them and instead test for keyword returned.

* [instagram] Improve test of InstagramProfilepageExtractor

Also check the count of media returned.

* [instagram] Several cleanup and improvements

- Change description, subcategories to generate a better description in
  docs/supportedsite.rst
- Remove not needed InstagramExtractor.__init__()
- Use text.parse_int() instead of directly using int() (the former is more
  robust)
- Use self.request().json() instead of using json.loads() the
  self.request().text()
- Add `pattern:' to check the URLs where we do not have a stable URLs.
  It seems that only the subdomain is not stable.

Thanks to @mikf!
2018-12-09 12:52:14 +01:00
Mike Fährmann
ba56827f36
[newgrounds] add user-, video-, image-extractors (#119) 2018-11-27 15:44:53 +01:00
Mike Fährmann
a0ae156edc
[pornreactor] add tag-, user-, post-extractors (#114) 2018-11-23 14:41:26 +01:00
Mike Fährmann
503d42a1c2
[joyreactor] add tag-, user-, post-extractors (#114) 2018-11-23 09:25:02 +01:00
Mike Fährmann
2cf3f53839
[yuki] add thread extractor (closes #111) 2018-09-28 12:47:32 +02:00
Mike Fährmann
c9290d8212
[wallhaven] add wallpaper and search extractors
todo:
- login support to gain access to NSFW wallpapers
- extractors for tag-, similar-, latest-listings
- skip() support
2018-09-17 21:26:13 +02:00
Mike Fährmann
69fd61ea86
[bobx] add gallery and model extractors 2018-09-13 20:13:12 +02:00
Mike Fährmann
3ecea4cf36
[hentaicafe] add chapter and manga extractors (#101) 2018-09-05 21:08:40 +02:00
Mike Fährmann
ab2bfaeb46
[ngomik] add replacement for 'subapics'
http://subapics.com/ got discontinued and replaced by http://ngomik.in/.

ngomik.in is still displaying a link to the "old site" showing a big
"Account Suspended" sign.
2018-08-23 15:29:53 +02:00
Mike Fährmann
3c90df6635
[piczel] add user, folder and image extractors 2018-08-08 10:53:01 +02:00
Mike Fährmann
df082e923c
[behance] add gallery extractor (#95) 2018-08-01 21:46:55 +02:00
Mike Fährmann
84854fcad7
[myportfolio] add user and gallery extractors (#95) 2018-07-19 18:56:45 +02:00
Mike Fährmann
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor 2018-07-14 09:53:16 +02:00
Mike Fährmann
55b0913412
[simplyhentai] add gallery extractor (#89) 2018-05-27 15:25:04 +02:00
Mike Fährmann
16e014baaa
[smugmug] added image and album extractor
just some initial code that still requires a lot of work ...

TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth

It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
2018-04-29 21:27:25 +02:00
Mike Fährmann
d96b3474e5
[puremashiro] remove module
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
2018-04-28 14:24:20 +02:00
Mike Fährmann
b44a296404
[gomanga] remove module
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
2018-04-28 14:24:21 +02:00
Mike Fährmann
dd49127408
[spectrumnexus] remove module
Site stopped hosting manga scans (http://view.thespectrum.net/)
2018-04-16 09:45:07 +02:00
Mike Fährmann
48a83a89e9
[loveisover] remove module
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
2018-04-09 16:05:15 +02:00
Mike Fährmann
dd314279fb
[test] add unit tests for extractor module functions 2018-03-25 11:49:42 +02:00
Mike Fährmann
723cc66bb1
[artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
d6ef52897c
[imgchili] remove module
All previously hosted images yield a 404
and the main page is just a logo.
2018-03-13 11:02:58 +01:00
Mike Fährmann
749fbbfa6c
[mangadex] add chapter- and manga-extractor 2018-03-05 18:37:21 +01:00
Mike Fährmann
edfd3d9fc9
[yeet] remove module
- archive.yeet.net returns a 500 server error
- yeet.net moved to yeet.rip, but the archive is gone
2018-02-23 01:05:41 +01:00
Mike Fährmann
54c36a8a34
[subapics] add chapter- and manga-extractor (#70) 2018-02-04 22:02:10 +01:00
Mike Fährmann
2dd3aeeeae
[komikcast] add chapter- and manga-extractor (#70) 2018-02-04 22:02:10 +01:00
Mike Fährmann
6a07e38366
implement extractor.add() and .add_module()
... as a public and non-hacky way to add (external) extractors to
gallery-dl's pool and make them available for extractor.find()
2018-02-02 00:01:41 +01:00
Mike Fährmann
1e93955170
[batoto] remove module
Site officially shut down on 2018.01.18
2018-01-23 21:37:32 +01:00
Mike Fährmann
91ed147cef
[oauth] use custom key/secret values during oauth:… 2018-01-16 17:39:46 +01:00
Mike Fährmann
40d35c87bc
[paheal] add tag- and post-extractors (closes #69) 2018-01-15 16:39:05 +01:00
Mike Fährmann
f10ffc0839
update extractor blacklist to also allow classes 2018-01-14 18:47:22 +01:00
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com 2018-01-09 17:54:37 +01:00
Mike Fährmann
5b094328b5
[puremashiro] add chapter- and manga-extractor (closes #66)
Also adds support for region subtags in language codes (e.g. en-us)
2018-01-07 21:50:43 +01:00
Leonardo Taccari
a8d2dde8b2 [slideshare] Add a new extractor for slideshare.net (#54) 2017-12-13 17:38:29 +01:00