Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 19:22:32 +01:00
Code Issues Releases Wiki Activity
3,298 Commits 6 Branches 164 Tags 16 MiB
7b05600056
Commit Graph

2401 Commits

Author SHA1 Message Date
Hans Christian Gunawan
334d690687
[hentaicosplays] Add extractor (#1473) 2021-04-18 20:28:00 +02:00
Mike Fährmann
82c32d25af
[500px] update query hashes 2021-04-15 17:28:31 +02:00
Mike Fährmann
de14b7ad7a
[slideshare] fix extraction 2021-04-15 17:15:59 +02:00
Mike Fährmann
bef3105121
[komikcast] fix extraction 2021-04-15 17:04:53 +02:00
Mike Fährmann
086925e685
[shopify] support omgmiamiswimwear.com (closes #1280) 2021-04-13 23:54:03 +02:00
thatfuckingbird
224b883ff4
[danbooru] add option for extended metadata extraction (#1458) 
* [danbooru] add option for extended metadata extraction

* appease linter

* [danbooru] update docs/configuration.rst

* [danbooru] rename extended-metadata -> metadata
 2021-04-13 23:41:30 +02:00
thatfuckingbird
dff03a6605
[booru] add an option to extract notes (only gelbooru for now) (#1457) 
* [booru] add an option to extract notes (currently implemented only for gelbooru)

* appease linter

* [gelbooru] rename "text" to "body" in note extraction

* add a code comment about reusing return value of _extended_tags
 2021-04-13 23:40:24 +02:00
Mike Fährmann
78d7ee3ef4
[yuki] remove module for yuki.la 2021-04-12 21:42:32 +02:00
Mike Fährmann
5a98bcec3a
[deviantart] improve folder name matching (fixes #1451) 2021-04-11 20:39:40 +02:00
thatfuckingbird
918b0441fb
[gelbooru] fix tag category extraction (#1455) 2021-04-10 19:05:00 +02:00
Mike Fährmann
fe6ce5b8f8
[erome] skip deleted albums (fixes #1447) 2021-04-09 15:24:18 +02:00
Mike Fährmann
457abf0e71
[deviantart] fix pagination for Eclipse results (fixes #1444) 
- don't crash on missing keys
- use fallback for invalid 'nextOffset' values
 2021-04-09 15:16:56 +02:00
Mike Fährmann
dee540050f
[8muses] fix JSON unobfuscation 
limit the characters that get modified,
leave non-ASCII characters alone
 2021-04-09 01:49:54 +02:00
Mike Fährmann
b869b3a9eb
[instagram] fetch media for incomplete GraphSidecar posts 
GraphSidecar results from /tagged pages don't contain
all media elements, only the first one.

(#1439)
 2021-04-09 00:37:16 +02:00
Mike Fährmann
b0686d2174
[instagram] update query hashes 2021-04-09 00:37:15 +02:00
Mike Fährmann
e8e3717b71
[instagram] add extractor for /tagged posts (#1439) 2021-04-09 00:37:08 +02:00
Mike Fährmann
abafe71e04
[exhentai] fix image limit detection (closes #1437) 
check for image limit message when downloading original files
 2021-04-08 21:33:41 +02:00
Mike Fährmann
52a7913abe
[artstation] download /4k/ images (#1422) 2021-04-07 21:50:16 +02:00
Christian Paul
41fbc20020
[webtoons]: Add cookie rstagGDPR_DE=true (#1431) 2021-04-07 21:42:55 +02:00
FollieHiyuki
e3b9f88540
Add manganelo extractor (#1415) 2021-04-02 21:01:31 +02:00
Mike Fährmann
fd858eed7b
[twitter] add 'user_likes' metadata field for liked tweets 
i.e. the 'screen_name' of the user whose liked tweets get extracted.

Ideally this would replace 'user' or at least be in the same format,
but that would break backwards compatibility or be impossible/too
complicated thanks to API result differences.

(#1421)
 2021-04-02 03:41:41 +02:00
Mike Fährmann
8d124a3766
[twitter] rename variables 2021-04-02 02:49:53 +02:00
Mike Fährmann
105f3c9666
[twitter] add extractor for direct image links (closes #1417) 2021-04-02 02:45:23 +02:00
Mike Fährmann
ec3d5d58a8
[vk] improve extractor (#474) 
- fetch all photos
- add 'metadata' option
- fix extracting photos without '?' in URL
 2021-04-01 14:35:56 +02:00
Mike Fährmann
ebd142e2a8
[twitter] don't use youtube-dl for cards when videos are disabled 
(#1416)
 2021-04-01 14:26:08 +02:00
Mike Fährmann
d5aad999dc
[tapas] implement login with username & password (#692) 2021-03-30 01:45:28 +02:00
Mike Fährmann
e9ec91c811
[exhentai] improve image limits check 
- check if current image is the '509 Bandwidth Exceeded' notification
  (https://ehgt.org/g/509.gif or https://exhentai.org/img/509.gif)
- remove 'limits' option
 2021-03-29 19:01:13 +02:00
Mike Fährmann
387fe415d5
unescape items in text.split_html() 2021-03-29 02:12:29 +02:00
Mike Fährmann
36291176bc
[pinterest] add 'search' extractor (#1411) 2021-03-29 01:41:28 +02:00
Mike Fährmann
058cc47e9b
[bcy] improve pagination 2021-03-28 23:08:26 +02:00
Mike Fährmann
ddd48ceee5
update extractor test results 2021-03-28 23:06:44 +02:00
Mike Fährmann
1a540fbe00
[komikcast] fix extraction 2021-03-28 21:18:58 +02:00
Mike Fährmann
5aa30c3669
[tapas] add 'series' and 'episode' extractors (#692) 2021-03-27 18:28:16 +01:00
Mike Fährmann
ccfa5a8694
[twitter] better error message when logging in with 2FA (#1409) 2021-03-27 18:26:37 +01:00
Mike Fährmann
214ecf62ce
[deviantart] fix arguments for search/popular results (#1408) 2021-03-27 18:26:10 +01:00
Magnus Boman
522d0a834c
[aryion] Unescape paths too (#1414) 
Without this you'll get paths like this:
  - Starcross - Ch. 2 "The Ins and Outs of Sarah"

This commit changes it to:
  - Starcross - Ch. 2 "The Ins and Outs of Sarah"
 2021-03-27 18:25:38 +01:00
beesdotjson
5ad615f0db
fix PixivFavoriteExtractor regex (#1405) 
* fix PixivFavoriteExtractor regex

* do not use lookbehind
 2021-03-25 14:59:33 +01:00
Mike Fährmann
62cfee4d28
[vk] initial support for albums (#474) 2021-03-23 19:02:16 +01:00
Mike Fährmann
0e601de67b
[sankaku] simplify 'pool' tags (#1388) 
normalize 'tags' and 'artist_tags' to a string-list
 2021-03-23 18:45:45 +01:00
Mike Fährmann
d085ade9d5
[sankaku] add 'tag_string' metadata field (#1388) 
The 'join()'ed version of 'tags'.
Handling lists in format strings isn't properly supported yet.
 2021-03-23 15:42:13 +01:00
Mike Fährmann
2dffd231b7
[sankaku] add enumeration index for books (#1388) 2021-03-23 15:32:54 +01:00
Mike Fährmann
139fb84108
[deviantart] fix username for 'watch' results (#794) 
before it'd use "/" as username
 2021-03-22 22:14:21 +01:00
Mike Fährmann
91c2e15da9
[deviantart] add support for posts from watched users (#794) 2021-03-22 19:25:04 +01:00
Mike Fährmann
03c20d8c8e
[deviantart] update 'watch' URL pattern (#794) 2021-03-21 22:48:06 +01:00
Mike Fährmann
2846235669
[twitter] allow specifying a custom format for user results 
(#1337)
 2021-03-21 22:26:26 +01:00
Mike Fährmann
dc23cfd684
[deviantart] use fallback for /intermediary/ URLs 
instead of checking availability with HEAD requests
 2021-03-20 00:10:53 +01:00
Mike Fährmann
b0438c8f99
Revert "[deviantart] extend 'extra' option" 
This reverts commit
5ad2b9c82b,
5c32a7bf58, and
83f465faca.

(#1387, #1356)
 2021-03-19 16:24:23 +01:00
Mike Fährmann
58b93635ee
[architizer] add 'firm' extractor (#1369) 2021-03-19 01:31:34 +01:00
Mike Fährmann
204523611c
[imgclick] use 'http://' for image URLs 
The TLS certificate for main.imgclick.net is invalid.
 2021-03-19 01:30:49 +01:00
Mike Fährmann
0b55f5ad84
[imgur] fix/improve rate limit handling (#1386) 
- also wait-and-retry on 429 status codes
- use infinite loop instead of recursive calls
- 'extractor.sleep()' -> 'extractor.wait()'
 2021-03-18 15:45:26 +01:00
Mike Fährmann
69ca4e29f1
[deviantart] add 'watch' extractor (#794) 2021-03-17 22:50:02 +01:00
Mike Fährmann
fcdda6128c
[mangastream] remove module 2021-03-16 23:52:36 +01:00
Mike Fährmann
c677ea19dd
[mangareader] remove module 2021-03-16 23:48:55 +01:00
Mike Fährmann
71523aaab6
[architizer] add 'project' extractor (#1369) 2021-03-16 03:24:29 +01:00
Mike Fährmann
3378b39719
[twitter] implement 'users' option (#1337) 2021-03-16 00:51:05 +01:00
Mike Fährmann
847e9b0ed7
[philomena] support post URLs without '/images/' 
e.g. 'derpibooru.org/1'
 2021-03-14 18:26:39 +01:00
Mike Fährmann
466966bf83
[hentaicafe] remove module 2021-03-14 17:19:57 +01:00
Mike Fährmann
97641cd151
[hentainexus] remove module 2021-03-14 17:19:57 +01:00
Mike Fährmann
c485d0a956
[philomena] add generalized extractors for philomena sites 
(closes #1379)
 2021-03-14 17:19:57 +01:00
Mike Fährmann
6be7df53da
[hentaifox] improve metadata extraction (fixes #1378) 2021-03-14 17:19:56 +01:00
Mike Fährmann
72fe9ac0f3
[gelbooru_v01] support some more boorus by default 
- https://drawfriends.booru.org/
- https://vidyart.booru.org/
- https://tlb.booru.org/
 2021-03-14 17:19:56 +01:00
tux93
10c279f285
Weasyl: Drop the &feature=submit part of the favourite extractor URL (#1374) 
It's optional and requiring it forces users to escape those URLs because
of the ampersand
 2021-03-12 16:56:04 +01:00
Mike Fährmann
4be27ff0fe
[nozomi] support '/index-N.html' URLs (closes #1365) 
and '/index-Popular-N.html'
 2021-03-11 01:06:47 +01:00
Mike Fährmann
780bac4c8a
[gelbooru] update video server (fixes #1368) 
from 'https://img2.gelbooru.com' to 'https://img3.gelbooru.com'
and provide fallback URLs
 2021-03-10 01:48:07 +01:00
Mike Fährmann
f8441e851a
[hentaifox] improve image extraction (fixes #1366) 
build image URLs from embedded JSON data
instead 0f rewriting thumbnail URLs
 2021-03-10 01:38:32 +01:00
Mike Fährmann
c7c3fef0bc
[exhentai] support '/tag/' URLs (closes #1363) 2021-03-08 22:40:51 +01:00
Mike Fährmann
90830daf85
[exhentai] improve 'favorites' extraction (closes #1360) 
add special cases for when the favorite count is 0 (Never) or 1 (Once)
 2021-03-07 22:40:25 +01:00
Mike Fährmann
83f465faca
[deviantart] refactor 'extra' (#1356) 
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
 2021-03-07 22:23:52 +01:00
Mike Fährmann
5c32a7bf58
[deviantart] allow selecting source for 'extra' (#1356) 
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
 2021-03-06 21:31:28 +01:00
Mike Fährmann
a677123abb
[instagram] recognize 'reels' as option for 'include' (#1329) 2021-03-06 18:50:19 +01:00
Mike Fährmann
94faf8c85a
add type check before applying 'browser' option (fixes #1358) 2021-03-06 18:15:32 +01:00
Mike Fährmann
7440d1f112
[pixiv] add 'translated-tags' option (closes #1354) 
(a lot more straight forward than I thought ...)
 2021-03-05 17:18:51 +01:00
Mike Fährmann
106cdc37c0
[instagram] support '/user/reels/' URLs (closes #1329) 2021-03-04 21:31:18 +01:00
Mike Fährmann
524ebb133e
[instagram] refactor reel handling 2021-03-04 20:30:28 +01:00
Mike Fährmann
9785c551bc
[500px] skip unavailable photos (#1335) 
instead of crashing with a KeyError exception
 2021-03-04 20:26:26 +01:00
Mike Fährmann
6cfc9613fe
update some code in Extractor constructor 
- combine '_init_headers' and '_emulate_browser' functionality
  into new '_init_session'
- add 'headers' and 'ciphers' options
 2021-03-03 23:13:29 +01:00
Mike Fährmann
f59e63669b
[hentaicafe] add 'search' and 'tag' extractors (#1345) 2021-03-03 23:09:34 +01:00
Mike Fährmann
38e66940c1
[tumblrgallery] simplify 2021-03-03 15:37:56 +01:00
Seonghyeon Cho
665499924d
Support naver webtoon (#1331) 
* Support naver webtoon (WIP)

* Apply patch

* Change filename format

* Fill test results

* Fill test result
 2021-03-03 15:21:13 +01:00
topozorra
a9119da4d4
support tumblrgallery.xyz (#1298) 
* support `tumblrgallery.xyz`

* fix format issues

* Refactor and add post and search page support

* Fix warnings

* Few improvments

* Better file names

* Fix linting errors

* move id closer to the begining of the file name

Co-authored-by: topozorra <none>
 2021-03-03 15:20:47 +01:00
Mike Fährmann
bff71cde80
implement 'util.unique_squence()' 2021-03-02 23:11:08 +01:00
Mike Fährmann
bae874f370
replace 'wait-min/-max' with 'sleep-request' 
on exhentai, idolcomplex, reactor
 2021-03-02 22:55:45 +01:00
Mike Fährmann
e165e6c265
[wallhaven] add 'collections' extractor (#1351) 2021-03-02 01:32:26 +01:00
Mike Fährmann
faf561b6ca
[wallhaven] add 'collection' extractor (#1351) 2021-03-01 17:12:11 +01:00
Mike Fährmann
5d3d94ba14
[wallhaven] refactor 2021-03-01 17:06:24 +01:00
Mike Fährmann
8821dceb79
use __import__() to dynamically load modules 2021-03-01 01:27:02 +01:00
Mike Fährmann
69ea781d32
[mangadex] improve caching of manga results 
'manga_id' being a string or integer are treated as two different keys
 2021-02-28 01:34:58 +01:00
Mike Fährmann
e58039358d
[mangadex] use 'api.mangadex.org' as default API server 
The caching issues seem to be gone.
(#1290, #1310)
 2021-02-28 01:31:50 +01:00
Mike Fährmann
fc15930266
[readcomiconline] download high quality image versions 
(fixes #1347)
 2021-02-28 01:11:32 +01:00
Mike Fährmann
f360778e60
[komikcast] fix extraction 2021-02-27 21:02:52 +01:00
Mike Fährmann
3df527ee2c
update extractor test results 2021-02-27 21:01:29 +01:00
Mike Fährmann
fe2ec9cf68
[patreon] reduce redirects when fetching campaign ID 2021-02-27 16:37:41 +01:00
Mike Fährmann
29ea54dc41
[patreon] use '"browser": "firefox"' by default (#1117) 2021-02-27 16:26:42 +01:00
Mike Fährmann
61fbbd2dae
[exhentai] rename metadata fields to match API results (#1325) 
- gallery_id    -> gid
- gallery_token -> token
- title_jp      -> title_jpn
- visible       -> expunged
- gallery_size  -> filesize
- count         -> filecount

Also changes the function of the 'metadata' option.
It is now boolean and causes extra data fields from the API to be added
instead of completely replacing the data from HTML when activated.
 2021-02-26 18:21:04 +01:00
Mike Fährmann
996bfe4d4b
[hentaicafe] fix manga extractor 
was broken since 993856b8
 2021-02-26 14:24:52 +01:00
Mike Fährmann
5d69e437d0
[twitter] add option to download all media from a conversation 
(fixes #1319)
 2021-02-26 13:50:46 +01:00
Mike Fährmann
cf5fa75d4c
add 'browser' option (#1117) 
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
 2021-02-26 13:41:27 +01:00
Mike Fährmann
e5735361ed
[exhentai] add 'metadata' option (#1325) 
to select between gallery metadata from 'api' or 'html'
 2021-02-23 00:23:42 +01:00
Mike Fährmann
8f095a0980
[exhentai] extract more metadata from gallery pages (#1325) 2021-02-22 22:58:31 +01:00
Mike Fährmann
ffce8d85e7
[cyberdrop] update 
- add test and archive_fmt
- extract more metadata
 2021-02-22 19:36:34 +01:00
Mike Fährmann
de0656941b
[twitter] add extractor for followed users (#1337) 
https://twitter.com/USER/following or
https://twitter.com/id:USERID/following
 2021-02-22 18:22:01 +01:00
Mike Fährmann
e39aea42cd
fix supportedsites.py for modules without docstring 
(fixes #1332)
 2021-02-21 22:57:37 +01:00
loragja
7b5ee922b7
cyberdrop extractor (#1328) 
* create cyberdrop extractor

* add cyberdrop to list of extractors

* fix formatting

* change class name from CyberdropExtractor to CyberdropAlbumExtractor

* add cyberdrop to list of supported sites

* attempt to clean up diff of supportedsites.rst

* replace regex with functions from text library
 2021-02-21 20:42:45 +01:00
Mike Fährmann
5ad2b9c82b
[deviantart] extend 'extra' option 
also download from embedded DeviantArt posts
 2021-02-21 19:27:04 +01:00
Mike Fährmann
6b0ecbf6bc
[hentainexus] add 'orignal' option (#1322) 2021-02-21 19:06:12 +01:00
Mike Fährmann
5542a11c46
[twitter] update GraphQL endpoints 2021-02-20 02:09:17 +01:00
Mike Fährmann
e1a12761d7
strip '/' from instance root URLs 2021-02-17 23:07:17 +01:00
Mike Fährmann
595bdaa4be
add extractors for gelbooru v0.1 sites 
- support https://illusioncards.booru.org/  (closes #426)
- support https://the-collection.booru.org/ (closes #767)
- support https://allgirl.booru.org/
- closes #234, closes #473, closes #1238

To get gallery-dl to recognize other sites running Gelbooru v0.1
(most sites on booru.org), add one or more entries to the
'gelbooru_v01' block in your config file. For example:

{
    "extractor": {
        "gelbooru_v01": {
            "rozenmaidenbooru": {"root": "http://rm.booru.org"},
            "drawfriendsbooru": {"root": "http://drawfriends.booru.org"}
        }
    }
}
 2021-02-17 02:36:27 +01:00
Mike Fährmann
59fd740b47
[tbib] add support for https://tbib.org/ (#473, closes #1082) 2021-02-17 00:28:25 +01:00
Mike Fährmann
08d7934c6e
move extractors from booru.py into their own gelbooru_v02 module 2021-02-17 00:26:24 +01:00
Mike Fährmann
d656892670
remove cloudflare.py 
The old IUAM challenge doesn't get used anymore, i.e. code to bypass it
is pointless, and the 'is_...()' checks are simple enough to directly
include them in 'extractor.request()'.
 2021-02-15 23:17:02 +01:00
Mike Fährmann
65ca923b4e
fix 'whitelist' option for BaseExtractor instances 2021-02-15 21:58:33 +01:00
Mike Fährmann
fbfcbcbf57
Merge branch '1.17.0' 2021-02-15 02:32:33 +01:00
Mike Fährmann
ba693d8686
[patreon] skip posts without view permission (#1316) 2021-02-14 16:03:11 +01:00
Mike Fährmann
dcbd995346
[vanillarock] fix metadata extraction 2021-02-14 15:44:22 +01:00
Mike Fährmann
4b1cda4cf7
[paheal] fix metadata extraction 2021-02-14 15:43:39 +01:00
Mike Fährmann
2919d78bfc
update extractor test results 2021-02-14 15:37:39 +01:00
Mike Fährmann
8974f0361c
[pixiv] update (#1304) 
- remove login with username & password
- require a refresh token
- add 'oauth:pixiv' functionality

See also:
- https://github.com/upbit/pixivpy/issues/158
- https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362
 2021-02-12 18:07:16 +01:00
Mike Fährmann
79c0fc249b
[mangadex] add 'api-server' option (#1309) 
and change the API server back to 'https://mangadex.org/api' for now
 2021-02-11 19:52:16 +01:00
Mike Fährmann
96a51ff169
[sankaku] update invalid-token detection (fixes #1309) 2021-02-11 19:49:24 +01:00
Mike Fährmann
23be48427c
[deviantart] fix 'folders' option (closes #1302) 
don't assume parent folders are listed before their children
 2021-02-10 17:45:30 +01:00
Mike Fährmann
ca6b0fc2ac
[imagehosts] cleanup 2021-02-10 17:36:45 +01:00
Mike Fährmann
95a66bdad6
[imgclick] add 'image' extractor (closes #1307) 
basically reverts b0e8daf415
 2021-02-10 17:22:12 +01:00
Mike Fährmann
fc78210725
[kemonoparty] include 'service' in directories and archive keys 2021-02-09 20:35:36 +01:00
Mike Fährmann
c386a9fabf
[kemonoparty] fix absolute file URLs 2021-02-09 20:29:39 +01:00
Mike Fährmann
7e7158e7c0
[kemonoparty] support URLs with non-numeric user and post IDs 
(fixes #1303)
 2021-02-09 19:37:22 +01:00
Mike Fährmann
e88d5bede8
[500px] update query hash 2021-02-08 22:40:02 +01:00
Mike Fährmann
280b1ac16d
[slideshare] fix extraction 2021-02-08 22:29:30 +01:00
Mike Fährmann
ae530f6365
[erome] add extractors for albums, users, searches (closes #409) 2021-02-07 22:58:19 +01:00
ticklebits
fa6d4d73c7
[hentaifox] support searching by group (#1294) 
Groups on hentaifox lists the items to download the same way as the other pages (artists, search, tag, etc). Added group to the pattern to search, and the test.
 2021-02-06 17:52:00 +01:00
Mike Fährmann
2cc1e04fe5
[kemonoparty] extract inline images (fixes #1286) 2021-02-05 20:10:20 +01:00
Mike Fährmann
56a8968435
remove 'Message.Metadata' (#866) 2021-01-31 02:12:37 +01:00
Mike Fährmann
1d145a6186
[mastodon] use cache for OAuth tokens (#616) 2021-01-31 01:38:23 +01:00
Mike Fährmann
a041a017d1
[pillowfort] ignore files without download URL (#846) 2021-01-30 21:50:56 +01:00
Mike Fährmann
23ff936d46
[nsfwalbum] use fallback for deleted content (fixes #1259) 2021-01-29 23:35:20 +01:00
Mike Fährmann
6b2bce3b7d
[behance] support 'video' modules (closes #1282) 
(requires youtube-dl to download from m3u8 manifests)
 2021-01-29 21:30:14 +01:00
Mike Fährmann
b0cff979b1
[inkbunny] raise NotFoundError for invalid/private submissions 
instead of crashing
 2021-01-28 17:05:52 +01:00
Mike Fährmann
e61b125fcc
[inkbunny] add 'sid' parameter to private file downloads 
(fixes #1281)
 2021-01-28 15:45:48 +01:00
Mike Fährmann
36bf76fa44
update 'oauth:mastodon:<instance>' code 2021-01-28 02:20:12 +01:00
Mike Fährmann
88fae99811
remove 'generate_extractors()' 2021-01-28 01:04:50 +01:00
Mike Fährmann
fa33f13453
[mastodon] update 
- inherit from BaseExtractor
- remove custom generate_extractors() and config()
- improve layout of MastodonAPI internals
 2021-01-27 23:49:01 +01:00
Mike Fährmann
231bcad614
[shopify] use BaseExtractor 2021-01-27 23:45:22 +01:00
Mike Fährmann
2de8ebc098
[moebooru] use BaseExtractor 2021-01-27 01:33:40 +01:00
Mike Fährmann
0978c1e184
[booru] use BaseExtractor 2021-01-27 01:33:01 +01:00
Mike Fährmann
c6cc86d7d0
[deviantart] update parameters for '/browse/popular' 
- limit results to 50 when also querying metadata (fixes #1267)
- remove deprecated 'category_path' parameter
 2021-01-26 23:35:32 +01:00
Mike Fährmann
993856b866
[foolslide] use BaseExtractor 2021-01-26 18:50:32 +01:00
Mike Fährmann
671a95cae5
[foolfuuka] use BaseExtractor 2021-01-26 18:48:37 +01:00
Mike Fährmann
745a114c61
[common] implement BaseExtractor class 
Should be used when the same extractor logic applies to different
instances/domains of several sites, e.g. FoolFuuka, Shopify, etc.

This will replace the functionality of 'generate_extractors()' in
a more efficient way, by condensing everything into 1 class and not
dynamically generating an extractor class for each instance.
 2021-01-26 03:48:02 +01:00
Mike Fährmann
c26de0929d
[deviantart] provide 'extension' for original file downloads 
(#1272)
 2021-01-25 19:11:00 +01:00
Mike Fährmann
24e8e398e0
[twitter] skip login if 'auth_token' cookie is present 2021-01-25 15:03:59 +01:00
Mike Fährmann
cdb0b02e30
[pillowfort] add 'reblogs' option (#846) 2021-01-25 15:03:59 +01:00
Mike Fährmann
7ca3bf7cb0
[pillowfort] add 'user' and 'post' extractors (#846) 2021-01-25 15:03:22 +01:00
Mike Fährmann
ebf417f31f
remove support for deprecated options 
- instagram.highlights
- metadata.bypost
- exec.final
 2021-01-24 22:30:16 +01:00
Mike Fährmann
1d13e48512
[unsplash] implement 'skip()' 2021-01-23 17:24:18 +01:00
Mike Fährmann
6cdbfb79e9
[photovogue] update (#1253) 2021-01-22 19:38:10 +01:00
Federico Ravasio
25297815bc
[photovogue] added portfolio extractor (#1253) 2021-01-22 19:36:13 +01:00
Mike Fährmann
0265fbda61
[mangakakalot] fix extraction 2021-01-22 17:24:18 +01:00
Mike Fährmann
7a096c443f
[unsplash] add 'format' option (#1197) 2021-01-22 17:23:46 +01:00
Mike Fährmann
3188ac16d1
[unsplash] add 'collection' extractor (#1197) 2021-01-21 22:30:01 +01:00
Mike Fährmann
247cc73446
[derpibooru] update 'date' parsing 2021-01-21 21:51:04 +01:00
Mike Fährmann
193dca2ce1
update extractor test results 2021-01-21 21:35:42 +01:00
Mike Fährmann
89ea1384fc
[unsplash] fix typo 2021-01-20 22:51:02 +01:00
Mike Fährmann
e5e591b848
[vipr] simplify and add test (#1258) 2021-01-20 21:40:04 +01:00
v-delta
e707e060cb
[vipr] add image extractor (#1258) 
* [vipr] add image extractor

Adds support for images hosted on https://vipr.im

* Fix codestyle issues
 2021-01-20 21:38:43 +01:00
Mike Fährmann
95e5911895
[twitter] match '/i/user/ID' URLs 2021-01-20 00:33:57 +01:00
Mike Fährmann
069b113cbf
[twitter] improve and fix retry after hitting rate limit 
- replace recursive call with infinite loop
- fix function arguments for recursive call
 2021-01-19 23:50:07 +01:00
Mike Fährmann
89a2bcbb2d
[furaffinity] add 'descriptions' option (#1231) 2021-01-19 19:09:29 +01:00
Mike Fährmann
36f281330a
[newgrounds] fix flash file extraction (closes #1257) 
… and add a 'flash' option to choose between flash and video formats.
 2021-01-19 17:48:14 +01:00
Mike Fährmann
534194bf92
[unsplash] add extractors (#1197) 
for
- single photos  (/photos/ID)
- user profiles  (/@USER)
- user likes     (/@USER/likes)
- search results (/s/photos/SEARCH)
 2021-01-19 02:23:39 +01:00
Mike Fährmann
1fc16cb8c5
[instagram] fix regex for '/saved' URLs (fixes #1251) 
The URL pattern erroneously had two '([^/?#]+)' capture groups,
which would split any username into 'usernam' for the first group
and 'e' for the ignored second group.
 2021-01-17 23:34:45 +01:00
Mike Fährmann
c008cb5100
[pixiv] add 'related' option (#1237) 2021-01-17 22:48:32 +01:00
Mike Fährmann
e9a75e27d9
[foolfuuka] stop search when results are exhausted (#1174) 2021-01-17 22:48:21 +01:00
Mike Fährmann
b0cf968115
[mangadex] update API URLs 
https://mangadex.org/thread/351011/9/#post_4238014
 2021-01-15 21:11:58 +01:00
Mike Fährmann
a6414c31d6
[kemonoparty] simplify (#1216) 
Use metadata from API responses as is and
don't try to detect duplicated by their original filename.
 2021-01-15 01:13:59 +01:00
Mike Fährmann
01b9ccd4de
[derpibooru] use the "Everything" filter by default (#1243) 
when neither 'api-key' nor 'filter' are set
 2021-01-14 18:29:09 +01:00
Mike Fährmann
1fdecfa269
[kemonoparty] use API endpoints (#1216) 2021-01-14 03:23:21 +01:00
Mike Fährmann
318876e4dd
[nozomi] add 'num' enumeration index (closes #1239) 2021-01-12 22:32:52 +01:00
Mike Fährmann
2da9068ea8
[sankaku] simplify login process 2021-01-12 00:15:22 +01:00
Mike Fährmann
e07dfc4fe5
[kemonoparty] add 'user' and 'post' extractors (#1216) 2021-01-11 22:17:08 +01:00
Mike Fährmann
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()' 
and add a 'size' argument
 2021-01-11 22:12:40 +01:00
Mike Fährmann
32fcc61b84
release version 1.16.2 2021-01-09 18:24:51 +01:00
Mike Fährmann
02bc59d75c
[hentainexus] fix extraction (fixes #1234) 
hentainexus is now hosting its images on wordpress, or at least it is
using wordpress' servers as cache:
https://i2.wp.com/images.hentainexus.com/gallery/2199754b23c191deb330c99c9bb43341/9576/002.png?filter=null
 2021-01-09 15:12:05 +01:00
Mike Fährmann
20bd9cd296
[wikiart] add extractor for single paintings (closes #1233) 
There is no API endpoint for single paintings from what I can tell,
so this uses the site's search.
 2021-01-08 23:19:00 +01:00
Mike Fährmann
e2d4ca4955
[deviantart] improve '--range' for favorites (closes #1226) 2021-01-08 22:57:35 +01:00
Mike Fährmann
56ccb9951a
[gfycat] add 'date' metadata field (#1138) 2021-01-08 17:45:09 +01:00
Mike Fährmann
f2b83b8578
[gfycat] convert IDs to lowercase 
Redgifs expects all IDs and names to be lowercase
and throws a 404 if an ID contains an uppercase letter.

Gfycat on the other hand doesn't care about case,
so it's fine to just convert all IDs.

(#1138)
 2021-01-08 17:41:45 +01:00
Mike Fährmann
b3bc646236
[redgifs] match embedded URLs 
https://redgifs.com/ifr/<ID>
 2021-01-08 16:01:01 +01:00
Mike Fährmann
98e0d21383
[instagram] categorize single highlight URLs as 'highlights' 
They were categorized as 'stories' before.

(fixes #1222)
 2021-01-08 15:56:27 +01:00
Mike Fährmann
fa8ee6eac4
[derpibooru] add search and gallery extractors (#862) 2021-01-07 18:05:32 +01:00
Mike Fährmann
3759d0cb42
[redgifs] fix search results 
The metadata for Redgifs search results got stripped down to a bare
minimum, including download URLs. (Clicking on search results on the
website itself is broken as well)

As a workaround, we make an extra call to '/v1/gfycats/<ID>'
for each search result entry to fetch the missing data.
 2021-01-06 18:16:06 +01:00
Mike Fährmann
8a88025dc4
[pinterest] support generic user URLs (#1205) 
i.e. https://www.pinterest.com/USERNAME

also renames 'BoardsExtractor' to 'UserExtractor'
 2021-01-02 02:36:53 +01:00
Mike Fährmann
56b460dcea
[foolfuuka] add 'search' extractors (#1174) 2021-01-02 02:34:06 +01:00
Mike Fährmann
fb64183d53
[foolfuuka] add 'board' extractors (closes #1044) 2021-01-01 19:33:35 +01:00
Mike Fährmann
b0beed7a06
[sankaku] add support for book searches (closes #1204) 2020-12-29 17:36:37 +01:00
Mike Fährmann
6cdbab07b5
[pinterest] add support for getting all boards of a user 
(#1205)
 2020-12-29 16:57:03 +01:00
Mike Fährmann
25074aec47
[twitter] fetch media from pinned tweets (#1203) 2020-12-29 16:27:43 +01:00
Mike Fährmann
2475176d99
[twitter] fetch tweets from 'homeConversation' entries 
When logged in, some entries returned by Twitter's API are so called
'homeConversation's (they would be regular tweet entries otherwise.)

Those weren't picked up before and resulted in missing files compared
to accessing a timeline as guest.

('/media' timelines and search results were not affected)
 2020-12-29 00:42:46 +01:00
Mike Fährmann
3af9350648
[twitter] update API calls 
- use 'https://twitter.com/i/api' for all requests
  except '/guest/activate.json'
- update (default) URL parameters
- update GraphQL endpoints
 2020-12-28 22:05:48 +01:00
Mike Fährmann
b656b829db
[twitter] fix login with username & password 
It is no longer possible to get an 'authenticity_token' from Twitter's
Javascript-free login form, which got disabled few days ago.

Generating a random 16 byte hex string client-side and sending that as
a cookie alongside the regular login form works just as well.
 2020-12-28 16:10:19 +01:00
Mike Fährmann
912eea29bc
update extractor test results 2020-12-27 17:41:08 +01:00
Mike Fährmann
47a7a51944
[sankaku] fix 'invalid_token' detection 2020-12-27 02:31:01 +01:00
Mike Fährmann
ba5df84f7e
[keenspot] improve redirect handling 
Before it would use http:// for all requests and
get a redirect to a https:// version if those are supported.

Now the redirect only happens once during the first request.
 2020-12-26 21:38:40 +01:00
Mike Fährmann
d781e6ac44
[e621] return pool posts in order (closes #1195) 
… and add a 'num' enumeration index.

A bit more code than the PR version, but it prints some helpful messages
and doesn't call 'metadata()' twice.
 2020-12-26 19:00:29 +01:00
Mike Fährmann
e7d446a8f7
[danbooru] slight code refactoring 2020-12-25 22:06:25 +01:00
Mike Fährmann
e41e2be2f9
[booru] split '_prepare_post()' 2020-12-24 01:13:54 +01:00
Mike Fährmann
53222445d5
[hentaicafe] simplify default filenames 2020-12-23 01:03:08 +01:00
Mike Fährmann
712c792fbe
[hentaicafe] prefer title of /hc.fyi/ pages (closes #1106) 2020-12-23 01:01:15 +01:00
Mike Fährmann
2c4d4a75db
[mangadex] respect 'chapter-reverse' settings (closes #1194) 
The extractor in question doesn't inherit from MangaExtractor
and therefore didn't do this automatically.
 2020-12-22 15:08:10 +01:00
Mike Fährmann
3bd08acc8f
[pixiv] output debug message on failed login attempt 
(#1192)
 2020-12-22 14:59:31 +01:00
Mike Fährmann
b58e605dc7
raise error when required username or password are missing 
do not try to login as 'None' (#1192)
 2020-12-22 14:40:18 +01:00
Mike Fährmann
b233531aaa
[sankaku] use '/posts' endpoint for single posts 2020-12-22 02:44:40 +01:00
Mike Fährmann
459a0af4f8
[sankaku] add support for sankaku.app URLs (closes #1193) 2020-12-22 01:57:53 +01:00
Mike Fährmann
371e9ca6df
[pinterest] implement video support (closes #1189) 2020-12-21 16:09:06 +01:00
Mike Fährmann
537742c0ee
[sankaku] normalize 'created_at' metadata (closes #1190) 2020-12-21 02:06:29 +01:00
Mike Fährmann
ae6748996a
[pornhub] update tests 2020-12-21 02:06:28 +01:00
Mike Fährmann
bf629a2818
[instagram] add 'include' option (closes #1180) 
Split the functionality of the old 'user' extractor into separate
'posts' and 'highlights' extractors, which respond to virtual URLs
('/<user>/posts' and '/<user>/highlights')
 2020-12-21 02:06:28 +01:00
Mike Fährmann
78061658ea
[booru] reduce exceptions caught during _prepare_post() 
don't catch HttpErrors etc.
 2020-12-21 02:05:59 +01:00
Mike Fährmann
212ae0c399
[mangapanda] remove module 
site now redirects to mangareader.net
 2020-12-20 17:42:15 +01:00
Mike Fährmann
337b118e25
[instagram] warn about private profiles (#1187) 2020-12-19 22:32:28 +01:00
Mike Fährmann
465015f75a
[sankaku] reimplement login support (#1176, #1182) 2020-12-17 16:12:59 +01:00
Mike Fährmann
8d2e4e5f13
[booru] improve error handling 
e.g. for posts without a valid 'file_url' (#1176)
 2020-12-17 01:16:45 +01:00
Mike Fährmann
1d753542c2
[hentainexus] fix extraction (fixes #1166) 2020-12-12 20:30:51 +01:00
Mike Fährmann
a00b60fbe7
[twitter] update 'x-csrf-token' header (fixes #1170) 
Twitter started using a bigger (80 instead of 16 bytes) CSRf token for
logged in users, and expects those to be used as 'x-csrf-token' header
when send via 'ct0' cookie.

Generating an 80 byte token ourselves doesn't work, and Twitter will
still insist on using its own.
 2020-12-11 13:46:58 +01:00
Mike Fährmann
b88c97b873
[instagram] add 'cursor' option (#1149) 
To enable at least 'some' way to continue downloading from the middle
of a user profile listing.
 2020-12-11 13:46:58 +01:00
Mike Fährmann
0d406c8daf
[common] restrict values used in 'generate_extractors()' 2020-12-11 13:46:47 +01:00
Mike Fährmann
b2c55f0a72
[sankaku] remove login support 
The old login method for 'https://chan.sankakucomplex.com/user/login'
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
 2020-12-08 21:05:47 +01:00
Mike Fährmann
7f3d811d7b
[moebooru] inherit from BooruExtractor 2020-12-08 18:34:56 +01:00
Mike Fährmann
a3a863fc13
[booru] add generalized extractors for *booru sites 
similar to cc15fbe7
 2020-12-08 18:34:30 +01:00
Mike Fährmann
5f23441e12
[piczel] update API URLs 2020-12-07 15:56:32 +01:00
Mike Fährmann
47114339a2
[webtoons] update 'ageGate' cookie 2020-12-07 14:56:32 +01:00
Mike Fährmann
4225f12783
[nozomi] handle empty 'date' fields (fixes #1163) 2020-12-07 00:08:53 +01:00
Mike Fährmann
2b93515ee0
[instagram] reimplement support for stories (#1149) 2020-12-06 21:32:10 +01:00
Mike Fährmann
ecdea799dd
[sankaku] use 'beta.sankakucomplex.com' API endpoints 2020-12-05 22:08:58 +01:00
Mike Fährmann
b3ecc89a9a
[instagram] use double quotes for strings when possible 2020-12-05 19:33:42 +01:00
Mike Fährmann
76285eb60d
[instagram] reimplement support for story highlights (#1149) 2020-12-05 19:13:00 +01:00
Mike Fährmann
8ca7f54750
rename '_request_…' variables 
- remove '_' at the beginning
- _request_last -> request_timestamp
 2020-12-05 00:09:15 +01:00
Mike Fährmann
15a122aff3
[instagram] update 'X-IG-WWW-Claim' headers 2020-12-04 20:58:34 +01:00
Mike Fährmann
e5d81bdc7b
[mangadex] handle 'external' chapters (closes #1154) 2020-12-04 20:56:30 +01:00
Mike Fährmann
447488fb18
[instagram] rewrite 
(#1113, #1122, #1128, #1130, #1149)

Rely on the results of GraphQL queries instead of requesting data
for each post separately via '/p/<shortcode>/?__a=1'.

This might result in some missing metadata, and there might be some
issues for '/channel/' and '/saved/' URLs, but at least downloading
from the regular post listings should work without issues and without
getting users blocked/banned.

TODO: reimplement support for stories
 2020-12-03 14:30:59 +01:00
Mike Fährmann
cc15fbe71a
[moebooru] add generalized extractors for moebooru sites 
- add support for sakugabooru.com (closes #1136)
- add support for lolibooru.moe   (closes #1050)

This allows users to dynamically add support for moebooru/myimouto
based sites by adding an entry to their config file
(like for foolslide, foolfuuka, etc)

For example:
{
    "extractor": {
        "moebooru": {
            "new-site-1": {"root": "https://site1.net"},
            "new-site-2": {"root": "https://www.site2.moe"}
        }
    }
}
 2020-12-01 22:27:18 +01:00
Mike Fährmann
43120407cc
[paheal] create directory for each post (closes #1147) 2020-12-01 12:14:55 +01:00
Mike Fährmann
63e61a0932
[twitter] update image URL format (#1145) 
use
'/<name>?format=<fmt>&name=<size>'
instead of the potentially deprecated
'/<name>.<fmt>:<size>'

but keep all of them as fallback URLs
 2020-12-01 11:53:51 +01:00
Mike Fährmann
ae6a1d5fbc
[mangoxo] fix extraction 2 2020-11-27 13:55:30 +01:00
Mike Fährmann
f6a684bc37
[hentainexus] update data decoding procedure (#1125) 2020-11-25 11:26:26 +01:00
Mike Fährmann
c57a918f4a
[e621] implement delay via '_request_interval_min' 2020-11-25 00:19:32 +01:00
Mike Fährmann
93ce7466e2
[2chan] skip external links 2020-11-24 16:41:47 +01:00
Mike Fährmann
b214e89b5c
[mangoxo] fix extraction 2020-11-24 12:50:46 +01:00
Mike Fährmann
578dcf805c
[mangapanda] don't force https:// 2020-11-21 20:24:37 +01:00
Mike Fährmann
102c482f5e
[reddit] skip invalid/failed gallery items (fixes #1127) 2020-11-21 17:34:38 +01:00
Mike Fährmann
174945d2b2
[hentainexus] fix extraction (fixes #1125) 2020-11-20 22:31:35 +01:00