Mike Fährmann
0bc492c0fa
add docs for 'event' and 'filename' options
...
from 9c3568c3ca59bd69
2020-11-25 12:12:41 +01:00
Mike Fährmann
f6a684bc37
[hentainexus] update data decoding procedure ( #1125 )
2020-11-25 11:26:26 +01:00
Mike Fährmann
c57a918f4a
[e621] implement delay via '_request_interval_min'
2020-11-25 00:19:32 +01:00
Mike Fährmann
93ce7466e2
[2chan] skip external links
2020-11-24 16:41:47 +01:00
Mike Fährmann
547107307e
fix 'Metadata' messages in result tests
2020-11-24 13:34:54 +01:00
Mike Fährmann
b214e89b5c
[mangoxo] fix extraction
2020-11-24 12:50:46 +01:00
Mike Fährmann
578dcf805c
[mangapanda] don't force https://
2020-11-21 20:24:37 +01:00
Mike Fährmann
102c482f5e
[reddit] skip invalid/failed gallery items ( fixes #1127 )
2020-11-21 17:34:38 +01:00
Mike Fährmann
174945d2b2
[hentainexus] fix extraction ( fixes #1125 )
2020-11-20 22:31:35 +01:00
Mike Fährmann
ca59bd691c
[postprocessor:metadata] add 'event' and 'filename' options
2020-11-20 22:29:11 +01:00
Mike Fährmann
9c3568c397
[postprocessor:exec] add 'event' option
...
and remove 'final' option -- use '"event": "finalize"' instead.
2020-11-19 02:30:48 +01:00
Mike Fährmann
9fffa9c343
rework post processor callbacks
2020-11-19 02:29:06 +01:00
Mike Fährmann
d6986be8b0
Move CI to GitHub Actions
2020-11-17 19:40:45 +01:00
Mike Fährmann
f99c6031e0
apply post processor blacklists/whitelists to basecategories
...
(#1103 )
2020-11-17 02:02:31 +01:00
Mike Fährmann
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
2020-11-17 00:34:07 +01:00
Mike Fährmann
ddfb4fd07a
[twitter] use ' https://twitter.com/i/api/ ' for logged in users
...
Doesn't seem to make a difference from what I can tell,
i.e. downloaded files are the same, but the website does it.
2020-11-16 11:26:37 +01:00
Mike Fährmann
42ccae53c4
[mangadex] switch to API v2
...
https://mangadex.org/api/v2/
https://mangadex.org/thread/351011
2020-11-16 11:05:17 +01:00
Mike Fährmann
ca44111726
[flickr] update
...
- ensure every photo has an 'owner' (#828 )
- change default directories to a more consistent schema
- create directory for each photo
2020-11-15 10:44:29 +01:00
Mike Fährmann
9b1bd09454
change 'extension-map' default
...
Replace all JPEG filename extensions with 'jpg'.
2020-11-14 22:40:31 +01:00
Mike Fährmann
e5438b8a29
release version 1.15.3
2020-11-13 15:50:05 +01:00
Mike Fährmann
de0c57886d
[twitter] add 'list-members' extractor ( closes #1096 )
2020-11-13 06:47:45 +01:00
Mike Fährmann
904ba08568
[gfycat] fix default filename format
2020-11-13 06:37:21 +01:00
Mike Fährmann
a46561bc16
[500px] update query hashes
2020-11-13 06:36:11 +01:00
Mike Fährmann
2e3a0dff21
[8kun] fix file URLs of older posts ( fixes #1101 )
2020-11-07 23:10:37 +01:00
Mike Fährmann
00825cddf5
[hentaifoundry] use scheme from input URL ( fixes #1095 )
...
Let the user choose between http and https,
instead of always forcing https.
2020-11-07 22:40:02 +01:00
Mike Fährmann
8a98d3549a
[weasyl] create directory for each favorite submission
...
(#1032 )
2020-11-07 18:47:55 +01:00
Mike Fährmann
91db8df1c7
[deviantart] add 'index_base36' metadata field ( closes #1099 )
...
This is the same ID as found in 'filename' without the 'd' in front,
which is just 'index' encoded in base36.
2020-11-07 18:39:50 +01:00
Mike Fährmann
b9bfa4c675
update extractor test results
2020-11-07 02:03:22 +01:00
Mike Fährmann
1b5b789401
[mangoxo] fix metadata extraction
2020-11-07 01:35:29 +01:00
Mike Fährmann
41d4968866
[twitter] add 'list' extractor ( #1096 )
2020-11-05 22:55:38 +01:00
Mike Fährmann
5d10520f4c
[twitter] update GraphQL endpoint & fix width/height entries
2020-11-05 22:53:29 +01:00
Mike Fährmann
bc7b1d91bc
fix rST markup in configuration.rst
...
[ci skip]
2020-11-02 15:32:29 +01:00
Mike Fährmann
9b2e5f72d6
[exhentai] update image URL parsing ( #1094 )
2020-11-02 15:28:54 +01:00
Mike Fährmann
e3480bc8de
implement 'extension-map' option ( #318 )
2020-11-02 15:27:07 +01:00
Mike Fährmann
98a4d86a01
[sankakucomplex] extract videos and embeds ( closes #308 )
2020-10-30 01:21:11 +01:00
Mike Fährmann
c3f01dc4e6
implement 'util.unique()'
2020-10-29 23:33:41 +01:00
Mike Fährmann
558cde139c
[paheal] fix extraction ( fixes #1088 )
2020-10-28 21:51:31 +01:00
Mike Fährmann
0211af7ca8
[hentaifoundry] update 'YII_CSRF_TOKEN' cookie handling
...
(fixes #1083 )
2020-10-28 21:49:03 +01:00
Mike Fährmann
d83b95fd28
[postprocessor:metadata] accept a string-list for 'content-format'
...
(closes #1080 )
2020-10-27 20:09:58 +01:00
Mike Fährmann
198c33ec36
also collect post processors from 'basecategory' entries
...
(fixes #1084 )
2020-10-27 19:56:48 +01:00
Mike Fährmann
350b1afe1c
speed up _list_classes() after iterating over all modules once
2020-10-26 22:18:15 +01:00
Mike Fährmann
5bcf28de93
add a 'extractor.modules' option
2020-10-25 03:05:10 +01:00
Mike Fährmann
18213dc5ba
release version 1.15.2
2020-10-24 18:57:29 +02:00
Mike Fährmann
de4a1e45c9
improve 'generate_csrf_token()'
...
no need to use hashlib.md5()
2020-10-24 02:56:40 +02:00
Mike Fährmann
b788712844
[fallenangels] fix extraction of '.5' chapters
2020-10-23 16:56:08 +02:00
Mike Fährmann
28d8541cb3
[mangafox] ensure download URLs have a scheme
2020-10-23 02:45:15 +02:00
Mike Fährmann
8e3a324c91
[mangakakalot] ignore "Go Home" buttons in chapter pages
2020-10-23 02:33:35 +02:00
Mike Fährmann
c14c5d82d6
[newgrounds] use generator for fallback URLs
2020-10-23 00:39:19 +02:00
Mike Fährmann
a09f42f6b3
improve filename_from_url() performance
...
Manually extracting the part between the last '/' and '?' instead of
relying on the standard libraries' 'urllib.parse.urlsplit()' increases
performance by ~400%.
urlsplit() : 3.64 secs per 1.000.000 iterations
partition(): 0.87 secs per 1.000.000 iterations
2020-10-23 00:14:06 +02:00
Mike Fährmann
968d3e8465
remove '&' from URL patterns
...
'/?&#' -> '/?#' and '?&#' -> '?#'
According to https://www.ietf.org/rfc/rfc3986.txt , URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
2020-10-22 23:31:25 +02:00
