Mike Fährmann
645b4627ef
[sankaku] update URL patterns
2023-11-24 02:41:52 +01:00
Mike Fährmann
c9a2be36d4
[sankaku] support '/posts/' tag search URLs ( #4740 )
2023-10-29 13:48:42 +01:00
Mike Fährmann
b52fd91ac6
[sankaku] support '/posts/' URLs ( #4688 )
2023-10-21 13:20:35 +02:00
Mike Fährmann
3ecb512722
send Referer headers by default
2023-09-19 00:02:04 +02:00
Mike Fährmann
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
2023-09-11 16:30:55 +02:00
Mike Fährmann
a60db454af
[sankaku] update/fix API headers
...
'Referer' and 'Origin' were both empty
2023-08-04 17:14:43 +02:00
Mike Fährmann
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
2023-07-22 01:20:50 +02:00
Mike Fährmann
3845c0256d
[sankaku] improve warnings for unavailable posts
2023-07-01 19:11:41 +02:00
Mike Fährmann
7f25cab56e
[sankaku] support post URLs with MD5 hashes ( #3952 )
2023-04-23 16:46:40 +02:00
Mike Fährmann
faca32a850
[sankaku] sanitize 'date:…' tags ( #1790 )
2023-04-19 20:09:11 +02:00
Mike Fährmann
107c60c973
[sankaku] update URL pattern ( #3523 )
...
match tag searches with language codes without a trailing slash
2023-01-18 21:38:01 +01:00
Mike Fährmann
b0cb4a1b9c
replace 'text.extract()' with 'text.extr()' where possible
2022-11-05 01:14:09 +01:00
Mike Fährmann
775895f44b
[booru] refactor 'tags' and 'notes' extraction
...
- move HTML request for post pages into its own function
- move gelbooru_v02.py notes extraction to gelbooru.py
since it only works there
- clean up some code
2022-10-31 12:01:19 +01:00
Mike Fährmann
5fd4374036
[sankaku] improve 429 and tag limit handling
2022-10-01 11:49:47 +02:00
Mike Fährmann
4089bceddd
[sankaku] implement 'refresh' option ( #2958 )
2022-09-30 19:55:48 +02:00
Mike Fährmann
850608551c
[sankaku] detect expired links ( #2958 )
2022-09-23 11:51:30 +02:00
Mike Fährmann
32c75d12e8
[sankaku] rewrite URLs to s.sankakucomplex.com ( #2746 )
2022-07-11 12:46:04 +02:00
Mike Fährmann
05d4a0215a
[sankaku] extend URL patterns ( fixes #2647 )
...
- support URLs with ISO 639-1 language codes
- support black.… and white.… subdomains
2022-06-01 21:31:11 +02:00
Mike Fährmann
211de95dd0
update extractor test results
2021-11-01 02:58:53 +01:00
Mike Fährmann
9ed13703cc
[sankaku] handle empty tags ( fixes #1617 )
2021-06-14 16:20:10 +02:00
Mike Fährmann
c5ca7905ce
add 'noop()' and 'identity()' functions
2021-05-04 19:27:17 +02:00
Mike Fährmann
6fa20d456b
[sankaku] update invalid-token detection ( fixes #1515 )
2021-04-30 22:04:45 +02:00
Mike Fährmann
bdfcc9c4b1
update extractor test results
2021-04-18 20:28:15 +02:00
Mike Fährmann
0e601de67b
[sankaku] simplify 'pool' tags ( #1388 )
...
normalize 'tags' and 'artist_tags' to a string-list
2021-03-23 18:45:45 +01:00
Mike Fährmann
d085ade9d5
[sankaku] add 'tag_string' metadata field ( #1388 )
...
The 'join()'ed version of 'tags'.
Handling lists in format strings isn't properly supported yet.
2021-03-23 15:42:13 +01:00
Mike Fährmann
2dffd231b7
[sankaku] add enumeration index for books ( #1388 )
2021-03-23 15:32:54 +01:00
Mike Fährmann
96a51ff169
[sankaku] update invalid-token detection ( fixes #1309 )
2021-02-11 19:49:24 +01:00
Mike Fährmann
2da9068ea8
[sankaku] simplify login process
2021-01-12 00:15:22 +01:00
Mike Fährmann
b0beed7a06
[sankaku] add support for book searches ( closes #1204 )
2020-12-29 17:36:37 +01:00
Mike Fährmann
47a7a51944
[sankaku] fix 'invalid_token' detection
2020-12-27 02:31:01 +01:00
Mike Fährmann
e41e2be2f9
[booru] split '_prepare_post()'
2020-12-24 01:13:54 +01:00
Mike Fährmann
b233531aaa
[sankaku] use '/posts' endpoint for single posts
2020-12-22 02:44:40 +01:00
Mike Fährmann
459a0af4f8
[sankaku] add support for sankaku.app URLs ( closes #1193 )
2020-12-22 01:57:53 +01:00
Mike Fährmann
537742c0ee
[sankaku] normalize 'created_at' metadata ( closes #1190 )
2020-12-21 02:06:29 +01:00
Mike Fährmann
465015f75a
[sankaku] reimplement login support ( #1176 , #1182 )
2020-12-17 16:12:59 +01:00
Mike Fährmann
8d2e4e5f13
[booru] improve error handling
...
e.g. for posts without a valid 'file_url' (#1176 )
2020-12-17 01:16:45 +01:00
Mike Fährmann
b2c55f0a72
[sankaku] remove login support
...
The old login method for 'https://chan.sankakucomplex.com/user/login '
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
2020-12-08 21:05:47 +01:00
Mike Fährmann
ecdea799dd
[sankaku] use 'beta.sankakucomplex.com' API endpoints
2020-12-05 22:08:58 +01:00
Mike Fährmann
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
2020-11-17 00:34:07 +01:00
Mike Fährmann
844793847c
update extractor test results
2020-10-11 18:15:41 +02:00
Mike Fährmann
4409d00141
embed error messages in StopExtraction exceptions
2019-10-28 16:39:49 +01:00
Mike Fährmann
7a5e78741c
[booru] build directory path for each file ( #385 )
2019-08-18 23:28:33 +02:00
Mike Fährmann
40637556fa
[ngomik] fix extraction
2019-07-28 10:53:46 +02:00
Mike Fährmann
7a99e85943
[kissmanga] fix download URLs and file extensions
...
The current Blogspot image URLs hosted on Kissmanga end with an
"invalid" query parameter (/000.png&upx=...), which doesn't get
recognized by 'spliturl()' and 'parseurl()' as such and gets therefore
included in the 'extension' field from 'text.nameext_from_url()'.
2019-06-28 20:34:43 +02:00
Mike Fährmann
74c2415138
[sankakucomplex] move article extractor to its own module ( #258 )
2019-05-27 23:49:23 +02:00
Mike Fährmann
1e3e15c4f3
[sankaku] add article extractor ( #258 )
2019-05-26 17:42:36 +02:00
Mike Fährmann
efa805c5d7
[sankaku] update pagination end condition ( fixes #265 )
...
Pagination over popular listings (`date:...+order:popular") never
terminates, not even on the site itself, and at some point returns the
same results over and over again.
2019-05-20 15:46:06 +02:00
Mike Fährmann
0b4be57a10
[sankaku] fix error when no tags available ( closes #259 )
...
[ci skip]
2019-05-14 23:40:07 +02:00
Mike Fährmann
aa8e366b90
[luscious] fix tag extraction
2019-05-14 17:35:52 +02:00
Mike Fährmann
a2af2d2965
adjust cache maxage values
2019-03-14 22:21:49 +01:00
