enduser420
039d06c8f6
[mangaread] add 'chapter' and 'manga' extractors
2022-11-13 16:00:34 +05:30
enduser420
93ea8ca8e3
[imxto] extract additional metadata ( #3175 )
2022-11-11 08:53:07 +01:00
Mike Fährmann
e3abab8629
[weibo] send 'Referer' headers ( #3188 )
2022-11-10 17:11:57 +01:00
Mike Fährmann
6423f990de
[realbooru] fix 'tags' extraction ( #2530 )
2022-11-10 17:04:02 +01:00
Mike Fährmann
ecad02cf3f
[realbooru] fix download URLs ( #2530 )
2022-11-10 13:29:35 +01:00
Mike Fährmann
a4ff20cf16
[downloader:http] fix issues from inaccurate 'time.sleep()'
...
(#3143 )
Reverts part of c59b98c8
2022-11-10 13:24:02 +01:00
Mike Fährmann
8d543b040d
update docs
...
- document '"mode": "jsonl"' for metadata pp
- remove 'username' and 'password' from 'instagram'
- add 'ytdl' example (#3180 )
2022-11-09 22:17:08 +01:00
Mike Fährmann
15cd114c9c
[twitter] update bookmarks pagination ( #3172 )
...
Do not stop when there aren't any tweets in a batch,
but only when the same cursor value appears twice in a row.
2022-11-09 20:40:51 +01:00
Mike Fährmann
550f90ab56
delay enabling .part files when 'http-metadata' is set
...
otherwise 'build_path' gets called before all metadata is collected
2022-11-09 13:23:52 +01:00
Mike Fährmann
20fbba9d7c
[exhentai] add metadata to search results ( #3181 )
...
'gallery_id' and 'gallery_token'
2022-11-09 11:40:27 +01:00
Mike Fährmann
6a0c5e34f4
[exhentai] fix pagination ( #3181 )
2022-11-09 11:15:34 +01:00
Mike Fährmann
05255f5be0
add 'default' argument to 'text.extr()'
2022-11-09 11:00:32 +01:00
Mike Fährmann
e326029355
build path when skipping archived files
...
fixes bug from 8124c16a
2022-11-09 10:33:09 +01:00
Mike Fährmann
171262c1b6
[instagram] remove login support
...
broken feature that I cannot get to work anymore
2022-11-08 21:48:31 +01:00
Mike Fährmann
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2022-11-08 17:03:24 +01:00
Mike Fährmann
39d9c362e4
include 'http-metadata' in '-K' output
2022-11-07 16:33:26 +01:00
Mike Fährmann
e2401c96ee
[postprocessor:metadata] add '"mode": "jsonl"'
2022-11-07 15:46:35 +01:00
Mike Fährmann
895f36e53b
[postprocessor:metadata] add 'open' and 'encoding' options
2022-11-07 15:37:22 +01:00
Mike Fährmann
93e6bd6847
[uploadir] use utf-8 filenames ( #3162 )
2022-11-05 19:03:48 +01:00
Mike Fährmann
870e6a48a0
implement 'http-metadata' option
...
or at least attempt to.
2022-11-05 18:29:29 +01:00
Mike Fährmann
b7a83ac726
[uploadir] update ( #3162 )
...
- prevent extra HTTP request from redirects
- add 'id' metadata field
- set 'filename_fmt' and 'archive_fmt'
2022-11-05 18:28:35 +01:00
Mike Fährmann
ccb80f1b8b
[uploadir] add support for 'uploadir.com' ( #3162 )
2022-11-05 14:25:09 +01:00
Mike Fährmann
b0cb4a1b9c
replace 'text.extract()' with 'text.extr()' where possible
2022-11-05 01:14:09 +01:00
Mike Fährmann
eb33e6cf2d
add 'text.extr()'
...
a stripped-down version of text.extract() that
- always returns a string (like 'extract_from')
- only returns a string
- does not deal with 'pos' arguments
- is ~20% faster
2022-11-04 21:37:36 +01:00
Mike Fährmann
597b63d922
move git head functionality to function in util.py
2022-11-04 19:49:39 +01:00
Mike Fährmann
4fd3c893fa
[booru] adjust/match '_tags' and '_notes' code
2022-11-04 19:49:39 +01:00
Mike Fährmann
88954aa2e4
[gelbooru_v02] implement 'notes' extraction
...
same code as for 'moebooru' works here as well
2022-11-04 19:49:39 +01:00
Mike Fährmann
942bc84962
add '--chunk-size' command-line option ( #3143 )
2022-11-04 19:49:39 +01:00
Mike Fährmann
79a9fc6e45
add '--user-agent' command-line option
2022-11-04 19:49:39 +01:00
Mike Fährmann
b357c2dcf8
build executables on GitHub Actions with Python 3.11
2022-11-04 19:49:39 +01:00
ClosedPort22
4e80d3210e
[tumblr] Fallback to gifv when possible ( #3095 ) ( #3159 )
2022-11-04 19:42:36 +01:00
thatfuckingbird
9d3f86dbcd
[twitter] update URL for syndication API ( #3160 )
...
Twitter changed the URL format to access tweet data through their syndication API.
2022-11-04 17:49:45 +01:00
enduser420
c01cad599a
[lolisafe] add support for xbunkr ( #3156 )
2022-11-04 16:13:52 +01:00
Allen
9fc142d27b
[mastodon] add "remote_instance" field ( #3119 )
...
Example Usage:
If the url is "mastodon:https://mastodon.example.org/@VoteChess@botsin.space the "remote_instance" will be "botsin.space"
...
"directory": ["mastodon", "{remote_instance|instance}", "{account[username]!l}"]
...
2022-11-02 17:09:38 +01:00
Mike Fährmann
bca9f965e5
[downloader:http] add 'chunk-size' option ( #3143 )
...
and double the previous default from 16384 (2**14) to 32768 (2**15)
2022-11-02 16:50:26 +01:00
Mike Fährmann
2a1cb403ee
Revert "[Deviantart] [ #1776 ] Remove the "you need session cookies to download mature scraps" warning ( #1777 )"
...
This reverts commit 1f02878351
2022-11-01 21:23:05 +01:00
Mike Fährmann
460095adca
update downloader tests
2022-11-01 18:48:35 +01:00
Mike Fährmann
0059e2bfe7
[downloader:http] add MIME type and signature for .avif files
2022-11-01 17:25:21 +01:00
Mike Fährmann
f687e64513
[downloader:http] refactor file signature checks
...
use functions/lambdas instead of startswith()
2022-11-01 17:09:13 +01:00
Mike Fährmann
86790da2d5
update Cloudflare IUAM detection
...
again
2022-10-31 18:33:52 +01:00
Mike Fährmann
c12a97bcde
[postprocessor] add 'post-after' event ( #3117 )
2022-10-31 14:35:48 +01:00
Mike Fährmann
775895f44b
[booru] refactor 'tags' and 'notes' extraction
...
- move HTML request for post pages into its own function
- move gelbooru_v02.py notes extraction to gelbooru.py
since it only works there
- clean up some code
2022-10-31 12:01:19 +01:00
Mike Fährmann
48bbe1ccf6
update CHANGELOG and README for 1.23.5
2022-10-30 18:42:12 +01:00
Luc Ritchie
0f9dfb7e62
[instagram] Fix AttributeError on user stories extraction ( #3123 )
2022-10-30 11:47:06 +01:00
Mike Fährmann
7a9cb6d7bf
update CHANGELOG and README for 1.23.4
2022-10-29 17:35:56 +02:00
Mike Fährmann
f81dd5297a
[skeb] fix extraction ( #3112 )
...
'completed_at' is no longer included in API responses
2022-10-29 15:07:21 +02:00
Mike Fährmann
b337e51e91
run flake8 on all .py files
2022-10-28 22:22:49 +02:00
enduser420
fb2dbb04e2
[moebooru] extract 'notes' ( #3094 )
2022-10-28 18:05:54 +02:00
Mike Fährmann
4e26bf98f5
[aibooru] support 'safe' subdomain ( #3110 )
2022-10-28 12:10:18 +02:00
Mike Fährmann
f037429fa4
attempt to improve '-K' output for lists
...
- use [N] instead if [] to indicate a Number needs to be placed there
- enumerate list items
2022-10-28 12:04:58 +02:00
