Mike Fährmann
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites ( #3734 )
...
add support for
- loudbooru.com (#3734 )
- booru.cavemanon.xyz (#3734 )
- giantessbooru.com (#943 )
- tentaclerape.net
2023-04-26 19:20:44 +02:00
Mike Fährmann
28419bf45a
[itchio] add 'game' extractor ( #3923 )
2023-04-26 19:20:43 +02:00
Mike Fährmann
3905f05f00
[postprocessor:metadata] support putting keys in quotes
...
for mode 'modify' and 'delete'
based on fe41a2b1
2023-04-25 14:30:18 +02:00
Mike Fährmann
7459e4abce
[postprocessor:metadata] fix traversing more than 1 level deep
...
for mode 'modify' and 'delete'
2023-04-25 14:17:25 +02:00
Mike Fährmann
5297ee0cd9
[tumblr] add 'day' extractor ( #3951 )
2023-04-24 22:01:47 +02:00
Mike Fährmann
de670bd7de
[tumblr] update pagination logic ( #2191 )
2023-04-24 20:07:10 +02:00
ClosedPort22
6f4a843fba
[downloader:http] release connection before logging messages
...
This allows connections to be properly released when using 'actions'
feature.
2023-04-24 23:59:36 +08:00
Mike Fährmann
98c9fdb414
[deviantart] revert e9353c63; retry downloads with private token
2023-04-23 21:10:16 +02:00
Mike Fährmann
5d7435e803
[nitter] extract user IDs from encoded banner URLs
...
still requires a banner to be present to begin with
2023-04-23 19:13:27 +02:00
Mike Fährmann
7f25cab56e
[sankaku] support post URLs with MD5 hashes ( #3952 )
2023-04-23 16:46:40 +02:00
Mike Fährmann
a05120412a
[oauth] catch exception from 'webbrowser.get()' ( #3947 )
...
It raises an exception instead of returning None
when no runnable browser is available.
2023-04-23 15:00:09 +02:00
Mike Fährmann
3fc2223893
merge #3935 : [reddit] match 'preview.redd.it' URLs
2023-04-21 20:09:20 +02:00
Mike Fährmann
1d505b39f8
[twitter] support 'profile-conversation' entries ( #3938 )
2023-04-21 15:08:50 +02:00
Mike Fährmann
aaf58a1259
[imgur] document 'client-id' option ( #3937 )
2023-04-21 15:08:50 +02:00
Mike Fährmann
202f5d86a7
[reddit] ignore 'id-max' value "zik0zj"/2147483647
...
(#3939 , #3862 , #3697 , #3606 , #3546 , #3521 , #3412 )
2023-04-21 15:08:50 +02:00
Mike Fährmann
8586ee81be
[nana] fix 'keyword' tests
2023-04-21 15:08:50 +02:00
ClosedPort22
cd4bfb0dd1
[reddit] match 'preview.redd.it' URLs
2023-04-20 15:54:09 +08:00
Mike Fährmann
faca32a850
[sankaku] sanitize 'date:…' tags ( #1790 )
2023-04-19 20:09:11 +02:00
Mike Fährmann
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors
...
(#731 , #2720 , #3812 )
2023-04-19 15:28:26 +02:00
Mike Fährmann
81bd2af83e
[2chen] update domain to sturdychan.help
2023-04-19 13:54:44 +02:00
Mike Fährmann
f500b45b5e
[twitter] improve 480bc34e
...
only check for double user assignment where necessary
2023-04-18 20:50:23 +02:00
Mike Fährmann
5b635f2317
[imxto] add 'gallery' extractor ( #1289 )
2023-04-17 20:49:09 +02:00
Mike Fährmann
359e31e462
[nozomi] update file URLs ( #3925 )
...
Static images are now only available in WebP format over the 'w'
subdomain. GIFs also got their own 'g' subdomain.
2023-04-17 15:42:42 +02:00
Mike Fährmann
2dfd4a3de2
[imagefap] extract 'categories' metadata and fix empty 'tags'
2023-04-17 14:49:50 +02:00
Mike Fährmann
480bc34e54
[twitter] do not overwrite previously assigned users ( #3922 )
2023-04-16 17:30:43 +02:00
Mike Fährmann
02ec5bb8e5
[imagefap] extract 'description' metadata ( #3905 )
2023-04-16 17:02:16 +02:00
Mike Fährmann
842f964c49
release version 1.25.2
2023-04-15 22:31:10 +02:00
Mike Fährmann
d253a3c542
merge #3841 : [urlshortener] add support for bit.ly & t.co
2023-04-15 18:08:21 +02:00
Mike Fährmann
5e63942b37
[urlshortener] update
2023-04-15 18:06:06 +02:00
Mike Fährmann
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
2023-04-15 17:09:22 +02:00
Mike Fährmann
c45f09d2a8
[imagechest] fix extraction ( #3914 )
2023-04-14 20:06:59 +02:00
Mike Fährmann
2cd4411ff8
[nitter] extract videos from 'source' elements ( #3912 )
2023-04-14 19:00:56 +02:00
Mike Fährmann
9501579279
[sexcom] fix fetching HD videos
2023-04-13 15:40:53 +02:00
Mike Fährmann
a2f7274eae
[sexcom] fix pagination ( #3906 )
2023-04-13 15:39:15 +02:00
Mike Fährmann
e9353c63d6
[deviantart] keep using private access tokens
...
for deviations returned from a private API call
also fixes a bug from 0a7eee3e
2023-04-13 14:46:06 +02:00
Mike Fährmann
e70af6a550
[hentaifoundry] do not update filters when cookies are provided
2023-04-13 14:16:53 +02:00
Mike Fährmann
9c29c904c7
[mastodon] try to get account IDs without access token
...
Try to query the public '/api/v1/accounts/lookup' endpoint
and fall back to '/v1/accounts/search' if it returns an error.
'/api/v1/accounts/lookup' is available since Mastodon v3.4.0.
The version of an instance can be found at '/api/v1/instance'.
2023-04-13 14:03:23 +02:00
Mike Fährmann
1614c5c4bf
[generic] write regular expressions without 'x' flags
2023-04-10 20:45:23 +02:00
Mike Fährmann
d84a617273
[hentaifoundry] fix setting content filters ( #3887 )
2023-04-09 18:04:49 +02:00
ClosedPort22
875485313f
[urlshortener] force HTTPS
2023-04-09 18:19:52 +08:00
Mike Fährmann
0a7eee3ee0
[deviantart] add 'public' option
2023-04-08 23:04:34 +02:00
Mike Fährmann
f5a59c4170
[twitter] add 'date_bookmarked' metadata ( #3816 )
2023-04-06 20:16:25 +02:00
Mike Fährmann
1c1f6fdc80
[twitter] fix regression from 160335ad
...
Tweets from 'homeConversation' or 'conversationthread' entries do not
contain a 'sortIndex' field. Accessing it raises a KeyError and would
erroneously get them labeled as 'deleted'.
2023-04-06 19:22:48 +02:00
Mike Fährmann
160335ad44
[twitter] add 'date_liked' metadata for liked Tweets ( #3816 )
2023-04-06 18:33:45 +02:00
Mike Fährmann
6d850ce629
[twitter] calculate 'date' from Tweet IDs
...
20 times faster than parsing 'created_at'
2023-04-05 22:29:14 +02:00
Mike Fährmann
25949bd767
merge #3871 : [hotleak] Fix downloading of creators whose name starts with a category name
2023-04-04 16:24:20 +02:00
Mike Fährmann
dbe06cdba1
[twitter] warn about 'withheld' Tweets and users ( #3864 )
2023-04-04 16:15:08 +02:00
Mike Fährmann
3cc1dd1572
[twitter] update query hashes
2023-04-03 23:20:20 +02:00
Mike Fährmann
3846ce0de5
[twitter] update to bookmark timeline v2 ( #3859 )
2023-04-03 22:46:12 +02:00
Mike Fährmann
34699fbf64
[deviantart:search] detect login redirects ( #3860 )
2023-04-03 19:37:12 +02:00
Mike Fährmann
e6cb92864a
[twitter] allow setting custom features per API endpoint
2023-04-03 16:18:31 +02:00
Balgden
4b141cce66
Fix indentation
2023-04-03 13:44:14 +00:00
Balgden
bbc5977121
Fix line length
2023-04-03 13:38:42 +00:00
Balgden
ffd30abcb3
[hotleak] Fix downloading of creators whose name starts with a category name
...
E.g. `hot4lexi` would start downloading the `hot` section by mistake
This happened because the regex had a negative lookahead for the category names, but didn't ensure that they where followed by either end-of-string or a slash.
2023-04-03 13:30:27 +00:00
Mike Fährmann
5ca9d55595
merge #3870 : [blogger] update 'sub' regex to get the highest resolution url
2023-04-03 14:47:18 +02:00
Mike Fährmann
fd7ce4c081
merge #3868 : [shopify] fix 'collection' extractor
2023-04-03 14:44:46 +02:00
Mike Fährmann
135ac9c302
merge #3854 : [twitter] fix: graphql_timeline_v2_bookmark_timeline cannot be null
2023-04-03 14:37:42 +02:00
enduser420
bbb1e34c34
[blogger] update sub regex
2023-04-03 12:43:58 +05:30
enduser420
96e3dd2128
[shopify] fix 'collection' extractor
2023-04-03 12:19:09 +05:30
Mike Fährmann
ac97aca99c
[realbooru] fix extraction
...
get file URLs from HTML pages
2023-04-02 20:45:16 +02:00
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata'
...
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
2023-04-02 20:11:52 +02:00
ClosedPort22
775d2ac999
[downloader:http] improve error logging when releasing connection
2023-03-31 20:08:38 +08:00
Amer Jazaerli
bebbff6578
fix: graphql_timeline_v2_bookmark_timeline cannot be null
...
twitter: 400 Bad Request (The following features cannot be null: graphql_timeline_v2_bookmark_timeline)
2023-03-31 00:06:49 +02:00
ClosedPort22
71b26adb9b
[urlshortener] add tinyurl.com as an example
2023-03-29 13:37:26 +08:00
Mike Fährmann
421db26aff
[bunkr] update domain to 'bunkr.la'
2023-03-28 20:10:36 +02:00
ClosedPort22
9e2a945013
[urlshortener] add support for bit.ly & t.co
2023-03-29 00:06:41 +08:00
Mike Fährmann
82f83c18e8
release version 1.25.1
2023-03-25 21:30:05 +01:00
Mike Fährmann
9b5e7ce8b9
[hiperdex] fix extraction
2023-03-25 18:18:27 +01:00
Mike Fährmann
89a67c45e0
[nitter] support nitter.it ( #3819 )
2023-03-25 13:29:22 +01:00
Mike Fährmann
88f29a751d
[nitter] skip broadcasts
...
instead of downloading an "Unsupported feature" HTML page
2023-03-25 13:09:24 +01:00
Mike Fährmann
1e013eba5a
[nitter] fix extraction for instances without user banners
2023-03-25 12:50:40 +01:00
Mike Fährmann
d94aa1ee02
[gelbooru] fix --range for favorites ( #3704 )
2023-03-23 22:58:13 +01:00
Mike Fährmann
1f82b00b8f
[gelbooru] fix and improve --range for pools
2023-03-23 18:22:46 +01:00
ClosedPort22
1a977f0f62
[downloader:http] handle exceptions in 'validate'
...
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
2023-03-23 19:57:13 +08:00
Mike Fährmann
197882cf12
[twitter] add 'hashtag' extractor ( #3783 )
2023-03-22 22:20:40 +01:00
Mike Fährmann
082d55de16
fix circular reference detection for -K
2023-03-21 23:46:36 +01:00
Mike Fährmann
2ab66ad899
update -K output to include quotes around keys
2023-03-21 22:28:04 +01:00
Mike Fährmann
fe41a2b159
[formatter] support putting keys in quotes
...
i.e. obj["key"] or obj['key']
as in f-strings
2023-03-21 22:06:54 +01:00
Mike Fährmann
46fdf46f21
[formatter] support loading an f-string from a template file
...
"\fTF ~/path/to/file.txt"
2023-03-20 22:05:33 +01:00
Mike Fährmann
1a4d4a799b
[formatter] support filesystem paths for \fM
2023-03-20 22:01:33 +01:00
Mike Fährmann
9789ebac52
[naverwebtoon] fix extraction ( #3729 )
2023-03-19 17:08:58 +01:00
Mike Fährmann
72f1f16eb2
[weibo] support 'mix_media_info' entries ( #3793 )
2023-03-18 15:19:25 +01:00
ClosedPort22
d4fb4ff47f
[twitter] extract TwitPic URLs in text ( #3792 )
...
also ignore previously seen URLs
2023-03-18 21:19:24 +08:00
Mike Fährmann
00f0233b28
[postprocessor:metadata] add 'skip' option ( #3786 )
2023-03-17 23:30:11 +01:00
Mike Fährmann
2bb937014f
[twitter] fall back to legacy /media endpoint when not logged in
2023-03-17 20:54:35 +01:00
Mike Fährmann
b68094d326
[twitter] support 'note_tweet's
2023-03-17 19:36:07 +01:00
Mike Fährmann
3dcabc97ed
[twitter] update API endpoints and parameters
2023-03-17 19:25:53 +01:00
Mike Fährmann
a1ca2404f9
add 'globals' instead of overwriting the default ( #3773 )
2023-03-16 18:37:00 +01:00
Mike Fährmann
dcb8af659a
[gelbooru] extract favorites without needing cookies ( #3704 )
...
TODO: fix --range
2023-03-15 19:21:35 +01:00
Mike Fährmann
b756dc13aa
[gelbooru] warn about missing cookies for favorites ( #3704 )
...
and add docstring so it shows up in --list-extractors
2023-03-15 14:58:55 +01:00
Mike Fährmann
17bd053d94
[hiperdex] fix extraction ( #3768 )
2023-03-15 14:28:03 +01:00
Mike Fährmann
f7ce33c85c
[output] set 'errors=replace' for output streams ( #3765 )
...
fixes regression from e480a933
2023-03-14 13:30:04 +01:00
Mike Fährmann
a14a2d6e59
release version 1.25.0
2023-03-11 21:05:28 +01:00
ClosedPort22
fcaeaf539c
[downloader:http] handle exceptions while consuming content
2023-03-11 21:36:37 +08:00
Mike Fährmann
4235d412c4
implement 'actions'
...
continuation of d37e7f48
2023-03-10 22:08:10 +01:00
Mike Fährmann
817fc0fbd1
[nitter] remove nitter.pussthecat.org
...
"Shutdown"
2023-03-09 23:48:39 +01:00
Mike Fährmann
67ec91cdbd
[downloader:http] change '_http_retry' to accept a Python function
...
and rename '_http_retry_codes' to '_http_retry'
(#3569 )
2023-03-09 23:30:15 +01:00
Mike Fährmann
175822e065
merge #3738 : [generic] add tests
2023-03-09 22:26:20 +01:00
Mike Fährmann
4883420e67
[generic] revert pattern change
2023-03-09 22:25:23 +01:00
ClosedPort22
df77271438
[downloader:http] add 'consume-content' option
...
* fix connection not being released when the response is neither
successful nor retried
* add the ability to consume the HTTP response body instead of closing
the connection
reference:
https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
2023-03-09 21:07:10 +08:00
Mike Fährmann
9037128315
[twitter] fix some 'original' retweets not downloading ( #3744 )
2023-03-08 18:33:19 +01:00
Mike Fährmann
ea3d95e7e8
merge #3740 : [deviantart] add support for fxdeviantart.com URLs
2023-03-08 17:30:06 +01:00
Mike Fährmann
9abcb2b6e5
update headers and ciphers for '"browser": "chrome"'
2023-03-08 17:19:59 +01:00
ClosedPort22
c489aecb3e
[deviantart] add support for fxdeviantart.com URLs
...
fxdeviantart.com is a service that fixes embeds on Discord, similar to
fxtwitter.com
2023-03-07 16:04:17 +08:00
ClosedPort22
34a7fab0e2
[generic] add support for IDNs
...
(internationalized domain name)
2023-03-06 22:42:36 +08:00
Mike Fährmann
c9a7345228
[newgrounds] prevent archive ID overlap ( #3681 )
...
add an 'i' and 'a' prefix to image and audio files
(/art/view/, /audio/listen/)
since their numeric ID may conflict with movies and other media
2023-03-06 15:03:49 +01:00
Mike Fährmann
8148c2a097
[downloader:ytdl] prevent exception on empty results
...
a7c7953107 (commitcomment-92042240)
2023-03-06 12:25:12 +01:00
Mike Fährmann
da9840a39d
[reddit] update 'videos' option ( #3712 )
...
- add 'dash' to directly extract DASH manifest URLs
(was default behavior since a7c79531a7c79531
2023-03-06 12:18:25 +01:00
Mike Fährmann
8f8b4de0e8
[ytdl] fix '--parse-metadata' ( #3663 )
2023-03-05 19:57:23 +01:00
Mike Fährmann
11df3a021d
[formatter] enclose f-strings with """ instead of '''
2023-03-03 20:03:27 +01:00
Mike Fährmann
baf41d7437
[misskey] update ( #3717 )
...
- add module docstring
- add options to docs/gallery-dl.conf
2023-03-03 15:52:50 +01:00
Mike Fährmann
7610d9cf82
merge #3675 : [pixiv] fix --write-tags for '"tags": "original"'
2023-03-02 21:48:31 +01:00
Mike Fährmann
6762d99515
merge #3717 : [misskey] add misskey extractors
2023-03-02 19:31:41 +01:00
Mike Fährmann
b8a702929d
[oauth] import extractor modules on demand
2023-03-02 18:39:54 +01:00
Mike Fährmann
dd88740ec7
replace remaining instances of base64 with binascii
2023-03-02 18:25:47 +01:00
enduser420
e1867cf5eb
[misskey] add 'renotes' and 'replies' options
2023-03-02 19:56:19 +05:30
enduser420
a95b5e0d8e
[misskey] add misskey extractors
2023-03-02 19:49:47 +05:30
Mike Fährmann
0d142e403c
[szurubooru] add 'tag' and 'post' extractors ( #3583 , #3713 )
2023-03-01 18:47:10 +01:00
Mike Fährmann
075c965512
add '--config-create' command-line option
...
(#2333 )
2023-03-01 14:49:40 +01:00
Mike Fährmann
26d06e0bb2
move executable check into util.py
2023-02-28 23:10:23 +01:00
Mike Fährmann
de2f35d068
simplify config.load()
2023-02-28 22:02:15 +01:00
Mike Fährmann
632d5d7745
allow loading config files in TOML format with --config-toml
2023-02-28 21:54:46 +01:00
Mike Fährmann
9e870eb930
rename --ignore-config to --config-ignore
...
--ignore-config still works as before,
but is no longer shown by --help
2023-02-28 21:07:42 +01:00
Mike Fährmann
d66257f2c8
improve option.Formatter performance
...
as always, only a very marginal difference,
but it still uses less resources than before
2023-02-28 20:57:28 +01:00
Mike Fährmann
d788e6c60c
implement 'globals' option
2023-02-28 18:18:55 +01:00
Mike Fährmann
b14f8d5817
[gelbooru] add 'favorite' extractor ( #3704 )
...
requires logged in cookies to work
2023-02-27 18:03:47 +01:00
Mike Fährmann
e480a93337
add 'output.stdout', '.stdin', and '.stderr' options
...
(#1621 , #2152 , #2529 )
Allow setting custom input/output encodings and options
without having to rely on Python's defaults.
2023-02-26 14:56:19 +01:00
Mike Fährmann
a70a3e5da6
[mangasee] extract 'author' and 'genre' metadata ( #3703 )
...
Both are lists/arrays. Use {author!S} or {genre:J, } to format them.
2023-02-26 14:12:19 +01:00
Mike Fährmann
6b03506655
[deviantart] allow searching when not logged in
2023-02-25 19:04:18 +01:00
Mike Fährmann
511a051705
[fanbox] fix crash with missing images ( #3673 )
2023-02-23 23:30:55 +01:00
Mike Fährmann
3fa456d989
[deviantart] remove mature scraps warning ( #3691 )
...
warn about private deviations
when paginating over eclipse results
2023-02-23 22:56:37 +01:00
Mike Fährmann
51301e0c31
replace remaining time.sleep() calls
...
with Extractor.sleep() or request_interval
2023-02-23 00:35:37 +01:00
Mike Fährmann
6ed4309aba
[deviantart] add 'gallery-search' extractor ( #1695 )
2023-02-22 17:25:31 +01:00
Mike Fährmann
3d8777fbc1
move user agent string to util.py
2023-02-22 11:09:17 +01:00
Mike Fährmann
56039d2456
add 'hash_md5' and 'hash_sha1' functions ( #3679 )
...
... to global eval namespace
2023-02-22 10:58:44 +01:00
Mike Fährmann
e1df7f73b1
[deviantart] add 'search' extractor
...
(#538 , #1264 , #2954 , #2970 , #3577 )
Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.
TODO: parse HTML search results
2023-02-20 20:54:46 +01:00
Gray Manley
f33ac885a6
[pixiv] fix tag write when set to original
2023-02-19 00:09:05 -06:00
Mike Fährmann
4f029ab38b
[pornpics] support '/pornstar' and '/channels' listings
...
- fix docstring (#3671 )
- simplify code
2023-02-18 22:38:21 +01:00
Mike Fährmann
cbe4769246
[danbooru] use gallery-dl UA ( #3665 )
...
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
2023-02-18 22:28:35 +01:00
Mike Fährmann
253ac08203
pre-define and use 'gallery-dö/<version>' UA string
2023-02-18 22:25:54 +01:00
Mike Fährmann
b4899c266f
merge #3656 : [deviantart] fix crash when handling deleted deviations in status updates
2023-02-18 13:11:47 +01:00
Mike Fährmann
bb11c2a576
merge #3662 : [redgifs] add 'collection' extractors
2023-02-18 13:09:17 +01:00
Mike Fährmann
884f1848d6
[redgifs] fix syntax for older Python versions
...
and update docs/supportedsites
2023-02-18 13:07:40 +01:00
Mike Fährmann
725baedad3
[deviantart] use '/collections/all' endpoint for favorites
...
(#3666 ,#3668)
2023-02-17 23:11:35 +01:00
Mike Fährmann
2bd8f2f4bd
[pornpics] add 'search' and 'tag' extractors
...
(#263 , #3544 , #3654 )
2023-02-17 23:03:14 +01:00
Mike Fährmann
79bc82884c
[pornpics] add 'gallery' extractor ( #263 , #3544 , #3654 )
2023-02-17 15:00:57 +01:00
Mike Fährmann
7bdc1d6d3d
[manganelo] update and fix metadata extraction
2023-02-16 22:31:18 +01:00
Mike Fährmann
363bb76dff
[manganelo] simplify URL pattern
2023-02-16 21:36:11 +01:00
enduser420
b28bd9789e
[redgifs] add 'collection' extractors
2023-02-16 20:10:22 +05:30
ClosedPort22
f4e211356d
[deviantart] slight refactor
2023-02-16 21:23:55 +08:00
