Mike Fährmann
e99a9b2aff
[twitter] improve 'cards-blacklist' ( #2875 )
...
allow blacklisting domains and 'name:domain',
where 'domain' depends on a card's 'vanity_url' value
2022-09-17 17:46:34 +02:00
Mike Fährmann
aaf6992bae
[twitter] fix new-style '/card_img/' URLs
2022-09-17 17:45:09 +02:00
Mike Fährmann
40baa77630
[twitter] provide proper 'date' for syndication results ( #2920 )
2022-09-17 14:11:43 +02:00
Mike Fährmann
46fe469c53
[tumblr] implement 'ratelimit' option ( #2919 )
2022-09-17 14:10:33 +02:00
Mike Fährmann
d0b73fec14
[flickr] add support for secure.flickr.com ( #2910 )
2022-09-14 16:19:27 +02:00
Mike Fährmann
35eddaa94e
[reddit] prevent exception with empty submission URLs ( #2913 )
2022-09-14 16:14:42 +02:00
Mike Fährmann
464ea90d14
[exhentai] guess extension for original files ( #2842 )
...
makes it possible to sometimes, when guessed correctly ('.jpg'),
skip an original file download without costing image limit points
2022-09-14 16:06:27 +02:00
Mike Fährmann
551fdf7ad7
[exhentai] move 509 check into its own function
2022-09-13 18:27:14 +02:00
Mike Fährmann
7a799df17f
[tumblr] pre-compile regular expressions
2022-09-13 17:50:48 +02:00
Mike Fährmann
73a52a95b0
update Cloudflare IUAM detection
2022-09-12 11:40:06 +02:00
Mike Fährmann
673b6f1218
[poipiku] use 'img-org.poipiku.com' as image domain ( #2796 )
2022-09-12 11:21:01 +02:00
Mike Fährmann
4ca1a6e5f3
[bunkr] fix extraction ( #2903 )
2022-09-09 18:09:52 +02:00
Mike Fährmann
8b76149521
[exhentai] improve 509.gif detection ( #2901 )
2022-09-09 18:09:52 +02:00
Mike Fährmann
2ed58029f9
{paheal[ add proper support for videos ( #2892 )
2022-09-04 13:30:48 +02:00
Mike Fährmann
444dfb4aa6
[instagram] add 'highlight_title' and 'date' metadata
...
to highlight posts (#2879 )
2022-09-03 16:21:26 +02:00
Mike Fährmann
7f764ebee6
[redgifs] "fix" download URLs ( #2884 )
2022-09-02 23:25:38 +02:00
Mike Fährmann
3cb8327c60
[zerochan] add 'metadata' option ( #2861 )
2022-09-02 23:25:19 +02:00
blankie
9745b48830
[tumblr] attempt to fetch high-quality inline images ( #2877 )
...
* [tumblr] attempt to fetch high-quality images (again)
Fixes #1846 , and fixes #1344
* slight refactor
* update configuration.rst entry
2022-08-31 10:53:50 +02:00
Mike Fährmann
daef91c925
[smugmug] update default API credentials ( #2881 )
...
The old key lacked v2 access and I'm unable to accept
the new terms of service since my old account got deleted
2022-08-31 10:28:25 +02:00
Mike Fährmann
4d78ca89db
[twitter] add 'cards-blacklist' option ( #2875 )
2022-08-31 10:28:25 +02:00
Mike Fährmann
4d7cb0bf56
[twitter] general support for unified cards ( #2875 )
...
just removing the 'type' check seems to work
2022-08-31 10:25:27 +02:00
Mike Fährmann
7ddfff957c
[twitter] support "image_website" unified cards ( #2875 )
2022-08-30 18:16:10 +02:00
Mike Fährmann
2eb0ddd083
[hitomi] fix error when number of tag results is multiple of 25
...
(#2870 )
2022-08-28 17:06:11 +02:00
Mike Fährmann
3cebf787c4
[slideshare] fix metadata extraction
2022-08-28 10:52:28 +02:00
Mike Fährmann
da11fb32d0
update extractor test results
2022-08-28 00:16:12 +02:00
Mike Fährmann
636d03df95
[nijie] reduce cache maxage to 90 days
2022-08-27 21:57:45 +02:00
Mike Fährmann
f375ec0ffa
[vsco] fix 'collection' extraction
2022-08-27 21:16:22 +02:00
Mike Fährmann
8672f8a2b9
[skeb] fix archive_ids for thumbnails and article images
...
8cf5981ded (commitcomment-82316040)
2022-08-27 16:46:53 +02:00
Mike Fährmann
69995d789b
Revert "[twitter] use '{author[name]' in default directory names"
...
This reverts commit 9ad3cdc5d8
2022-08-27 15:11:59 +02:00
Mike Fährmann
946643c23c
[hitomi] use maxage for gg.js cache ( #2863 )
...
cached values become invalid after 1-2 hours
2022-08-26 17:57:17 +02:00
Mike Fährmann
d508b2c049
[gelbooru] implement 'pool' pagination ( #2853 )
2022-08-26 17:57:17 +02:00
Mike Fährmann
67a2efb885
[rule34] implement 'pool' pagination ( #2853 )
2022-08-26 17:57:17 +02:00
Mike Fährmann
70dc4ce911
[skeb] ignore article images with empty URL
...
8cf5981ded (commitcomment-81980633)
2022-08-26 17:57:17 +02:00
Mike Fährmann
f362d4a3c7
[e621] fix 'popular' extraction
2022-08-26 17:57:17 +02:00
Mike Fährmann
7e385ed63e
[foolfuuka] update domains
...
- remove nyafuu
- add rozenarcana (https://archive.alice.al/ )
- add tokyochronos (https://www.tokyochronos.net )
2022-08-26 17:57:17 +02:00
Mike Fährmann
6ba72b6bc6
[twitter] ignore invalid user entries ( #2850 )
2022-08-26 17:57:17 +02:00
blankie
e4cff67aaa
[tumblr] add count metadata field ( #2804 )
...
Fixes #2778
2022-08-18 18:24:37 +02:00
enduser420
574e38a287
[kemonoparty] add 'favorites' option ( #2826 ) ( #2831 )
...
* [kemonoparty] add 'favorites' option (#2826 )
* [kemonoparty] add regex for the url parameter and fallback on the config
option
* [kemonoparty] simplify
2022-08-18 18:01:42 +02:00
Mike Fährmann
a799fae2df
[catbox] add 'album' extractor ( #2410 )
...
adapted from https://github.com/mikf/gallery-dl/pull/2805
- rewrite using GalleryExtractor
- extract more metadata
- match lolisafe names
- add test
2022-08-18 18:00:24 +02:00
Mike Fährmann
264f1336ad
[twitter] unescape '+' in search queries ( #2226 )
...
... and do not raise exception if searched user does not exist
2022-08-17 22:20:26 +02:00
Mike Fährmann
21ff77fea0
[zerochan] extract more metadata for single posts
...
Neither HTML pages nor RSS feed entries have *all* metadata.
It might be necessary to do 1-2 extra HTTP requests to grab everything.
2022-08-14 17:26:29 +02:00
Mike Fährmann
391aecf219
[instagram] provide 'date' for directories ( #2830 )
2022-08-14 17:24:48 +02:00
Martin Juhasz
9ea521bec5
[oauth] add host config option ( #2806 )
2022-08-14 17:08:01 +02:00
Mike Fährmann
9ad3cdc5d8
[twitter] use '{author[name]' in default directory names
...
with the changes to 'user' (749802c7
2022-08-12 11:36:55 +02:00
Mike Fährmann
1c89ccb27d
[weibo] prevent errors when paginating over album entries ( #2817 )
2022-08-11 12:22:14 +02:00
Mike Fährmann
a27b17481f
[tumblr] restrict condition for calling _original_image
2022-08-11 12:20:39 +02:00
Mike Fährmann
df1c643dda
[tumblr] attempt to extract full-resolution photos
...
- for photos with apparent width == 2048 or height == 3072
- can be disabled with 'original' option
2022-08-10 20:01:46 +02:00
Mike Fährmann
a69f198ac9
[wallhaven] implement 'metadata' option ( #2803 )
2022-08-08 21:46:36 +02:00
Mike Fährmann
033faf67ec
[poipiku] update filter for static images ( #2796 )
2022-08-01 12:36:19 +02:00
Mike Fährmann
b9a14b5758
[poipiku] fix posts with more than one image ( #2796 )
2022-08-01 12:28:52 +02:00
Mike Fährmann
0f5826e884
[weibo] prevent exception for missing 'playback_list' ( #2792 )
2022-07-30 16:49:08 +02:00
Mike Fährmann
81a37d21d3
[twitter] simplify 'user' assignment
2022-07-29 20:26:22 +02:00
Mike Fährmann
8a70b94245
[twitter] implement constant 'user' for tweet URLs
2022-07-29 19:44:29 +02:00
Mike Fährmann
8cf5981ded
[skeb] add option to download 'article' images ( #1031 )
2022-07-29 16:32:00 +02:00
Mike Fährmann
43ec315a7f
[deviantart] use public access token for journals ( #2702 )
...
and retry with a private token if needed
2022-07-29 16:18:09 +02:00
Mike Fährmann
3f08a91131
[bunkr] fix extraction ( #2788 )
...
... again
2022-07-29 15:53:46 +02:00
Mike Fährmann
5038893cdd
[blogger] emit metadata for posts without files ( #2789 )
2022-07-29 13:38:39 +02:00
Mike Fährmann
98af5a0409
[zerochan] implement login with username & password ( #1434 )
2022-07-29 12:56:20 +02:00
Mike Fährmann
3a8addfe45
[zerochan] add 'tag' and 'image' extractors ( #1434 )
2022-07-27 22:58:23 +02:00
Mike Fährmann
e660e48a60
[vk] prevent exceptions for broken/invalid photos ( #2774 )
2022-07-27 18:52:43 +02:00
Mike Fährmann
f559943d77
[instagram] fix empty 'params' in '_pagination_api()'
2022-07-27 13:02:24 +02:00
Mike Fährmann
1540d0e695
[twitter] use filter:links ( #2766 )
2022-07-27 12:17:43 +02:00
Mike Fährmann
8d0801ad8e
[twitter] fall back to unfiltered search ( #2766 )
2022-07-27 12:16:53 +02:00
Marius Kaufmann
0aa8345a13
[mastodon] allow downloading without access token ( #2782 )
...
Most mastodon instances allow accessing /api/v1/accounts/XXXX/statuses and api/v1/statuses/XXXX without an API access token.
This commit allows users to download at least some links from such a mastodon instance that does not already have access tokens hard-coded into the extractor.
User extractor only works on links that include the user id such as https://mastodon.tld/@id:12345 . Status links work as-is.
2022-07-27 12:07:06 +02:00
thatfuckingbird
ea5ffb19a6
fanbox: download cover images in original size ( #2784 )
2022-07-27 10:53:04 +02:00
Chew Shee Yang
977d53b640
[Instagram] Add support for user's saved collection ( #2769 )
...
* [Instagram] Add support for user's saved collection
* [Instagram] Run formatter
* [Instagram] Simplify collection_id retrieval and add metadata
* [Instagram] Fix bug when params is not passed to _pagination_api
2022-07-27 10:49:45 +02:00
blankie
5b63df46c0
[tumblr] attempt to get higher-quality images ( #2761 )
2022-07-27 10:47:43 +02:00
blankie
59b16b3f70
[artstation] add 'num' and 'count' metadata fields ( #2764 )
2022-07-19 14:25:07 +02:00
Mike Fährmann
eb68d45544
add global 'warnings' option ( #2762 )
2022-07-18 22:20:30 +02:00
Mike Fährmann
f225247670
[gelbooru] add support for api_key and user_id ( #2767 )
2022-07-18 18:46:31 +02:00
Mike Fährmann
77bdd8fe0f
[twitter] implement constant 'user' for 'from:…' searches
2022-07-17 19:14:32 +02:00
Mike Fährmann
a267a05a3f
[twitter] update 'quote_id' and 'quote_by'
...
- 'quote_id' is now non-null for quoted Tweets and has the ID of the
quoting Tweet, instead the other way round like before
- 'quote_by' is now the 'screen_name' of the quoting user
(was the same the new 'quote_id' is now)
2022-07-17 18:50:21 +02:00
Mike Fährmann
749802c7bd
[twitter] update 'user' and 'author' fields
...
- 'author' is always the user who authored a tweet
- 'user' is always the user specified in the input URL
or equal to 'author' when the former is not given
2022-07-17 17:04:24 +02:00
Mike Fährmann
a566e63cdf
[tumblr] support '/blog/view' URLs ( #2760 )
2022-07-15 15:22:54 +02:00
Mike Fährmann
46f11a3118
[bunkr] fix extraction ( #2732 )
...
move bunkr.is code to its own module
2022-07-15 13:00:57 +02:00
Mike Fährmann
baf3815ebd
[nozomi] small code optimizations
2022-07-14 14:59:11 +02:00
blankie
836402bf58
[twitter] unescape content ( #2756 ) ( #2757 )
...
Fixes #2756
2022-07-13 19:45:14 +02:00
Mike Fährmann
62cc47755b
[nozomi] reduce memory consumption during searches ( #2754 )
...
only load and use the entire 'index.nozomi' database
if there are only negative search terms
2022-07-13 17:16:10 +02:00
Mike Fährmann
467a2a4d35
[instagram] add 'pinned' metadata field ( #2752 )
...
'pinned' is a list of user IDs for which a post is pinned
and empty if not pinned anywhere.
2022-07-13 15:54:08 +02:00
Mike Fährmann
fe2b3d57d4
[komikcast] update domain
2022-07-12 23:07:58 +02:00
Mike Fährmann
4e11ca737e
[hentaifoundry] fix metadata extraction
2022-07-12 22:19:22 +02:00
Mike Fährmann
f2e59cc906
[slideshare] fix 'description' extraction
2022-07-12 18:38:44 +02:00
Mike Fährmann
31e868fca1
[khinsider] extract 'platform' metadata
2022-07-12 18:31:31 +02:00
Mike Fährmann
c6a9bab019
update extractor test results
2022-07-12 15:49:22 +02:00
Mike Fährmann
539e3bbed9
[weibo] handle invalid/broken status objects
2022-07-12 15:49:09 +02:00
Mike Fährmann
32c75d12e8
[sankaku] rewrite URLs to s.sankakucomplex.com ( #2746 )
2022-07-11 12:46:04 +02:00
Mike Fährmann
d5ded11aa8
[pixiv] fix default filenames for backgrounds
2022-07-11 12:45:38 +02:00
Mike Fährmann
e1f501ed14
[mangakakalot] update domain
2022-07-11 00:29:25 +02:00
Mike Fährmann
2dc57637cf
[foolfuuka] remove archive.wakarimasen.moe
2022-07-10 23:13:49 +02:00
Mike Fährmann
98744977cf
[itaku] fix 'date' parsing
2022-07-10 20:45:51 +02:00
Mike Fährmann
b590774f67
[twitter] add 'count' metadata field ( #2741 )
2022-07-10 14:37:04 +02:00
Mike Fährmann
7c0505868c
[kemonoparty] ensure all files have an 'extension' ( #2740 )
2022-07-10 13:53:07 +02:00
Mike Fährmann
e4f48cc810
make it easier to disable default 'browser' settings
...
Previously it was necessary to set 'browser' to a non-empty, non-string
value to disable any default 'browser' value.
Now '-o browser=' or '-o browser=false' is enough.
2022-07-07 11:17:43 +02:00
Mike Fährmann
92b75bcdce
limit path length for --write-pages output on Windows ( #2733 )
2022-07-06 18:56:23 +02:00
Mike Fährmann
311e9383af
[pinterest] handle section pins with separate extractors ( #2684 )
2022-07-03 18:12:16 +02:00
Mike Fährmann
1d14928bd9
[twitter] ignore previously seen Tweets ( #2712 )
...
occurs primarily for /with_replies results when logged in
2022-07-03 16:13:53 +02:00
Mike Fährmann
4b2a0a0eda
[twitter] implement 'strategy' option ( #2712 )
...
to be able to better control what Tweets get used an returned
for twitter.com/USER URLs.
2022-07-03 14:29:15 +02:00
Mike Fährmann
c794777600
[newgrounds] prevent exception on empty results ( #2727 )
2022-07-03 11:44:46 +02:00
Mike Fährmann
36ead45546
[itaku] fix caching bug ( #1842 )
...
ItakuApi.user() would always return the first user it was called with,
regardless of its 'username' argument.
2022-07-01 21:00:39 +02:00
Mike Fährmann
127a190c94
[itaku] categorize sections by group ( #1842 )
2022-06-29 22:34:07 +02:00
