4122 Commits

Author	SHA1	Message	Date
Mike Fährmann	095e5ded6f	[reddit] support comment embeds (#5366)	2024-04-01 23:35:42 +02:00
Mike Fährmann	64948f2c09	[foolfuuka] improve 'board' pattern & support pages (#5408)	2024-04-01 22:31:25 +02:00
Mike Fährmann	0c17884673	store 'match' and 'groups' in Extractor objects	2024-04-01 03:07:52 +02:00
Mike Fährmann	bdbc8be5f3	[twitter] prevent crash when extracting 'birdwatch' metadata (#5403)	2024-03-30 18:13:38 +01:00
Mike Fährmann	ef0c90414c	[wikimedia] suppress exception for entries without 'imageinfo' (#5384)	2024-03-26 15:33:26 +01:00
Mike Fährmann	9cce461627	[kemonoparty] add 'announcements' option (#5262) ... https://github.com/mikf/gallery-dl/issues/5262#issuecomment-2015919188	2024-03-26 15:20:14 +01:00
Mike Fährmann	72ac2c750d	[kemonoparty:favorite] support 'sort' and 'order' query params (#5375)	2024-03-26 02:27:36 +01:00
Mike Fährmann	d1d017ab5d	merge #5372: [twitter] match Tweet URLs with query parameters (#5371) ... fixes regression introduced in 40c05535	2024-03-25 22:01:46 +01:00
fireattack	423599ce95	[twitter] fix pattern for single tweet (#5371) ... - Add optional slash - Update tests to include some non-standard tweet URLs	2024-03-25 21:57:35 +01:00
Mike Fährmann	15a4bc2584	[kemonoparty] fix KeyError for empty files (#5368)	2024-03-24 02:21:38 +01:00
Mike Fährmann	925123e007	[deviantart] handle CloudFront blocks in general (#5363) ... This was already done for non-OAuth requests (#655) but CF is now blocking OAuth API requests as well.	2024-03-23 15:50:24 +01:00
Mike Fährmann	31e7ca73b6	[gelbooru] add 'order-posts' option for favorites (#5220)	2024-03-23 13:30:09 +01:00
Mike Fährmann	4b6f47e571	[pornhub:gif] extract 'viewkey' and 'timestamp' metadata (#4463) ... https://github.com/mikf/gallery-dl/issues/4463#issuecomment-2014550302	2024-03-22 18:00:20 +01:00
Mike Fährmann	9f73fac5ef	[steamgriddb] raise proper exception for deleted assets	2024-03-22 17:53:06 +01:00
Mike Fährmann	fe9171508f	[kemonoparty] fix exception for '/revision/' URLs ... caused by 03a9ce98	2024-03-22 03:10:12 +01:00
Mike Fährmann	a650fd3177	[deviantart] improve 'index' extraction for stash files (#5335)	2024-03-22 02:53:54 +01:00
Mike Fährmann	1d6260f456	[bunkr] remove 'description' metadata ... album descriptions are no longer available on album pages and the previous code erroneously returned just '0'	2024-03-22 02:14:41 +01:00
Mike Fährmann	32262a048b	[idolcomplex] fix metadata extraction ... - replace legacy 'id' vales with alphanumeric ones, since the former are no longer available - approximate 'vote_average', since the real value is no longer available - fix 'vote_count'	2024-03-22 01:43:05 +01:00
Mike Fährmann	77ab015df2	[idolcomplex] support new pool URLs	2024-03-22 01:38:25 +01:00
Mike Fährmann	1b34d5ac40	[subscribestar] fix 'date' metadata	2024-03-22 00:45:09 +01:00
wankio	56f2d5a5f2	fapello.py Fullsize image ... by remove ".md" and ".th" in image url, it will download fullsize of images	2024-03-21 18:08:18 +01:00
wankio	eb673a7204	Update fapello.py ... get fullsize image instead resized	2024-03-19 02:48:53 +07:00
Mike Fährmann	6d93295fea	[gelbooru] add 'date_favorited' metadata field	2024-03-18 20:46:11 +01:00
Mike Fährmann	0d69af94d5	[gelbooru] detect returned favorites order (#5220)	2024-03-18 20:45:06 +01:00
Mike Fährmann	e1c51c0dfb	[vipergirls] add 'domain' option (#4166)	2024-03-18 17:36:16 +01:00
Mike Fährmann	ae116812cc	[vipergirls] add 'like' option (#4166)	2024-03-18 17:36:16 +01:00
Mike Fährmann	da6ba60331	[bluesky] add 'instance' metadata field (#4438)	2024-03-18 17:36:16 +01:00
Mike Fährmann	5c1f5861b6	[flickr] add 'contexts' option (#5324)	2024-03-18 17:36:16 +01:00
Mike Fährmann	26bc2d55f4	[hiperdex] update URL patterns & fix 'manga' metadata (#5340)	2024-03-18 17:36:16 +01:00
Mike Fährmann	8e694d85c4	[twitter] add 'birdwatch' metadata field (#5317) ... should probably get a better name, but this is what it's called internally by Twitter	2024-03-18 17:36:02 +01:00
Mike Fährmann	b8e7be225c	merge #5333: [imagefap] fix folder extractor	2024-03-15 23:46:43 +01:00
Herp	99c53f7fa8	Fix imagefap extrcator	2024-03-15 23:44:25 +01:00
Mike Fährmann	03a9ce9832	[kemonoparty] add 'order-revisions' option (#5334)	2024-03-15 23:06:05 +01:00
Mike Fährmann	1418c0ce38	[kemonoparty] add 'revision_count' metadata field (#5334)	2024-03-15 22:28:15 +01:00
Mike Fährmann	ddee5ae7c4	[gofile] fix extraction	2024-03-15 20:34:54 +01:00
Mike Fährmann	5716430c35	[deviantart:stash] recognize 'deviantart.com/stash/…' URLs	2024-03-15 18:14:55 +01:00
Mike Fährmann	76683c5f5c	[deviantart:stash] fix 'index' metadata (#5335)	2024-03-15 18:10:59 +01:00
Mike Fährmann	5158cbb4c1	[weibo] rework pagination logic (#4168) ... don't automatically stop when receiving an empty status list shouldn't improve 'tabtype=feed' results, but at least 'tabtype=album' ones and others using cursors won't end prematurely	2024-03-14 00:06:25 +01:00
Mike Fährmann	d53db6e11a	[twitter] handle "account is temporarily locked" errors (#5300) ... - display proper error message - add 'locked' option	2024-03-13 22:34:25 +01:00
Mike Fährmann	108abab537	[twitter] add 'protected' metadata field (#5327) ... for 'author' and 'user'	2024-03-13 14:46:03 +01:00
Mike Fährmann	dde822e69a	[instagram] change 'posts are private' exception to a warning (#5322)	2024-03-12 18:58:15 +01:00
Mike Fährmann	e40973118d	merge #5323: [instagram] default 'likes' to '0'	2024-03-12 18:47:24 +01:00
Mike Fährmann	b188cb2729	merge #5321: [mastodon] fix handling null 'moved' account field	2024-03-12 18:45:53 +01:00
Martin Contento	c210c7a4ed	[instagram] default posts like_count to zero ... I don't know when/why this happens and for private account reasons can't add an example here, but i had this error out and it feels like a safe fix to default to `0` here.	2024-03-12 15:12:07 +00:00
blankie	225d849139	[mastodon] fix handling null 'moved' account field	2024-03-12 11:44:25 +11:00
Mike Fährmann	106dfdb4c3	cleanup sleep-request retry delay code ... more lines but easier to read I'd say	2024-03-11 21:38:06 +01:00
Mike Fährmann	4565de3c8e	[reddit] provide 'gif' and 'mp4' variant fallbacks (#5315)	2024-03-10 20:50:55 +01:00
Mike Fährmann	5842e4928d	[imgur] fail downloads when redirected to 'removed.png' (#5308)	2024-03-09 23:35:23 +01:00
Mike Fährmann	ac4e29f70a	[lensdump] support more direct link formats (#5293)	2024-03-09 23:33:58 +01:00
Mike Fährmann	146459056c	[reddit] provide 'fallback_url' as video fallback (#5296)	2024-03-07 15:58:01 +01:00
Mike Fährmann	d3003f8531	merge #5270: [imagefap] add 'folder' metadata	2024-03-07 01:31:40 +01:00
Mike Fährmann	05331f9cf1	[imagefap] flake8, cleanup, tests	2024-03-07 01:29:19 +01:00
Mike Fährmann	40c0553523	[twitter] add 'quotes' extractor (#5262) ... https://github.com/mikf/gallery-dl/issues/5262#issuecomment-1981571924 It's implemented as a search for 'quoted_tweet_id:…' on Twitter.	2024-03-07 00:52:50 +01:00
Mike Fährmann	790c0ffb8d	[lensdump] recognize direct image links (#5293	2024-03-06 22:56:57 +01:00
Mike Fährmann	6d9e3c0eb1	[skeb] add extractor for followed users (#5290) ... needs 'Authorization' header from browser session -o headers.Authorization="Bearer ey…"	2024-03-06 22:43:01 +01:00
Mike Fährmann	ace16f00f5	[weibo] fix retweets (#2825, #3874, #5263) ... - handle 快转 retweets - disable 'retweets' by default - skip all retweet media when 'retweets' are disabled - extract all retweet media when 'retweets' is set to "original"	2024-03-06 19:36:53 +01:00
Mike Fährmann	0676a9d6ec	[weibo] fix 'livephoto' filename extensions (#5287)	2024-03-06 19:36:32 +01:00
Mike Fährmann	db507e30c7	[pixiv] fix novel text extraction (#5285) ... change to '/webview/v2/novel' since '/v1/novel/text' does not work anymore	2024-03-06 02:31:26 +01:00
Mike Fährmann	296f20e630	[warosu] fix 'board_name' metadata	2024-03-06 01:28:47 +01:00
Mike Fährmann	24873c2724	[warosu] fix crash for threads with deleted posts (#5289)	2024-03-06 01:27:45 +01:00
Mike Fährmann	f296067797	[naver] unescape post 'title' and 'description'	2024-03-06 00:46:19 +01:00
Mike Fährmann	a71cdab53e	merge #5126: [naver] fix EUC-KR encoding issue in old image URLs	2024-03-06 00:22:33 +01:00
Mike Fährmann	a8d3efbb99	[naver] simplify code + add test	2024-03-06 00:21:23 +01:00
Johann Hong	f64fb8f239	[naver] EUC-KR encoding issue in old image URLs Fix ... Around October 2010, the image server URL format and file name encoding changed from EUC-KR to UTF-8. Modified to detect old URL format and decode image URLs into EUC-KR - (lint with flake8) Customize conditions Wrap lines smaller than 79 characters - (lint with flake8) Customize conditions (2nd try) - One import per line - Indent on consecutive lines - (lint with flake8) Customize conditions (3rd try) - E128 continuation line under-indented for visual indent - E123 closing bracket does not match indentation of opening bracket's line - Update naver.py Check encoding for all image URLs	2024-03-06 00:21:23 +01:00
Mike Fährmann	7b28418f69	[naver] recognize '.naver' URLs ... https://blog.naver.com/PostView.naver?…	2024-03-05 22:30:29 +01:00
Mike Fährmann	a767832332	[deviantart:avatar] ignore default avatars (#5276)	2024-03-04 23:11:30 +01:00
Mike Fährmann	0cbc910905	[deviantart:avatar] fix 'index' for avatars without '?' (#5276)	2024-03-04 22:31:35 +01:00
Mike Fährmann	6482bbc525	[bluesky] handle different 'embed' structure	2024-03-03 20:41:01 +01:00
Mike Fährmann	1115dccd0d	[bluesky] fix feeds and lists ... bug introduced in 495c9ee1	2024-03-03 20:22:34 +01:00
Mike Fährmann	1a9b9aa310	[artstation] support video clips (#2566, #3309, #3911) ... - add 'videos' and 'previews' options - fix 403 errors for video previews	2024-03-03 18:00:45 +01:00
termvacycurtocs	f8b037ed40	[Imagefap] Add folder metadata ... [Imagefap] Add "folder" metadata when downloading a folder or user profile. No additional request is made to the server. Use for example with the following configuration : "parent-metadata": true "directory":["{category}", "{uploader}", "{folder}", "{gallery_id} {title}"]	2024-03-02 22:15:45 +01:00
Mike Fährmann	982880615d	[deviantart] prevent unnecessary API requests (#4995) ... … when using 'comments-avatars' This also has the added benefit of making it possible to download comment avatars from users without a valid user profile entry, like deleted users.	2024-03-02 21:59:16 +01:00
Mike Fährmann	25d2854272	[deviantart] add 'comments-avatars' option (#4995)	2024-03-02 21:59:16 +01:00
Mike Fährmann	218ec1a9ee	[instagram] raise proper error for missing 'reels_media' (#5257)	2024-03-02 21:58:59 +01:00
Mike Fährmann	82c73c77b0	[redgifs] make 'date' available for directories (#5262) ... https://github.com/mikf/gallery-dl/issues/5262#issuecomment-1973975415	2024-03-01 23:39:16 +01:00
Mike Fährmann	cf9e99c07b	[artstation] support collections (#146) ... https://github.com/mikf/gallery-dl/issues/146#issuecomment-1972101003	2024-03-01 20:21:21 +01:00
Mike Fährmann	32ec695195	merge #5256: [wikimedia] add azurlane.koumakan.jp	2024-02-29 21:50:24 +01:00
thatfuckingbird	88a06df165	[wikimedia] add azurlane.koumakan.jp to presets	2024-02-29 19:28:50 +01:00
Mike Fährmann	1db0a587f3	[nitter] ignore invalid Tweets (#5253) ... like "Load newest"	2024-02-29 16:31:37 +01:00
Mike Fährmann	a00b171d4e	[bluesky] wait unitl 'RateLimit-Reset' on 429 responses	2024-02-28 18:13:16 +01:00
Mike Fährmann	7d874e2497	[bluesky] improve API error messages	2024-02-28 14:45:36 +01:00
Mike Fährmann	495c9ee126	[bluesky] add 'reposts' option (#4438, #5248)	2024-02-27 18:42:29 +01:00
Mike Fährmann	c8b591303f	[paheal] cleanup	2024-02-27 02:27:20 +01:00
Mike Fährmann	8a11b72253	remove extractor/test.py (#4504)	2024-02-27 01:37:57 +01:00
Mike Fährmann	fde9e25c9f	[tests:kemonoparty] '.party' -> '.su'	2024-02-26 22:25:04 +01:00
Mike Fährmann	311a21bfb2	[bluesky] fix '/follows' not spawning child extractors (#5246)	2024-02-26 15:38:31 +01:00
Mike Fährmann	d3dca68225	[xvideos] fix galleries with more than 500 images (#5244)	2024-02-26 15:36:41 +01:00
Mike Fährmann	13443f40a3	[xvideos] support '/channels/' URLs (#5244)	2024-02-26 00:08:37 +01:00
Mike Fährmann	c60ebc6519	[deviantart] improve fetching extended metadata (#5175) ... use multiple metadata API calls per chunk of deviations if necessary	2024-02-25 03:36:00 +01:00
Mike Fährmann	cc6b9e4c18	[zerochan] use API by default (#3669) ... add 'pagination' option	2024-02-25 00:36:14 +01:00
Mike Fährmann	a2b55d5dde	[skeb] retry 429 responses containing a 'request_key' cookie (#5210)	2024-02-24 00:54:15 +01:00
Mike Fährmann	b4c46de4b8	merge #5224: [artstation] update URL patterns to recognize usernames with dashes	2024-02-21 14:41:02 +01:00
blankie	962f55cc68	[artstation] fix handling usernames with dashes	2024-02-21 17:39:37 +11:00
Mike Fährmann	fe7e2281ac	[nijie] increase default delay between requests (#5221) ... 1-2s is not enough	2024-02-20 18:19:49 +01:00
Mike Fährmann	a34312e3ac	[instagram] make accessing 'like_count' non-fatal (#5218)	2024-02-19 19:24:51 +01:00
Mike Fährmann	741fd00cec	[deviantart] extend 'metadata' option (#5175) ... alloe fetching extended metadata in addition to the usual 'description', 'tags', etc by setting 'metadata' to a list of 'camera', 'stats', 'submission', 'collection', and 'gallery' for example "metadata": "stats,submission"	2024-02-18 23:14:14 +01:00
Mike Fährmann	8a63801311	[vsco] add 'spaces' extractor (#5202) ... for spaces listed on a user page	2024-02-17 18:20:48 +01:00
Mike Fährmann	ccb413df71	[wikimedia] support 'pidgi.net' and 'bulbapedia.bulbagarden.net' (#5205, #5206)	2024-02-17 17:35:10 +01:00
Mike Fährmann	7033cc14e9	[vsco] add 'space' extractor (#5202)	2024-02-17 01:54:05 +01:00
Mike Fährmann	770aec922d	[fapachi] ignore empty entries	2024-02-16 22:43:37 +01:00
Mike Fährmann	ee7c054855	[bluesky] add 'search' extractor (#4438) ... Both https://bsky.app/search?q=QUERY and https://bsky.app/search/QUERY are recognized as search URLs, where QUERY gets forwarded unmodified as 'q' parameter for app.bsky.feed.searchPosts . User searches are not supported yet.	2024-02-16 15:58:47 +01:00
Mike Fährmann	91e5c4fdfe	[bluesky] add 'avatar' and 'background' extractors (#4438)	2024-02-16 15:41:19 +01:00
Mike Fährmann	24c1317e0d	[batoto] fix crash when manga/chapter contains a '-' (#5200)	2024-02-16 00:10:08 +01:00
Mike Fährmann	0abd9723af	[bluesky] add 'metadata' option (#4438) ... allow extracting 'user' metadata and make 'facets' extraction optional	2024-02-15 23:30:16 +01:00
Mike Fährmann	7e036ea290	[bluesky] add 'depth' option (#4438) ... and reduce default depth and parentHeight values	2024-02-15 22:26:05 +01:00
Mike Fährmann	42335ea880	[zerochan] fix skipping every other post	2024-02-15 02:51:01 +01:00
Mike Fährmann	c97b92cc35	[fanbox] add 'home' and 'supporting' extractors (#5138)	2024-02-14 23:25:39 +01:00
Mike Fährmann	04e4ffc64c	[deviantart] combine 'png' option with 'quality' (#4846) ... "quality": "png" to download PNGs instead og JPEGs	2024-02-14 22:07:29 +01:00
Mike Fährmann	9cc4ec2c58	[deviantart] add 'png' option (#4846)	2024-02-14 01:03:15 +01:00
Mike Fährmann	966c8608e6	[deviantart] move image content extraction into separate function	2024-02-14 00:30:06 +01:00
Mike Fährmann	1d1ffe3317	[pornpics] update 'channel' extraction & add test ... change 'channel' to a list, since extracting both 'channel' and 'channels' does not really work with text.extract_from()	2024-02-13 23:48:46 +01:00
cc1234	32472d7d6c	Add support for multi channels	2024-02-13 18:34:04 +00:00
Mike Fährmann	139ff3f6ab	[kemonoparty] add 'posts' extractor (#5194)	2024-02-13 15:41:34 +01:00
Mike Fährmann	814ad9321e	[deviantart] skip locked/blurred posts (#4567, #5193)	2024-02-13 14:15:12 +01:00
Mike Fährmann	f7f8ef8684	[twitter] support communities (#4913)	2024-02-13 01:30:23 +01:00
Mike Fährmann	cae77e85f8	[twitter] update query hashes ... ... as well as 'variables' and 'features' values also remove unused legacy API code	2024-02-12 23:19:13 +01:00
Mike Fährmann	06cb518d97	[bunkr] fix extraction (#5088, #5151, #5153) ... - remove legacy code - map legacy domains to bunkr.sk - use input URL domain for newer domains - update tests (some files got slightly modified or deleted)	2024-02-11 22:36:03 +01:00
Mike Fährmann	dcc6e3f65c	merge #5134: [bunkr] add new bunkr domains (#5130)	2024-02-11 21:10:06 +01:00
Mike Fährmann	4641937ca3	[imagetwist] add 'gallery' extractor (#5190)	2024-02-11 18:41:02 +01:00
Mike Fährmann	fde82ab0ce	[imagechest] add 'user' extractor (#5143)	2024-02-11 18:38:33 +01:00
Mike Fährmann	4474cea31b	merge #5187: [skeb] add 'num' and 'count' metadata fields	2024-02-10 19:36:59 +01:00
Mike Fährmann	4cfceb23cb	[skeb] rename 'data' -> 'file' & add tests	2024-02-10 19:35:50 +01:00
Mike Fährmann	c83d0a1596	[weibo] add 'gifs' option (#5183)	2024-02-10 18:17:07 +01:00
blankie	f9a8e8cacf	[skeb] add 'num' and 'count' metadata fields	2024-02-10 21:51:23 +11:00
Mike Fährmann	af61d2b037	[wikimedia] combine most wikimedia.org sites (#1443) ... add wikidata.org and wikivoyage.org	2024-02-10 03:00:58 +01:00
Mike Fährmann	c7d17f1111	[bluesky] extract 'hashtags', 'mentions', and 'uris' metadata (#4438)	2024-02-10 00:01:55 +01:00
Mike Fährmann	55bbd49a0e	[bluesky] download images in original resolution (#4438) ... at least up to 2000 px	2024-02-09 21:33:33 +01:00
Mike Fährmann	6414dc6bca	[idolcomplex] fix pagination for tags containing ':' (#5171)	2024-02-09 17:51:08 +01:00
Mike Fährmann	5c2a2321a2	[bluesky] update refresh token after using it (#4438)	2024-02-08 22:33:34 +01:00
Mike Fährmann	9c10be54fb	[bluesky] add 'following' extractor (#4438)	2024-02-08 21:58:17 +01:00
Mike Fährmann	86ce35d6a1	[bluesky] simplify 'pattern'	2024-02-08 21:28:21 +01:00
Mike Fährmann	da292ded4e	[bluesky] add 'list' extractor (#4438)	2024-02-08 21:24:07 +01:00
Mike Fährmann	004bf7bb38	[bluesky] add 'feed' extractor (#4438)	2024-02-08 21:01:44 +01:00
Mike Fährmann	6aea818d4e	[bluesky] allow using DIDs as user handles (#4438)	2024-02-08 20:15:54 +01:00
Mike Fährmann	aee5580c62	[idolcomplex] extract 'id_alnum' metadata (#5171)	2024-02-08 18:29:54 +01:00
Mike Fährmann	cf7d6be2d4	[bluesky] initial support (#4438, #4708, #4722, #5047)	2024-02-07 19:09:33 +01:00
Mike Fährmann	6ef143ea31	[idolcomplex] support alphanumeric post IDs (#5171)	2024-02-07 14:57:13 +01:00
Mike Fährmann	6e928300bc	[flickr] handle non-JSON errors (#5131)	2024-02-06 21:22:10 +01:00
Mike Fährmann	90ac6d7375	[wikimedia] use '/api.php' as default API path	2024-02-06 00:36:51 +01:00
Mike Fährmann	d7823b9f81	[pinterest] fix section URLs for boards with /?# in name (#5104)	2024-02-05 15:54:06 +01:00
Mike Fährmann	de752eb7b1	[naverwebtoon] support '/webtoon/' paths for all comics (#5123)	2024-02-04 21:38:46 +01:00
Jeff Mercado	d9d0601ab1	break up line to fit 80 char	2024-01-29 20:31:58 -08:00
Jeff Mercado	6bcd3c9380	[bunkr] add new bunkr domains (#5130)	2024-01-29 20:25:33 -08:00
Mike Fährmann	62d6f5f8d2	[luscious] fix IndexError for files without thumbnail (#5122)	2024-01-28 01:43:29 +01:00
Mike Fährmann	22647c2626	[naverwebtoon] fix 'title' for comics with empty tags (#5120)	2024-01-27 16:24:03 +01:00
Mike Fährmann	3433481dd2	[gofile] update 'website_token' extraction	2024-01-27 01:10:14 +01:00
Mike Fährmann	1f7101d606	[archivedmoe] fix thebarchive webm URLs (#5116)	2024-01-27 00:24:41 +01:00
Mike Fährmann	34a4ddc399	[sankaku] add 'id-format' option (#5073)	2024-01-26 17:56:08 +01:00
Mike Fährmann	afd20ef42c	[kemonoparty] implement filtering duplicate revisions (#5013) ... set 'revisions' to '"unique"' to have it ignore duplicate revisions	2024-01-26 14:44:15 +01:00
Mike Fährmann	c28475d325	[kemonoparty] fix deleting 'name' in orginal objects (#5103) ... ... when computing 'revision_hash' regression caused by 3d68eda4 dict.copy() only creates a shallow copy I know that and still managed to get I wrong ...	2024-01-25 23:46:19 +01:00