895 Commits

Author	SHA1	Message	Date
Mike Fährmann	7033cc14e9	[vsco] add 'space' extractor (#5202)	2024-02-17 01:54:05 +01:00
Mike Fährmann	c9efccc959	[tests] update extractor results	2024-02-16 22:42:06 +01:00
Mike Fährmann	c413834dfc	[bluesky] extend tests	2024-02-16 16:30:02 +01:00
Mike Fährmann	24c1317e0d	[batoto] fix crash when manga/chapter contains a '-' (#5200)	2024-02-16 00:10:08 +01:00
Mike Fährmann	0abd9723af	[bluesky] add 'metadata' option (#4438) ... allow extracting 'user' metadata and make 'facets' extraction optional	2024-02-15 23:30:16 +01:00
Mike Fährmann	c97b92cc35	[fanbox] add 'home' and 'supporting' extractors (#5138)	2024-02-14 23:25:39 +01:00
Mike Fährmann	04e4ffc64c	[deviantart] combine 'png' option with 'quality' (#4846) ... "quality": "png" to download PNGs instead og JPEGs	2024-02-14 22:07:29 +01:00
Mike Fährmann	9cc4ec2c58	[deviantart] add 'png' option (#4846)	2024-02-14 01:03:15 +01:00
Mike Fährmann	1d1ffe3317	[pornpics] update 'channel' extraction & add test ... change 'channel' to a list, since extracting both 'channel' and 'channels' does not really work with text.extract_from()	2024-02-13 23:48:46 +01:00
Mike Fährmann	139ff3f6ab	[kemonoparty] add 'posts' extractor (#5194)	2024-02-13 15:41:34 +01:00
Mike Fährmann	814ad9321e	[deviantart] skip locked/blurred posts (#4567, #5193)	2024-02-13 14:15:12 +01:00
Mike Fährmann	f7f8ef8684	[twitter] support communities (#4913)	2024-02-13 01:30:23 +01:00
Mike Fährmann	8f27f43d4d	[tests] implement explicitly disabling auth	2024-02-13 00:08:27 +01:00
Mike Fährmann	06cb518d97	[bunkr] fix extraction (#5088, #5151, #5153) ... - remove legacy code - map legacy domains to bunkr.sk - use input URL domain for newer domains - update tests (some files got slightly modified or deleted)	2024-02-11 22:36:03 +01:00
Mike Fährmann	4641937ca3	[imagetwist] add 'gallery' extractor (#5190)	2024-02-11 18:41:02 +01:00
Mike Fährmann	fde82ab0ce	[imagechest] add 'user' extractor (#5143)	2024-02-11 18:38:33 +01:00
Mike Fährmann	4474cea31b	merge #5187: [skeb] add 'num' and 'count' metadata fields	2024-02-10 19:36:59 +01:00
Mike Fährmann	4cfceb23cb	[skeb] rename 'data' -> 'file' & add tests	2024-02-10 19:35:50 +01:00
Mike Fährmann	c83d0a1596	[weibo] add 'gifs' option (#5183)	2024-02-10 18:17:07 +01:00
Mike Fährmann	af61d2b037	[wikimedia] combine most wikimedia.org sites (#1443) ... add wikidata.org and wikivoyage.org	2024-02-10 03:00:58 +01:00
Mike Fährmann	c7d17f1111	[bluesky] extract 'hashtags', 'mentions', and 'uris' metadata (#4438)	2024-02-10 00:01:55 +01:00
Mike Fährmann	55bbd49a0e	[bluesky] download images in original resolution (#4438) ... at least up to 2000 px	2024-02-09 21:33:33 +01:00
Mike Fährmann	aee5580c62	[idolcomplex] extract 'id_alnum' metadata (#5171)	2024-02-08 18:29:54 +01:00
Mike Fährmann	cf7d6be2d4	[bluesky] initial support (#4438, #4708, #4722, #5047)	2024-02-07 19:09:33 +01:00
Mike Fährmann	6ef143ea31	[idolcomplex] support alphanumeric post IDs (#5171)	2024-02-07 14:57:13 +01:00
Mike Fährmann	d7823b9f81	[pinterest] fix section URLs for boards with /?# in name (#5104)	2024-02-05 15:54:06 +01:00
Mike Fährmann	de752eb7b1	[naverwebtoon] support '/webtoon/' paths for all comics (#5123)	2024-02-04 21:38:46 +01:00
Mike Fährmann	22647c2626	[naverwebtoon] fix 'title' for comics with empty tags (#5120)	2024-01-27 16:24:03 +01:00
Mike Fährmann	1f7101d606	[archivedmoe] fix thebarchive webm URLs (#5116)	2024-01-27 00:24:41 +01:00
Mike Fährmann	34a4ddc399	[sankaku] add 'id-format' option (#5073)	2024-01-26 17:56:08 +01:00
Mike Fährmann	afd20ef42c	[kemonoparty] implement filtering duplicate revisions (#5013) ... set 'revisions' to '"unique"' to have it ignore duplicate revisions	2024-01-26 14:44:15 +01:00
Mike Fährmann	c28475d325	[kemonoparty] fix deleting 'name' in orginal objects (#5103) ... ... when computing 'revision_hash' regression caused by 3d68eda4 dict.copy() only creates a shallow copy I know that and still managed to get I wrong ...	2024-01-25 23:46:19 +01:00
Mike Fährmann	beacfa7436	[bunkr] update domain to 'bunkr.sk' (#5114)	2024-01-25 23:45:41 +01:00
Mike Fährmann	0d3af0d35b	[tests] ignore 'ytdl' categories when import fails (#5095)	2024-01-21 15:31:12 +01:00
Mike Fährmann	f3ad91b44f	[bunkr] update domain (#5088)	2024-01-21 03:00:57 +01:00
Mike Fährmann	c7a42880ab	[wikimedia] support fandom wikis (#1443, #2677, #3378) ... Wikis hosted on fandom.com are just wikimedia instances and support its API.	2024-01-21 00:52:02 +01:00
blankie	df718887c2	[webtoons] fix extracting comic and episode name with commas	2024-01-21 09:50:27 +11:00
Mike Fährmann	0d367ce1b9	[tests] update extractor results	2024-01-20 18:02:36 +01:00
Mike Fährmann	9ca6117c67	[hbrowse] remove module ... website gone	2024-01-20 02:53:44 +01:00
Mike Fährmann	375eefb886	[chevereto] remove 'pixl.li' ... "Pixl is closing down" "All images will be deleted January 1st."	2024-01-20 02:21:40 +01:00
Mike Fährmann	b0a441f1e3	[nitter] remove 'nitter.lacontrevoie.fr' ... "Fermeture de Nitter / Closing down Nitter"	2024-01-19 19:34:16 +01:00
Mike Fährmann	a1c1e80f67	[giantessbooru] update domain	2024-01-19 14:21:56 +01:00
Mike Fährmann	2007cb2f59	[tests] check extractor category values	2024-01-19 14:21:09 +01:00
Mike Fährmann	93b4120e77	[gelbooru] support 'all' and empty tag (#5076)	2024-01-18 21:49:33 +01:00
Mike Fährmann	a416d4c3d5	[sankaku] support post URLs with alphanumeric IDs (#5073)	2024-01-18 16:23:14 +01:00
Mike Fährmann	ea553a1d55	[wikimedia] generalize (#1443) ... - support mediawiki.org - support mariowiki.com (#3660) - combine code into a single extractor (use prefix as subcategory) - handle non-wiki instances - unescape titles	2024-01-18 15:36:16 +01:00
Mike Fährmann	c3c1635ef3	[wikimedia] update ... - rewrite using BaseExtractor - support most Wiki* domains - update docs/supportedsites - add tests	2024-01-17 00:08:06 +01:00
Mike Fährmann	3d68eda4ab	[kemonoparty] add 'revision_hash' metadata (#4706, #4727, #5013) ... A SHA1 hexdigest of other relevant metadata fields like title, content, file and attachment URLs. This value does NOT reflect which revisions are listed on the website. Neither does 'edited' or any other metadata field (combinations).	2024-01-16 00:38:10 +01:00
Mike Fährmann	799a8206ad	merge #5061: [webtoons] extract more metadata ... - author_name - comic_name - episode_name - username	2024-01-15 18:27:12 +01:00
Mike Fährmann	8ffa0cd3c8	[webtoons] small optimization ... don't extract the entire 'author_area' and avoid creating a second 'text.extract_from()' object	2024-01-15 18:24:47 +01:00
Mike Fährmann	68196589c4	[2ch] update ... - simplify extractor code - more metadata - add tests	2024-01-15 04:09:05 +01:00
Mike Fährmann	69726fc82c	[tests] skip tests requiring auth when non is provided	2024-01-14 22:47:16 +01:00
blankie	bb446b1598	[webtoons] extract more metadata	2024-01-14 19:26:49 +11:00
Mike Fährmann	355b909f46	merge #5041: [steamgriddb] add support (#5033)	2024-01-13 00:59:15 +01:00
Mike Fährmann	71e2c3e5a2	merge #5037: [hatenablog] add support (#5036)	2024-01-13 00:57:21 +01:00
Mike Fährmann	b97af09e03	[tests] include URL in failure report	2024-01-12 03:23:21 +01:00
Mike Fährmann	58e0665fbc	[tests] load config from external file	2024-01-12 03:21:44 +01:00
Mike Fährmann	2dcfb012ea	[patreon] download 'm3u8' manifests with ytdl	2024-01-12 02:33:27 +01:00
Mike Fährmann	2191e29e14	[nijie] fix image URL for single image posts (#5049)	2024-01-11 05:07:38 +01:00
Mike Fährmann	39904c9e4e	[deviantart:avatar] add 'formats' option (#4995)	2024-01-10 17:13:34 +01:00
Mike Fährmann	887ade30a5	[batoto] support more mirror domains (#5042)	2024-01-09 18:02:49 +01:00
blankie	2ccb7d3bd3	[steamgriddb] add support	2024-01-09 17:12:56 +11:00
blankie	2cfe788f93	[hatenablog] fix extractor naming errors	2024-01-09 01:42:57 +11:00
blankie	61f3b2f820	[hatenablog] add support	2024-01-09 01:29:47 +11:00
Mike Fährmann	657ed93a22	[batoto] improve v2 manga URL pattern ... and add tests	2024-01-07 22:23:30 +01:00
Mike Fährmann	33f228756a	[mangadex] add 'list' extractor (#5025) ... supports listing manga and chapters from list feed	2024-01-07 02:59:35 +01:00
Mike Fährmann	c25bdbae91	[komikcast] fix 'manga' extractor (#5027)	2024-01-06 14:19:44 +01:00
Mike Fährmann	8e1a2b5446	[komikcast] update domain to 'komikcast.lol' (#5027)	2024-01-06 02:16:43 +01:00
Mike Fährmann	a441249ea2	merge #4979: [batoto] add 'chapter' and 'manga' extractors (#1434, #2111)	2024-01-06 01:53:26 +01:00
Mike Fährmann	b11c352d66	[bato] rename to 'batoto' ... to use the same category name as the previous bato.to site	2024-01-06 01:49:34 +01:00
Mike Fährmann	3aa24c3744	[bato] simplify and update	2024-01-06 01:10:04 +01:00
Mike Fährmann	11150a7d72	[nudecollect] remove module	2024-01-05 21:32:04 +01:00
Mike Fährmann	c158927c38	merge #5016: [zzup] add 'gallery' extractor (#4517, #4604, #4659, #4863)	2024-01-05 21:25:46 +01:00
Mike Fährmann	217fa7f8a1	include 'test/results' in flake8 checks	2024-01-05 18:16:33 +01:00
Mike Fährmann	e61f016465	[szurubooru] support 'snootbooru.com' (#5023)	2024-01-05 17:56:39 +01:00
Mike Fährmann	b4bcf40278	[weibo] fix AttributeError in 'user' extractor (#5022) ... yet another bug caused by a383eca7	2024-01-05 17:18:33 +01:00
Mike Fährmann	0ab0a10d2d	[jpgfish] update domain	2024-01-05 02:27:20 +01:00
enduser420	0f30136109	[zzup] add 'gallery' extractor	2024-01-04 21:38:59 +05:30
Mike Fährmann	7eaf648f2e	[fanbox] add 'metadata' option (#4921) ... extracts 'plan' and extended 'user' metadata	2024-01-04 15:01:33 +01:00
Mike Fährmann	4f3671458e	[deviantart] add 'avatar' and 'background' extractors (#4995)	2024-01-03 00:07:55 +01:00
Mike Fährmann	63f649cd92	[idolcomplex] fix extraction & update URL patterns (#5002)	2024-01-01 17:38:32 +01:00
Mike Fährmann	7aa1c9671b	[tests] fix 'invalid escape sequence' warnings	2024-01-01 16:12:43 +01:00
Mike Fährmann	b6903a4c90	[nijie] add 'count' metadata field ... https://github.com/mikf/gallery-dl/issues/146#issuecomment-1812849102	2023-12-30 22:25:59 +01:00
Mike Fährmann	b93b351db9	merge #4962: [poringa] add support (#4675)	2023-12-30 20:39:35 +01:00
Mike Fährmann	9f21c839ad	[poringa] improvements and fixes ... - add 'num' and 'count' metadata fields - prevent crash for "private" posts - prevent crash when there's no 'main-info' - update tests	2023-12-30 20:37:09 +01:00
Mike Fährmann	caceb14fc2	[tests] fail when a results file contains syntax errors ... or is otherwise not importable	2023-12-30 17:26:57 +01:00
Mike Fährmann	085411f3f1	[rule34] recognize URLs with 'www' subdomain (#4984)	2023-12-30 16:07:56 +01:00
Antonio	e348da7a06	[poringa] add support	2023-12-27 00:07:23 -06:00
bug-assassin	74c225f94e	[bato] add support	2023-12-26 22:33:33 -05:00
Mike Fährmann	f9544194c0	[paheal] restore 'extension' metadata (#4976)	2023-12-26 16:09:26 +01:00
Mike Fährmann	77d46e6f0c	[lynxchan] update 'bbw-chan' domain (#4970)	2023-12-25 15:29:05 +01:00
Mike Fährmann	108c978073	merge #4919: [postmill] add support (#4917)	2023-12-23 15:23:56 +01:00
Mike Fährmann	2a60645095	[deviantart] set 'is_original' for intermediary URLs to 'false'	2023-12-22 14:49:10 +01:00
Mike Fährmann	01bb75f6cb	merge #4945: {shimmie2[ support 'rule34hentai.net' (#861, #4789)	2023-12-22 00:10:26 +01:00
Mike Fährmann	79e4606893	[rule34hentai] cleanup ... - fix using 'self._posts_rule34hentai' - fix 'file_url' for posts - update docs/supportedsites - add tests	2023-12-22 00:01:36 +01:00
Mike Fährmann	627ed794a2	[danbooru] provide 'tags' as list (#4942) ... keep the old 'tag_string' values around, similar to sankaku a lot of repeat code ... would be a lot less bad if "".split(" ") returned an empty list	2023-12-21 14:39:38 +01:00
Mike Fährmann	99aa923322	[inkbunny] improve '/submissionsviewall.php' patterns (#4934) ... allow 'mode=…' to be in any position don't require it to be somewhere in the middle	2023-12-16 19:21:20 +01:00
Mike Fährmann	3f9c113d78	[mastodon] Support non-numeric status IDs (#4936)	2023-12-16 01:52:31 +01:00
Mike Fährmann	2852404e49	[inkbunny] add 'unread' extractor (#4934)	2023-12-15 21:20:12 +01:00
Mike Fährmann	a37b7759bc	[myhentaigallery] recognize '/g/' URLs (#4920)	2023-12-12 20:02:28 +01:00
blankie	fbe14a2745	[postmill] add support	2023-12-12 21:36:52 +11:00
Mike Fährmann	bf74eb5c46	merge #4886: [urlgalleries] add 'gallery' extractor (#919, #1184, #2905)	2023-12-08 22:55:58 +01:00
Mike Fährmann	ade93c5397	[urlgalleries] add tests	2023-12-08 22:55:16 +01:00
Mike Fährmann	4eb3590103	[nijie] fix image URLs of multi-image posts (#4876)	2023-12-05 17:48:50 +01:00
Mike Fährmann	c83fbe6c2d	merge #4855: [nitter] fix video extraction (#4853)	2023-11-27 18:39:05 +01:00
Mike Fährmann	1137d72d48	[tests] skip test_init for BaseExtractor classes without instances	2023-11-27 18:36:15 +01:00
Mike Fährmann	625e94fa7d	update extractor test results ... still not everything, but good enough for now	2023-11-27 18:30:53 +01:00
enduser420	1e9bacd169	[nitter] fix video extraction	2023-11-27 21:58:06 +05:30
Mike Fährmann	95c1dfb089	[tests] swap assertEqual argument order ... before this, it would show test failures as + test value - extracted value when it should be the other way round	2023-11-27 01:06:13 +01:00
Mike Fährmann	bdb3ce7217	[foolslide] remove 'powermanga.org'	2023-11-26 23:19:05 +01:00
Mike Fährmann	f9dac43be9	[warosu] fix file URLs	2023-11-24 02:44:55 +01:00
Mike Fährmann	645b4627ef	[sankaku] update URL patterns	2023-11-24 02:41:52 +01:00
Mike Fährmann	119755a5a3	[tests] implement skipping/failing tests when pressing ctrl+c	2023-11-24 00:48:37 +01:00
Mike Fährmann	1ae43d8123	merge #4841: [fapello] support '.su' TLD (#4840)	2023-11-22 20:18:32 +01:00
Mike Fährmann	e1404827a6	[pixeldrain] add 'file' and 'album' extractors (#4839)	2023-11-22 19:01:19 +01:00
enduser420	2402162e8a	[fapello] support '.su' TLD	2023-11-22 19:35:43 +05:30
Mike Fährmann	725c8dd55a	[tmohentai] 'categories' -> 'genres' ... quite likely that the site meant 'genres' by "Genders"	2023-11-21 22:11:43 +01:00
Mike Fährmann	ce7c4cb544	merge #4832: [tmohentai] add 'gallery' extractor (#4808)	2023-11-21 20:25:49 +01:00
Mike Fährmann	c4a201ed42	[tmohentai] simplify + tests	2023-11-21 20:24:07 +01:00
Mike Fährmann	e17a48fe56	[blogger] inherit from BaseExtractor ... - support www.micmicidol.club (#4759)	2023-11-21 16:52:25 +01:00
Mike Fährmann	0fa85360a0	merge #4812: [erome] add 'count' metadata field	2023-11-20 22:42:02 +01:00
Mike Fährmann	a43cf78bb7	[erome] tests	2023-11-20 22:41:12 +01:00
Mike Fährmann	07cb584231	[behance] add 'modules' option (#4799)	2023-11-17 22:54:38 +01:00
Mike Fährmann	ea78f67860	[downloader:http] skip files not passing filesize-min/-max (#4821) ... instead of failing the download	2023-11-17 22:54:20 +01:00
Mike Fährmann	3f591d5a4e	[mastodon] update test results	2023-11-11 21:24:07 +01:00
Mike Fährmann	6402f2950f	[pp:metadata] ignore non-string tag values (#4764)	2023-11-04 17:33:14 +01:00
Mike Fährmann	007c433677	[patreon] support 'id:<campaign_id>' in place of a user name ... https://patreon.com/id:12345 … and remove 'campaign-id' config option	2023-11-04 00:17:41 +01:00
Mike Fährmann	43a3d93467	merge #4755: [twitter] recognize fixupx.com URLs	2023-11-02 15:33:29 +01:00
Mike Fährmann	cdf77e326f	[twitter] add test for fixupx.com	2023-11-02 15:32:48 +01:00
Mike Fährmann	fc8f86bf24	[hitomi] recognize 'imageset' gallery URLs (#4756)	2023-11-02 15:29:44 +01:00
Mike Fährmann	72b18d701f	represent util.NONE as 'null' in JSON output ... was '"None"' before	2023-11-02 15:23:28 +01:00
Mike Fährmann	68e72a836c	[exhentai] fix extraction (#4730) ... - update to new API response layout - use proper API server URL - fix 'filesize' metadata	2023-10-30 13:38:49 +01:00
Mike Fährmann	fd8f58ad76	[behance] unescape embed URLs (#4742)	2023-10-30 13:38:49 +01:00
Mike Fährmann	c9a2be36d4	[sankaku] support '/posts/' tag search URLs (#4740)	2023-10-29 13:48:42 +01:00
Mike Fährmann	218295a4c6	[twitter] fix avatars without 'date' information (#4696)	2023-10-27 17:58:02 +02:00
Mike Fährmann	d0effcae20	[kemonoparty] add 'revision_index' metadata field (#4727)	2023-10-26 22:26:38 +02:00
Mike Fährmann	3bbaa875f1	[kemonoparty] fix parsing of non-standard 'dates' (#4676)	2023-10-26 21:50:18 +02:00
Mike Fährmann	a09df34bcf	merge #4714: [4archive] add 'thread' and 'board' extractors ... (#1262, #2418, #4400, #4710)	2023-10-25 20:12:07 +02:00
enduser420	acb713b95a	[4archive] update	2023-10-25 23:08:45 +05:30
Mike Fährmann	6766877524	merge #4693: [reddit] support Reddit Mobile share links	2023-10-25 17:54:32 +02:00
Mike Fährmann	1042278bec	[misskey] support 'misskey.design' (#4713)	2023-10-25 17:47:03 +02:00
enduser420	c0714d5585	[4archive] add 'thread' and 'board' extractors	2023-10-24 23:05:28 +05:30
inty	b68aad3dab	[reddit] implement Reddit Mobile share links	2023-10-22 10:38:05 +00:00
Mike Fährmann	7958ab1946	[newgrounds] support 'imageData' files (#4642)	2023-10-21 13:22:55 +02:00
Mike Fährmann	b52fd91ac6	[sankaku] support '/posts/' URLs (#4688)	2023-10-21 13:20:35 +02:00
Mike Fährmann	b2c3db3e24	[bunkr] add extractor for media URLs (#4684)	2023-10-20 15:22:44 +02:00
Mike Fährmann	6e830ffc9e	[kemonoparty] support post searches (#3385, #4057)	2023-10-19 23:06:06 +02:00
Mike Fährmann	aaf539009b	[kemonoparty] initial support for post revisions (#4498, #4597) ... - single revision https://kemono.party/SERVICE/user/12345/post/12345/revision/12345 - all revisions https://kemono.party/SERVICE/user/12345/post/12345/revisions	2023-10-19 22:32:51 +02:00
Mike Fährmann	174191cb79	[kemonoparty] restore discord pagination (#4676)	2023-10-19 21:57:27 +02:00
Mike Fährmann	c9a976d8a6	[kemonoparty] various updates and fixes (#4676, #4681) ... - fix pagination - fix 'date' metadata - fix discord channel API endpoint	2023-10-19 17:36:16 +02:00