796 Commits

Author	SHA1	Message	Date
Mike Fährmann	a37b7759bc	[myhentaigallery] recognize '/g/' URLs (#4920)	2023-12-12 20:02:28 +01:00
blankie	fbe14a2745	[postmill] add support	2023-12-12 21:36:52 +11:00
Mike Fährmann	bf74eb5c46	merge #4886: [urlgalleries] add 'gallery' extractor (#919, #1184, #2905)	2023-12-08 22:55:58 +01:00
Mike Fährmann	ade93c5397	[urlgalleries] add tests	2023-12-08 22:55:16 +01:00
Mike Fährmann	4eb3590103	[nijie] fix image URLs of multi-image posts (#4876)	2023-12-05 17:48:50 +01:00
Mike Fährmann	c83fbe6c2d	merge #4855: [nitter] fix video extraction (#4853)	2023-11-27 18:39:05 +01:00
Mike Fährmann	1137d72d48	[tests] skip test_init for BaseExtractor classes without instances	2023-11-27 18:36:15 +01:00
Mike Fährmann	625e94fa7d	update extractor test results ... still not everything, but good enough for now	2023-11-27 18:30:53 +01:00
enduser420	1e9bacd169	[nitter] fix video extraction	2023-11-27 21:58:06 +05:30
Mike Fährmann	95c1dfb089	[tests] swap assertEqual argument order ... before this, it would show test failures as + test value - extracted value when it should be the other way round	2023-11-27 01:06:13 +01:00
Mike Fährmann	bdb3ce7217	[foolslide] remove 'powermanga.org'	2023-11-26 23:19:05 +01:00
Mike Fährmann	f9dac43be9	[warosu] fix file URLs	2023-11-24 02:44:55 +01:00
Mike Fährmann	645b4627ef	[sankaku] update URL patterns	2023-11-24 02:41:52 +01:00
Mike Fährmann	119755a5a3	[tests] implement skipping/failing tests when pressing ctrl+c	2023-11-24 00:48:37 +01:00
Mike Fährmann	1ae43d8123	merge #4841: [fapello] support '.su' TLD (#4840)	2023-11-22 20:18:32 +01:00
Mike Fährmann	e1404827a6	[pixeldrain] add 'file' and 'album' extractors (#4839)	2023-11-22 19:01:19 +01:00
enduser420	2402162e8a	[fapello] support '.su' TLD	2023-11-22 19:35:43 +05:30
Mike Fährmann	725c8dd55a	[tmohentai] 'categories' -> 'genres' ... quite likely that the site meant 'genres' by "Genders"	2023-11-21 22:11:43 +01:00
Mike Fährmann	ce7c4cb544	merge #4832: [tmohentai] add 'gallery' extractor (#4808)	2023-11-21 20:25:49 +01:00
Mike Fährmann	c4a201ed42	[tmohentai] simplify + tests	2023-11-21 20:24:07 +01:00
Mike Fährmann	e17a48fe56	[blogger] inherit from BaseExtractor ... - support www.micmicidol.club (#4759)	2023-11-21 16:52:25 +01:00
Mike Fährmann	0fa85360a0	merge #4812: [erome] add 'count' metadata field	2023-11-20 22:42:02 +01:00
Mike Fährmann	a43cf78bb7	[erome] tests	2023-11-20 22:41:12 +01:00
Mike Fährmann	07cb584231	[behance] add 'modules' option (#4799)	2023-11-17 22:54:38 +01:00
Mike Fährmann	ea78f67860	[downloader:http] skip files not passing filesize-min/-max (#4821) ... instead of failing the download	2023-11-17 22:54:20 +01:00
Mike Fährmann	3f591d5a4e	[mastodon] update test results	2023-11-11 21:24:07 +01:00
Mike Fährmann	6402f2950f	[pp:metadata] ignore non-string tag values (#4764)	2023-11-04 17:33:14 +01:00
Mike Fährmann	007c433677	[patreon] support 'id:<campaign_id>' in place of a user name ... https://patreon.com/id:12345 … and remove 'campaign-id' config option	2023-11-04 00:17:41 +01:00
Mike Fährmann	43a3d93467	merge #4755: [twitter] recognize fixupx.com URLs	2023-11-02 15:33:29 +01:00
Mike Fährmann	cdf77e326f	[twitter] add test for fixupx.com	2023-11-02 15:32:48 +01:00
Mike Fährmann	fc8f86bf24	[hitomi] recognize 'imageset' gallery URLs (#4756)	2023-11-02 15:29:44 +01:00
Mike Fährmann	72b18d701f	represent util.NONE as 'null' in JSON output ... was '"None"' before	2023-11-02 15:23:28 +01:00
Mike Fährmann	68e72a836c	[exhentai] fix extraction (#4730) ... - update to new API response layout - use proper API server URL - fix 'filesize' metadata	2023-10-30 13:38:49 +01:00
Mike Fährmann	fd8f58ad76	[behance] unescape embed URLs (#4742)	2023-10-30 13:38:49 +01:00
Mike Fährmann	c9a2be36d4	[sankaku] support '/posts/' tag search URLs (#4740)	2023-10-29 13:48:42 +01:00
Mike Fährmann	218295a4c6	[twitter] fix avatars without 'date' information (#4696)	2023-10-27 17:58:02 +02:00
Mike Fährmann	d0effcae20	[kemonoparty] add 'revision_index' metadata field (#4727)	2023-10-26 22:26:38 +02:00
Mike Fährmann	3bbaa875f1	[kemonoparty] fix parsing of non-standard 'dates' (#4676)	2023-10-26 21:50:18 +02:00
Mike Fährmann	a09df34bcf	merge #4714: [4archive] add 'thread' and 'board' extractors ... (#1262, #2418, #4400, #4710)	2023-10-25 20:12:07 +02:00
enduser420	acb713b95a	[4archive] update	2023-10-25 23:08:45 +05:30
Mike Fährmann	6766877524	merge #4693: [reddit] support Reddit Mobile share links	2023-10-25 17:54:32 +02:00
Mike Fährmann	1042278bec	[misskey] support 'misskey.design' (#4713)	2023-10-25 17:47:03 +02:00
enduser420	c0714d5585	[4archive] add 'thread' and 'board' extractors	2023-10-24 23:05:28 +05:30
inty	b68aad3dab	[reddit] implement Reddit Mobile share links	2023-10-22 10:38:05 +00:00
Mike Fährmann	7958ab1946	[newgrounds] support 'imageData' files (#4642)	2023-10-21 13:22:55 +02:00
Mike Fährmann	b52fd91ac6	[sankaku] support '/posts/' URLs (#4688)	2023-10-21 13:20:35 +02:00
Mike Fährmann	b2c3db3e24	[bunkr] add extractor for media URLs (#4684)	2023-10-20 15:22:44 +02:00
Mike Fährmann	6e830ffc9e	[kemonoparty] support post searches (#3385, #4057)	2023-10-19 23:06:06 +02:00
Mike Fährmann	aaf539009b	[kemonoparty] initial support for post revisions (#4498, #4597) ... - single revision https://kemono.party/SERVICE/user/12345/post/12345/revision/12345 - all revisions https://kemono.party/SERVICE/user/12345/post/12345/revisions	2023-10-19 22:32:51 +02:00
Mike Fährmann	174191cb79	[kemonoparty] restore discord pagination (#4676)	2023-10-19 21:57:27 +02:00
Mike Fährmann	c9a976d8a6	[kemonoparty] various updates and fixes (#4676, #4681) ... - fix pagination - fix 'date' metadata - fix discord channel API endpoint	2023-10-19 17:36:16 +02:00
Mike Fährmann	bfdc07632a	[deviantart] expand nested comment replies (#4653)	2023-10-17 19:40:53 +02:00
Mike Fährmann	9bc5ad4784	[tests] implement 'len:'	2023-10-17 19:25:31 +02:00
Mike Fährmann	a1977a698e	[tests] fix spurious failures in '_assert_isotime()'	2023-10-16 18:16:48 +02:00
Mike Fährmann	390d14dbcc	[chevereto] support 'img.kiwi' and 'deltaporno.com' (#4664, #1381)	2023-10-16 18:14:30 +02:00
Mike Fährmann	727c8eec6c	merge #4667: [redgifs] fix 'niches' extraction (#4666)	2023-10-16 14:20:01 +02:00
Mike Fährmann	2911ed1240	[chevereto] add generic extractors (#4664) ... - support jpgfish - support pixl.li / pixl.is (#3179, #4357)	2023-10-16 14:15:39 +02:00
enduser420	db3363ac0b	[redgifs] fix 'niches' extraction	2023-10-16 16:51:30 +05:30
Mike Fährmann	ade8347ead	[kemonoparty] fix DM dates	2023-10-15 19:54:28 +02:00
Mike Fährmann	6dfe200ae4	[kemonoparty] support discord URLs with channel IDs (#4662)	2023-10-15 19:45:22 +02:00
Mike Fährmann	c6a3892210	[imgbb] update username extraction (#4626)	2023-10-14 20:55:39 +02:00
Mike Fährmann	13ce3a9acb	[warosu] fix extraction (#4634)	2023-10-13 23:03:39 +02:00
Mike Fährmann	c4c4e4d2f4	[newgrounds] improve 'art-image' extraction (#4642) ... - download files in original resolution - replace .webp with extension of first file	2023-10-13 20:10:55 +02:00
Mike Fährmann	833dce141f	[fantia] add 'content_count' and 'content_num' metadata fields (#4627)	2023-10-13 20:10:55 +02:00
Mike Fährmann	2d41702762	[deviantart] implement '"group": "skip"' (#4630)	2023-10-12 22:14:20 +02:00
Mike Fährmann	a9c3442d4e	[deviantart] add a couple 'deactivated account' test URLs	2023-10-12 21:40:10 +02:00
Mike Fährmann	2974b8e3c8	[moebooru] add 'metadata' option (#4646) ... for extended 'pool' metadata	2023-10-12 21:34:25 +02:00
Mike Fährmann	67ba4ee842	[pp:exec] support more replacement fields for '--exec' (#4633) ... - {_directory} - {_filename} - {_path} (alias for {})	2023-10-09 12:50:10 +02:00
Mike Fährmann	9a008523ac	[hentaifoundry] fix '.swf' file downloads (#4641)	2023-10-09 11:45:55 +02:00
Mike Fährmann	15f940819b	[newgrounds] support 'art-image' files (#4642)	2023-10-09 11:20:10 +02:00
Mike Fährmann	efaab4fbfa	[twitter] fix crash due to missing 'source' (#4620) ... regression caused by 06aaedde	2023-10-04 23:01:04 +02:00
Mike Fährmann	84fbbd96aa	[shimmie2] remove 'meme.museum'	2023-10-02 20:41:25 +02:00
Mike Fährmann	0b150d45db	[tests] add 'msg' arguments to assert statements	2023-10-01 13:52:00 +02:00
Mike Fährmann	27da3f2958	[tests] re-implement filtering by basecategory	2023-10-01 13:31:23 +02:00
Mike Fährmann	c7bd9925d9	[tests] use fallback URLs for content tests (#3163)	2023-09-30 21:00:55 +02:00
Mike Fährmann	b92645cd37	[bunkr] fix extraction (#4514, #4532, #4529, #4540)	2023-09-30 18:05:12 +02:00
Mike Fährmann	bd3f7a5bbc	[tests] support one regex per URL for #pattern	2023-09-28 21:56:09 +02:00
Mike Fährmann	0c5d8b1505	[deviantart] re-add 'quality' option and 'intermediary' transform	2023-09-24 17:36:05 +02:00
Mike Fährmann	dbd820d7c5	[tests] allow checking for exact URL results	2023-09-24 01:52:47 +02:00
Mike Fährmann	642998504d	[tests] support 'range()' for #count and metadata checks	2023-09-24 01:52:40 +02:00
Mike Fährmann	1e31fce37b	[pillowfort] support '/tagged/' URLs (#4570)	2023-09-23 00:11:01 +02:00
Mike Fährmann	1d2fd0b831	[pillowfort] extract 'b2_lg_url' media (#4570)	2023-09-23 00:05:26 +02:00
Mike Fährmann	50e2ebaff0	[danbooru] support 'donmai.moe' URLs	2023-09-22 20:58:38 +02:00
Mike Fährmann	918ba4f847	[redgifs] match gfycat image URLs (#4558)	2023-09-22 18:02:55 +02:00
Mike Fährmann	2cd801232b	fix --range causing crashes (#4557) ... regression caused by a383eca7	2023-09-22 16:28:20 +02:00
Mike Fährmann	27ec653991	fix bug in test_init and update example URLs	2023-09-14 13:27:03 +02:00
Mike Fährmann	24a1d46391	[mastodon] support '/@USER/following' URLs ... Previously, only '/users/USER/following' got matched.	2023-09-13 23:42:51 +02:00
Mike Fährmann	ac00d47a16	update test/test_results.py	2023-09-13 14:54:25 +02:00
Mike Fährmann	65b6011cc5	update test/test_extractor.py	2023-09-11 17:20:06 +02:00
Mike Fährmann	a833c244c8	add exported extractor results	2023-09-10 14:45:01 +02:00
Mike Fährmann	93a7a89cf6	[formatter] use value of last alternative (#4492) ... fixes {fieldname|''} evaluating to the value of 'keywords-default' instead of an empty string	2023-09-05 17:53:27 +02:00
Mike Fährmann	f2de70f254	[gfycat] remove module	2023-09-04 18:27:11 +02:00
Mike Fährmann	d319777a24	[tests] skip 'test_init_ytdl' on Python<3.6 ... It passes without error in a Python 3.4/3.5 venv on my own machine, but fails for some inexplicable reason on Github Actions.	2023-08-10 23:34:49 +02:00
Mike Fährmann	0ef1fcab20	[postprocessor] update 'finalize' events ... Add 'finalize-error' and 'finalize-success' events that trigger depending on whether error(s) did or did not happen. 'finalize' itself now always triggers regardless of error status. (was supposed to have the same behavior as the new 'finalize-success')	2023-08-10 19:46:37 +02:00
Mike Fährmann	d50c312ff0	prevent test failure when there's no 'ytdl' module (#4364) ... split of ytdl into its own test function and skip it when there's an ImportError similar to test_ytdl.py	2023-07-29 13:48:31 +02:00
Mike Fährmann	48ef062867	fix issues with 'Extractor.finalize()' ... - prevent crash in InstagramUserExtractor (#4359) - call it at the end of every DownloadJob - add it to tests	2023-07-29 13:43:27 +02:00
Mike Fährmann	255d08b79e	add test for 'Extractor.initialize()' (#4359)	2023-07-28 16:58:16 +02:00
Mike Fährmann	a383eca7f6	decouple extractor initialization ... Introduce an 'initialize()' function that does the actual init (session, cookies, config options) and can called separately from the constructor __init__(). This allows, for example, to adjust config access inside a Job before most of it already happened when calling 'extractor.find()'.	2023-07-25 22:16:16 +02:00
Mike Fährmann	f0203b7559	[postprocessor:python] add tests	2023-07-24 15:22:57 +02:00
Mike Fährmann	d97b8c2fba	consistent cookie-related names ... - rename every cookie variable or method to 'cookies_*' - simplify '.session.cookies' to just '.cookies' - more consistent 'login()' structure	2023-07-22 01:20:50 +02:00
Mike Fährmann	c5565f79f7	merge #4096: [danbooru] add support for booru.borvar.art instance	2023-07-18 18:33:08 +02:00
Mike Fährmann	63326e3168	[danbooru] add tests for booruvar	2023-07-18 18:29:57 +02:00
Mike Fährmann	5171d8975c	[E621] support 'e6ai.net' (#4320)	2023-07-18 18:16:30 +02:00
Mike Fährmann	7444fc125b	[gfycat] implement login support (#3770, #4271) ... For the record: '/webtoken' and '/weblogin' are not the same ...	2023-07-06 18:56:34 +02:00
Mike Fährmann	25c5a6ffcb	no f-strings	2023-06-25 14:01:26 +02:00
Mike Fährmann	ec64cbefeb	[postprocessor:exec] add tests	2023-06-21 23:54:35 +02:00
Mike Fährmann	ce93c460a6	[formatter] implement 'H' conversion (#4164) ... to remove HTML tags and unescape HTML entities	2023-06-15 13:07:51 +02:00
Mike Fährmann	deff3b434d	[vipergirls] implement login support (#4166)	2023-06-13 21:05:09 +02:00
Mike Fährmann	69865dcc05	[formatter] implement slicing strings as bytes (#4087) ... prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes the string to bytes in filesystem encoding before applying the slice	2023-05-22 18:30:45 +02:00
Mike Fährmann	df11214281	[ytdl] improve --xff/--geo-bypass detection (#3989) ... check if --xff is supported in a try-except block and select expected results accordingly	2023-05-01 18:26:37 +02:00
Mike Fährmann	aa731c4298	[ytdl] run yt-dlp tests with latest code from master (#3989) ... Only use PyPI version for Python 3.6, since that's no longer supported by the current codebase.	2023-05-01 16:42:57 +02:00
Mike Fährmann	43f4bd9faa	[ytdl] fix tests ... tests pass with latest Git HEAD, but not with the current PyPI version	2023-04-29 18:05:45 +02:00
Mike Fährmann	61a65d5bb9	[ytdl] fix crash due to --geo-bypass deprecation (#3975)	2023-04-29 17:25:38 +02:00
Mike Fährmann	a96745368e	"fix" tests on Python 3.4 and 3.5 ... can't rely on dict insertion order	2023-04-26 19:31:27 +02:00
Mike Fährmann	3905f05f00	[postprocessor:metadata] support putting keys in quotes ... for mode 'modify' and 'delete' based on fe41a2b1	2023-04-25 14:30:18 +02:00
Mike Fährmann	7459e4abce	[postprocessor:metadata] fix traversing more than 1 level deep ... for mode 'modify' and 'delete'	2023-04-25 14:17:25 +02:00
Mike Fährmann	2edcdee32f	[downloader:http] add MIME type and signature for .heic files ... (#3915) https://github.com/strukturag/libheif/issues/83	2023-04-15 17:09:22 +02:00
Mike Fährmann	082d55de16	fix circular reference detection for -K	2023-03-21 23:46:36 +01:00
Mike Fährmann	2ab66ad899	update -K output to include quotes around keys	2023-03-21 22:28:04 +01:00
Mike Fährmann	fe41a2b159	[formatter] support putting keys in quotes ... i.e. obj["key"] or obj['key'] as in f-strings	2023-03-21 22:06:54 +01:00
Mike Fährmann	46fdf46f21	[formatter] support loading an f-string from a template file ... "\fTF ~/path/to/file.txt"	2023-03-20 22:05:33 +01:00
Mike Fährmann	1a4d4a799b	[formatter] support filesystem paths for \fM	2023-03-20 22:01:33 +01:00
Mike Fährmann	00f0233b28	[postprocessor:metadata] add 'skip' option (#3786)	2023-03-17 23:30:11 +01:00
Mike Fährmann	8f8b4de0e8	[ytdl] fix '--parse-metadata' (#3663)	2023-03-05 19:57:23 +01:00
Mike Fährmann	7610d9cf82	merge #3675: [pixiv] fix --write-tags for '"tags": "original"'	2023-03-02 21:48:31 +01:00
Mike Fährmann	83e7a25b6b	extend OAuth tests	2023-03-02 17:26:51 +01:00
Mike Fährmann	d788e6c60c	implement 'globals' option	2023-02-28 18:18:55 +01:00
Mike Fährmann	56039d2456	add 'hash_md5' and 'hash_sha1' functions (#3679) ... ... to global eval namespace	2023-02-22 10:58:44 +01:00
Mike Fährmann	e1df7f73b1	[deviantart] add 'search' extractor ... (#538, #1264, #2954, #2970, #3577) Requires login to fetch any results, since the API endpoint raises an error for not logged in requests. TODO: parse HTML search results	2023-02-20 20:54:46 +01:00
Gray Manley	38a6389e2c	Fix lint.	2023-02-20 00:33:30 -06:00
Gray Manley	56cbae92ec	Use more pythony naming.	2023-02-19 06:14:34 -06:00
Gray Manley	8e2ba4f32e	Add test.	2023-02-19 06:13:21 -06:00
Mike Fährmann	dd884b02ee	replace json.loads with direct calls to JSONDecoder.decode	2023-02-09 15:22:00 +01:00
Mike Fährmann	b7337d810e	[postprocessor:metadata] add 'sort' and 'separators' options	2023-02-07 18:28:14 +01:00
Mike Fährmann	3436c6b117	[postprocessor:metadata] speed up JSON encoding	2023-02-06 12:35:28 +01:00
Mike Fährmann	925b467496	split e621 from danbooru module (#3425)	2023-02-03 19:24:31 +01:00
Mike Fährmann	c2bc70593e	implement ability to load external extractor classes ... - -X/--extractors - extractor.module-sources	2023-01-30 23:10:10 +01:00
ClosedPort22	b6706b373a	[downloader:http] add signature checks for some formats ... also add the MIME type for .obj files	2023-01-15 23:40:55 +08:00
Mike Fährmann	71d3143c35	fix bug in test_extractors.py ... pattern matching tests would succeed if there is exactly one match but for the wrong extractor	2023-01-08 15:35:05 +01:00
Mike Fährmann	fa144f38ed	[ytdl} fix dfe4f00c for legacy yt-dlp	2023-01-04 21:42:22 +01:00
Mike Fährmann	dfe4f00ca2	[ytdl] update for yt-dlp changes	2023-01-04 13:12:24 +01:00
Mike Fährmann	d651d45239	implement specifying ranges in slice notation (#918, #2865) ... e.g. - '1:101' or ':101' or ':101:' for files 1 to 100 - '1::2' or '::2' for every second file - '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96 (the second argument specifies the first index NOT included)	2022-12-27 18:21:12 +01:00
Mike Fährmann	3616adfc75	implement '--range' with Python ranges	2022-12-26 18:32:34 +01:00
Mike Fährmann	1800bd7d14	allow '*-filter' options to be a list of expressions	2022-12-23 22:20:21 +01:00
Mike Fährmann	43c211f1a7	extend and rename util.CustomNone	2022-12-06 22:08:51 +01:00
Mike Fährmann	42481aed59	[formatter] implement 'S' format specifier (#3266) ... to Sort lists	2022-11-21 21:44:42 +01:00
Mike Fährmann	6e08ad26f7	update downloader tests	2022-11-16 22:59:18 +01:00
Mike Fährmann	05255f5be0	add 'default' argument to 'text.extr()'	2022-11-09 11:00:32 +01:00
Mike Fährmann	8124c16a50	split 'build_path' from 'set_filename' and 'set_extension' ... Do not automatically build a new path when setting file metadata or updating its extension.	2022-11-08 17:03:24 +01:00
Mike Fährmann	eb33e6cf2d	add 'text.extr()' ... a stripped-down version of text.extract() that - always returns a string (like 'extract_from') - only returns a string - does not deal with 'pos' arguments - is ~20% faster	2022-11-04 21:37:36 +01:00