158 Commits

Author	SHA1	Message	Date
Mike Fährmann	d83b95fd28	[postprocessor:metadata] accept a string-list for 'content-format' ... (closes #1080)	2020-10-27 20:09:58 +01:00
Mike Fährmann	fd0685d9b5	[postprocessor:zip] defer zip file creation (fixes #968) ... don't try to create zip files on postprocessor construction, wait until directory creation during file download,	2020-08-31 21:53:18 +02:00
Mike Fährmann	c8787647ed	add global WINDOWS bool	2020-05-19 22:32:53 +02:00
Mike Fährmann	ece73b5b2a	make 'path' and 'keywords' available in logging messages ... Wrap all loggers used by job, extractor, downloader, and postprocessor objects into a (custom) LoggerAdapter that provides access to the underlying job, extractor, pathfmt, and kwdict objects and their properties. __init__() signatures for all downloader and postprocessor classes have been changed to take the current Job object as their first argument, instead of the current extractor or pathfmt. (#574, #575)	2020-05-18 19:04:51 +02:00
Mike Fährmann	4a3d2405de	[postprocessor:ugoira] small optimization ... Use tuples instead of lists when extending the list of command-line arguments.	2020-02-07 17:59:28 +01:00
Mike Fährmann	254f7c3999	implement a post-processor module to compare file versions ... (#530)	2020-01-18 21:08:25 +01:00
Mike Fährmann	760b9b4db4	add remove_file() and remove_directory() helpers ... these functions call os.unlink() or os.rmdir() while catching and suppressing potential OSErrors	2020-01-18 00:21:26 +01:00
Mike Fährmann	87c8b89ddd	[postprocessor:metadata] add 'directory' option (#520)	2020-01-02 20:58:10 +01:00
Mike Fährmann	63e6993716	merge 'bypost' functionality into metadata postprocessor	2019-12-16 17:19:23 +01:00
Gio	c0b9ad678d	Separate metadata from handle_url into handle_metadata, commenting	2019-12-09 16:02:15 -06:00
Gio	c20bb5c338	Naming convention, as per travis.	2019-12-09 01:05:01 -06:00
Gio	cfc70a97ab	Added an additional channel for downloading the metadata of an entire post or gallery.	2019-12-09 00:56:27 -06:00
Mike Fährmann	26d2334550	[postprocessor:metadata] rename 'format' to 'content-format' ... Just to be consistent with the other 'extension-format' option name, and only 'format' is also still accepted.	2019-11-30 17:27:49 +01:00
Mike Fährmann	a412531451	[postprocessor:metadata] implement 'extension-format' option ... closes #477	2019-11-30 17:26:17 +01:00
Mike Fährmann	3fc1e12949	[postprocessor:metadata] filter private entries ... i.e. keys starting with an underscore	2019-11-21 16:58:44 +01:00
Mike Fährmann	9e88e7a344	[postprocessor:exec] improve (#421, #413) ... - add 'final' option - include job status in pp finalization - improve and extend documentation	2019-11-03 21:45:45 +01:00
Mike Fährmann	ad81c07204	[postprocessor] match logger names of downloader modules ... The logger name for a postprocessor object got changed to "postprocessor.<module-name>" instead of just "postprocessor"	2019-10-06 23:30:18 +02:00
Mike Fährmann	03bc8adfc7	[postprocessor:exec] run after file moved to target location ... (#421)	2019-10-06 23:12:22 +02:00
Mike Fährmann	35958bebd4	[postprocessor:exec] fix filename quoting on Windows (#421)	2019-10-06 15:09:00 +02:00
Mike Fährmann	b06c372e4d	[postprocessor:exec] improve; add command-line option (#421)	2019-10-05 23:46:55 +02:00
Mike Fährmann	5fcebb69c2	[postprocessor:ugoira] improve error messages (#406)	2019-08-31 21:55:42 +02:00
Mike Fährmann	60c8e090da	[postprocessor:zip] fix archive names (closes #397) ... Remove the trailing path separator introduced in 3284c62 before adding the archive's filename extension. [ci skip]	2019-08-24 23:14:26 +02:00
Mike Fährmann	3284c62f22	ensure PathFormat.directory ends with a path separator ... ... plus some other small optimizations	2019-08-20 00:25:13 +02:00
Mike Fährmann	2495b99347	[postprocessor:classify] improve path generation (fixes #138) ... It still doesn't work for converted ugoira animations thanks to how those files are handled, but everything else, including files with unknown or changing file extension, now works as it should.	2019-08-19 23:05:28 +02:00
Mike Fährmann	5f8621b29d	improve output of active post processor modules	2019-08-15 13:31:04 +02:00
Mike Fährmann	0bb873757a	update PathFormat class ... - change 'has_extension' from a simple flag/bool to a field that contains the original filename extension - rename 'keywords' to 'kwdict' and some other stuff as well - inline 'adjust_path()' - put enumeration index before filename extension (#306)	2019-08-12 21:40:37 +02:00
Mike Fährmann	a90280f4e7	[postprocessor:zip] add 'mode' option (#355)	2019-07-29 16:51:26 +02:00
Mike Fährmann	c41ff9441e	improve find() for downloaders and postprocessors	2019-07-15 16:33:03 +02:00
Mike Fährmann	16c582aaf9	implement 'mtime' post-processor (#332) ... This can set a file's modification time according to a UNIX timestamp or a datetime object from its metadata.	2019-07-14 22:39:17 +02:00
Mike Fährmann	a01f99728c	[postprocessor:zip] delete empty archives when done (#316)	2019-06-19 18:14:33 +02:00
Mike Fährmann	b1db194c14	[reactor] update and improve ... - split 'tags' into a list - parse 'date' into a datetime object - fix webm/mp4 URLs	2019-05-09 23:24:49 +02:00
Mike Fährmann	523ebc9b0b	Fix serialization of 'datetime' objects in '--write-metadata' ... Simplified universal serialization support in json.dump() can be achieved by passing 'default=str', which was already the case in DataJob.run() for -j/--dump-json, but not for the 'metadata' post-processor. This commit introduces util.dump_json() that (more or less) unifies the JSON output procedure of both --write-metadata and --dump-json. (#251, #252)	2019-05-09 16:49:22 +02:00
Mike Fährmann	7973419b54	restrict downloader and postprocessor module imports	2019-04-16 18:09:30 +02:00
Mike Fährmann	8aba2bdebf	[postprocessor:metadata] add 'tags' and 'custom' modes (#135)	2019-01-17 21:18:12 +01:00
Mike Fährmann	5f38ac9609	[postprocessor:exec] add a better error message (#155)	2019-01-13 13:59:11 +01:00
Mike Fährmann	38500ad697	[postprocessor:metadata] first implementation (#135)	2019-01-05 16:39:05 +01:00
Mike Fährmann	79f6755c60	[postprocessor:classify] handle missing "extension" (#138)	2018-12-11 20:10:02 +01:00
Mike Fährmann	d3d7f01543	add 'prepare()' step for post-processors ... This allows post-processors to modify the destination path before checking if a file already exists.	2018-10-18 22:32:03 +02:00
Mike Fährmann	09d2f3e5e7	[postprocessor:ugoira] improve libx264 detection	2018-09-24 13:12:54 +02:00
Mike Fährmann	55f5c87160	[postprocessor:ugoira] add 'libx264-prevent-odd' option ... A rather crude workaround for "width/height not divisible by 2" errors when using libx264.	2018-09-21 20:01:14 +02:00
Mike Fährmann	e9ae6fd080	improve downloader/postprocessor module loading ... - handle arguments of any type without propagating an exception - prevent potential security risk through relative imports	2018-09-05 16:39:40 +02:00
Mike Fährmann	6ecb36d88c	[postprocessor:ugoira] add 'ffmpeg-output' option	2018-08-31 17:37:35 +02:00
Mike Fährmann	02a4a67f6d	[postprocessor:ugoira] support danbooru sources	2018-08-27 20:58:45 +02:00
Mike Fährmann	54a0d72dc8	[postprocessor:ugoira] improve frame rate handling ... By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped frames if the source requires a higher frame rate than that. This commit adds a `framerate` option (default "auto"), which allows to automatically assign a (more or less) fitting frame rate based on delays between ugoira frames and avoids dropped frames.	2018-07-23 21:33:20 +02:00
Mike Fährmann	4eb94aca17	[postprocessor:ugoira] pass '-f' if not present	2018-06-22 13:26:17 +02:00
Mike Fährmann	0c1c4557dd	[postprocessor:ugoira] add option for two-pass encoding	2018-06-20 18:48:10 +02:00
Mike Fährmann	eb7a1f3b98	[pixiv] rework ugoira handling ... Frame information now gets attached to the ZIP file's keyword dict instead of being written to a separate text file.	2018-06-18 17:57:57 +02:00
Mike Fährmann	613b692275	[postprocessor:ugoira] add a few options ... - ffmpeg-location: path to the ffmpeg (or avconv) executable - ffmpeg-args: additional command line args for ffmpeg - extension: filename extension of the resulting video file	2018-06-16 12:03:53 +02:00
Mike Fährmann	a444755979	[postprocessor] add 'ugoira' to convert pixiv animations to webm	2018-06-15 20:28:59 +02:00
Mike Fährmann	eec081dd3e	[postprocessor:zip] delete directory (#85)	2018-06-11 18:08:12 +02:00
Mike Fährmann	baccf8a958	improve postprocessor handling ... - add pathfmt argument for __init__() - add finalization step - add option to keep or delete zipped files	2018-06-08 17:39:02 +02:00
Mike Fährmann	2628911ba0	[pp:exec] add 'async' option	2018-06-07 23:35:18 +02:00
Mike Fährmann	37d97ff02c	[pp:classify] use temppath	2018-06-06 21:11:20 +02:00
Mike Fährmann	97189e50cd	[pp:zip] use temppath; add options	2018-06-06 20:49:52 +02:00
Mike Fährmann	d0ae3ed52c	[postprocessor] add 'zip' to write files to a ZIP archive ... (#85)	2018-05-22 16:54:17 +02:00
Mike Fährmann	ca4008e1c1	[postprocessor] add 'classify' to sort downloads by fileext	2018-05-22 16:21:17 +02:00
Mike Fährmann	d378c0a323	[postprocessor] add 'exec' to execute user-defined processes	2018-05-22 15:00:31 +02:00
Mike Fährmann	76c32d58e5	[postprocessor] initial code	2018-05-22 14:59:22 +02:00