Mike Fährmann
ece73b5b2a
make 'path' and 'keywords' available in logging messages
...
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.
__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.
(#574 , #575 )
2020-05-18 19:04:51 +02:00
Mike Fährmann
4a3d2405de
[postprocessor:ugoira] small optimization
...
Use tuples instead of lists when extending the list of
command-line arguments.
2020-02-07 17:59:28 +01:00
Mike Fährmann
254f7c3999
implement a post-processor module to compare file versions
...
(#530 )
2020-01-18 21:08:25 +01:00
Mike Fährmann
760b9b4db4
add remove_file() and remove_directory() helpers
...
these functions call os.unlink() or os.rmdir()
while catching and suppressing potential OSErrors
2020-01-18 00:21:26 +01:00
Mike Fährmann
87c8b89ddd
[postprocessor:metadata] add 'directory' option ( #520 )
2020-01-02 20:58:10 +01:00
Mike Fährmann
63e6993716
merge 'bypost' functionality into metadata postprocessor
2019-12-16 17:19:23 +01:00
Gio
c0b9ad678d
Separate metadata from handle_url into handle_metadata, commenting
2019-12-09 16:02:15 -06:00
Gio
c20bb5c338
Naming convention, as per travis.
2019-12-09 01:05:01 -06:00
Gio
cfc70a97ab
Added an additional channel for downloading the metadata of an entire post or gallery.
2019-12-09 00:56:27 -06:00
Mike Fährmann
26d2334550
[postprocessor:metadata] rename 'format' to 'content-format'
...
Just to be consistent with the other 'extension-format' option name,
and only 'format' is also still accepted.
2019-11-30 17:27:49 +01:00
Mike Fährmann
a412531451
[postprocessor:metadata] implement 'extension-format' option
...
closes #477
2019-11-30 17:26:17 +01:00
Mike Fährmann
3fc1e12949
[postprocessor:metadata] filter private entries
...
i.e. keys starting with an underscore
2019-11-21 16:58:44 +01:00
Mike Fährmann
9e88e7a344
[postprocessor:exec] improve ( #421 , #413 )
...
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
2019-11-03 21:45:45 +01:00
Mike Fährmann
ad81c07204
[postprocessor] match logger names of downloader modules
...
The logger name for a postprocessor object got changed to
"postprocessor.<module-name>" instead of just
"postprocessor"
2019-10-06 23:30:18 +02:00
Mike Fährmann
03bc8adfc7
[postprocessor:exec] run after file moved to target location
...
(#421 )
2019-10-06 23:12:22 +02:00
Mike Fährmann
35958bebd4
[postprocessor:exec] fix filename quoting on Windows ( #421 )
2019-10-06 15:09:00 +02:00
Mike Fährmann
b06c372e4d
[postprocessor:exec] improve; add command-line option ( #421 )
2019-10-05 23:46:55 +02:00
Mike Fährmann
5fcebb69c2
[postprocessor:ugoira] improve error messages ( #406 )
2019-08-31 21:55:42 +02:00
Mike Fährmann
60c8e090da
[postprocessor:zip] fix archive names ( closes #397 )
...
Remove the trailing path separator introduced in 3284c62
2019-08-24 23:14:26 +02:00
Mike Fährmann
3284c62f22
ensure PathFormat.directory ends with a path separator
...
... plus some other small optimizations
2019-08-20 00:25:13 +02:00
Mike Fährmann
2495b99347
[postprocessor:classify] improve path generation ( fixes #138 )
...
It still doesn't work for converted ugoira animations thanks to how
those files are handled, but everything else, including files with
unknown or changing file extension, now works as it should.
2019-08-19 23:05:28 +02:00
Mike Fährmann
5f8621b29d
improve output of active post processor modules
2019-08-15 13:31:04 +02:00
Mike Fährmann
0bb873757a
update PathFormat class
...
- change 'has_extension' from a simple flag/bool to a field that
contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306 )
2019-08-12 21:40:37 +02:00
Mike Fährmann
a90280f4e7
[postprocessor:zip] add 'mode' option ( #355 )
2019-07-29 16:51:26 +02:00
Mike Fährmann
c41ff9441e
improve find() for downloaders and postprocessors
2019-07-15 16:33:03 +02:00
Mike Fährmann
16c582aaf9
implement 'mtime' post-processor ( #332 )
...
This can set a file's modification time according to a UNIX timestamp
or a datetime object from its metadata.
2019-07-14 22:39:17 +02:00
Mike Fährmann
a01f99728c
[postprocessor:zip] delete empty archives when done ( #316 )
2019-06-19 18:14:33 +02:00
Mike Fährmann
b1db194c14
[reactor] update and improve
...
- split 'tags' into a list
- parse 'date' into a datetime object
- fix webm/mp4 URLs
2019-05-09 23:24:49 +02:00
Mike Fährmann
523ebc9b0b
Fix serialization of 'datetime' objects in '--write-metadata'
...
Simplified universal serialization support in json.dump() can be achieved
by passing 'default=str', which was already the case in DataJob.run()
for -j/--dump-json, but not for the 'metadata' post-processor.
This commit introduces util.dump_json() that (more or less) unifies the
JSON output procedure of both --write-metadata and --dump-json.
(#251 , #252 )
2019-05-09 16:49:22 +02:00
Mike Fährmann
7973419b54
restrict downloader and postprocessor module imports
2019-04-16 18:09:30 +02:00
Mike Fährmann
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes ( #135 )
2019-01-17 21:18:12 +01:00
Mike Fährmann
5f38ac9609
[postprocessor:exec] add a better error message ( #155 )
2019-01-13 13:59:11 +01:00
Mike Fährmann
38500ad697
[postprocessor:metadata] first implementation ( #135 )
2019-01-05 16:39:05 +01:00
Mike Fährmann
79f6755c60
[postprocessor:classify] handle missing "extension" ( #138 )
2018-12-11 20:10:02 +01:00
Mike Fährmann
d3d7f01543
add 'prepare()' step for post-processors
...
This allows post-processors to modify the destination path before
checking if a file already exists.
2018-10-18 22:32:03 +02:00
Mike Fährmann
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection
2018-09-24 13:12:54 +02:00
Mike Fährmann
55f5c87160
[postprocessor:ugoira] add 'libx264-prevent-odd' option
...
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
2018-09-21 20:01:14 +02:00
Mike Fährmann
e9ae6fd080
improve downloader/postprocessor module loading
...
- handle arguments of any type without propagating an exception
- prevent potential security risk through relative imports
2018-09-05 16:39:40 +02:00
Mike Fährmann
6ecb36d88c
[postprocessor:ugoira] add 'ffmpeg-output' option
2018-08-31 17:37:35 +02:00
Mike Fährmann
02a4a67f6d
[postprocessor:ugoira] support danbooru sources
2018-08-27 20:58:45 +02:00
Mike Fährmann
54a0d72dc8
[postprocessor:ugoira] improve frame rate handling
...
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.
This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
2018-07-23 21:33:20 +02:00
Mike Fährmann
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present
2018-06-22 13:26:17 +02:00
Mike Fährmann
0c1c4557dd
[postprocessor:ugoira] add option for two-pass encoding
2018-06-20 18:48:10 +02:00
Mike Fährmann
eb7a1f3b98
[pixiv] rework ugoira handling
...
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
2018-06-18 17:57:57 +02:00
Mike Fährmann
613b692275
[postprocessor:ugoira] add a few options
...
- ffmpeg-location: path to the ffmpeg (or avconv) executable
- ffmpeg-args: additional command line args for ffmpeg
- extension: filename extension of the resulting video file
2018-06-16 12:03:53 +02:00
Mike Fährmann
a444755979
[postprocessor] add 'ugoira' to convert pixiv animations to webm
2018-06-15 20:28:59 +02:00
Mike Fährmann
eec081dd3e
[postprocessor:zip] delete directory ( #85 )
2018-06-11 18:08:12 +02:00
Mike Fährmann
baccf8a958
improve postprocessor handling
...
- add pathfmt argument for __init__()
- add finalization step
- add option to keep or delete zipped files
2018-06-08 17:39:02 +02:00
Mike Fährmann
2628911ba0
[pp:exec] add 'async' option
2018-06-07 23:35:18 +02:00
Mike Fährmann
37d97ff02c
[pp:classify] use temppath
2018-06-06 21:11:20 +02:00
