1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 11:12:40 +01:00
Commit Graph

137 Commits

Author SHA1 Message Date
Mike Fährmann
91308140ec
make 'generate_token()' compatible with Python 3.4 2021-01-14 03:48:10 +01:00
Mike Fährmann
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()'
and add a 'size' argument
2021-01-11 22:12:40 +01:00
Mike Fährmann
79501a356f
fix crash when 'path-restrict' is an object/dict
This basically reverts commit 5818c928

(#1234)
2021-01-10 00:13:48 +01:00
Mike Fährmann
5d4494b15f
add "ascii" as a special 'path-restrict' value 2021-01-09 02:41:20 +01:00
Mike Fährmann
5818c928c4
refactor 'path-restrict' parsing 2021-01-09 02:33:42 +01:00
Mike Fährmann
aac00a2024
add 'd' conversion for format strings
to convert a timestamp to a formattable 'datetime' object.

For example '{created_at!d:%Y-%m-%d}'
transforms the timestamp in 'created_at' into a 'datetime' object
and then formats its content using '%Y-%m-%d' as template.

1262304000 -> datetime(2010, 1, 1) -> "2010-01-01"
2021-01-09 01:58:44 +01:00
Mike Fährmann
511d8d3fa3
increase SQLite connection timeouts (#1173) 2020-12-19 20:15:07 +01:00
Mike Fährmann
9b1bd09454
change 'extension-map' default
Replace all JPEG filename extensions with 'jpg'.
2020-11-14 22:40:31 +01:00
Mike Fährmann
e3480bc8de
implement 'extension-map' option (#318) 2020-11-02 15:27:07 +01:00
Mike Fährmann
c3f01dc4e6
implement 'util.unique()' 2020-10-29 23:33:41 +01:00
Mike Fährmann
de4a1e45c9
improve 'generate_csrf_token()'
no need to use hashlib.md5()
2020-10-24 02:56:40 +02:00
Mike Fährmann
ec61696316
add 't' format string conversion (closes #1065)
to Trim whitespace from the beginning and end of strings.
Example: '{field!t}' becomes 'foo' for 'field' == "  \nfoo\t\r"
2020-10-16 00:37:22 +02:00
Mike Fährmann
1b1cf01d0d
add a general 'generate_csrf_token()' function 2020-10-15 15:14:18 +02:00
Mike Fährmann
d5fa716d89
fix crash when using 'skip=false' and archive (fixes #1023)
Separating the archive check from pathfmt.exists() in b5243297
had some unintended side effects.

It is also not possible to monkey-patch a dunder method like
__contains__ because of the special method lookup that gets
performed for them.
2020-09-23 19:07:40 +02:00
Mike Fährmann
65744a7a31
use alternative for all falsey values in format strings
… and not just None (#525)

It would be better to consistently use None for all non-existent
fields and/or fields without a valid value, but this is a good
enough workaround for now.
2020-09-19 22:02:47 +02:00
Mike Fährmann
b5243297ff
write skipped files to archive (closes #550) 2020-09-03 18:37:38 +02:00
Mike Fährmann
4d8b3e4f70
defer directory creation (fixes #722)
Only call os.makedirs() before a file is getting downloaded,
and not immediately for every Directory message.
2020-07-04 22:15:23 +02:00
Mike Fährmann
1ae1df0d27
update '--write-pages' (#737)
- fix infinite recursion for responses with multiple entries in
  'history'
- hide values of Set-Cookie headers
- only write the response content by default
  (use '-o write-pages=all' to also include HTTP headers)
2020-06-18 15:07:30 +02:00
Mike Fährmann
1fcf938f9c
implement a general 'delete_items()' function 2020-06-06 23:49:49 +02:00
Mike Fährmann
ddc253cf9a
implement a 'path-replace' option (#662, #755) 2020-05-25 22:21:58 +02:00
Mike Fährmann
15c3d29062
move dump_response() into a separate function (#737) 2020-05-25 22:21:58 +02:00
Mike Fährmann
bc53302ad6
extend 'path-restrict' option
Allow its value to be a JSON object / Python dict that specifies
a mapping from invalid/unwanted input characters to specific
output characters.

For example {"/": "-", "*": "+"} will transform
"foo / ***bar***" into "foo - +++bar+++"

(closes #662, #755)
2020-05-25 22:21:56 +02:00
Mike Fährmann
3201fe3521
add global SENTINEL object 2020-05-19 22:32:53 +02:00
Mike Fährmann
c8787647ed
add global WINDOWS bool 2020-05-19 22:32:53 +02:00
Mike Fährmann
ece73b5b2a
make 'path' and 'keywords' available in logging messages
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.

__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.

(#574, #575)
2020-05-18 19:04:51 +02:00
Mike Fährmann
abbd8fbbd9
reset filenames on empty file extensions (#733) 2020-05-18 19:04:50 +02:00
Mike Fährmann
38bc6430d3
[downloader:http] don't overwrite existing '_mtime' fields 2020-04-10 23:08:03 +02:00
Mike Fährmann
9159cb8fb3
remove trailing dots and spaces from directory names (#647) 2020-03-19 21:12:18 +01:00
Mike Fährmann
90e4c645ba
[formatter] allow multiple "special" format specifiers (#595)
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
2020-02-16 21:47:08 +01:00
Mike Fährmann
219c4cc78c
[formatter] allow for numeric list and string indices 2020-02-15 22:46:22 +01:00
Mike Fährmann
7d1da614d9
[formatter] implement field name alternatives (#525)
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
2020-02-15 17:58:21 +01:00
Mike Fährmann
56f1c96168
implement 'parent-directory' option (#551) 2020-01-29 18:32:37 +01:00
Mike Fährmann
2a9be48511
improve util.load/save_cookiestxt() and add tests
- take a file object as argument instead of an filename
- accept whitespace before comments ("   # comment")
- map expiration "0" to None and not the number 0
2020-01-25 23:02:15 +01:00
Mike Fährmann
c1a6862863
implement functions to load/save cookies.txt files (closes #586)
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
2020-01-21 21:59:36 +01:00
Mike Fährmann
760b9b4db4
add remove_file() and remove_directory() helpers
these functions call os.unlink() or os.rmdir()
while catching and suppressing potential OSErrors
2020-01-18 00:21:26 +01:00
Mike Fährmann
b2d542ad40
improve PathFormat._enum_file()
open only one try-except block for the whole loop,
instead of one for each iteration in os.path.exists()
2020-01-18 00:21:25 +01:00
Mike Fährmann
025f6e3398
add fallback for missing WITHOUT ROWID support (#553) 2020-01-03 22:58:28 +01:00
Mike Fährmann
58391d492d
cache archive keys generated in __contains__() (#524)
To avoid writing a different key to the archive than what was checked
against before the file download.
2019-12-20 16:43:08 +01:00
Mike Fährmann
0f1538af78
split filename formatting into its own function 2019-11-29 22:32:07 +01:00
Mike Fährmann
3fc1e12949
[postprocessor:metadata] filter private entries
i.e. keys starting with an underscore
2019-11-21 16:58:44 +01:00
Mike Fährmann
d5e3910270
adjust 'util.raises()' 2019-10-28 15:06:17 +01:00
Mike Fährmann
c887493a80
overhaul exception stuff 2019-10-27 23:53:37 +01:00
Mike Fährmann
776e9e073f
close archive on job completion (#417) 2019-09-10 22:43:51 +02:00
Mike Fährmann
0ce98169b8
improve path generation
- fix 'abspath()' results for Python <3.7 (closes #402)
  - 'abspath()' in Python 3.7+ removes trailing path separators
  - in Python <3.7 it doesn't
- filter empty path segments
2019-08-28 23:25:18 +02:00
Mike Fährmann
3284c62f22
ensure PathFormat.directory ends with a path separator
... plus some other small optimizations
2019-08-20 00:25:13 +02:00
Mike Fährmann
e77a656437
optimize directory path generation
- use str.join() instead of os.path.join()
  (less "features", but 10x as fast)
- cache directory formatters
- detect and optimize field access for 1-element format strings
2019-08-19 15:56:20 +02:00
Mike Fährmann
454bf1ebf9
preserve enumeration index after 'set_extension()' (#306) 2019-08-16 23:12:33 +02:00
Mike Fährmann
f5039b897f
replace DownloadArchive.check() with __contains__()
Interestingly enough, 'a in obj' is slightly faster than
'obj.check(a)' and is also nicer to look at, I think.
2019-08-16 23:12:32 +02:00
Mike Fährmann
5a210991b6
Remove control characters from filesystem paths
- add 'path-remove' option to specify the set of characters that
 should be removed
- rename 'restrict-filenames' to 'path-restrict'
- #348, #380
2019-08-16 23:12:16 +02:00
Mike Fährmann
0bb873757a
update PathFormat class
- change 'has_extension' from a simple flag/bool to a field that
  contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306)
2019-08-12 21:40:37 +02:00