Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-25 20:22:36 +01:00
Code Issues Releases Wiki Activity
6,071 Commits 6 Branches 164 Tags 15 MiB
a09d9edaa6
Commit Graph

80 Commits

Author SHA1 Message Date
Mike Fährmann
5bc3657c59
[util] implement 'compile_filter()' (#5262) 
https://github.com/mikf/gallery-dl/issues/5262#issuecomment-2477029728

allow (theoretically*) all filter expression statements
to be a list of individual filters

(*) except for 'filename' and 'directory' conditionals,
as dict keys cannot be lists
 2024-11-14 22:47:36 +01:00
Mike Fährmann
0b99d9e6b9
[util] add "defaultdict" filters-environment 
allows accessing undefined values without raising an exception,
but preserves other errors like TypeError, AttributeError, etc
 2024-11-14 22:47:25 +01:00
Mike Fährmann
4667833195
[util] add 'std' object to global eval namespace (#6330) 
allows accessing standard library modules (and other external modules)
in a more straightforward manner than '__import__(...)'

* std.os.getcwd()
* std["os"].getcwd()
 2024-10-17 21:08:07 +02:00
Mike Fährmann
3946fe5ac4
[cookies] return loaded cookies as list 
don't set_cookie() them immediately into a CookieJar
also, give some more consistent names to chrome/chromium functions
 2024-10-14 14:24:27 +02:00
Mike Fährmann
2c7a0c3ca8
add alternatives for deprecated utc datetime functions 2024-09-19 20:47:05 +02:00
Mike Fährmann
01e8433889
fix tests & syntax for older Python versions 2024-08-15 18:16:55 +02:00
Mike Fährmann
5208c0d28a
[util] extend CustomNone with an __index__ method (#6009) 
- Make it compatible with functions expecting integer arguments
- Simplify and reuse some method definitions
 2024-08-13 16:05:54 +02:00
Mike Fährmann
ad73789256
[util] extend CustomNone with arithmetic operators (#6007) 2024-08-13 09:41:12 +02:00
Mike Fährmann
d7a2c73274
[util] let a CustomNone instance be equal to itself 2024-07-26 20:56:01 +02:00
Mike Fährmann
f7383a56f8
wrap filters/conditionals in a try-except block 
allows accessing undefined fields without exception or locals().get(…)
but hides mistakes/typos/etc by evaluating to False without feedback

performance loss compared to the previous version without try-except
is negligible (~20ns for me)
 2024-07-12 22:51:11 +02:00
Mike Fährmann
7614bc458e
[util] extend CustomNone with comparison operators 2024-06-05 16:49:30 +02:00
Mike Fährmann
ba062712ad
[tests] '__main__' -> "__main__" 2024-02-27 02:10:05 +01:00
Mike Fährmann
72b18d701f
represent util.NONE as 'null' in JSON output 
was '"None"' before
 2023-11-02 15:23:28 +01:00
Mike Fährmann
d788e6c60c
implement 'globals' option 2023-02-28 18:18:55 +01:00
Mike Fährmann
56039d2456
add 'hash_md5' and 'hash_sha1' functions (#3679) 
... to global eval namespace
 2023-02-22 10:58:44 +01:00
Mike Fährmann
d651d45239
implement specifying ranges in slice notation (#918, #2865) 
e.g.
- '1:101'   or ':101' or ':101:'  for files 1 to 100
- '1::2'    or '::2'              for every second file
- '1:101:5' or ':101:5'           for files 1, 6, 11, ..., 91, 96

(the second argument specifies the first index NOT included)
 2022-12-27 18:21:12 +01:00
Mike Fährmann
3616adfc75
implement '--range' with Python ranges 2022-12-26 18:32:34 +01:00
Mike Fährmann
1800bd7d14
allow '*-filter' options to be a list of expressions 2022-12-23 22:20:21 +01:00
Mike Fährmann
43c211f1a7
extend and rename util.CustomNone 2022-12-06 22:08:51 +01:00
Mike Fährmann
c0051d7d4c
fix test 2022-08-01 21:40:35 +02:00
Mike Fährmann
dd3a6a9fd1
make 'enumerate_reversed()' work with generators (#2795) 2022-08-01 14:08:44 +02:00
Mike Fährmann
c4b9f7bab8
update functions working with cookies.txt files 
- rename
  - load_cookiestxt -> cookiestxt_load
  - save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
  instead of storing them in a list first
- other unnoticeable performance increases
 2022-05-06 13:21:29 +02:00
Mike Fährmann
ca3a364db7
fix build_duration_func() (#2533) 
for extractors with request_interval_min > 0
 2022-04-27 20:28:14 +02:00
Mike Fährmann
7fe54bab2a
attempt to fix some issues with 'contains()' (#2446) 
add a third argument that gets used
when the values o search are given as a string
 2022-04-08 14:40:26 +02:00
Mike Fährmann
d78a2c7163
re.escape() arguments for 'contains()' (#2446) 2022-04-07 15:35:54 +02:00
Mike Fährmann
413b77757b
implement 'contains()' (#2446) 
and add it to globals() in compiled expressions for --filter etc
 2022-03-30 16:18:33 +02:00
Mike Fährmann
29db716a63
implement 'datetime_to_timestamp()' 
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
 2022-03-23 22:36:01 +01:00
Mike Fährmann
8295bc6d97
fix loading/storing cookies without domain 2022-03-19 15:14:55 +01:00
Mike Fährmann
64cf26eaf4
allow specifying sleep-* options as string 
either as single value or as range: "3.5", "2.1 - 5.0"
 2021-12-18 23:28:56 +01:00
Mike Fährmann
010d65dcec
extend blacklist/whitelist syntax (#2025) 
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'

For example
  "blacklist": "imgur,*:tag,gfycat:user" or
  "blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors  with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho),
and all 'gfycat' user extractors.
 2021-11-23 20:31:43 +01:00
Mike Fährmann
c22ff97743
remove 'unit' argument from 'util.format_value()' 2021-09-28 23:07:55 +02:00
Mike Fährmann
74145467dd
move 'util.Formatter' into its own 'formatter' module 2021-09-27 02:37:04 +02:00
Mike Fährmann
292fffc83c
add 'j' format string conversion 
to convert to a JSON formatted string
 2021-08-28 01:19:36 +02:00
Mike Fährmann
2792ed6e4b
implement 'util.format_value()' 2021-07-26 02:11:22 +02:00
Mike Fährmann
9e42cd58ea
replace ChainPredicate class with 'functools.partial' 2021-07-20 20:21:32 +02:00
Mike Fährmann
0179581340
add 'T' format string conversion (#1646) 
to convert 'date'/datetime to timestamp
 2021-06-25 22:35:45 +02:00
Mike Fährmann
0abad8bc12
implement 'compile_expression()' 2021-06-03 22:34:58 +02:00
Mike Fährmann
8fd8126117
fix ISO 639-1 code for Japanese 
"jp" -> "ja"
 2021-05-22 16:07:04 +02:00
Mike Fährmann
c5ca7905ce
add 'noop()' and 'identity()' functions 2021-05-04 19:27:17 +02:00
Mike Fährmann
bff71cde80
implement 'util.unique_squence()' 2021-03-02 23:11:08 +01:00
Mike Fährmann
91308140ec
make 'generate_token()' compatible with Python 3.4 2021-01-14 03:48:10 +01:00
Mike Fährmann
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()' 
and add a 'size' argument
 2021-01-11 22:12:40 +01:00
Mike Fährmann
aac00a2024
add 'd' conversion for format strings 
to convert a timestamp to a formattable 'datetime' object.

For example '{created_at!d:%Y-%m-%d}'
transforms the timestamp in 'created_at' into a 'datetime' object
and then formats its content using '%Y-%m-%d' as template.

1262304000 -> datetime(2010, 1, 1) -> "2010-01-01"
 2021-01-09 01:58:44 +01:00
Mike Fährmann
c3f01dc4e6
implement 'util.unique()' 2020-10-29 23:33:41 +01:00
Mike Fährmann
ec61696316
add 't' format string conversion (closes #1065) 
to Trim whitespace from the beginning and end of strings.
Example: '{field!t}' becomes 'foo' for 'field' == "  \nfoo\t\r"
 2020-10-16 00:37:22 +02:00
Mike Fährmann
65744a7a31
use alternative for all falsey values in format strings 
… and not just None (#525)

It would be better to consistently use None for all non-existent
fields and/or fields without a valid value, but this is a good
enough workaround for now.
 2020-09-19 22:02:47 +02:00
Mike Fährmann
5df8f2959b
insert local directory into PYTHONPATH when running tests 2020-05-02 01:15:50 +02:00
Mike Fährmann
90e4c645ba
[formatter] allow multiple "special" format specifiers (#595) 
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
 2020-02-16 21:47:08 +01:00
Mike Fährmann
219c4cc78c
[formatter] allow for numeric list and string indices 2020-02-15 22:46:22 +01:00
Mike Fährmann
7d1da614d9
[formatter] implement field name alternatives (#525) 
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
 2020-02-15 17:58:21 +01:00