Mike Fährmann
390d14dbcc
[chevereto] support 'img.kiwi' and 'deltaporno.com' ( #4664 , #1381 )
2023-10-16 18:14:30 +02:00
Mike Fährmann
727c8eec6c
merge #4667 : [redgifs] fix 'niches' extraction ( #4666 )
2023-10-16 14:20:01 +02:00
Mike Fährmann
2911ed1240
[chevereto] add generic extractors ( #4664 )
...
- support jpgfish
- support pixl.li / pixl.is (#3179 , #4357 )
2023-10-16 14:15:39 +02:00
enduser420
db3363ac0b
[redgifs] fix 'niches' extraction
2023-10-16 16:51:30 +05:30
Mike Fährmann
ade8347ead
[kemonoparty] fix DM dates
2023-10-15 19:54:28 +02:00
Mike Fährmann
6dfe200ae4
[kemonoparty] support discord URLs with channel IDs ( #4662 )
2023-10-15 19:45:22 +02:00
Mike Fährmann
c6a3892210
[imgbb] update username extraction ( #4626 )
2023-10-14 20:55:39 +02:00
Mike Fährmann
13ce3a9acb
[warosu] fix extraction ( #4634 )
2023-10-13 23:03:39 +02:00
Mike Fährmann
c4c4e4d2f4
[newgrounds] improve 'art-image' extraction ( #4642 )
...
- download files in original resolution
- replace .webp with extension of first file
2023-10-13 20:10:55 +02:00
Mike Fährmann
833dce141f
[fantia] add 'content_count' and 'content_num' metadata fields ( #4627 )
2023-10-13 20:10:55 +02:00
Mike Fährmann
2d41702762
[deviantart] implement '"group": "skip"' ( #4630 )
2023-10-12 22:14:20 +02:00
Mike Fährmann
a9c3442d4e
[deviantart] add a couple 'deactivated account' test URLs
2023-10-12 21:40:10 +02:00
Mike Fährmann
2974b8e3c8
[moebooru] add 'metadata' option ( #4646 )
...
for extended 'pool' metadata
2023-10-12 21:34:25 +02:00
Mike Fährmann
67ba4ee842
[pp:exec] support more replacement fields for '--exec' ( #4633 )
...
- {_directory}
- {_filename}
- {_path} (alias for {})
2023-10-09 12:50:10 +02:00
Mike Fährmann
9a008523ac
[hentaifoundry] fix '.swf' file downloads ( #4641 )
2023-10-09 11:45:55 +02:00
Mike Fährmann
15f940819b
[newgrounds] support 'art-image' files ( #4642 )
2023-10-09 11:20:10 +02:00
Mike Fährmann
efaab4fbfa
[twitter] fix crash due to missing 'source' ( #4620 )
...
regression caused by 06aaedde
2023-10-04 23:01:04 +02:00
Mike Fährmann
84fbbd96aa
[shimmie2] remove 'meme.museum'
2023-10-02 20:41:25 +02:00
Mike Fährmann
0b150d45db
[tests] add 'msg' arguments to assert statements
2023-10-01 13:52:00 +02:00
Mike Fährmann
27da3f2958
[tests] re-implement filtering by basecategory
2023-10-01 13:31:23 +02:00
Mike Fährmann
c7bd9925d9
[tests] use fallback URLs for content tests ( #3163 )
2023-09-30 21:00:55 +02:00
Mike Fährmann
b92645cd37
[bunkr] fix extraction ( #4514 , #4532 , #4529 , #4540 )
2023-09-30 18:05:12 +02:00
Mike Fährmann
bd3f7a5bbc
[tests] support one regex per URL for #pattern
2023-09-28 21:56:09 +02:00
Mike Fährmann
0c5d8b1505
[deviantart] re-add 'quality' option and 'intermediary' transform
2023-09-24 17:36:05 +02:00
Mike Fährmann
dbd820d7c5
[tests] allow checking for exact URL results
2023-09-24 01:52:47 +02:00
Mike Fährmann
642998504d
[tests] support 'range()' for #count and metadata checks
2023-09-24 01:52:40 +02:00
Mike Fährmann
1e31fce37b
[pillowfort] support '/tagged/' URLs ( #4570 )
2023-09-23 00:11:01 +02:00
Mike Fährmann
1d2fd0b831
[pillowfort] extract 'b2_lg_url' media ( #4570 )
2023-09-23 00:05:26 +02:00
Mike Fährmann
50e2ebaff0
[danbooru] support 'donmai.moe' URLs
2023-09-22 20:58:38 +02:00
Mike Fährmann
918ba4f847
[redgifs] match gfycat image URLs ( #4558 )
2023-09-22 18:02:55 +02:00
Mike Fährmann
2cd801232b
fix --range causing crashes ( #4557 )
...
regression caused by a383eca7
2023-09-22 16:28:20 +02:00
Mike Fährmann
27ec653991
fix bug in test_init and update example URLs
2023-09-14 13:27:03 +02:00
Mike Fährmann
24a1d46391
[mastodon] support '/@USER/following' URLs
...
Previously, only '/users/USER/following' got matched.
2023-09-13 23:42:51 +02:00
Mike Fährmann
ac00d47a16
update test/test_results.py
2023-09-13 14:54:25 +02:00
Mike Fährmann
65b6011cc5
update test/test_extractor.py
2023-09-11 17:20:06 +02:00
Mike Fährmann
a833c244c8
add exported extractor results
2023-09-10 14:45:01 +02:00
Mike Fährmann
93a7a89cf6
[formatter] use value of last alternative ( #4492 )
...
fixes {fieldname|''} evaluating to the value of 'keywords-default'
instead of an empty string
2023-09-05 17:53:27 +02:00
Mike Fährmann
f2de70f254
[gfycat] remove module
2023-09-04 18:27:11 +02:00
Mike Fährmann
d319777a24
[tests] skip 'test_init_ytdl' on Python<3.6
...
It passes without error in a Python 3.4/3.5 venv on my own machine,
but fails for some inexplicable reason on Github Actions.
2023-08-10 23:34:49 +02:00
Mike Fährmann
0ef1fcab20
[postprocessor] update 'finalize' events
...
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
2023-08-10 19:46:37 +02:00
Mike Fährmann
d50c312ff0
prevent test failure when there's no 'ytdl' module ( #4364 )
...
split of ytdl into its own test function and
skip it when there's an ImportError similar to test_ytdl.py
2023-07-29 13:48:31 +02:00
Mike Fährmann
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
2023-07-29 13:43:27 +02:00
Mike Fährmann
255d08b79e
add test for 'Extractor.initialize()' ( #4359 )
2023-07-28 16:58:16 +02:00
Mike Fährmann
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
2023-07-25 22:16:16 +02:00
Mike Fährmann
f0203b7559
[postprocessor:python] add tests
2023-07-24 15:22:57 +02:00
Mike Fährmann
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
2023-07-22 01:20:50 +02:00
Mike Fährmann
c5565f79f7
merge #4096 : [danbooru] add support for booru.borvar.art instance
2023-07-18 18:33:08 +02:00
Mike Fährmann
63326e3168
[danbooru] add tests for booruvar
2023-07-18 18:29:57 +02:00
Mike Fährmann
5171d8975c
[E621] support 'e6ai.net' ( #4320 )
2023-07-18 18:16:30 +02:00
Mike Fährmann
7444fc125b
[gfycat] implement login support ( #3770 , #4271 )
...
For the record: '/webtoken' and '/weblogin' are not the same ...
2023-07-06 18:56:34 +02:00
Mike Fährmann
25c5a6ffcb
no f-strings
2023-06-25 14:01:26 +02:00
Mike Fährmann
ec64cbefeb
[postprocessor:exec] add tests
2023-06-21 23:54:35 +02:00
Mike Fährmann
ce93c460a6
[formatter] implement 'H' conversion ( #4164 )
...
to remove HTML tags and unescape HTML entities
2023-06-15 13:07:51 +02:00
Mike Fährmann
deff3b434d
[vipergirls] implement login support ( #4166 )
2023-06-13 21:05:09 +02:00
Mike Fährmann
69865dcc05
[formatter] implement slicing strings as bytes ( #4087 )
...
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
2023-05-22 18:30:45 +02:00
Mike Fährmann
df11214281
[ytdl] improve --xff/--geo-bypass detection ( #3989 )
...
check if --xff is supported in a try-except block
and select expected results accordingly
2023-05-01 18:26:37 +02:00
Mike Fährmann
aa731c4298
[ytdl] run yt-dlp tests with latest code from master ( #3989 )
...
Only use PyPI version for Python 3.6, since that's no longer supported
by the current codebase.
2023-05-01 16:42:57 +02:00
Mike Fährmann
43f4bd9faa
[ytdl] fix tests
...
tests pass with latest Git HEAD, but not with the current PyPI version
2023-04-29 18:05:45 +02:00
Mike Fährmann
61a65d5bb9
[ytdl] fix crash due to --geo-bypass deprecation ( #3975 )
2023-04-29 17:25:38 +02:00
Mike Fährmann
a96745368e
"fix" tests on Python 3.4 and 3.5
...
can't rely on dict insertion order
2023-04-26 19:31:27 +02:00
Mike Fährmann
3905f05f00
[postprocessor:metadata] support putting keys in quotes
...
for mode 'modify' and 'delete'
based on fe41a2b1
2023-04-25 14:30:18 +02:00
Mike Fährmann
7459e4abce
[postprocessor:metadata] fix traversing more than 1 level deep
...
for mode 'modify' and 'delete'
2023-04-25 14:17:25 +02:00
Mike Fährmann
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
2023-04-15 17:09:22 +02:00
Mike Fährmann
082d55de16
fix circular reference detection for -K
2023-03-21 23:46:36 +01:00
Mike Fährmann
2ab66ad899
update -K output to include quotes around keys
2023-03-21 22:28:04 +01:00
Mike Fährmann
fe41a2b159
[formatter] support putting keys in quotes
...
i.e. obj["key"] or obj['key']
as in f-strings
2023-03-21 22:06:54 +01:00
Mike Fährmann
46fdf46f21
[formatter] support loading an f-string from a template file
...
"\fTF ~/path/to/file.txt"
2023-03-20 22:05:33 +01:00
Mike Fährmann
1a4d4a799b
[formatter] support filesystem paths for \fM
2023-03-20 22:01:33 +01:00
Mike Fährmann
00f0233b28
[postprocessor:metadata] add 'skip' option ( #3786 )
2023-03-17 23:30:11 +01:00
Mike Fährmann
8f8b4de0e8
[ytdl] fix '--parse-metadata' ( #3663 )
2023-03-05 19:57:23 +01:00
Mike Fährmann
7610d9cf82
merge #3675 : [pixiv] fix --write-tags for '"tags": "original"'
2023-03-02 21:48:31 +01:00
Mike Fährmann
83e7a25b6b
extend OAuth tests
2023-03-02 17:26:51 +01:00
Mike Fährmann
d788e6c60c
implement 'globals' option
2023-02-28 18:18:55 +01:00
Mike Fährmann
56039d2456
add 'hash_md5' and 'hash_sha1' functions ( #3679 )
...
... to global eval namespace
2023-02-22 10:58:44 +01:00
Mike Fährmann
e1df7f73b1
[deviantart] add 'search' extractor
...
(#538 , #1264 , #2954 , #2970 , #3577 )
Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.
TODO: parse HTML search results
2023-02-20 20:54:46 +01:00
Gray Manley
38a6389e2c
Fix lint.
2023-02-20 00:33:30 -06:00
Gray Manley
56cbae92ec
Use more pythony naming.
2023-02-19 06:14:34 -06:00
Gray Manley
8e2ba4f32e
Add test.
2023-02-19 06:13:21 -06:00
Mike Fährmann
dd884b02ee
replace json.loads with direct calls to JSONDecoder.decode
2023-02-09 15:22:00 +01:00
Mike Fährmann
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options
2023-02-07 18:28:14 +01:00
Mike Fährmann
3436c6b117
[postprocessor:metadata] speed up JSON encoding
2023-02-06 12:35:28 +01:00
Mike Fährmann
925b467496
split e621 from danbooru module ( #3425 )
2023-02-03 19:24:31 +01:00
Mike Fährmann
c2bc70593e
implement ability to load external extractor classes
...
- -X/--extractors
- extractor.module-sources
2023-01-30 23:10:10 +01:00
ClosedPort22
b6706b373a
[downloader:http] add signature checks for some formats
...
also add the MIME type for .obj files
2023-01-15 23:40:55 +08:00
Mike Fährmann
71d3143c35
fix bug in test_extractors.py
...
pattern matching tests would succeed
if there is exactly one match
but for the wrong extractor
2023-01-08 15:35:05 +01:00
Mike Fährmann
fa144f38ed
[ytdl} fix dfe4f00c for legacy yt-dlp
2023-01-04 21:42:22 +01:00
Mike Fährmann
dfe4f00ca2
[ytdl] update for yt-dlp changes
2023-01-04 13:12:24 +01:00
Mike Fährmann
d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2022-12-27 18:21:12 +01:00
Mike Fährmann
3616adfc75
implement '--range' with Python ranges
2022-12-26 18:32:34 +01:00
Mike Fährmann
1800bd7d14
allow '*-filter' options to be a list of expressions
2022-12-23 22:20:21 +01:00
Mike Fährmann
43c211f1a7
extend and rename util.CustomNone
2022-12-06 22:08:51 +01:00
Mike Fährmann
42481aed59
[formatter] implement 'S' format specifier ( #3266 )
...
to Sort lists
2022-11-21 21:44:42 +01:00
Mike Fährmann
6e08ad26f7
update downloader tests
2022-11-16 22:59:18 +01:00
Mike Fährmann
05255f5be0
add 'default' argument to 'text.extr()'
2022-11-09 11:00:32 +01:00
Mike Fährmann
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2022-11-08 17:03:24 +01:00
Mike Fährmann
eb33e6cf2d
add 'text.extr()'
...
a stripped-down version of text.extract() that
- always returns a string (like 'extract_from')
- only returns a string
- does not deal with 'pos' arguments
- is ~20% faster
2022-11-04 21:37:36 +01:00
Mike Fährmann
460095adca
update downloader tests
2022-11-01 18:48:35 +01:00
Mike Fährmann
f037429fa4
attempt to improve '-K' output for lists
...
- use [N] instead if [] to indicate a Number needs to be placed there
- enumerate list items
2022-10-28 12:04:58 +02:00
thatfuckingbird
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2022-10-19 11:53:59 +02:00
Mike Fährmann
b57015cf0a
[postprocessor:metadata] assume 'mode: custom' when format is set
...
{"name": "metadata", "format": "foobar"}
will now implicitly use mode:custom and no longer mode:json like before
2022-10-04 22:35:26 +02:00
