Mike Fährmann
11421cf940
[skeb] fix '429 Too Many Requests' errors ( #5766 )
...
Introduce '_handle_429' method to make it easier for Extractors to react
to 429 errors regardless of 'sleep-429' settings.
2024-06-21 00:12:05 +02:00
Mike Fährmann
60b4541199
improve a1bb3279, fix oauth:pixiv ( #5757 )
...
Check 'input' option only when required.
This also fixes an exception in oauth:pixiv caused by using the same
'_input' name as a method defined there.
2024-06-18 16:50:04 +02:00
Mike Fährmann
a1bb32792b
do not try to read from stdin when it is non-interactive ( #5733 )
...
add '--no-input' command-line option and 'input' config file option
to allow users to manually configure this
2024-06-16 18:31:39 +02:00
Mike Fährmann
5d3d03a1f1
fix 6cfbc107
...
the former condition would return True for 2.31.*
6cfbc1071f (commitcomment-142642913)
2024-06-02 18:16:53 +02:00
Mike Fährmann
6cfbc1071f
workaround for requests 2.32.3 ( #5665 )
...
manually call 'load_default_certs()' for SSLContexts
in custom HTTPAdapter instances
2024-06-01 16:02:18 +02:00
Mike Fährmann
28039229fe
[common] use 'create_urllib3_context' for creating SSLContexts
...
enables dumping TLS session keys by setting SSLKEYLOGFILE (#5215 )
as well as other potentially useful settings.
2024-05-10 22:59:29 +02:00
Mike Fährmann
33006fe126
[common] disable 'check_hostname' for non-urllib3 SSLContexts
...
e.g. when 'browser' is set to a non-empty value and gallery-dl creates
its own SSLContext instance instead of using requests' and urllib3's
defaults.
urllib3 disables this option for its default contexts,
since it does this check on its own.
Fixes "ValueError: Cannot set verify_mode to CERT_NONE when
check_hostname is enabled" when using --no-check-certificate.
(#3614 , #4891 , #5576 )
2024-05-10 18:20:08 +02:00
Mike Fährmann
d11ec00908
[common] fix _cfgpath for BaseExtractor objects
...
After the changes in 0c17884673
2024-05-01 16:00:07 +02:00
Mike Fährmann
a7d8cbab0e
[common] show full URL in Extractor.request() error messages
2024-04-18 15:45:36 +02:00
Mike Fährmann
a5071c9ca0
[common] fix NameError in Extractor.request()
...
… when accessing 'code' after an requests exception was raised.
Caused by the changes in 566472f080
2024-04-18 15:42:53 +02:00
Mike Fährmann
566472f080
[common] add 'sleep-429' option ( #5160 )
2024-04-16 18:41:28 +02:00
Mike Fährmann
923c6f3214
[common] simplify 'status_code' check in Extractor.request()
2024-04-16 18:39:47 +02:00
Mike Fährmann
68f4208251
[common] update Extractor.wait() message format
2024-04-16 17:51:14 +02:00
Mike Fährmann
b38a917355
[common] add Extractor.input() method
2024-04-16 00:02:48 +02:00
Mike Fährmann
0d72789aa3
merge #5461 : [cookies] use tempfile when saving cookies.txt files
2024-04-13 19:02:39 +02:00
Mike Fährmann
63ac06643f
compute tempfile path only once
2024-04-13 18:59:18 +02:00
Mike Fährmann
0c17884673
store 'match' and 'groups' in Extractor objects
2024-04-01 03:07:52 +02:00
Mike Fährmann
106dfdb4c3
cleanup sleep-request retry delay code
...
more lines but easier to read I'd say
2024-03-11 21:38:06 +01:00
Mike Fährmann
89066844f4
add 'config_instance' method
...
to allow for a more streamlined access to BaseExtractor instance options
2024-01-18 03:20:36 +01:00
Mike Fährmann
f36dafad06
improve 'include' handling ( #4982 )
...
- remove spaces when given as string
- warn about invalid vales
2023-12-28 19:07:04 +01:00
Luc Ritchie
7dd79eee93
save cookies to tempfile, then rename
...
avoids wiping the cookies file if the disk is full
2023-12-11 00:47:42 -05:00
Mike Fährmann
6a4218aa23
handle 'json' parameter in Extractor.request() manually
...
Mainly to allow passing custom classes like util.LazyPrompt,
but also to simplify and streamline how requests handles it.
2023-12-06 22:13:13 +01:00
Mike Fährmann
9dd5cb8c8a
interactively prompt for passwords on login when none is provided
2023-12-06 22:12:59 +01:00
Mike Fährmann
34a387b6e2
support 'metadata-*' names for '*-metadata' options
...
For example, instead of 'url-metadata' it is now also possible to use
'metadata-url' as option name.
- metadata-url
- metadata-path
- metadata-http
- metadata-version
- metadata-parent
2023-11-18 23:52:10 +01:00
Mike Fährmann
61d6558322
[exhentai] try to avoid 'DH_KEY_TOO_SMALL' errors ( #1021 , #4593 )
2023-11-04 17:30:27 +01:00
Mike Fährmann
eb230e4b77
[nsfwalbum] disable Referer headers by default ( #4598 )
2023-10-01 13:55:17 +02:00
Mike Fährmann
3ecb512722
send Referer headers by default
2023-09-19 00:02:04 +02:00
Mike Fährmann
4cdab8074e
update/fix --list-extractors
2023-09-11 17:32:59 +02:00
Mike Fährmann
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
2023-09-11 16:30:55 +02:00
Mike Fährmann
ceb59e176f
fix default Firefox user agent string
...
note to self: do not trust some random third-party website
2023-09-02 22:22:23 +02:00
Mike Fährmann
a4f7f7da17
add '_dump()' convenience method to Extractor
2023-08-06 17:03:09 +02:00
Mike Fährmann
48ef062867
fix issues with 'Extractor.finalize()'
...
- prevent crash in InstagramUserExtractor (#4359 )
- call it at the end of every DownloadJob
- add it to tests
2023-07-29 13:43:27 +02:00
Mike Fährmann
ed21908fda
initial support for child extractor options
...
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
2023-07-28 17:07:25 +02:00
Mike Fährmann
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
2023-07-25 22:16:16 +02:00
Mike Fährmann
d97b8c2fba
consistent cookie-related names
...
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
2023-07-22 01:20:50 +02:00
Mike Fährmann
ceebacc9e1
remove 'pyopenssl' option
2023-07-19 20:44:07 +02:00
Mike Fährmann
5b59a0d143
update default User-Agent header to Firefox 115 ESR
2023-07-05 15:12:50 +02:00
Mike Fährmann
856f6c10cd
allow for GalleryExtractors to skip loading gallery_url
2023-05-22 22:29:30 +02:00
Mike Fährmann
3ca5dac8b6
extend 'cookies-update' functionality
...
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser
To convert browser cookies to cookies.txt format:
gallery-dl --cookies-fr chromium \
-o cookies-update=cookies.txt \
--no-download \
http://example.org/file.jpg
2023-05-04 15:10:47 +02:00
Mike Fährmann
bc6d65d203
implement 'Extractor.config_deprecated()'
...
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
2023-05-04 10:49:14 +02:00
Mike Fährmann
076380e079
remove '*' indicating keyword-only arguments
...
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
2023-05-02 22:23:33 +02:00
Mike Fährmann
9abcb2b6e5
update headers and ciphers for '"browser": "chrome"'
2023-03-08 17:19:59 +01:00
Mike Fährmann
00b94946b3
[instagram] show -o cursor=… after every error ( #3440 )
2023-01-23 13:00:44 +01:00
Mike Fährmann
80a2ff2d38
support setting 'write-pages' to "ALL"
...
to show authentication header, cookies, etc
2023-01-14 22:34:46 +01:00
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2023-01-14 17:25:30 +01:00
Mike Fährmann
9695c4e88d
emit debug logging message when loading cookies from file
...
attempt nr. 2
no idea how I managed to remove 6514828da918ce29
2023-01-06 11:13:44 +01:00
Mike Fährmann
a918ce29b5
run tests on ubuntu-20.04
...
and remove Python 3.4, since that's no longer available
on this test runner
2023-01-05 13:33:27 +01:00
Mike Fährmann
6514828d4e
emit debug logging message when loading cookies from file
2023-01-05 12:40:22 +01:00
Mike Fährmann
9f06e79868
implement '"user-agent": "browser"' ( #2636 )
2022-11-13 19:17:39 +01:00
Mike Fährmann
86790da2d5
update Cloudflare IUAM detection
...
again
2022-10-31 18:33:52 +01:00
