Mike Fährmann
0dacb2b24c
[downloader:http] remove 'pyopenssl' import ( #5156 )
2024-02-04 21:06:54 +01:00
Mike Fährmann
ea78f67860
[downloader:http] skip files not passing filesize-min/-max ( #4821 )
...
instead of failing the download
2023-11-17 22:54:20 +01:00
ClosedPort22
5448268d5c
[downloader:http] close connection when file already exists ( #3748 )
2023-08-08 23:35:43 +08:00
Mike Fährmann
c182094ebf
merge #3748 : [downloader:http] add 'consume-content' option
2023-04-26 23:03:18 +02:00
ClosedPort22
6f4a843fba
[downloader:http] release connection before logging messages
...
This allows connections to be properly released when using 'actions'
feature.
2023-04-24 23:59:36 +08:00
Mike Fährmann
2edcdee32f
[downloader:http] add MIME type and signature for .heic files
...
(#3915 )
https://github.com/strukturag/libheif/issues/83
2023-04-15 17:09:22 +02:00
ClosedPort22
775d2ac999
[downloader:http] improve error logging when releasing connection
2023-03-31 20:08:38 +08:00
ClosedPort22
1a977f0f62
[downloader:http] handle exceptions in 'validate'
...
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
2023-03-23 19:57:13 +08:00
ClosedPort22
fcaeaf539c
[downloader:http] handle exceptions while consuming content
2023-03-11 21:36:37 +08:00
Mike Fährmann
67ec91cdbd
[downloader:http] change '_http_retry' to accept a Python function
...
and rename '_http_retry_codes' to '_http_retry'
(#3569 )
2023-03-09 23:30:15 +01:00
ClosedPort22
df77271438
[downloader:http] add 'consume-content' option
...
* fix connection not being released when the response is neither
successful nor retried
* add the ability to consume the HTTP response body instead of closing
the connection
reference:
https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
2023-03-09 21:07:10 +08:00
Mike Fährmann
8148c2a097
[downloader:ytdl] prevent exception on empty results
...
a7c7953107 (commitcomment-92042240)
2023-03-06 12:25:12 +01:00
Mike Fährmann
d16873941c
[downloader:http] use 'time.monotonic()'
2023-01-31 15:32:12 +01:00
Mike Fährmann
ec9ff7640d
merge #3535 : [downloader:http] add signature checks for .blend, .obj, and .clip files
2023-01-16 15:09:10 +01:00
ClosedPort22
b6706b373a
[downloader:http] add signature checks for some formats
...
also add the MIME type for .obj files
2023-01-15 23:40:55 +08:00
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2023-01-14 17:25:30 +01:00
Mike Fährmann
c0d7d2be35
[downloader:http] add 'validate' option
2023-01-11 15:37:40 +01:00
Mike Fährmann
80102fa367
[downloader:http] add 'retry-codes' option ( #3313 )
2022-12-01 11:08:23 +01:00
Mike Fährmann
b4253f69c9
[downloader:http] fix ZeroDivisionError ( #3328 )
...
ensure 'time_elapsed' only get used as divisor
when it is greater than zero
2022-11-30 21:56:18 +01:00
Mike Fährmann
f87cfa5f66
[downloader:http] add signature check for .mp4 files
2022-11-16 21:45:26 +01:00
Mike Fährmann
a4ff20cf16
[downloader:http] fix issues from inaccurate 'time.sleep()'
...
(#3143 )
Reverts part of c59b98c8
2022-11-10 13:24:02 +01:00
Mike Fährmann
550f90ab56
delay enabling .part files when 'http-metadata' is set
...
otherwise 'build_path' gets called before all metadata is collected
2022-11-09 13:23:52 +01:00
Mike Fährmann
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
...
Do not automatically build a new path
when setting file metadata or updating its extension.
2022-11-08 17:03:24 +01:00
Mike Fährmann
39d9c362e4
include 'http-metadata' in '-K' output
2022-11-07 16:33:26 +01:00
Mike Fährmann
870e6a48a0
implement 'http-metadata' option
...
or at least attempt to.
2022-11-05 18:29:29 +01:00
Mike Fährmann
bca9f965e5
[downloader:http] add 'chunk-size' option ( #3143 )
...
and double the previous default from 16384 (2**14) to 32768 (2**15)
2022-11-02 16:50:26 +01:00
Mike Fährmann
0059e2bfe7
[downloader:http] add MIME type and signature for .avif files
2022-11-01 17:25:21 +01:00
Mike Fährmann
f687e64513
[downloader:http] refactor file signature checks
...
use functions/lambdas instead of startswith()
2022-11-01 17:09:13 +01:00
Mike Fährmann
1aae9f2b71
[downloader:ytdl] update _set_outtmpl() ( fixes #2692 )
...
bf1824b391
2022-06-20 11:32:02 +02:00
Mike Fährmann
c0c1277c5f
[downloader:http] support sending POST data ( #2433 )
...
by setting the '_http_data' metadata field for a file
needed in addition to be3492776b
2022-03-23 21:48:38 +01:00
Mike Fährmann
be3492776b
[downloader:http] support using a different method than GET ( #2433 )
...
by setting the '_http_method' metadata field for a file
2022-03-20 10:09:05 +01:00
Mike Fährmann
47cf05c4ab
refactor proxy handling code ( #2357 )
...
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
- add 'downloader.proxy' option
- '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
now has the same effect as youtube-dl's '--geo-verification-proxy'
2022-03-10 23:55:35 +01:00
Mike Fährmann
c0fddcefc5
[downloader:ytdl] make ImportErrors non-fatal ( #2273 )
2022-02-08 19:30:29 +01:00
Mike Fährmann
ebd3d5c1cc
[bunkr] fix .mp4 downloads ( closes #2239 )
2022-01-28 23:21:16 +01:00
Mike Fährmann
f4e3cee6ac
use yt-dlp by default ( #1850 , #2028 )
2021-11-29 18:24:26 +01:00
Mike Fährmann
19403a7fff
[downloader:ytdl] prevent crash in '_progress_hook()' ( #1680 )
...
'speed' is not guaranteed to be defined or convertible to 'int'
2021-11-12 18:54:04 +01:00
Mike Fährmann
efa178cc91
[ytdl] implement parsing ytdl command-line options ( #1680 )
...
- adds 'config-file' and 'cmdline-args' options
for both ytdl downloader and extractor
- create 'ytdl' helper module, which combines YoutubeDL creation
and option parsing.
- most likely a buggy mess due to incompatibilities between the
original youtube-dl and yt-dlp.
2021-11-07 02:44:11 +01:00
Mike Fährmann
232ab626a7
[downloader:ytdl] prevent crash in '_progress_hook()'
...
https://github.com/mikf/gallery-dl/discussions/1964#discussioncomment-1516702
2021-10-21 22:57:04 +02:00
Mike Fährmann
d0761454b1
implement a download progress indicator ( #1519 )
2021-09-28 22:48:58 +02:00
Mike Fährmann
b5b1cf22b7
[downloader:http] reorder HTTP header sources
...
so that any header can be overwritten by a user, except Range
2021-08-05 23:01:54 +02:00
Mike Fährmann
f5b097165e
[ytdl] transfer YoutubeDL objects to downloader ( #1680 )
...
allows specifying downloader-specific options per subcategory
but overwrites all downloader.ytdl settings
2021-07-16 15:40:54 +02:00
Mike Fährmann
fc19010808
[downloader:ytdl] fix 'outtmpl' setting for yt_dlp ( #1680 )
...
yt_dlp supports multiple outtmpl settings for different file types and
uses its 'outtmpl_dict' for that.
2021-07-16 15:05:16 +02:00
Mike Fährmann
e622e004f0
[ytdl] improve module imports ( #1680 )
...
Apply 'extractor.ytdl.module' for every URL, not just the first.
2021-07-14 03:08:00 +02:00
Mike Fährmann
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
...
(#1680 , #878 )
Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
2021-07-10 20:55:47 +02:00
Mike Fährmann
221015e586
[downloader:http] disable filename extension changes for ugoira
...
(#1507 )
2021-04-27 01:29:09 +02:00
Mike Fährmann
1a38fae785
add option to use different youtube-dl modules ( fixes #1330 )
...
by setting the 'downloader.ytdl.module' value. For example
{
"downloader": {
"ytdl": {
"module": "yt_dlp"
}
}
}
or '-o module=yt_dlp'
2021-03-01 03:10:42 +01:00
Mike Fährmann
8821dceb79
use __import__() to dynamically load modules
2021-03-01 01:27:02 +01:00
Mike Fährmann
cf5fa75d4c
add 'browser' option ( #1117 )
...
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
2021-02-26 13:41:27 +01:00
Mike Fährmann
560277394e
[downloader:http] add 'headers' option ( #1322 )
2021-02-21 19:13:39 +01:00
Mike Fährmann
a228bb3a5f
[downloader:http] support callbacks to validate responses
2021-01-29 22:15:21 +01:00
