1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 18:53:21 +01:00
Commit Graph

124 Commits

Author SHA1 Message Date
Mike Fährmann
550f90ab56
delay enabling .part files when 'http-metadata' is set
otherwise 'build_path' gets called before all metadata is collected
2022-11-09 13:23:52 +01:00
Mike Fährmann
8124c16a50
split 'build_path' from 'set_filename' and 'set_extension'
Do not automatically build a new path
when setting file metadata or updating its extension.
2022-11-08 17:03:24 +01:00
Mike Fährmann
39d9c362e4
include 'http-metadata' in '-K' output 2022-11-07 16:33:26 +01:00
Mike Fährmann
870e6a48a0
implement 'http-metadata' option
or at least attempt to.
2022-11-05 18:29:29 +01:00
Mike Fährmann
bca9f965e5
[downloader:http] add 'chunk-size' option (#3143)
and double the previous default from 16384 (2**14) to 32768 (2**15)
2022-11-02 16:50:26 +01:00
Mike Fährmann
0059e2bfe7
[downloader:http] add MIME type and signature for .avif files 2022-11-01 17:25:21 +01:00
Mike Fährmann
f687e64513
[downloader:http] refactor file signature checks
use functions/lambdas instead of startswith()
2022-11-01 17:09:13 +01:00
Mike Fährmann
1aae9f2b71
[downloader:ytdl] update _set_outtmpl() (fixes #2692)
bf1824b391
2022-06-20 11:32:02 +02:00
Mike Fährmann
c0c1277c5f
[downloader:http] support sending POST data (#2433)
by setting the '_http_data' metadata field for a file

needed in addition to be3492776b
to download files with POST requests
2022-03-23 21:48:38 +01:00
Mike Fährmann
be3492776b
[downloader:http] support using a different method than GET (#2433)
by setting the '_http_method' metadata field for a file
2022-03-20 10:09:05 +01:00
Mike Fährmann
47cf05c4ab
refactor proxy handling code (#2357)
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
2022-03-10 23:55:35 +01:00
Mike Fährmann
c0fddcefc5
[downloader:ytdl] make ImportErrors non-fatal (#2273) 2022-02-08 19:30:29 +01:00
Mike Fährmann
ebd3d5c1cc
[bunkr] fix .mp4 downloads (closes #2239) 2022-01-28 23:21:16 +01:00
Mike Fährmann
f4e3cee6ac
use yt-dlp by default (#1850, #2028) 2021-11-29 18:24:26 +01:00
Mike Fährmann
19403a7fff
[downloader:ytdl] prevent crash in '_progress_hook()' (#1680)
'speed' is not guaranteed to be defined or convertible to 'int'
2021-11-12 18:54:04 +01:00
Mike Fährmann
efa178cc91
[ytdl] implement parsing ytdl command-line options (#1680)
- adds 'config-file' and 'cmdline-args' options
  for both ytdl downloader and extractor
- create 'ytdl' helper module, which combines YoutubeDL creation
  and option parsing.
- most likely a buggy mess due to incompatibilities between the
  original youtube-dl and yt-dlp.
2021-11-07 02:44:11 +01:00
Mike Fährmann
232ab626a7
[downloader:ytdl] prevent crash in '_progress_hook()'
https://github.com/mikf/gallery-dl/discussions/1964#discussioncomment-1516702
2021-10-21 22:57:04 +02:00
Mike Fährmann
d0761454b1
implement a download progress indicator (#1519) 2021-09-28 22:48:58 +02:00
Mike Fährmann
b5b1cf22b7
[downloader:http] reorder HTTP header sources
so that any header can be overwritten by a user, except Range
2021-08-05 23:01:54 +02:00
Mike Fährmann
f5b097165e
[ytdl] transfer YoutubeDL objects to downloader (#1680)
allows specifying downloader-specific options per subcategory
but overwrites all downloader.ytdl settings
2021-07-16 15:40:54 +02:00
Mike Fährmann
fc19010808
[downloader:ytdl] fix 'outtmpl' setting for yt_dlp (#1680)
yt_dlp supports multiple outtmpl settings for different file types and
uses its 'outtmpl_dict' for that.
2021-07-16 15:05:16 +02:00
Mike Fährmann
e622e004f0
[ytdl] improve module imports (#1680)
Apply 'extractor.ytdl.module' for every URL, not just the first.
2021-07-14 03:08:00 +02:00
Mike Fährmann
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl
(#1680, #878)

Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
2021-07-10 20:55:47 +02:00
Mike Fährmann
221015e586
[downloader:http] disable filename extension changes for ugoira
(#1507)
2021-04-27 01:29:09 +02:00
Mike Fährmann
1a38fae785
add option to use different youtube-dl modules (fixes #1330)
by setting the 'downloader.ytdl.module' value. For example

{
    "downloader": {
        "ytdl": {
            "module": "yt_dlp"
        }
    }
}

or '-o module=yt_dlp'
2021-03-01 03:10:42 +01:00
Mike Fährmann
8821dceb79
use __import__() to dynamically load modules 2021-03-01 01:27:02 +01:00
Mike Fährmann
cf5fa75d4c
add 'browser' option (#1117)
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
2021-02-26 13:41:27 +01:00
Mike Fährmann
560277394e
[downloader:http] add 'headers' option (#1322) 2021-02-21 19:13:39 +01:00
Mike Fährmann
a228bb3a5f
[downloader:http] support callbacks to validate responses 2021-01-29 22:15:21 +01:00
Mike Fährmann
0594821fcd
[downloader:http] add MIME type and signature for .ico files
(closes #1211)
2021-01-01 16:07:33 +01:00
Mike Fährmann
476d563ec2
[downloader:http] add MIME type and signature for .swf files 2020-12-11 14:21:04 +01:00
Mike Fährmann
fe0265c7a5
[downloader.http] small improvements to file signature list
- specify multiple entries for gif, mp3, zip
- add entries for pdf
2020-12-08 21:20:18 +01:00
Mike Fährmann
1a4b61f7eb
[downloader:http] fix issues with chunked transfer encoding
(fixes #1144)
2020-11-30 01:10:45 +01:00
Mike Fährmann
536c088462
[downloader:http] improve 'adjust-extensions' (#776)
Check file headers against a list of file signatures before
downloading the whole file and writing it to disk.

The file signature check needs some improvements (*),
but it produces usable results for the most part.

(*)
- 'webp', 'wav', and others start with 'RFFI'
- 'svg' uses the same "signature" as all XML documents
- 'webm' has the same signature as 'mkv' files
- only 'mp3' files in an ID3v2 container get recognized
2020-11-29 20:55:35 +01:00
Mike Fährmann
f6fd449b59
reduce wait time growth rate from exponential to linear
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
2020-09-06 22:38:25 +02:00
Mike Fährmann
ac3036ef56
add 'filesize-min' and 'filesize-max' options (closes #780) 2020-09-03 18:21:04 +02:00
Mike Fährmann
34929f673f
readd 'session' to base downloader class (fixes #768) 2020-05-20 20:04:46 +02:00
Mike Fährmann
ece73b5b2a
make 'path' and 'keywords' available in logging messages
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.

__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.

(#574, #575)
2020-05-18 19:04:51 +02:00
Mike Fährmann
f8661c6578
[downloader:ytdl] fix file extensions when merging into mkv 2020-05-13 22:35:33 +02:00
Mike Fährmann
dba87ca99e
[downloader:ytdl] change 'forward-cookies' default to 'false'
There are currently no situations where forwarding gallery-dl's cookies
to youtube-dl is necessary, and it only causes problems when forcing
youtube-dl for Twitter video downloads while logged in.
2020-05-12 20:17:08 +02:00
Mike Fährmann
19a7afdd9b
[downloader:http] add MIME types for .psd files (closes #714) 2020-04-29 23:01:42 +02:00
Mike Fährmann
38bc6430d3
[downloader:http] don't overwrite existing '_mtime' fields 2020-04-10 23:08:03 +02:00
Mike Fährmann
115fd2c6f2
"fix" incomplete MIME types (#632)
e-/exhentai's original image downloads currently send
incomplete/invalid Content-Type headers, "jpg" instead
of "image/jpg" etc, since the last update.
(https://forums.e-hentai.org/index.php?showtopic=236113)

This change prepends any Content-Type value missing a
media type specification with "image/", transforming it
into a valid MIME type.

(A global solution to a local problem, but it shouldn't
 cause any issues anywhere else)
2020-03-03 21:21:57 +01:00
Mike Fährmann
adcd7cb24a
[downloader:http] add another MIME type for '.rar' files (#628) 2020-03-01 20:42:13 +01:00
Mike Fährmann
380b693fad
[downloader:http] add more MIME types for '.bmp' files (#621) 2020-02-23 16:51:04 +01:00
Mike Fährmann
760b9b4db4
add remove_file() and remove_directory() helpers
these functions call os.unlink() or os.rmdir()
while catching and suppressing potential OSErrors
2020-01-18 00:21:26 +01:00
Mike Fährmann
200aea308a
[downloader:common] enable 'job'/'extractor' for logging messages
(#574)
2020-01-12 21:41:16 +01:00
Mike Fährmann
c4702ec9b6
simplify some logging calls 2019-12-10 21:30:08 +01:00
Mike Fährmann
c59b98c81b
[downloader:http] improve rate limit handling
- Move the download "logic" with rate limit checks into its own
  method that only gets used if a rate limit should be enforced
- Fix an issue where suspending gallery-dl during a download would
  basically ignore the rate limit for the remaining download when
  resuming its execution.
2019-12-09 20:34:22 +01:00
Mike Fährmann
bbbafc1c24
[downloader:http] catch both possible SSLException instances
With pyOpenSSL installed, but disabled, the SSLError exception
would be set to the one from pyOpenSSL, which could never get raised.

This commit solves this problem by catching both, the native SSLError
exception as well as the one from pyOpenSSL (if available.1)
2019-12-09 20:34:10 +01:00