1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 18:53:21 +01:00
Commit Graph

657 Commits

Author SHA1 Message Date
Mike Fährmann
a14b63d941
support selecting a domain for '--cookies-from-browser'
for example 'gallery-dl --cookies-from-browser firefox/twitter.com'
2023-05-06 15:09:49 +02:00
Mike Fährmann
3ca5dac8b6
extend 'cookies-update' functionality
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser

To convert browser cookies to cookies.txt format:
  gallery-dl --cookies-fr chromium \
             -o cookies-update=cookies.txt \
             --no-download \
             http://example.org/file.jpg
2023-05-04 15:10:47 +02:00
Mike Fährmann
79f47f98dd
[nana] remove module
permanently gone since 2023-03-13
2023-05-03 18:05:53 +02:00
Mike Fährmann
c182094ebf
merge #3748: [downloader:http] add 'consume-content' option 2023-04-26 23:03:18 +02:00
Mike Fährmann
b81ce381f0
fix backticks for 'extractor.redgifs.format' docs
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1518331927
2023-04-21 23:12:01 +02:00
Mike Fährmann
aaf58a1259
[imgur] document 'client-id' option (#3937) 2023-04-21 15:08:50 +02:00
Mike Fährmann
0a7eee3ee0
[deviantart] add 'public' option 2023-04-08 23:04:34 +02:00
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata'
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
2023-04-02 20:11:52 +02:00
Mike Fährmann
00f0233b28
[postprocessor:metadata] add 'skip' option (#3786) 2023-03-17 23:30:11 +01:00
Mike Fährmann
a1ca2404f9
add 'globals' instead of overwriting the default (#3773) 2023-03-16 18:37:00 +01:00
ClosedPort22
fcaeaf539c
[downloader:http] handle exceptions while consuming content 2023-03-11 21:36:37 +08:00
ClosedPort22
df77271438
[downloader:http] add 'consume-content' option
* fix connection not being released when the response is neither
  successful nor retried
* add the ability to consume the HTTP response body instead of closing
  the connection

reference:

https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
2023-03-09 21:07:10 +08:00
Mike Fährmann
da9840a39d
[reddit] update 'videos' option (#3712)
- add 'dash' to directly extract DASH manifest URLs
  (was default behavior since a7c79531)
- change default strategy back to before a7c79531
- disable 'Falling back on generic information extractor' warning
2023-03-06 12:18:25 +01:00
enduser420
e1867cf5eb [misskey] add 'renotes' and 'replies' options 2023-03-02 19:56:19 +05:30
Mike Fährmann
0d142e403c
[szurubooru] add 'tag' and 'post' extractors (#3583, #3713) 2023-03-01 18:47:10 +01:00
Mike Fährmann
d788e6c60c
implement 'globals' option 2023-02-28 18:18:55 +01:00
Mike Fährmann
e480a93337
add 'output.stdout', '.stdin', and '.stderr' options
(#1621, #2152, #2529)

Allow setting custom input/output encodings and options
without having to rely on Python's defaults.
2023-02-26 14:56:19 +01:00
Mike Fährmann
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options 2023-02-07 18:28:14 +01:00
Mike Fährmann
762a68996b
implement 'archive-pragma' option 2023-02-05 17:00:31 +01:00
Mike Fährmann
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction
(#3425)
2023-02-04 21:20:38 +01:00
Mike Fährmann
925b467496
split e621 from danbooru module (#3425) 2023-02-03 19:24:31 +01:00
Mike Fährmann
1ae48a54f8
[twitter] add 'transform' option 2023-02-02 22:01:36 +01:00
Mike Fährmann
78d3960a31
[postprocessor:exec] implement archive options (#3584) 2023-02-01 13:57:16 +01:00
Mike Fährmann
4572c122d8
document 'extractor.module-sources' 2023-01-30 23:32:36 +01:00
Mike Fährmann
f58215705a
add '-O/--postprocessor-option' command-line option (#3565) 2023-01-26 14:59:24 +01:00
ClosedPort22
3b32671fbd
[deviantart] add extractor for status updates
extract user status updates using the '/user/statuses/' endpoint
2023-01-19 11:30:48 +08:00
Mike Fährmann
4e86aaa4db
document 'extractor.headers' and 'extractor.ciphers' 2023-01-16 14:49:56 +01:00
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option (#3313)
do not retry 429 and 430 by default
2023-01-14 17:25:30 +01:00
Mike Fährmann
512abeb4ae
[booru] add 'url' option 2023-01-13 16:32:32 +01:00
Mike Fährmann
c87bd1a752
[danbooru] extend 'metadata' option
make it possible to specify a custom list of metadata includes
2023-01-13 16:20:01 +01:00
Mike Fährmann
c0d7d2be35
[downloader:http] add 'validate' option 2023-01-11 15:37:40 +01:00
Mike Fährmann
cb9d100bac
update Authentication support in docs
https://github.com/mikf/gallery-dl/pull/3075#issuecomment-1374723412
2023-01-08 16:57:31 +01:00
Mike Fährmann
362cd6991b
[pixiv] implement 'metadata-bookmark' option (#3417) 2023-01-07 23:19:43 +01:00
Mike Fährmann
0895e6afee
merge #3462: [docs] Update links and fix field typo 2023-01-07 15:22:03 +01:00
Mike Fährmann
4b9bb632f2
[docs] update links 2023-01-07 15:21:40 +01:00
Mike Fährmann
ac78712618
merge #3496: [docs] update OAuth guide in README.rst 2023-01-06 13:36:23 +01:00
Mike Fährmann
2f31d21509
merge #3455: [twitter] apply tweet type checks before uniqueness check 2023-01-06 13:32:50 +01:00
Mike Fährmann
64435cc37e
[twitter] update docs 2023-01-06 13:32:08 +01:00
Mike Fährmann
b125fcafc5
update docs/configuration.rst
- document metadata.indent
- add concrete mappings to object types
- fix lists for code examples
- fix typos
2023-01-06 13:07:33 +01:00
ClosedPort22
c94692f39b
[docs] document 'access-token' for Mastodon instances 2023-01-06 13:20:41 +08:00
Mike Fährmann
4fd6da474f
merge #3473: [twitter] fix crash when using 'expand' and 'syndication' 2023-01-05 14:19:47 +01:00
Mike Fährmann
818f6fa71f
merge #3453: [twitter] document #1396 and #2712 2023-01-05 14:18:21 +01:00
ClosedPort22
455481ad76
[twitter] improve documentation
This includes:
- The automatic expansion of conversations (#2712)
- How to retrieve age-restricted content from search results (#1396)
2023-01-05 12:14:42 +08:00
Mike Fährmann
6933727b45
merge #3483: [twitter] implement 'syndication=extended' 2023-01-04 17:36:17 +01:00
Mike Fährmann
9116398c1c
[pinterest] add 'domain' option (#3484)
use input URL domain by default
2023-01-04 17:20:14 +01:00
ClosedPort22
6853b14be3
[twitter] apply suggestions from code review
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2023-01-02 21:03:01 +08:00
Mike Fährmann
4611237f8c
merge #3457: [danbooru] extract uploader metadata (if option is set) 2023-01-01 21:20:39 +01:00
Mike Fährmann
0ed5986062
document custom 'output.mode' (#2529) 2022-12-30 17:14:42 +01:00
ClosedPort22
7c8eab8d52
[twitter] implement 'syndication=extended'
to be able to fetch extended user metadata
2022-12-30 20:48:41 +08:00
ClosedPort22
be3286206a
[twitter] assume 'conversation_id' when using syndication
not possible to expand replies at the momemt
2022-12-30 13:57:37 +08:00
ClosedPort22
ce8dbb1ccc
[twitter] fix crash when using 'expand' and 'syndication'
caused by KeyError: 'conversation_id_str'
2022-12-30 12:45:44 +08:00
Mike Fährmann
d651d45239
implement specifying ranges in slice notation (#918, #2865)
e.g.
- '1:101'   or ':101' or ':101:'  for files 1 to 100
- '1::2'    or '::2'              for every second file
- '1:101:5' or ':101:5'           for files 1, 6, 11, ..., 91, 96

(the second argument specifies the first index NOT included)
2022-12-27 18:21:12 +01:00
pi_allen
64902f518e [docs] Update links and fix field typo 2022-12-26 07:46:41 +01:00
ClosedPort22
5d5ea58887
[twitter] update docs for 'quoted' & 'unique' 2022-12-25 15:59:26 +08:00
blankie
f82ee93676
[danbooru] extract uploader metadata (if metadata is set) 2022-12-25 11:00:59 +07:00
Mike Fährmann
1800bd7d14
allow '*-filter' options to be a list of expressions 2022-12-23 22:20:21 +01:00
Mike Fährmann
73ab5d84c0
update docs/configuration.rst 2022-12-20 17:30:46 +01:00
ClosedPort22
dd4a4a3fa6
[e621] softcode the pagination threshold 2022-12-17 21:06:47 +08:00
ClosedPort22
d0ad6d0e67
[e621] implement manual pagination mode 2022-12-16 16:15:36 +08:00
Mike Fährmann
6b6f886dcf
[bunkr] update domain (#3391)
and improve bunkr/app.bunkr handling
2022-12-11 23:18:01 +01:00
Mike Fährmann
dfe7b23579
support Firefox containers for --cookies-from-browser (#3346) 2022-12-09 19:43:55 +01:00
Mike Fährmann
80102fa367
[downloader:http] add 'retry-codes' option (#3313) 2022-12-01 11:08:23 +01:00
ClosedPort22
b14b33f19e
Implement version-metadata option (#3201) 2022-11-27 16:09:42 +01:00
Mike Fährmann
6cb12f513b
[nitter] support quoted Tweets
- distinguish between regular and quoted Tweets and media
- add 'quoted' option and metadata field
2022-11-26 11:23:03 +01:00
Mike Fährmann
a41d093bb1
[nitter] add 'retweets' option (#3278) 2022-11-25 19:53:28 +01:00
Mike Fährmann
e31d12139c
[nitter] add 'videos' option (#3279)
with the same semantics as for twitter
2022-11-24 22:56:01 +01:00
Mike Fährmann
501d9bccfe
[artstation] add 'max-posts' option (#3270) 2022-11-23 22:00:18 +01:00
Mike Fährmann
b1ad6f2289
[artstation] add 'pro-first' option (#3273) 2022-11-23 21:45:20 +01:00
Mike Fährmann
6b97dcf2e0
[postprocessor:metadata] add 'private' option 2022-11-20 15:27:36 +01:00
Mike Fährmann
6379157543
[instagram] use REST API by default
regardless of logged in status
2022-11-17 17:15:38 +01:00
enduser420
7897f68225
[wallhaven] update 'user' extractor (#3226)
* [wallhaven] update 'user' extractor

* [wallhaven] update 'configuration.rst'

add 'extractor.wallhaven.include' entry

* [wallhaven] add 'wallhaven.include' in gallery-dl.conf
2022-11-15 17:35:52 +01:00
Mike Fährmann
9f06e79868
implement '"user-agent": "browser"' (#2636) 2022-11-13 19:17:39 +01:00
Mike Fährmann
8d543b040d
update docs
- document '"mode": "jsonl"' for metadata pp
- remove 'username' and 'password' from 'instagram'
- add 'ytdl' example (#3180)
2022-11-09 22:17:08 +01:00
Mike Fährmann
171262c1b6
[instagram] remove login support
broken feature that I cannot get to work anymore
2022-11-08 21:48:31 +01:00
Mike Fährmann
895f36e53b
[postprocessor:metadata] add 'open' and 'encoding' options 2022-11-07 15:37:22 +01:00
Mike Fährmann
870e6a48a0
implement 'http-metadata' option
or at least attempt to.
2022-11-05 18:29:29 +01:00
Mike Fährmann
bca9f965e5
[downloader:http] add 'chunk-size' option (#3143)
and double the previous default from 16384 (2**14) to 32768 (2**15)
2022-11-02 16:50:26 +01:00
Mike Fährmann
c12a97bcde
[postprocessor] add 'post-after' event (#3117) 2022-10-31 14:35:48 +01:00
Mike Fährmann
7c6af27eb8
[tumblr] add 'fallback-*' options (#2957)
specifically 'fallback-delay' and 'fallback-retries'
and change default number of retries to 2 (down from 3)
2022-10-26 13:59:09 +02:00
Mike Fährmann
cf86f68864
[instagram] add 'avatar' extractor (#929, #1097, #2992) 2022-10-19 10:58:42 +02:00
Mike Fährmann
52d1eb928d
[pixiv] extend 'metadata' option (#3057)
make it usable for all 'pixiv' extractors
2022-10-16 15:32:31 +02:00
Mike Fährmann
0714274f1f
[instagram] remove 'channel' extractor 2022-10-15 21:13:58 +02:00
Mike Fährmann
f1f89b2436
[tumblr] add 'offset' option 2022-10-11 10:54:23 +02:00
Mike Fährmann
15a4d7696f
[docs] update 'adjust-extensions' explanation (#3031) 2022-10-11 10:48:28 +02:00
Mike Fährmann
7f30a0d7a7
add 'path-extended' option (#3021) 2022-10-08 15:41:39 +02:00
Mike Fährmann
14717f3fc9
[deviantart] add 'group' option (#3018)
disabling this option allows to better download from deleted accounts
2022-10-08 15:41:21 +02:00
enduser420
f7ba19a1c0
[nana] add 'nana' extractors (#2967) 2022-10-04 09:23:24 +02:00
Mike Fährmann
4089bceddd
[sankaku] implement 'refresh' option (#2958) 2022-09-30 19:55:48 +02:00
Mike Fährmann
aa49bf13d2
[instagram] add 'api' option 2022-09-26 22:05:29 +02:00
Mike Fährmann
ac45ed2764
[skeb] implement 'filters' option (#2945) 2022-09-26 15:17:28 +02:00
Mike Fährmann
0393e59535
[newgrounds] add 'games' extractor (#2955) 2022-09-24 12:34:37 +02:00
Mike Fährmann
e99a9b2aff
[twitter] improve 'cards-blacklist' (#2875)
allow blacklisting domains and 'name:domain',
where 'domain' depends on a card's 'vanity_url' value
2022-09-17 17:46:34 +02:00
Mike Fährmann
46fe469c53
[tumblr] implement 'ratelimit' option (#2919) 2022-09-17 14:10:33 +02:00
Mike Fährmann
b36125333f
[postprocessor:zip] implement 'files' option (#2872) 2022-09-09 11:41:27 +02:00
Mike Fährmann
3cb8327c60
[zerochan] add 'metadata' option (#2861) 2022-09-02 23:25:19 +02:00
blankie
9745b48830
[tumblr] attempt to fetch high-quality inline images (#2877)
* [tumblr] attempt to fetch high-quality images (again)

Fixes #1846, and fixes #1344

* slight refactor

* update configuration.rst entry
2022-08-31 10:53:50 +02:00
Mike Fährmann
4d78ca89db
[twitter] add 'cards-blacklist' option (#2875) 2022-08-31 10:28:25 +02:00
Mike Fährmann
f16fbe9f93
document 'extractor.twitter.expand' (#2848) 2022-08-30 18:16:20 +02:00
enduser420
574e38a287
[kemonoparty] add 'favorites' option (#2826) (#2831)
* [kemonoparty] add 'favorites' option (#2826)

* [kemonoparty] add regex for the url parameter and fallback on the config
option

* [kemonoparty] simplify
2022-08-18 18:01:42 +02:00
Martin Juhasz
9ea521bec5
[oauth] add host config option (#2806) 2022-08-14 17:08:01 +02:00
Mike Fährmann
df1c643dda
[tumblr] attempt to extract full-resolution photos
- for photos with apparent width == 2048 or height == 3072
- can be disabled with 'original' option
2022-08-10 20:01:46 +02:00
Mike Fährmann
a69f198ac9
[wallhaven] implement 'metadata' option (#2803) 2022-08-08 21:46:36 +02:00
Mike Fährmann
7d1a95ada6
implement 'path-metadata' option (#2734) 2022-07-30 12:31:45 +02:00
Mike Fährmann
8cf5981ded
[skeb] add option to download 'article' images (#1031) 2022-07-29 16:32:00 +02:00
Mike Fährmann
98af5a0409
[zerochan] implement login with username & password (#1434) 2022-07-29 12:56:20 +02:00
Mike Fährmann
0c73914848
[postprocessor:metadata] implement 'mode: modify' (#2640) 2022-07-19 12:24:26 +02:00
Mike Fährmann
f3de6b7a87
[postprocessor:metadata] implement 'mode: delete' (#2640) 2022-07-19 00:57:29 +02:00
Mike Fährmann
eb68d45544
add global 'warnings' option (#2762) 2022-07-18 22:20:30 +02:00
Mike Fährmann
f225247670
[gelbooru] add support for api_key and user_id (#2767) 2022-07-18 18:46:31 +02:00
Mike Fährmann
74865adae5
implement 'format-separator' option (#2737)
a global option, that servers as a workaround for shortcomings due to
lack of a proper format string parser
2022-07-10 13:31:43 +02:00
bradenhilton
117eeefda0
[postprocessor:mtime] add 'value' option (#2739) 2022-07-08 20:56:01 +02:00
Mike Fährmann
1d14928bd9
[twitter] ignore previously seen Tweets (#2712)
occurs primarily for /with_replies results when logged in
2022-07-03 16:13:53 +02:00
Mike Fährmann
4b2a0a0eda
[twitter] implement 'strategy' option (#2712)
to be able to better control what Tweets get used an returned
for twitter.com/USER URLs.
2022-07-03 14:29:15 +02:00
Mike Fährmann
124b5c6410
[postprocessor:ugoira] enable 'mtime' by default (#2714) 2022-07-01 20:59:19 +02:00
Mike Fährmann
3a5d5c3a91
update default User-Agent header to Firefox 102 ESR
snd update headers and ciphers for "browser": "firefox"
2022-06-28 17:38:58 +02:00
Mike Fährmann
9d8e99af80
[itaku] support videos (#1842) 2022-06-20 19:47:53 +02:00
Mike Fährmann
9c8d895d19
[twitter] implement 'csrf' option (#2676) 2022-06-13 18:36:39 +02:00
Mike Fährmann
6db77d4656
[weibo] support '?tabtype=video' listings (#2601) 2022-06-12 17:55:23 +02:00
Mike Fährmann
4b78bd423f
[paheal] add 'metadata' option (#2641) 2022-06-04 16:05:49 +02:00
Mike Fährmann
57508d3bb7
[weibo] support all different 'tabtype' listings (#686, #2601) 2022-06-03 16:36:22 +02:00
HRXN
6d0aa8c0f3
[docs/configuration.rst] Fix small typo (#2649) 2022-06-01 18:31:39 +02:00
Mike Fährmann
e0ac358aa5
[gofile] fix 401 Unauthorized errors (#2632) 2022-06-01 13:02:34 +02:00
Mike Fährmann
4bf5bc2403
[weibo] support 'livephoto' entries (#2146) 2022-05-31 15:35:24 +02:00
Mike Fährmann
5b43faffed
[postprocessor:metadata] write to stdout by setting filename to "-"
(#2624)
2022-05-30 21:17:31 +02:00
Mike Fährmann
603af48265
implement 'output.ansi' option (#2628) 2022-05-29 19:15:25 +02:00
Mike Fährmann
915dba8345
[twitter] improve results for regular user URLs
- continuation of 3346f58a
- use media timeline results (or tweet timeline if retweets are enabled)
  plus search results starting from the last tweet id of the first
  timeline, similar to how Twitter Media Downloader operates
- the old behavior can be forced by appending '/tweets' to a user URL,
  like with '/media' (https://twitter.com/USER/tweets)
  although there should be no need to ever do that
2022-05-23 18:33:52 +02:00
Mike Fährmann
9df4e0f65b
[twitter] disable 'cards' by default 2022-05-21 15:39:25 +02:00
Mike Fährmann
79dce8ae68
[weasyl] implement 'metadata' option (#2610) 2022-05-20 22:32:35 +02:00
Mike Fährmann
6b6eb0b8f6
[lolisafe] implement 'domain' option (#2575) 2022-05-10 12:17:59 +02:00
Mike Fährmann
6742f3bc1e
implement --cookies-from-browser (#1606)
most of the code is adapted from yt-dlp's implementation
and *should* work the same.
2022-05-07 23:06:37 +02:00
Mike Fährmann
61887c895b
implement 'output.colors' options (#2532) 2022-05-02 12:41:14 +02:00
Mike Fährmann
84756982e9
[pixiv] implement 'include' option
- split 'user' extractor and its 'avatar' and 'background' options into
  separate extractors ('artworks', 'avatar', 'background')
- avatars can now be downloaded with
  https://www.pixiv.net/en/users/ID/avatar
  as URL and will use a proper archive key; similar for backgrounds
- options for the 'user' subcategory must be moved to 'artworks' to have
  the same effect as before
2022-05-02 09:03:35 +02:00
Mike Fährmann
d11e2191ae
[nijie] support /history_nuita.php listings (closes #2541) 2022-05-02 09:03:34 +02:00
Mike Fährmann
61afa973b7
use extended format strings for archive keys 2022-04-28 20:26:12 +02:00
Mike Fährmann
9e6ff42a9d
[pixiv] implement 'background' option (#623, #1124, #2495) 2022-04-21 13:53:02 +02:00
Mike Fährmann
97e8a15295
[deviantart] implement 'pagination' option (#2488) 2022-04-18 18:08:01 +02:00
Mike Fährmann
12bd9ba33a
[readcomiconline] add 'quality' option (#2467) 2022-04-15 18:10:37 +02:00
Mike Fährmann
24ea1398bc
[postprocessor:ugoira] do not auto-select image2 demuxer (#2492) 2022-04-12 21:47:47 +02:00
Mike Fährmann
1171911dc3
[twitter] add 'syndication' option (#2354)
to fetch age-restricted content using Twitter's  syndication API
2022-04-01 16:56:47 +02:00
loragja
7e545a3ae9
[gofile] add gofile.io extractor (#2364)
* Add gofile extractor

* add gofile extractor to module list

* add support for tiny monitors and ancient python versions

* seriously, f-strings are not *that* new...

* i love flake8 :)

* add 'api-token' and 'recursive' options
* add tests
2022-03-29 17:31:57 +02:00
Mike Fährmann
5c2d953766
[postprocessor:ugoira] auto-select demuxer by default 2022-03-26 21:10:59 +01:00
Mike Fährmann
37d584a9b2
[hitomi] update metadata extraction (fixes #2444)
remove 'hitomi.metadata' option, as it is no longer necessary
to make additional HTTP requests to fetch all metadata.
2022-03-26 12:46:18 +01:00
Mike Fährmann
e718dd7b32
[postprocessor:ugoira] support setting timecodes with mkvmerge
by selecting the "mkvmerge" demuxer

(#1550)
2022-03-26 12:41:55 +01:00
Mike Fährmann
71bba774da
respect 'output.private' in '-K/--list-keywords' output 2022-03-25 22:19:37 +01:00
Mike Fährmann
ba69fb669d
[kemonoparty] add 'duplicates' option (closes #2440) 2022-03-24 11:58:38 +01:00
Mike Fährmann
40ce505420
[postprocessor:ugoira] add 'mtime' option (#2307) 2022-03-21 10:02:29 +01:00
Mike Fährmann
9bd27b1b8d
[postprocessor:metadata] implement archive options (#2421)
'archive', 'archive-format', and 'archive-prefix'
2022-03-20 21:16:46 +01:00
Mike Fährmann
f8230dde43
[instagram] add 'previews' option (#2135) 2022-03-19 15:26:40 +01:00
Mike Fährmann
fc277fa45f
[seiga] require authentication with 'user_session' cookie (#2372)
Login with username & password would now require entering a 2FA token.

see also 7b009cc893
2022-03-11 02:10:15 +01:00
Mike Fährmann
47cf05c4ab
refactor proxy handling code (#2357)
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
2022-03-10 23:55:35 +01:00
Mike Fährmann
acbb0f5a57
clarify 'metadata.mtime' usage and shortcomings (#2307) 2022-03-08 21:11:06 +01:00
Mike Fährmann
bfa5e61900
[patreon] add explicit 'image_large' file type (#2257)
to allow more control over when and if to download 'large_url' images

4fee3a0e52 forced them to be downloaded
instead of regular images, even though 'large_url' images are most likely
an upscaled version of the original.
2022-03-06 17:07:13 +01:00
Alice
f1cab23724
[skeb] add 'sent-requests' option (#2322) (#2330)
* Update skeb.py

* Update configuration.rst

* flake8
2022-02-28 22:42:15 +01:00
Mike Fährmann
4385a34e05
[twitter] fix handling of 429 responses (fixes #2339)
Twitter doesn't return a valid JSON response for 429 errors anymore.
2022-02-28 16:42:55 +01:00
Mike Fährmann
59749555b1
[postprocessor:mtime] add 'event' option (#2307) 2022-02-22 23:27:40 +01:00
Mike Fährmann
7958995398
[postprocessor:metadata] implement 'mtime' option (#2307) 2022-02-22 23:02:13 +01:00
Mike Fährmann
254a5b26e0
[twibooru] add extractors for searches, galleries, and posts
(#2219)
2022-02-18 23:43:57 +01:00
Mike Fährmann
f5b2a9fcc6
add 'signals-ignore' option (#2296) 2022-02-13 23:00:50 +01:00
Mike Fährmann
f1c853c6ef
[furaffinity] add 'layout' option (#2277)
to be able to force gallery-dl to parse according to a specific layout
in case its auto-detect fails
2022-02-11 00:28:47 +01:00
Mike Fährmann
86fa412b47
[hitomi] add 'format' option (#2260)
default is 'webp' since downloading original files is no longer allowed
2022-02-03 23:32:19 +01:00
Mike Fährmann
492436f936
[twitter] add 'warnings' option (#2258)
disable reporting any non-fatal errors by default
2022-02-02 18:37:19 +01:00
Mike Fährmann
de754590e0
add --source-address command-line option (closes #2206) 2022-01-21 17:07:56 +01:00
Mike Fährmann
f2e8aedd74
[twitter] changes to 'cards' option
- change default value to 'true'
- only invoke youtube-dl for cards unsupported by gallery
  when 'cards' is set to "ytdl"

"cards": true   --> only download card images
"cards": "ytdl" --> download card images and
                    use youtube_dl on otherwise unsupported cards
2022-01-15 22:02:57 +01:00
Mike Fährmann
3c79c9b271
document extended blacklist/whitelist syntax (#2025)
and not just in the commit message of 010d65dc
2022-01-06 23:36:57 +01:00
Mike Fährmann
fa7d92f7a9
add docs for 'extractor.generic.enabled' 2021-12-29 22:46:17 +01:00
Mike Fährmann
64cf26eaf4
allow specifying sleep-* options as string
either as single value or as range: "3.5", "2.1 - 5.0"
2021-12-18 23:28:56 +01:00
Mike Fährmann
62692c6842
[exhentai] add 'source' option
setting it to "hitomi" downloads the corresponding gallery from
hitomi.la; might be extended to other sources in the future
2021-12-16 23:16:19 +01:00
Mike Fährmann
099ed72de7
[hitomi] disable extra 'metadata' by default
safes one HTTP request that not needed with default filename settings
2021-12-16 22:21:07 +01:00
Mike Fährmann
f587458a3c
[twitter] include '4096x4096' as a default image fallback
(closes #2107, closes #1881)
2021-12-15 23:19:30 +01:00
Mike Fährmann
f4e3cee6ac
use yt-dlp by default (#1850, #2028) 2021-11-29 18:24:26 +01:00
Mike Fährmann
f1b142e993
{kemonoparty[ change default 'files' order to attachments,file,inline
(#1991)
2021-11-29 04:41:30 +01:00
Alice
612850438e
[skeb] add 'thumbnails' option (#2047) (#2051) 2021-11-23 21:16:42 +01:00
Mike Fährmann
11a3d96d13
[mangadex] load additional metadata using includes[] directives
- always provide 'artist', 'author', and 'group' metadata fields (#2049)
- remove 'metadata' option
2021-11-22 01:16:33 +01:00
Mike Fährmann
c67756e187
[kemonoparty] add 'dms' option (#2008) 2021-11-20 23:36:16 +01:00
Mike Fährmann
9bc83af3a6
[kemonoparty] 'postfile' -> 'file' (#1991)
to stay consistent with the existing file types for kemono
2021-11-19 01:50:48 +01:00
Mike Fährmann
d433735750
[kemonoparty] skip duplicate files (#2032, #1991, #1899)
Extract the SHA-256 file hash from URLs
and skip files with the same hash in the same post.

- provide a 'hash' metadata field (empty string if not available)
- remove 'patreon-skip-file' option
2021-11-17 22:44:15 +01:00
Mike Fährmann
d4ec245554
[kemonoparty] implement a 'files' option (#1991)
similar to 8d676151
2021-11-17 22:43:41 +01:00
Mike Fährmann
7c50f473e4
[ytdl] document 'cmdline-args' and 'config-file' options (#1680) 2021-11-12 18:54:04 +01:00
Mike Fährmann
b315a0ecef
[redgifs] update to API v2 (#1984) 2021-11-04 21:31:20 +01:00
Mike Fährmann
f0fc3b0ba1
[kemonoparty] add 'comments' option (#1980) 2021-11-03 23:02:13 +01:00
Mike Fährmann
9156e90f1f
[twitter] add 'pinned' option 2021-10-29 22:10:58 +02:00
Mike Fährmann
06b414c9a3
[redgifs] 'gfyId' -> 'id' (#1984) 2021-10-29 02:05:39 +02:00
Mike Fährmann
02a247f4e5
[deviantart] full resolution for non-downloadable images (#293)
Many thanks to @Ironchest337 for discovering this method
and providing a well-documented implementation.
2021-10-24 21:11:12 +02:00
Mike Fährmann
8bb442f20d
[redgifs][gfycat] provide fallback URLs (fixes #1962)
and extend the 'format' option
2021-10-22 22:47:29 +02:00
Mike Fährmann
8d676151b7
[patreon] implement 'files' option (#1935) 2021-10-17 04:17:10 +02:00
Mike Fährmann
373d3e1c57
[seisoparty] implement login with username & password (#1906) 2021-10-08 22:44:31 +02:00
Mike Fährmann
d93b5474c3
[mangadex] update parameter handling for API requests
- move common parameters into '_pagination()'
- add 'ratings' (#1908) and 'api-parameters' options
2021-10-05 19:46:48 +02:00
Mike Fährmann
cd66c3c415
[twitter] add 'size' option (#1881) 2021-10-05 19:14:54 +02:00
Mike Fährmann
df8050b81d
[postprocessor:compare] add 'equal' option (#1592)
Move functionality from cdd72e14 to its own option,
where it can be used with any 'action'
2021-10-05 03:03:37 +02:00
Mike Fährmann
cdd72e1413
[postprocessor:compare] extend 'action' option (#1592)
allow setting it to "abort", "terminate", or "exit" as with 'skip'
2021-10-01 20:18:36 +02:00
Mike Fährmann
8c29a6e491
add example for 'url-metadata' option (closes #1884) 2021-09-28 22:59:37 +02:00
Mike Fährmann
d0761454b1
implement a download progress indicator (#1519) 2021-09-28 22:48:58 +02:00
Mike Fährmann
140a7bfdcd
update --clear-cache command in deviantart client-id instructions 2021-09-23 18:02:19 +02:00
Mike Fährmann
e69ee41f25
implement 'page-reverse' option (#1854) 2021-09-23 18:02:19 +02:00
Mike Fährmann
7b05600056
[pinterest] only mention cookies for authentication (#1856)
regular login is broken due to CAPTCHA
2021-09-16 16:43:05 +02:00
Mike Fährmann
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options (#1835)
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
2021-09-14 17:40:05 +02:00
Mike Fährmann
bd845303ad
implement a way to shorten filenames with east-asian characters
(#1377)

Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
2021-09-13 21:38:33 +02:00
Mike Fährmann
4ec11af6a4
[kemonoparty] implement login with username & password (#1824) 2021-09-09 01:06:25 +02:00
Mike Fährmann
44f572c27f
[deviantart] implement a 'auto-unwatch' option (#1466, #1757) 2021-09-08 00:27:28 +02:00
Mike Fährmann
96fec14ef7
[deviantart] rename 'watch' option to 'auto-watch'
(#1466, #1757)

Similar reason as in e05a96db.
'watch' is already used by the WatchExtractor class.
2021-08-29 21:10:53 +02:00