Mike Fährmann
4e86aaa4db
document 'extractor.headers' and 'extractor.ciphers'
2023-01-16 14:49:56 +01:00
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option ( #3313 )
...
do not retry 429 and 430 by default
2023-01-14 17:25:30 +01:00
Mike Fährmann
512abeb4ae
[booru] add 'url' option
2023-01-13 16:32:32 +01:00
Mike Fährmann
c87bd1a752
[danbooru] extend 'metadata' option
...
make it possible to specify a custom list of metadata includes
2023-01-13 16:20:01 +01:00
Mike Fährmann
c0d7d2be35
[downloader:http] add 'validate' option
2023-01-11 15:37:40 +01:00
Mike Fährmann
4833ec323e
[imagefap] add 'folder' extractor ( #3504 )
2023-01-08 16:57:31 +01:00
Mike Fährmann
cb9d100bac
update Authentication support in docs
...
https://github.com/mikf/gallery-dl/pull/3075#issuecomment-1374723412
2023-01-08 16:57:31 +01:00
Mike Fährmann
362cd6991b
[pixiv] implement 'metadata-bookmark' option ( #3417 )
2023-01-07 23:19:43 +01:00
Mike Fährmann
0895e6afee
merge #3462 : [docs] Update links and fix field typo
2023-01-07 15:22:03 +01:00
Mike Fährmann
4b9bb632f2
[docs] update links
2023-01-07 15:21:40 +01:00
Mike Fährmann
d1dd52349a
merge #3189 : [tcbscans] add 'chapter' and 'manga' extractors
2023-01-06 13:42:27 +01:00
Mike Fährmann
ac78712618
merge #3496 : [docs] update OAuth guide in README.rst
2023-01-06 13:36:23 +01:00
Mike Fährmann
2f31d21509
merge #3455 : [twitter] apply tweet type checks before uniqueness check
2023-01-06 13:32:50 +01:00
Mike Fährmann
64435cc37e
[twitter] update docs
2023-01-06 13:32:08 +01:00
Mike Fährmann
b125fcafc5
update docs/configuration.rst
...
- document metadata.indent
- add concrete mappings to object types
- fix lists for code examples
- fix typos
2023-01-06 13:07:33 +01:00
enduser420
e8541a131d
[tcbscans] add 'chapter' and 'manga' extractors
2023-01-06 16:16:31 +05:30
Mike Fährmann
3c03928d75
simplify 'docs/options.md' generation
...
use the plain text output from '--help'
instead of trying to generate 'fancy' HTML with Markdown restrictions
2023-01-06 11:21:47 +01:00
ClosedPort22
c94692f39b
[docs] document 'access-token' for Mastodon instances
2023-01-06 13:20:41 +08:00
Mike Fährmann
4fd6da474f
merge #3473 : [twitter] fix crash when using 'expand' and 'syndication'
2023-01-05 14:19:47 +01:00
Mike Fährmann
818f6fa71f
merge #3453 : [twitter] document #1396 and #2712
2023-01-05 14:18:21 +01:00
ClosedPort22
455481ad76
[twitter] improve documentation
...
This includes:
- The automatic expansion of conversations (#2712 )
- How to retrieve age-restricted content from search results (#1396 )
2023-01-05 12:14:42 +08:00
Mike Fährmann
f29ba089ff
merge #3474 : [fanleaks] add 'post' and 'model' extractors
2023-01-04 17:39:17 +01:00
Mike Fährmann
6933727b45
merge #3483 : [twitter] implement 'syndication=extended'
2023-01-04 17:36:17 +01:00
Mike Fährmann
9116398c1c
[pinterest] add 'domain' option ( #3484 )
...
use input URL domain by default
2023-01-04 17:20:14 +01:00
Mike Fährmann
f1a715dbcd
improve docs/options.md
...
- use non-breaking hyphens
- don't replace '…' in examples
- don't force width for long option column
2023-01-04 14:41:05 +01:00
Mike Fährmann
294108c90a
[pinterest] support 'All Pins' boards ( #2855 , #3484 )
2023-01-03 19:11:20 +01:00
Mike Fährmann
13c16490bd
handle subcategory names ending with 'y'
2023-01-03 18:51:42 +01:00
Mike Fährmann
50d89d4acb
docs/options.md: use a separate table for each option group
2023-01-03 16:04:48 +01:00
Mike Fährmann
7de6b7f489
add list of command-line options as docs/options.md ( #3464 )
2023-01-02 19:14:16 +01:00
ClosedPort22
6853b14be3
[twitter] apply suggestions from code review
...
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2023-01-02 21:03:01 +08:00
Mike Fährmann
4611237f8c
merge #3457 : [danbooru] extract uploader metadata (if option is set)
2023-01-01 21:20:39 +01:00
Mike Fährmann
e7522482bb
merge #3463 : [lynxchan] support 'bbw-chan.nl'
2023-01-01 15:50:09 +01:00
Mike Fährmann
0ed5986062
document custom 'output.mode' ( #2529 )
2022-12-30 17:14:42 +01:00
enduser420
5a740ef78b
[fanleaks] add 'post' and 'model' extractors
2022-12-30 19:24:05 +05:30
ClosedPort22
7c8eab8d52
[twitter] implement 'syndication=extended'
...
to be able to fetch extended user metadata
2022-12-30 20:48:41 +08:00
ClosedPort22
be3286206a
[twitter] assume 'conversation_id' when using syndication
...
not possible to expand replies at the momemt
2022-12-30 13:57:37 +08:00
ClosedPort22
ce8dbb1ccc
[twitter] fix crash when using 'expand' and 'syndication'
...
caused by KeyError: 'conversation_id_str'
2022-12-30 12:45:44 +08:00
Mike Fährmann
d651d45239
implement specifying ranges in slice notation ( #918 , #2865 )
...
e.g.
- '1:101' or ':101' or ':101:' for files 1 to 100
- '1::2' or '::2' for every second file
- '1:101:5' or ':101:5' for files 1, 6, 11, ..., 91, 96
(the second argument specifies the first index NOT included)
2022-12-27 18:21:12 +01:00
enduser420
527bb2c4ab
[lynxchan/bbw-chan] add 'thread' and 'board' extractors
2022-12-26 20:59:42 +05:30
pi_allen
64902f518e
[docs] Update links and fix field typo
2022-12-26 07:46:41 +01:00
ClosedPort22
5d5ea58887
[twitter] update docs for 'quoted' & 'unique'
2022-12-25 15:59:26 +08:00
blankie
f82ee93676
[danbooru] extract uploader metadata (if metadata is set)
2022-12-25 11:00:59 +07:00
Mike Fährmann
1800bd7d14
allow '*-filter' options to be a list of expressions
2022-12-23 22:20:21 +01:00
lx30011
895b41f1ac
[jschan] add generic jschan extractor
2022-12-23 00:32:52 +01:00
Mike Fährmann
73ab5d84c0
update docs/configuration.rst
2022-12-20 17:30:46 +01:00
Mike Fährmann
bee354c264
Merge pull request #3415 from enduser420/extractor/fapello
...
[fapello] add 'post', 'user' and 'path' extractors
2022-12-17 20:05:46 +01:00
ClosedPort22
dd4a4a3fa6
[e621] softcode the pagination threshold
2022-12-17 21:06:47 +08:00
enduser420
e5076ba056
[fapello] add 'post', 'user' and 'path' extractors
2022-12-16 16:53:32 +05:30
ClosedPort22
d0ad6d0e67
[e621] implement manual pagination mode
2022-12-16 16:15:36 +08:00
Mike Fährmann
1317625ec4
[webmshare] add 'video' extractor ( #2410 )
2022-12-14 19:59:07 +01:00
enduser420
41bf236d36
[lynxchan] add generic extractors for lynxchan imageboards ( #3394 )
...
* [lynxchan] add generic extractors for lynxchan imageboards
includes kohlchan.net, endchan.org:wq
* [lynxchan] set pop default to empty tuple
* Apply suggestions from code review
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2022-12-12 13:13:39 +01:00
Mike Fährmann
3c75c3bbc4
[soundgasm] add 'user' extractor ( #3384 )
...
based on code from PR #3388 by @enduser420
2022-12-12 12:23:59 +01:00
Mike Fährmann
2952add4a8
[reddit] increase 'id-max' default value ( #3397 )
...
to float("inf")
2022-12-12 12:10:06 +01:00
Mike Fährmann
6b6f886dcf
[bunkr] update domain ( #3391 )
...
and improve bunkr/app.bunkr handling
2022-12-11 23:18:01 +01:00
Mike Fährmann
eb94568e1f
[soundgasm] add 'audio' extractor ( #3384 )
2022-12-09 23:19:07 +01:00
Mike Fährmann
dfe7b23579
support Firefox containers for --cookies-from-browser ( #3346 )
2022-12-09 19:43:55 +01:00
Mike Fährmann
4a3a1f4c87
[komikcast] update domain and fix extraction
2022-12-06 22:00:23 +01:00
Mike Fährmann
a42ba25ca1
[foolslide] remove 'kireicake'
...
site redirects to (unclaimed) mangadex group
2022-12-05 19:47:56 +01:00
Mike Fährmann
86f0597c95
[kissgoddess] remove module
...
site does not host albums anymore
2022-12-05 19:28:50 +01:00
Mike Fährmann
fceaee3c4f
[lolisafe] remove zz.ht
2022-12-04 16:21:13 +01:00
enduser420
54844944ab
[pixhost] add 'gallery' support ( #3353 )
2022-12-04 16:19:14 +01:00
enduser420
213676c785
[fapachi] add 'post' and 'user' extractors ( #3347 )
...
* [fapachi] add 'post' and 'user' extractors
* [fapachi] add 'keyword' to test
* [fapachi] remove whitespaces
2022-12-02 13:54:29 +01:00
Mike Fährmann
80102fa367
[downloader:http] add 'retry-codes' option ( #3313 )
2022-12-01 11:08:23 +01:00
ClosedPort22
b14b33f19e
Implement version-metadata option ( #3201 )
2022-11-27 16:09:42 +01:00
Mike Fährmann
6cb12f513b
[nitter] support quoted Tweets
...
- distinguish between regular and quoted Tweets and media
- add 'quoted' option and metadata field
2022-11-26 11:23:03 +01:00
Mike Fährmann
a41d093bb1
[nitter] add 'retweets' option ( #3278 )
2022-11-25 19:53:28 +01:00
Mike Fährmann
e31d12139c
[nitter] add 'videos' option ( #3279 )
...
with the same semantics as for twitter
2022-11-24 22:56:01 +01:00
Mike Fährmann
501d9bccfe
[artstation] add 'max-posts' option ( #3270 )
2022-11-23 22:00:18 +01:00
Mike Fährmann
b1ad6f2289
[artstation] add 'pro-first' option ( #3273 )
2022-11-23 21:45:20 +01:00
Mike Fährmann
42481aed59
[formatter] implement 'S' format specifier ( #3266 )
...
to Sort lists
2022-11-21 21:44:42 +01:00
Mike Fährmann
6b97dcf2e0
[postprocessor:metadata] add 'private' option
2022-11-20 15:27:36 +01:00
Mike Fährmann
08fd1ff835
[twitter] add 'avatar' and 'background' extractors ( #349 , #3023 )
2022-11-18 23:06:22 +01:00
Mike Fährmann
6379157543
[instagram] use REST API by default
...
regardless of logged in status
2022-11-17 17:15:38 +01:00
enduser420
7897f68225
[wallhaven] update 'user' extractor ( #3226 )
...
* [wallhaven] update 'user' extractor
* [wallhaven] update 'configuration.rst'
add 'extractor.wallhaven.include' entry
* [wallhaven] add 'wallhaven.include' in gallery-dl.conf
2022-11-15 17:35:52 +01:00
enduser420
5a68b5cb3c
[wallhaven] add 'user' extractor ( #3213 )
2022-11-15 12:25:12 +01:00
Mike Fährmann
6c153750fa
[nitter] add extractors for Nitter instances ( #2696 )
2022-11-15 11:44:16 +01:00
Mike Fährmann
9f06e79868
implement '"user-agent": "browser"' ( #2636 )
2022-11-13 19:17:39 +01:00
enduser420
039d06c8f6
[mangaread] add 'chapter' and 'manga' extractors
2022-11-13 16:00:34 +05:30
Mike Fährmann
70c7fbe89a
[instagram] add 'guide' extractor ( #3192 )
2022-11-11 20:21:41 +01:00
Mike Fährmann
8d543b040d
update docs
...
- document '"mode": "jsonl"' for metadata pp
- remove 'username' and 'password' from 'instagram'
- add 'ytdl' example (#3180 )
2022-11-09 22:17:08 +01:00
Mike Fährmann
171262c1b6
[instagram] remove login support
...
broken feature that I cannot get to work anymore
2022-11-08 21:48:31 +01:00
Mike Fährmann
895f36e53b
[postprocessor:metadata] add 'open' and 'encoding' options
2022-11-07 15:37:22 +01:00
Mike Fährmann
870e6a48a0
implement 'http-metadata' option
...
or at least attempt to.
2022-11-05 18:29:29 +01:00
Mike Fährmann
ccb80f1b8b
[uploadir] add support for 'uploadir.com' ( #3162 )
2022-11-05 14:25:09 +01:00
enduser420
c01cad599a
[lolisafe] add support for xbunkr ( #3156 )
2022-11-04 16:13:52 +01:00
Mike Fährmann
bca9f965e5
[downloader:http] add 'chunk-size' option ( #3143 )
...
and double the previous default from 16384 (2**14) to 32768 (2**15)
2022-11-02 16:50:26 +01:00
Mike Fährmann
c12a97bcde
[postprocessor] add 'post-after' event ( #3117 )
2022-10-31 14:35:48 +01:00
Mike Fährmann
58d97188b4
[mastodon] add 'bookmark' extractor ( #3109 )
2022-10-26 21:28:50 +02:00
Mike Fährmann
7c6af27eb8
[tumblr] add 'fallback-*' options ( #2957 )
...
specifically 'fallback-delay' and 'fallback-retries'
and change default number of retries to 2 (down from 3)
2022-10-26 13:59:09 +02:00
enduser420
ac6111e693
[mangasee] add support for 'mangalife' ( #3086 )
2022-10-21 16:46:36 +02:00
Mike Fährmann
a7d23f1484
[vichan] add generic extractors for vichan imageboards
...
includes 8kun.top, smuglo.li, and wikieat.club
2022-10-21 14:40:45 +02:00
thatfuckingbird
062ef238a6
add support for aibooru (using danbooru extractor) ( #3075 )
2022-10-19 11:53:59 +02:00
enduser420
0163ca86f7
[smugloli] add smugloli extractors ( #3060 )
2022-10-19 11:25:18 +02:00
Mike Fährmann
cf86f68864
[instagram] add 'avatar' extractor ( #929 , #1097 , #2992 )
2022-10-19 10:58:42 +02:00
Mike Fährmann
618c81afdf
[ngomik] remove module
...
"Access denied"
2022-10-19 10:47:25 +02:00
Mike Fährmann
52d1eb928d
[pixiv] extend 'metadata' option ( #3057 )
...
make it usable for all 'pixiv' extractors
2022-10-16 15:32:31 +02:00
Mike Fährmann
0714274f1f
[instagram] remove 'channel' extractor
2022-10-15 21:13:58 +02:00
Mike Fährmann
3b369ce3d1
[nijie] add 'followed' extractor ( #3048 )
2022-10-14 14:59:18 +02:00
Mike Fährmann
c4a62a48ae
[nijie] add 'feed' extractor ( #3048 )
2022-10-14 12:03:00 +02:00
Mike Fährmann
f1f89b2436
[tumblr] add 'offset' option
2022-10-11 10:54:23 +02:00
Mike Fährmann
15a4d7696f
[docs] update 'adjust-extensions' explanation ( #3031 )
2022-10-11 10:48:28 +02:00
Mike Fährmann
1696f68a68
[8chan] add 'thread' and 'board' extractors ( #2938 )
2022-10-11 10:47:19 +02:00
Mike Fährmann
560f7b41d8
[vk] add 'tagged' extractor ( #2997 )
2022-10-08 15:41:39 +02:00
Mike Fährmann
7f30a0d7a7
add 'path-extended' option ( #3021 )
2022-10-08 15:41:39 +02:00
Mike Fährmann
14717f3fc9
[deviantart] add 'group' option ( #3018 )
...
disabling this option allows to better download from deleted accounts
2022-10-08 15:41:21 +02:00
enduser420
f0321f423d
[2chen] Add 2chen.moe extractor ( #2707 )
...
* [2chen] Add 2chen.moe extractor
* change "==" to is
* fix for "test_unique_pattern_matches"
* fix regex pattern and group matching
* fix regex again
* [2chen] add 'reply_no' and 'hash' metadata and change 'filename_fmt'
also made an entry in supportedsites.md
* [2chen] unescape 'title'
* [2chen] partition() -> rpartition()
* [2chen] extract 'date' and 'name' metadata
* [2chen] remove 'offset' argument
* [2chen] do some changes
* [2chen] do some more changes
* [2chen] unescape 'name' and 'filename'
2022-10-04 22:18:13 +02:00
enduser420
f7ba19a1c0
[nana] add 'nana' extractors ( #2967 )
2022-10-04 09:23:24 +02:00
Mike Fährmann
4089bceddd
[sankaku] implement 'refresh' option ( #2958 )
2022-09-30 19:55:48 +02:00
Mike Fährmann
63e0924927
[pixiv] add 'series' extractor ( #2964 )
2022-09-27 23:24:03 +02:00
Mike Fährmann
aa49bf13d2
[instagram] add 'api' option
2022-09-26 22:05:29 +02:00
Mike Fährmann
ac45ed2764
[skeb] implement 'filters' option ( #2945 )
2022-09-26 15:17:28 +02:00
Mike Fährmann
08b0c4c0b1
[docs] add kemono text content example ( #2963 )
2022-09-25 14:07:41 +02:00
Mike Fährmann
0393e59535
[newgrounds] add 'games' extractor ( #2955 )
2022-09-24 12:34:37 +02:00
Mike Fährmann
4b2a006871
[skeb] add 'search' extractor ( #2945 )
2022-09-21 17:57:55 +02:00
Mike Fährmann
eef50c1f28
[blogger] split 'search' extractor ( #2930 )
2022-09-19 21:01:21 +02:00
enduser420
bd846abba0
[hotleak] add hotleak extractor ( #2909 ) ( #2890 )
2022-09-18 13:37:16 +02:00
Mike Fährmann
e99a9b2aff
[twitter] improve 'cards-blacklist' ( #2875 )
...
allow blacklisting domains and 'name:domain',
where 'domain' depends on a card's 'vanity_url' value
2022-09-17 17:46:34 +02:00
Mike Fährmann
46fe469c53
[tumblr] implement 'ratelimit' option ( #2919 )
2022-09-17 14:10:33 +02:00
Mike Fährmann
b36125333f
[postprocessor:zip] implement 'files' option ( #2872 )
2022-09-09 11:41:27 +02:00
Mike Fährmann
3cb8327c60
[zerochan] add 'metadata' option ( #2861 )
2022-09-02 23:25:19 +02:00
blankie
9745b48830
[tumblr] attempt to fetch high-quality inline images ( #2877 )
...
* [tumblr] attempt to fetch high-quality images (again)
Fixes #1846 , and fixes #1344
* slight refactor
* update configuration.rst entry
2022-08-31 10:53:50 +02:00
Mike Fährmann
4d78ca89db
[twitter] add 'cards-blacklist' option ( #2875 )
2022-08-31 10:28:25 +02:00
Mike Fährmann
8839b0d2ee
add section about global replacement fields to formatting.md
...
(#2862 )
2022-08-30 21:32:22 +02:00
Mike Fährmann
f16fbe9f93
document 'extractor.twitter.expand' ( #2848 )
2022-08-30 18:16:20 +02:00
Mike Fährmann
7e385ed63e
[foolfuuka] update domains
...
- remove nyafuu
- add rozenarcana (https://archive.alice.al/ )
- add tokyochronos (https://www.tokyochronos.net )
2022-08-26 17:57:17 +02:00
Mike Fährmann
67bad04dda
[formatter] add 'g' conversion to sluGify a string ( #2410 )
2022-08-26 17:57:17 +02:00
enduser420
3d87cedc58
[jpgchurch] rework the image extractor
...
now the image extractor can recognize if an image if from an album
also removed some unnecessary methods
2022-08-26 15:58:19 +05:30
enduser420
574e38a287
[kemonoparty] add 'favorites' option ( #2826 ) ( #2831 )
...
* [kemonoparty] add 'favorites' option (#2826 )
* [kemonoparty] add regex for the url parameter and fallback on the config
option
* [kemonoparty] simplify
2022-08-18 18:01:42 +02:00
Mike Fährmann
a799fae2df
[catbox] add 'album' extractor ( #2410 )
...
adapted from https://github.com/mikf/gallery-dl/pull/2805
- rewrite using GalleryExtractor
- extract more metadata
- match lolisafe names
- add test
2022-08-18 18:00:24 +02:00
Martin Juhasz
9ea521bec5
[oauth] add host config option ( #2806 )
2022-08-14 17:08:01 +02:00
Mike Fährmann
df1c643dda
[tumblr] attempt to extract full-resolution photos
...
- for photos with apparent width == 2048 or height == 3072
- can be disabled with 'original' option
2022-08-10 20:01:46 +02:00
Mike Fährmann
a69f198ac9
[wallhaven] implement 'metadata' option ( #2803 )
2022-08-08 21:46:36 +02:00
Mike Fährmann
7d1a95ada6
implement 'path-metadata' option ( #2734 )
2022-07-30 12:31:45 +02:00
Mike Fährmann
8cf5981ded
[skeb] add option to download 'article' images ( #1031 )
2022-07-29 16:32:00 +02:00
Mike Fährmann
98af5a0409
[zerochan] implement login with username & password ( #1434 )
2022-07-29 12:56:20 +02:00
Mike Fährmann
3a8addfe45
[zerochan] add 'tag' and 'image' extractors ( #1434 )
2022-07-27 22:58:23 +02:00
Chew Shee Yang
977d53b640
[Instagram] Add support for user's saved collection ( #2769 )
...
* [Instagram] Add support for user's saved collection
* [Instagram] Run formatter
* [Instagram] Simplify collection_id retrieval and add metadata
* [Instagram] Fix bug when params is not passed to _pagination_api
2022-07-27 10:49:45 +02:00
Mike Fährmann
0c73914848
[postprocessor:metadata] implement 'mode: modify' ( #2640 )
2022-07-19 12:24:26 +02:00
Mike Fährmann
f3de6b7a87
[postprocessor:metadata] implement 'mode: delete' ( #2640 )
2022-07-19 00:57:29 +02:00
Mike Fährmann
eb68d45544
add global 'warnings' option ( #2762 )
2022-07-18 22:20:30 +02:00
Mike Fährmann
f225247670
[gelbooru] add support for api_key and user_id ( #2767 )
2022-07-18 18:46:31 +02:00
Mike Fährmann
46f11a3118
[bunkr] fix extraction ( #2732 )
...
move bunkr.is code to its own module
2022-07-15 13:00:57 +02:00
Mike Fährmann
fe2b3d57d4
[komikcast] update domain
2022-07-12 23:07:58 +02:00
Mike Fährmann
2dc57637cf
[foolfuuka] remove archive.wakarimasen.moe
2022-07-10 23:13:49 +02:00
Mike Fährmann
74865adae5
implement 'format-separator' option ( #2737 )
...
a global option, that servers as a workaround for shortcomings due to
lack of a proper format string parser
2022-07-10 13:31:43 +02:00
bradenhilton
117eeefda0
[postprocessor:mtime] add 'value' option ( #2739 )
2022-07-08 20:56:01 +02:00
Mike Fährmann
90ae48c40c
[formatter] implement 'O' format specifier ( #2736 )
...
to apply a UTC offset to 'date' values and other datetime objects
2022-07-08 12:51:03 +02:00
Mike Fährmann
1d14928bd9
[twitter] ignore previously seen Tweets ( #2712 )
...
occurs primarily for /with_replies results when logged in
2022-07-03 16:13:53 +02:00
Mike Fährmann
4b2a0a0eda
[twitter] implement 'strategy' option ( #2712 )
...
to be able to better control what Tweets get used an returned
for twitter.com/USER URLs.
2022-07-03 14:29:15 +02:00
Mike Fährmann
124b5c6410
[postprocessor:ugoira] enable 'mtime' by default ( #2714 )
2022-07-01 20:59:19 +02:00
