Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Code Issues Releases Wiki Activity
5,063 Commits 6 Branches 164 Tags 16 MiB
ade93c5397
Commit Graph

1090 Commits

Author SHA1 Message Date
Mike Fährmann
c76f0f3a1b
[misskey] update 
- rename to 'MisskeyFavoriteExtractor'
- add 'access-token' option to docs
- add test URLs for other instances
- simplify 'pattern'
 2023-05-23 22:17:17 +02:00
Mike Fährmann
3fca455b82
[pixiv] add 'embeds' option (#1241) 2023-05-23 12:14:06 +02:00
Mike Fährmann
9810ab35af
[docs] update formatting.md 
- add element access with quotes
- fix \f formatting
- remove note about typing \f in a shell (9e6c9813)
 2023-05-22 22:24:23 +02:00
Mike Fährmann
69865dcc05
[formatter] implement slicing strings as bytes (#4087) 
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
 2023-05-22 18:30:45 +02:00
Mike Fährmann
e6f55d1555
[imagechest] add API support and 'access-token' option (#4065) 2023-05-20 17:00:27 +02:00
Mike Fährmann
77abcf5ab3
[gofile] automatically fetch 'website-token' by default 
the hardcoded token changed yet again
 2023-05-20 17:00:27 +02:00
Mike Fährmann
e3fed9bd17
[tcbscans] update domain to 'tcbscans.com' (#4080) 2023-05-20 13:59:35 +02:00
Mike Fährmann
a83983c651
[instagram] add 'order-posts' option (#4017, #3993) 2023-05-18 22:50:12 +02:00
Mike Fährmann
d680623db3
[instagram] add 'order-files' option (#4017, #3993) 2023-05-18 22:34:33 +02:00
Mike Fährmann
5037013e2b
[gofile] update 'website-token' (#4056) 2023-05-15 14:59:44 +02:00
Mike Fährmann
20dc13f832
[pixiv] initial 'novel' support (#1241, #4044) 
supported URLs are
- https://www.pixiv.net/novel/show.php?id=<ID>
- https://www.pixiv.net/novel/series/<ID>
- https://www.pixiv.net/en/users/<ID>/novels
 2023-05-12 16:34:08 +02:00
Mike Fährmann
1406f7125f
[4chanarchives] add 'thread' and 'board' extractors (#4012) 2023-05-06 20:45:57 +02:00
Mike Fährmann
285391df43
add '-C' as short option for '--cookies' 
and put cookie options into their own section
 2023-05-06 20:45:44 +02:00
Mike Fährmann
b9b1cdd71b
add '--cookies-export' command-line option 2023-05-06 17:37:49 +02:00
Mike Fährmann
8520de57f0
[imgur] add 'favorite-folder' extractor (#4016) 2023-05-06 15:10:13 +02:00
Mike Fährmann
a14b63d941
support selecting a domain for '--cookies-from-browser' 
for example 'gallery-dl --cookies-from-browser firefox/twitter.com'
 2023-05-06 15:09:49 +02:00
Mike Fährmann
3ca5dac8b6
extend 'cookies-update' functionality 
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser

To convert browser cookies to cookies.txt format:
  gallery-dl --cookies-fr chromium \
             -o cookies-update=cookies.txt \
             --no-download \
             http://example.org/file.jpg
 2023-05-04 15:10:47 +02:00
Mike Fährmann
79f47f98dd
[nana] remove module 
permanently gone since 2023-03-13
 2023-05-03 18:05:53 +02:00
Mike Fährmann
0e74df1de8
[420chan] remove module 
offline since 2022-06-01
 2023-05-03 17:46:21 +02:00
Mike Fährmann
0c46758a93
[foolslide] remove 'sensescans.com' 
group moved to mangadex
https://mangadex.org/group/1071e71d-cc55-4fa6-81d1-4b5913a2fde5/sense-scans
 2023-05-02 20:09:04 +02:00
Mike Fährmann
a08fdfac6e
[foolfuuka] add 'archive.palanq.win' 2023-05-02 19:58:55 +02:00
Mike Fährmann
1870df8b23
[foolfuuka] remove 'tokyochronos.net' 2023-05-02 19:25:50 +02:00
Mike Fährmann
ef4e2d8178
[foolfuuka] remove 'archive.alice.al' 2023-05-02 19:23:26 +02:00
Mike Fährmann
c182094ebf
merge #3748: [downloader:http] add 'consume-content' option 2023-04-26 23:03:18 +02:00
Mike Fährmann
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites (#3734) 
add support for
- loudbooru.com       (#3734)
- booru.cavemanon.xyz (#3734)
- giantessbooru.com   (#943)
- tentaclerape.net
 2023-04-26 19:20:44 +02:00
Mike Fährmann
28419bf45a
[itchio] add 'game' extractor (#3923) 2023-04-26 19:20:43 +02:00
Mike Fährmann
5297ee0cd9
[tumblr] add 'day' extractor (#3951) 2023-04-24 22:01:47 +02:00
Mike Fährmann
b81ce381f0
fix backticks for 'extractor.redgifs.format' docs 
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1518331927
 2023-04-21 23:12:01 +02:00
Mike Fährmann
aaf58a1259
[imgur] document 'client-id' option (#3937) 2023-04-21 15:08:50 +02:00
Mike Fährmann
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors 
(#731, #2720, #3812)
 2023-04-19 15:28:26 +02:00
Mike Fährmann
81bd2af83e
[2chen] update domain to sturdychan.help 2023-04-19 13:54:44 +02:00
Mike Fährmann
5b635f2317
[imxto] add 'gallery' extractor (#1289) 2023-04-17 20:49:09 +02:00
Mike Fährmann
d253a3c542
merge #3841: [urlshortener] add support for bit.ly & t.co 2023-04-15 18:08:21 +02:00
Mike Fährmann
5e63942b37
[urlshortener] update 2023-04-15 18:06:06 +02:00
Mike Fährmann
0a7eee3ee0
[deviantart] add 'public' option 2023-04-08 23:04:34 +02:00
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata' 
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
 2023-04-02 20:11:52 +02:00
ClosedPort22
71b26adb9b
[urlshortener] add tinyurl.com as an example 2023-03-29 13:37:26 +08:00
Mike Fährmann
421db26aff
[bunkr] update domain to 'bunkr.la' 2023-03-28 20:10:36 +02:00
ClosedPort22
9e2a945013
[urlshortener] add support for bit.ly & t.co 2023-03-29 00:06:41 +08:00
Mike Fährmann
89a67c45e0
[nitter] support nitter.it (#3819) 2023-03-25 13:29:22 +01:00
Mike Fährmann
197882cf12
[twitter] add 'hashtag' extractor (#3783) 2023-03-22 22:20:40 +01:00
Mike Fährmann
46fdf46f21
[formatter] support loading an f-string from a template file 
"\fTF ~/path/to/file.txt"
 2023-03-20 22:05:33 +01:00
Mike Fährmann
1a4d4a799b
[formatter] support filesystem paths for \fM 2023-03-20 22:01:33 +01:00
Mike Fährmann
00f0233b28
[postprocessor:metadata] add 'skip' option (#3786) 2023-03-17 23:30:11 +01:00
Mike Fährmann
a1ca2404f9
add 'globals' instead of overwriting the default (#3773) 2023-03-16 18:37:00 +01:00
Mike Fährmann
17bd053d94
[hiperdex] fix extraction (#3768) 2023-03-15 14:28:03 +01:00
ClosedPort22
fcaeaf539c
[downloader:http] handle exceptions while consuming content 2023-03-11 21:36:37 +08:00
Mike Fährmann
817fc0fbd1
[nitter] remove nitter.pussthecat.org 
"Shutdown"
 2023-03-09 23:48:39 +01:00
ClosedPort22
df77271438
[downloader:http] add 'consume-content' option 
* fix connection not being released when the response is neither
  successful nor retried
* add the ability to consume the HTTP response body instead of closing
  the connection

reference:

https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
 2023-03-09 21:07:10 +08:00
Mike Fährmann
da9840a39d
[reddit] update 'videos' option (#3712) 
- add 'dash' to directly extract DASH manifest URLs
  (was default behavior since a7c79531)
- change default strategy back to before a7c79531
- disable 'Falling back on generic information extractor' warning
 2023-03-06 12:18:25 +01:00
Mike Fährmann
baf41d7437
[misskey] update (#3717) 
- add module docstring
- add options to docs/gallery-dl.conf
 2023-03-03 15:52:50 +01:00
enduser420
e1867cf5eb [misskey] add 'renotes' and 'replies' options 2023-03-02 19:56:19 +05:30
enduser420
a95b5e0d8e [misskey] add misskey extractors 2023-03-02 19:49:47 +05:30
Mike Fährmann
0d142e403c
[szurubooru] add 'tag' and 'post' extractors (#3583, #3713) 2023-03-01 18:47:10 +01:00
Mike Fährmann
075c965512
add '--config-create' command-line option 
(#2333)
 2023-03-01 14:49:40 +01:00
Mike Fährmann
632d5d7745
allow loading config files in TOML format with --config-toml 2023-02-28 21:54:46 +01:00
Mike Fährmann
d788e6c60c
implement 'globals' option 2023-02-28 18:18:55 +01:00
Mike Fährmann
b14f8d5817
[gelbooru] add 'favorite' extractor (#3704) 
requires logged in cookies to work
 2023-02-27 18:03:47 +01:00
Mike Fährmann
e480a93337
add 'output.stdout', '.stdin', and '.stderr' options 
(#1621, #2152, #2529)

Allow setting custom input/output encodings and options
without having to rely on Python's defaults.
 2023-02-26 14:56:19 +01:00
Mike Fährmann
6ed4309aba
[deviantart] add 'gallery-search' extractor (#1695) 2023-02-22 17:25:31 +01:00
Mike Fährmann
e1df7f73b1
[deviantart] add 'search' extractor 
(#538, #1264, #2954, #2970, #3577)

Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.

TODO: parse HTML search results
 2023-02-20 20:54:46 +01:00
Mike Fährmann
bb11c2a576
merge #3662: [redgifs] add 'collection' extractors 2023-02-18 13:09:17 +01:00
Mike Fährmann
884f1848d6
[redgifs] fix syntax for older Python versions 
and update docs/supportedsites
 2023-02-18 13:07:40 +01:00
Mike Fährmann
2bd8f2f4bd
[pornpics] add 'search' and 'tag' extractors 
(#263, #3544, #3654)
 2023-02-17 23:03:14 +01:00
Mike Fährmann
79bc82884c
[pornpics] add 'gallery' extractor (#263, #3544, #3654) 2023-02-17 15:00:57 +01:00
Mike Fährmann
bd5d08abbc
[catbox] add 'file' extractor (#3570) 2023-02-15 19:50:26 +01:00
Mike Fährmann
2a53e6445c
[bunkr] update domain (#3636) 2023-02-09 19:28:32 +01:00
Mike Fährmann
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options 2023-02-07 18:28:14 +01:00
Mike Fährmann
762a68996b
implement 'archive-pragma' option 2023-02-05 17:00:31 +01:00
Mike Fährmann
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction 
(#3425)
 2023-02-04 21:20:38 +01:00
Mike Fährmann
925b467496
split e621 from danbooru module (#3425) 2023-02-03 19:24:31 +01:00
Mike Fährmann
1ae48a54f8
[twitter] add 'transform' option 2023-02-02 22:01:36 +01:00
Mike Fährmann
78d3960a31
[postprocessor:exec] implement archive options (#3584) 2023-02-01 13:57:16 +01:00
Mike Fährmann
0f7e6c422a
merge #3596: [shopify] support ohpolly.com 2023-01-31 20:13:51 +01:00
enduser420
fcf7030b85 [shopify] support ohpolly.com 2023-02-01 00:31:12 +05:30
Mike Fährmann
4572c122d8
document 'extractor.module-sources' 2023-01-30 23:32:36 +01:00
Mike Fährmann
c2bc70593e
implement ability to load external extractor classes 
- -X/--extractors
- extractor.module-sources
 2023-01-30 23:10:10 +01:00
Mike Fährmann
13a90969c7
merge #3575: [nudecollect] add 'image' and 'album' extractors 2023-01-28 16:04:47 +01:00
Mike Fährmann
abc3619feb
[lexica] add 'search' extractor (#3567) 2023-01-28 16:00:32 +01:00
Mike Fährmann
f58215705a
add '-O/--postprocessor-option' command-line option (#3565) 2023-01-26 14:59:24 +01:00
enduser420
2a5903dc16 [nudecollect] add 'image' and 'album' extractors 2023-01-26 17:25:33 +05:30
Mike Fährmann
58c008e30a
[hiperdex] update domain (#3572) 2023-01-26 12:01:16 +01:00
Mike Fährmann
caae8fefe1
merge #3541: [deviantart] add extractor for status updates 2023-01-23 13:54:51 +01:00
ClosedPort22
3b32671fbd
[deviantart] add extractor for status updates 
extract user status updates using the '/user/statuses/' endpoint
 2023-01-19 11:30:48 +08:00
enduser420
5cb263fdd2 [wikifeet/wikifeetx] add 'gallery' extractor 2023-01-16 21:08:45 +05:30
Mike Fährmann
4e86aaa4db
document 'extractor.headers' and 'extractor.ciphers' 2023-01-16 14:49:56 +01:00
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00
Mike Fährmann
512abeb4ae
[booru] add 'url' option 2023-01-13 16:32:32 +01:00
Mike Fährmann
c87bd1a752
[danbooru] extend 'metadata' option 
make it possible to specify a custom list of metadata includes
 2023-01-13 16:20:01 +01:00
Mike Fährmann
c0d7d2be35
[downloader:http] add 'validate' option 2023-01-11 15:37:40 +01:00
Mike Fährmann
4833ec323e
[imagefap] add 'folder' extractor (#3504) 2023-01-08 16:57:31 +01:00
Mike Fährmann
cb9d100bac
update Authentication support in docs 
https://github.com/mikf/gallery-dl/pull/3075#issuecomment-1374723412
 2023-01-08 16:57:31 +01:00
Mike Fährmann
362cd6991b
[pixiv] implement 'metadata-bookmark' option (#3417) 2023-01-07 23:19:43 +01:00
Mike Fährmann
0895e6afee
merge #3462: [docs] Update links and fix field typo 2023-01-07 15:22:03 +01:00
Mike Fährmann
4b9bb632f2
[docs] update links 2023-01-07 15:21:40 +01:00
Mike Fährmann
d1dd52349a
merge #3189: [tcbscans] add 'chapter' and 'manga' extractors 2023-01-06 13:42:27 +01:00
Mike Fährmann
ac78712618
merge #3496: [docs] update OAuth guide in README.rst 2023-01-06 13:36:23 +01:00
Mike Fährmann
2f31d21509
merge #3455: [twitter] apply tweet type checks before uniqueness check 2023-01-06 13:32:50 +01:00
Mike Fährmann
64435cc37e
[twitter] update docs 2023-01-06 13:32:08 +01:00
Mike Fährmann
b125fcafc5
update docs/configuration.rst 
- document metadata.indent
- add concrete mappings to object types
- fix lists for code examples
- fix typos
 2023-01-06 13:07:33 +01:00
enduser420
e8541a131d [tcbscans] add 'chapter' and 'manga' extractors 2023-01-06 16:16:31 +05:30
Mike Fährmann
3c03928d75
simplify 'docs/options.md' generation 
use the plain text output from '--help'
instead of trying to generate 'fancy' HTML with Markdown restrictions
 2023-01-06 11:21:47 +01:00
ClosedPort22
c94692f39b
[docs] document 'access-token' for Mastodon instances 2023-01-06 13:20:41 +08:00
Mike Fährmann
4fd6da474f
merge #3473: [twitter] fix crash when using 'expand' and 'syndication' 2023-01-05 14:19:47 +01:00
Mike Fährmann
818f6fa71f
merge #3453: [twitter] document #1396 and #2712 2023-01-05 14:18:21 +01:00
ClosedPort22
455481ad76
[twitter] improve documentation 
This includes:
- The automatic expansion of conversations (#2712)
- How to retrieve age-restricted content from search results (#1396)
 2023-01-05 12:14:42 +08:00
Mike Fährmann
f29ba089ff
merge #3474: [fanleaks] add 'post' and 'model' extractors 2023-01-04 17:39:17 +01:00
Mike Fährmann
6933727b45
merge #3483: [twitter] implement 'syndication=extended' 2023-01-04 17:36:17 +01:00
Mike Fährmann
9116398c1c
[pinterest] add 'domain' option (#3484) 
use input URL domain by default
 2023-01-04 17:20:14 +01:00
Mike Fährmann
f1a715dbcd
improve docs/options.md 
- use non-breaking hyphens
- don't replace '…' in examples
- don't force width for long option column
 2023-01-04 14:41:05 +01:00
Mike Fährmann
294108c90a
[pinterest] support 'All Pins' boards (#2855, #3484) 2023-01-03 19:11:20 +01:00
Mike Fährmann
13c16490bd
handle subcategory names ending with 'y' 2023-01-03 18:51:42 +01:00
Mike Fährmann
50d89d4acb
docs/options.md: use a separate table for each option group 2023-01-03 16:04:48 +01:00
Mike Fährmann
7de6b7f489
add list of command-line options as docs/options.md (#3464) 2023-01-02 19:14:16 +01:00
ClosedPort22
6853b14be3
[twitter] apply suggestions from code review 
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
 2023-01-02 21:03:01 +08:00
Mike Fährmann
4611237f8c
merge #3457: [danbooru] extract uploader metadata (if option is set) 2023-01-01 21:20:39 +01:00
Mike Fährmann
e7522482bb
merge #3463: [lynxchan] support 'bbw-chan.nl' 2023-01-01 15:50:09 +01:00
Mike Fährmann
0ed5986062
document custom 'output.mode' (#2529) 2022-12-30 17:14:42 +01:00
enduser420
5a740ef78b [fanleaks] add 'post' and 'model' extractors 2022-12-30 19:24:05 +05:30
ClosedPort22
7c8eab8d52
[twitter] implement 'syndication=extended' 
to be able to fetch extended user metadata
 2022-12-30 20:48:41 +08:00
ClosedPort22
be3286206a
[twitter] assume 'conversation_id' when using syndication 
not possible to expand replies at the momemt
 2022-12-30 13:57:37 +08:00
ClosedPort22
ce8dbb1ccc
[twitter] fix crash when using 'expand' and 'syndication' 
caused by KeyError: 'conversation_id_str'
 2022-12-30 12:45:44 +08:00
Mike Fährmann
d651d45239
implement specifying ranges in slice notation (#918, #2865) 
e.g.
- '1:101'   or ':101' or ':101:'  for files 1 to 100
- '1::2'    or '::2'              for every second file
- '1:101:5' or ':101:5'           for files 1, 6, 11, ..., 91, 96

(the second argument specifies the first index NOT included)
 2022-12-27 18:21:12 +01:00
enduser420
527bb2c4ab [lynxchan/bbw-chan] add 'thread' and 'board' extractors 2022-12-26 20:59:42 +05:30
pi_allen
64902f518e [docs] Update links and fix field typo 2022-12-26 07:46:41 +01:00
ClosedPort22
5d5ea58887
[twitter] update docs for 'quoted' & 'unique' 2022-12-25 15:59:26 +08:00
blankie
f82ee93676
[danbooru] extract uploader metadata (if metadata is set) 2022-12-25 11:00:59 +07:00
Mike Fährmann
1800bd7d14
allow '*-filter' options to be a list of expressions 2022-12-23 22:20:21 +01:00
lx30011
895b41f1ac [jschan] add generic jschan extractor 2022-12-23 00:32:52 +01:00
Mike Fährmann
73ab5d84c0
update docs/configuration.rst 2022-12-20 17:30:46 +01:00
Mike Fährmann
bee354c264
Merge pull request #3415 from enduser420/extractor/fapello 
[fapello] add 'post', 'user' and 'path' extractors
 2022-12-17 20:05:46 +01:00
ClosedPort22
dd4a4a3fa6
[e621] softcode the pagination threshold 2022-12-17 21:06:47 +08:00
enduser420
e5076ba056 [fapello] add 'post', 'user' and 'path' extractors 2022-12-16 16:53:32 +05:30
ClosedPort22
d0ad6d0e67
[e621] implement manual pagination mode 2022-12-16 16:15:36 +08:00
Mike Fährmann
1317625ec4
[webmshare] add 'video' extractor (#2410) 2022-12-14 19:59:07 +01:00
enduser420
41bf236d36
[lynxchan] add generic extractors for lynxchan imageboards (#3394) 
* [lynxchan] add generic extractors for lynxchan imageboards

includes kohlchan.net, endchan.org:wq

* [lynxchan] set pop default to empty tuple

* Apply suggestions from code review

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
 2022-12-12 13:13:39 +01:00
Mike Fährmann
3c75c3bbc4
[soundgasm] add 'user' extractor (#3384) 
based on code from PR #3388 by @enduser420
 2022-12-12 12:23:59 +01:00
Mike Fährmann
2952add4a8
[reddit] increase 'id-max' default value (#3397) 
to float("inf")
 2022-12-12 12:10:06 +01:00
Mike Fährmann
6b6f886dcf
[bunkr] update domain (#3391) 
and improve bunkr/app.bunkr handling
 2022-12-11 23:18:01 +01:00
Mike Fährmann
eb94568e1f
[soundgasm] add 'audio' extractor (#3384) 2022-12-09 23:19:07 +01:00
Mike Fährmann
dfe7b23579
support Firefox containers for --cookies-from-browser (#3346) 2022-12-09 19:43:55 +01:00
Mike Fährmann
4a3a1f4c87
[komikcast] update domain and fix extraction 2022-12-06 22:00:23 +01:00
Mike Fährmann
a42ba25ca1
[foolslide] remove 'kireicake' 
site redirects to (unclaimed) mangadex group
 2022-12-05 19:47:56 +01:00
Mike Fährmann
86f0597c95
[kissgoddess] remove module 
site does not host albums anymore
 2022-12-05 19:28:50 +01:00
Mike Fährmann
fceaee3c4f
[lolisafe] remove zz.ht 2022-12-04 16:21:13 +01:00
enduser420
54844944ab
[pixhost] add 'gallery' support (#3353) 2022-12-04 16:19:14 +01:00
enduser420
213676c785
[fapachi] add 'post' and 'user' extractors (#3347) 
* [fapachi] add 'post' and 'user' extractors

* [fapachi] add 'keyword' to test

* [fapachi] remove whitespaces
 2022-12-02 13:54:29 +01:00
Mike Fährmann
80102fa367
[downloader:http] add 'retry-codes' option (#3313) 2022-12-01 11:08:23 +01:00
ClosedPort22
b14b33f19e
Implement version-metadata option (#3201) 2022-11-27 16:09:42 +01:00
Mike Fährmann
6cb12f513b
[nitter] support quoted Tweets 
- distinguish between regular and quoted Tweets and media
- add 'quoted' option and metadata field
 2022-11-26 11:23:03 +01:00