Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 18:53:21 +01:00
Code Issues Releases Wiki Activity
6,075 Commits 6 Branches 164 Tags 15 MiB
ddd325b435
Commit Graph

1346 Commits

Author SHA1 Message Date
Mike Fährmann
75697dfb26
implement -e/--error-file as a logging handler 
similar to --write-unsupported
 2023-12-10 00:06:39 +01:00
Mike Fährmann
9a8dc6b02b
[exhentai] add 'fallback-retries' option (#4792) 2023-12-09 02:01:59 +01:00
Mike Fährmann
bf74eb5c46
merge #4886: [urlgalleries] add 'gallery' extractor (#919, #1184, #2905) 2023-12-08 22:55:58 +01:00
Mike Fährmann
042a9da451
add 'output.errorfile' config option 2023-12-08 17:29:54 +01:00
Mike Fährmann
9dd5cb8c8a
interactively prompt for passwords on login when none is provided 2023-12-06 22:12:59 +01:00
Mike Fährmann
99b76628f7
implement '-e/--error-file' command-line option (#4732) 
copying per-URL options from regular, read-only input files
does currently not work
 2023-12-05 20:49:51 +01:00
jsouthgb
1770c31e63 [urlgalleries] add support 2023-12-05 07:07:06 -05:00
Mike Fährmann
bdb3ce7217
[foolslide] remove 'powermanga.org' 2023-11-26 23:19:05 +01:00
Mike Fährmann
4e15513415
[docs] fix 'Example' -> 'Default' from b43be672 
copy-paste mistake
 2023-11-22 22:14:30 +01:00
Mike Fährmann
b43be67206
[exhentai] add 'gp' option (#4576) 2023-11-22 20:15:00 +01:00
Mike Fährmann
cb9a1176e6
[pixeldrain] add 'api-key' option (#4839) 2023-11-22 19:13:51 +01:00
Mike Fährmann
e1404827a6
[pixeldrain] add 'file' and 'album' extractors (#4839) 2023-11-22 19:01:19 +01:00
Mike Fährmann
ce7c4cb544
merge #4832: [tmohentai] add 'gallery' extractor (#4808) 2023-11-21 20:25:49 +01:00
Mike Fährmann
c4a201ed42
[tmohentai] simplify + tests 2023-11-21 20:24:07 +01:00
Mike Fährmann
e17a48fe56
[blogger] inherit from BaseExtractor 
- support www.micmicidol.club (#4759)
 2023-11-21 16:52:25 +01:00
Mike Fährmann
aea15f6d17
add 'metadata-extractor' option (#4549) 2023-11-20 22:16:15 +01:00
Mike Fährmann
34a387b6e2
support 'metadata-*' names for '*-metadata' options 
For example, instead of 'url-metadata' it is now also possible to use
'metadata-url' as option name.

- metadata-url
- metadata-path
- metadata-http
- metadata-version
- metadata-parent
 2023-11-18 23:52:10 +01:00
jsouthgb
286d0cb098 [tmohentai] add support 2023-11-17 19:34:34 -05:00
Mike Fährmann
07cb584231
[behance] add 'modules' option (#4799) 2023-11-17 22:54:38 +01:00
Mike Fährmann
8bf161e574
reorder post processing options shown by --help 2023-11-16 17:37:15 +01:00
Mike Fährmann
168331d147
replace '--ugoira-conv' etc with a general '--ugoira' 
update --ugoira webm to use the same FFmpeg args as Danbooru

--ugoira-conv          -> --ugoira vp8
--ugoira-conv-lossless -> --ugoira vp9-lossless
--ugoira-conv-copy     -> --ugoira copy

(--ugoira-conv and co still work as before,
 but --help now lists only --ugoira)
 2023-11-16 17:15:17 +01:00
Mike Fährmann
97357e65ee
replace '--mtime-from-date' with a more generic '--mtime' 
--mtime-from-date -> --mtime date
for the same effect as before

(--mtime-from-date also still works,
 but --help now lists only --mtime)
 2023-11-15 15:21:22 +01:00
Mike Fährmann
51e377e612
add '--cbz' command-line option 2023-11-14 23:10:31 +01:00
Mike Fährmann
4700051562
rework and extend input file processing (#4732) 
- add 2 command-line options to modify input file contents
  - -I/--input-file-comment
  - -x/--input-file-delete
- implement InputManager class
- move code from util.py to __init__.py
  (mainly to avoid import cycles)
 2023-11-14 20:50:11 +01:00
Mike Fährmann
7a0f145cbe
[twitter] ignore promoted Tweets (#4790, #3894) 
add 'ads' option in case someone actually wants to
download promoted content for whatever reason
 2023-11-10 23:46:46 +01:00
Mike Fährmann
f4e61fd1d5
reword 'reddit.client-id' instructions (#4749) 2023-11-04 00:52:18 +01:00
Mike Fährmann
007c433677
[patreon] support 'id:<campaign_id>' in place of a user name 
https://patreon.com/id:12345
… and remove 'campaign-id' config option
 2023-11-04 00:17:41 +01:00
Mike Fährmann
a09df34bcf
merge #4714: [4archive] add 'thread' and 'board' extractors 
(#1262, #2418, #4400, #4710)
 2023-10-25 20:12:07 +02:00
Mike Fährmann
d2874c7724
[4archive] docs/supportedsites 2023-10-25 20:11:14 +02:00
Mike Fährmann
6766877524
merge #4693: [reddit] support Reddit Mobile share links 2023-10-25 17:54:32 +02:00
Mike Fährmann
1042278bec
[misskey] support 'misskey.design' (#4713) 2023-10-25 17:47:03 +02:00
Mike Fährmann
12a800ce21
[patreon] improve 'campaign_id' handling (#4699, #4715) 
- add ways to directly specify a 'campaign_id'
  - 'campaign-id' config option
  - 'c' or 'campaign_id' URL query parameter
- more descriptive error messages
- show 'campaign_id' value in debug log
 2023-10-25 17:18:06 +02:00
inty
b68aad3dab [reddit] implement Reddit Mobile share links 2023-10-22 10:38:05 +00:00
Mike Fährmann
95a74be2a5
release version 1.26.1 2023-10-21 16:32:52 +02:00
Mike Fährmann
0d52b775cb
[kemonoparty] add 'revisions' option (#4498, #4597) 2023-10-20 15:20:49 +02:00
Mike Fährmann
390d14dbcc
[chevereto] support 'img.kiwi' and 'deltaporno.com' (#4664, #1381) 2023-10-16 18:14:30 +02:00
Mike Fährmann
2911ed1240
[chevereto] add generic extractors (#4664) 
- support jpgfish
- support pixl.li / pixl.is (#3179, #4357)
 2023-10-16 14:15:39 +02:00
Mike Fährmann
2d41702762
[deviantart] implement '"group": "skip"' (#4630) 2023-10-12 22:14:20 +02:00
Mike Fährmann
992e86ec94
[deviantart] disable 'jwt' (#4652) 2023-10-12 21:37:10 +02:00
Mike Fährmann
2974b8e3c8
[moebooru] add 'metadata' option (#4646) 
for extended 'pool' metadata
 2023-10-12 21:34:25 +02:00
Mike Fährmann
67ba4ee842
[pp:exec] support more replacement fields for '--exec' (#4633) 
- {_directory}
- {_filename}
- {_path} (alias for {})
 2023-10-09 12:50:10 +02:00
Mike Fährmann
84fbbd96aa
[shimmie2] remove 'meme.museum' 2023-10-02 20:41:25 +02:00
Mike Fährmann
0c5d8b1505
[deviantart] re-add 'quality' option and 'intermediary' transform 2023-09-24 17:36:05 +02:00
Mike Fährmann
20d1683c47
[deviantart] fix JWT replacement (#293, #4548, #4563) 
And again, a huge thank you to @Ironchest337
for discovering this.
 2023-09-24 15:06:16 +02:00
Mike Fährmann
82296b1f05
[reddit] add 'previews' option (#4322) 
another way to disable this new behavior
 2023-09-22 18:10:44 +02:00
Mike Fährmann
9d8317d963
[deviantart] disable JWT updates (#4548, #4563) 
back to lowres images ...
 2023-09-22 17:13:05 +02:00
Mike Fährmann
3ecb512722
send Referer headers by default 2023-09-19 00:02:04 +02:00
Mike Fährmann
6178177227
[twitter] fix '_extractor' of following results (#4536) 
regression from 20ed647f
 2023-09-15 23:04:30 +02:00
Mike Fährmann
f2de70f254
[gfycat] remove module 2023-09-04 18:27:11 +02:00
Mike Fährmann
6eca1fab9b
[gelbooru_v02] support 'xbooru.com' (#4493) 2023-09-03 15:39:02 +02:00
Mike Fährmann
23bac772f2
[jpgfish] update domain to 'jpg1.su' (#4494) 2023-09-03 15:03:47 +02:00
Mike Fährmann
ceb59e176f
fix default Firefox user agent string 
note to self: do not trust some random third-party website
 2023-09-02 22:22:23 +02:00
Mike Fährmann
a783c4f0fe
[pornhub] add 'gif' support (#4463) 2023-08-29 19:34:27 +02:00
Mike Fährmann
d12a5e440a
update docs/supportedsites 2023-08-24 15:01:26 +02:00
Mike Fährmann
5ed245317d
[exhentai] add 'fav' option (#4409) 
The name 'favorite' is already taken as extractor subcategory
 2023-08-23 23:58:11 +02:00
Mike Fährmann
410f783a33
implement 'subconfigs' option (#4440) 2023-08-21 21:18:40 +02:00
Mike Fährmann
c1c73c0b0e
[pp:ugoira] add '"framerate": "uniform"' (#4421) 2023-08-21 19:43:47 +02:00
Mike Fährmann
70bdf32a88
[pp:ugoira] extend 'ffmpeg-output' (#4421) 
- when setting this option to a string value,
  pass -hide-banner and -loglevel to FFmpeg
- change default to "error"
 2023-08-21 18:19:53 +02:00
Mike Fährmann
84089c7368
[docs] add note about 'path-restrict' implementation 
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1671892152
 2023-08-11 18:00:11 +02:00
Mike Fährmann
f9fb276e81
[postprocessor] add 'prepare-after' event (#4083) 2023-08-10 21:28:48 +02:00
Mike Fährmann
0ef1fcab20
[postprocessor] update 'finalize' events 
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.

'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
 2023-08-10 19:46:37 +02:00
Mike Fährmann
9d67655397
add "ascii+" as a special 'path-restrict' value (#4371) 2023-08-01 19:33:53 +02:00
Mike Fährmann
1ece3b92ff
[mangadex] allow multiple values for 'lang' (#4093) 
This was already possible by setting 'lang' to a list of strings,
but now it can also be done as a more command-line friendly string.

-o lang=fr,it
 2023-07-26 17:39:27 +02:00
Mike Fährmann
c6d70c7483
[docs] fix typo (#4350) 2023-07-26 14:24:19 +02:00
Mike Fährmann
7cd5130e88
[docs] small fixes/updates 2023-07-24 14:27:37 +02:00
Mike Fährmann
54d974deb0
add 'python' post processor 
similar to 'exec' but calls a Python function
 2023-07-24 14:06:14 +02:00
Mike Fährmann
ceebacc9e1
remove 'pyopenssl' option 2023-07-19 20:44:07 +02:00
Mike Fährmann
0ba8d1f168
merge #4312: [redgifs] add 'niches' extractor 2023-07-18 18:36:15 +02:00
Mike Fährmann
c5565f79f7
merge #4096: [danbooru] add support for booru.borvar.art instance 2023-07-18 18:33:08 +02:00
Mike Fährmann
63326e3168
[danbooru] add tests for booruvar 2023-07-18 18:29:57 +02:00
Mike Fährmann
5171d8975c
[E621] support 'e6ai.net' (#4320) 2023-07-18 18:16:30 +02:00
Mike Fährmann
90231f2d5a
[twitter] add 'tweet-endpoint' option (#4307) 
use the newer TweetResultByRestId only for guests by default
 2023-07-18 17:19:32 +02:00
Mike Fährmann
20ed647f6f
[twitter] add 'user' extractor and 'include' option (#4275) 2023-07-18 16:42:55 +02:00
Mike Fährmann
86be197d11
[twitter] remove '/search/adaptive.json' 2023-07-18 15:45:37 +02:00
enduser420
12cd85658b [redgifs] add 'niches' extractor 2023-07-16 21:22:06 +05:30
Mike Fährmann
ffbbbd3baf
[gelbooru_v01] 'vidyart' -> 'vidyart2' 2023-07-14 15:09:39 +02:00
Mike Fährmann
2dd6942d1c
[jpgfish] update domain to 'jpeg.pet' 2023-07-13 23:21:01 +02:00
Mike Fährmann
1137b89ed4
[lineblog] remove module 
"LINE BLOGは2023年6月29日をもちましてサービスを終了いたしました"
 2023-07-13 20:46:04 +02:00
Mike Fährmann
86560fe0cd
[bcy] remove module 
"The website was shut down on July 12, 2023"
https://danbooru.donmai.us/wiki_pages/bcy
 2023-07-13 20:46:04 +02:00
Mike Fährmann
fc43c74694
[docs] update reddit client-id instructions (#4292) 
add clearing cached access tokens and getting a new refresh token
 2023-07-13 20:43:36 +02:00
Mike Fährmann
f079d9a703
[reddit] notify users about registering an oauth application 
(#4292, #4253, #3943)
 2023-07-12 21:43:00 +02:00
Mike Fährmann
7444fc125b
[gfycat] implement login support (#3770, #4271) 
For the record: '/webtoken' and '/weblogin' are not the same ...
 2023-07-06 18:56:34 +02:00
Mike Fährmann
5b59a0d143
update default User-Agent header to Firefox 115 ESR 2023-07-05 15:12:50 +02:00
Mike Fährmann
1bf9f52c99
[twitter] add 'ratelimit' option (#4251) 2023-07-04 18:17:32 +02:00
Mike Fährmann
f86fdf64a6
[twitter] use GraphQL search by default (#4264) 2023-07-04 17:55:22 +02:00
Mike Fährmann
6ae3101fd0
[mangapark] add 'source' option (#3969) 2023-07-02 15:07:22 +02:00
Mike Fährmann
c45a913bfd
[flickr] add 'exif' option 2023-07-01 19:19:39 +02:00
Mike Fährmann
ccbc1a1d55
[flickr] add 'metadata' option (#4227) 2023-06-26 16:49:48 +02:00
Mike Fährmann
c1cce4a80b
[twitter] extend 'conversations' option (#4211) 2023-06-24 21:34:34 +02:00
Mike Fährmann
f50b81d3e9
[docs] add novel-related keys to 'pixiv.include' (#4111) 2023-06-19 15:01:53 +02:00
Mike Fährmann
ce93c460a6
[formatter] implement 'H' conversion (#4164) 
to remove HTML tags and unescape HTML entities
 2023-06-15 13:07:51 +02:00
Mike Fährmann
deff3b434d
[vipergirls] implement login support (#4166) 2023-06-13 21:05:09 +02:00
Mike Fährmann
aad5e6490c
merge #4159: [bunkr] update domain to bunkrr.su 2023-06-08 17:02:54 +02:00
Mike Fährmann
e0522ffb3d
[bunkr] update 2023-06-08 17:01:04 +02:00
Mike Fährmann
e04796e04b
merge #3447: [jschan] add generic extractors for jschan imageboards 2023-06-08 16:52:26 +02:00
Mike Fährmann
a90974178d
[jpgfish] update domain to 'jpg.pet' (#4138) 2023-06-03 15:49:56 +02:00
Mike Fährmann
54cf1fa3e7
[twitter] use GraphQL search endpoint (#3942) 
for guest users; selectable with 'search-endpoint' option.

adapted from 9c7b888ffa
 2023-06-01 21:37:31 +02:00
Mike Fährmann
0cf7282fa0
[pixiv] add 'full-series' option for novels (#4111) 2023-06-01 13:07:20 +02:00
Mike Fährmann
b286efefcc
[pixiv] add 'novel-bookmark' extractor (#4111) 2023-05-28 16:30:17 +02:00
Mike Fährmann
28f6487c64
[instagram] add 'metadata' option (#3107) 2023-05-27 15:51:13 +02:00
Mike Fährmann
58f7480d46
[lensdump] update 
- update docs/supportedsites.md
- add GPL2 header
- use BASE_PATTERN
- improve LensdumpImageExtractor
 2023-05-26 23:39:17 +02:00
Mike Fährmann
148bdc04a4
merge #2719: [jpgfish] add 'jpgfish' extractors 2023-05-25 23:04:46 +02:00
Mike Fährmann
2b1f875ef4
[jpgchurch] update to 'jpgfish' 2023-05-25 18:58:51 +02:00
Mike Fährmann
5f86527cbe
merge #2781: [mangaread] Add Mangaread extractor 2023-05-25 17:03:28 +02:00
Mike Fährmann
f3cca50b9e
[mangadex] update links to API docs 2023-05-24 12:32:13 +02:00
Mike Fährmann
65a9f4b124
merge #3950: [misskey] add 'favorite' extractor 2023-05-23 22:18:32 +02:00
Mike Fährmann
c76f0f3a1b
[misskey] update 
- rename to 'MisskeyFavoriteExtractor'
- add 'access-token' option to docs
- add test URLs for other instances
- simplify 'pattern'
 2023-05-23 22:17:17 +02:00
Mike Fährmann
3fca455b82
[pixiv] add 'embeds' option (#1241) 2023-05-23 12:14:06 +02:00
Mike Fährmann
9810ab35af
[docs] update formatting.md 
- add element access with quotes
- fix \f formatting
- remove note about typing \f in a shell (9e6c9813)
 2023-05-22 22:24:23 +02:00
Mike Fährmann
69865dcc05
[formatter] implement slicing strings as bytes (#4087) 
prefixing a slice '[10:30]' with a lowercase b '[b10:30]' encodes
the string to bytes in filesystem encoding before applying the slice
 2023-05-22 18:30:45 +02:00
Mike Fährmann
e6f55d1555
[imagechest] add API support and 'access-token' option (#4065) 2023-05-20 17:00:27 +02:00
Mike Fährmann
77abcf5ab3
[gofile] automatically fetch 'website-token' by default 
the hardcoded token changed yet again
 2023-05-20 17:00:27 +02:00
Mike Fährmann
e3fed9bd17
[tcbscans] update domain to 'tcbscans.com' (#4080) 2023-05-20 13:59:35 +02:00
Mike Fährmann
a83983c651
[instagram] add 'order-posts' option (#4017, #3993) 2023-05-18 22:50:12 +02:00
Mike Fährmann
d680623db3
[instagram] add 'order-files' option (#4017, #3993) 2023-05-18 22:34:33 +02:00
Mike Fährmann
5037013e2b
[gofile] update 'website-token' (#4056) 2023-05-15 14:59:44 +02:00
Mike Fährmann
20dc13f832
[pixiv] initial 'novel' support (#1241, #4044) 
supported URLs are
- https://www.pixiv.net/novel/show.php?id=<ID>
- https://www.pixiv.net/novel/series/<ID>
- https://www.pixiv.net/en/users/<ID>/novels
 2023-05-12 16:34:08 +02:00
Mike Fährmann
1406f7125f
[4chanarchives] add 'thread' and 'board' extractors (#4012) 2023-05-06 20:45:57 +02:00
Mike Fährmann
285391df43
add '-C' as short option for '--cookies' 
and put cookie options into their own section
 2023-05-06 20:45:44 +02:00
Mike Fährmann
b9b1cdd71b
add '--cookies-export' command-line option 2023-05-06 17:37:49 +02:00
Mike Fährmann
8520de57f0
[imgur] add 'favorite-folder' extractor (#4016) 2023-05-06 15:10:13 +02:00
Mike Fährmann
a14b63d941
support selecting a domain for '--cookies-from-browser' 
for example 'gallery-dl --cookies-from-browser firefox/twitter.com'
 2023-05-06 15:09:49 +02:00
Mike Fährmann
3ca5dac8b6
extend 'cookies-update' functionality 
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser

To convert browser cookies to cookies.txt format:
  gallery-dl --cookies-fr chromium \
             -o cookies-update=cookies.txt \
             --no-download \
             http://example.org/file.jpg
 2023-05-04 15:10:47 +02:00
Mike Fährmann
79f47f98dd
[nana] remove module 
permanently gone since 2023-03-13
 2023-05-03 18:05:53 +02:00
Mike Fährmann
0e74df1de8
[420chan] remove module 
offline since 2022-06-01
 2023-05-03 17:46:21 +02:00
Mike Fährmann
0c46758a93
[foolslide] remove 'sensescans.com' 
group moved to mangadex
https://mangadex.org/group/1071e71d-cc55-4fa6-81d1-4b5913a2fde5/sense-scans
 2023-05-02 20:09:04 +02:00
Mike Fährmann
a08fdfac6e
[foolfuuka] add 'archive.palanq.win' 2023-05-02 19:58:55 +02:00
Mike Fährmann
1870df8b23
[foolfuuka] remove 'tokyochronos.net' 2023-05-02 19:25:50 +02:00
Mike Fährmann
ef4e2d8178
[foolfuuka] remove 'archive.alice.al' 2023-05-02 19:23:26 +02:00
Mike Fährmann
c182094ebf
merge #3748: [downloader:http] add 'consume-content' option 2023-04-26 23:03:18 +02:00
Mike Fährmann
7865067d19
[shimmie2] add generic extractors for Shimmie2 sites (#3734) 
add support for
- loudbooru.com       (#3734)
- booru.cavemanon.xyz (#3734)
- giantessbooru.com   (#943)
- tentaclerape.net
 2023-04-26 19:20:44 +02:00
Mike Fährmann
28419bf45a
[itchio] add 'game' extractor (#3923) 2023-04-26 19:20:43 +02:00
Mike Fährmann
5297ee0cd9
[tumblr] add 'day' extractor (#3951) 2023-04-24 22:01:47 +02:00
Mike Fährmann
b81ce381f0
fix backticks for 'extractor.redgifs.format' docs 
https://github.com/mikf/gallery-dl/issues/146#issuecomment-1518331927
 2023-04-21 23:12:01 +02:00
Mike Fährmann
aaf58a1259
[imgur] document 'client-id' option (#3937) 2023-04-21 15:08:50 +02:00
Mike Fährmann
6f1e34ec69
[vipergirls] add 'thread' and 'post' extractors 
(#731, #2720, #3812)
 2023-04-19 15:28:26 +02:00
Mike Fährmann
81bd2af83e
[2chen] update domain to sturdychan.help 2023-04-19 13:54:44 +02:00
Mike Fährmann
5b635f2317
[imxto] add 'gallery' extractor (#1289) 2023-04-17 20:49:09 +02:00
Mike Fährmann
d253a3c542
merge #3841: [urlshortener] add support for bit.ly & t.co 2023-04-15 18:08:21 +02:00
Mike Fährmann
5e63942b37
[urlshortener] update 2023-04-15 18:06:06 +02:00
Mike Fährmann
0a7eee3ee0
[deviantart] add 'public' option 2023-04-08 23:04:34 +02:00
Mike Fährmann
75666cf9c3
[danbooru] reduce API requests for fetching extended 'metadata' 
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
 2023-04-02 20:11:52 +02:00
ClosedPort22
71b26adb9b
[urlshortener] add tinyurl.com as an example 2023-03-29 13:37:26 +08:00
Mike Fährmann
421db26aff
[bunkr] update domain to 'bunkr.la' 2023-03-28 20:10:36 +02:00
ClosedPort22
9e2a945013
[urlshortener] add support for bit.ly & t.co 2023-03-29 00:06:41 +08:00
Mike Fährmann
89a67c45e0
[nitter] support nitter.it (#3819) 2023-03-25 13:29:22 +01:00
Mike Fährmann
197882cf12
[twitter] add 'hashtag' extractor (#3783) 2023-03-22 22:20:40 +01:00
Mike Fährmann
46fdf46f21
[formatter] support loading an f-string from a template file 
"\fTF ~/path/to/file.txt"
 2023-03-20 22:05:33 +01:00
Mike Fährmann
1a4d4a799b
[formatter] support filesystem paths for \fM 2023-03-20 22:01:33 +01:00
Mike Fährmann
00f0233b28
[postprocessor:metadata] add 'skip' option (#3786) 2023-03-17 23:30:11 +01:00
Mike Fährmann
a1ca2404f9
add 'globals' instead of overwriting the default (#3773) 2023-03-16 18:37:00 +01:00
Mike Fährmann
17bd053d94
[hiperdex] fix extraction (#3768) 2023-03-15 14:28:03 +01:00
ClosedPort22
fcaeaf539c
[downloader:http] handle exceptions while consuming content 2023-03-11 21:36:37 +08:00
Mike Fährmann
817fc0fbd1
[nitter] remove nitter.pussthecat.org 
"Shutdown"
 2023-03-09 23:48:39 +01:00
ClosedPort22
df77271438
[downloader:http] add 'consume-content' option 
* fix connection not being released when the response is neither
  successful nor retried
* add the ability to consume the HTTP response body instead of closing
  the connection

reference:

https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
 2023-03-09 21:07:10 +08:00
Mike Fährmann
da9840a39d
[reddit] update 'videos' option (#3712) 
- add 'dash' to directly extract DASH manifest URLs
  (was default behavior since a7c79531)
- change default strategy back to before a7c79531
- disable 'Falling back on generic information extractor' warning
 2023-03-06 12:18:25 +01:00
Mike Fährmann
baf41d7437
[misskey] update (#3717) 
- add module docstring
- add options to docs/gallery-dl.conf
 2023-03-03 15:52:50 +01:00
enduser420
e1867cf5eb [misskey] add 'renotes' and 'replies' options 2023-03-02 19:56:19 +05:30
enduser420
a95b5e0d8e [misskey] add misskey extractors 2023-03-02 19:49:47 +05:30
Mike Fährmann
0d142e403c
[szurubooru] add 'tag' and 'post' extractors (#3583, #3713) 2023-03-01 18:47:10 +01:00
Mike Fährmann
075c965512
add '--config-create' command-line option 
(#2333)
 2023-03-01 14:49:40 +01:00
Mike Fährmann
632d5d7745
allow loading config files in TOML format with --config-toml 2023-02-28 21:54:46 +01:00
Mike Fährmann
d788e6c60c
implement 'globals' option 2023-02-28 18:18:55 +01:00
Mike Fährmann
b14f8d5817
[gelbooru] add 'favorite' extractor (#3704) 
requires logged in cookies to work
 2023-02-27 18:03:47 +01:00
Mike Fährmann
e480a93337
add 'output.stdout', '.stdin', and '.stderr' options 
(#1621, #2152, #2529)

Allow setting custom input/output encodings and options
without having to rely on Python's defaults.
 2023-02-26 14:56:19 +01:00
Mike Fährmann
6ed4309aba
[deviantart] add 'gallery-search' extractor (#1695) 2023-02-22 17:25:31 +01:00
Mike Fährmann
e1df7f73b1
[deviantart] add 'search' extractor 
(#538, #1264, #2954, #2970, #3577)

Requires login to fetch any results, since the API endpoint raises an
error for not logged in requests.

TODO: parse HTML search results
 2023-02-20 20:54:46 +01:00
Mike Fährmann
bb11c2a576
merge #3662: [redgifs] add 'collection' extractors 2023-02-18 13:09:17 +01:00
Mike Fährmann
884f1848d6
[redgifs] fix syntax for older Python versions 
and update docs/supportedsites
 2023-02-18 13:07:40 +01:00
Mike Fährmann
2bd8f2f4bd
[pornpics] add 'search' and 'tag' extractors 
(#263, #3544, #3654)
 2023-02-17 23:03:14 +01:00
Mike Fährmann
79bc82884c
[pornpics] add 'gallery' extractor (#263, #3544, #3654) 2023-02-17 15:00:57 +01:00
Mike Fährmann
bd5d08abbc
[catbox] add 'file' extractor (#3570) 2023-02-15 19:50:26 +01:00
Mike Fährmann
2a53e6445c
[bunkr] update domain (#3636) 2023-02-09 19:28:32 +01:00
Mike Fährmann
b7337d810e
[postprocessor:metadata] add 'sort' and 'separators' options 2023-02-07 18:28:14 +01:00
Mike Fährmann
762a68996b
implement 'archive-pragma' option 2023-02-05 17:00:31 +01:00
Mike Fährmann
bbf0911a46
[e621] implement 'notes' and 'pools' metadata extraction 
(#3425)
 2023-02-04 21:20:38 +01:00
Mike Fährmann
925b467496
split e621 from danbooru module (#3425) 2023-02-03 19:24:31 +01:00
Mike Fährmann
1ae48a54f8
[twitter] add 'transform' option 2023-02-02 22:01:36 +01:00
Mike Fährmann
78d3960a31
[postprocessor:exec] implement archive options (#3584) 2023-02-01 13:57:16 +01:00
Mike Fährmann
0f7e6c422a
merge #3596: [shopify] support ohpolly.com 2023-01-31 20:13:51 +01:00
enduser420
fcf7030b85 [shopify] support ohpolly.com 2023-02-01 00:31:12 +05:30
Mike Fährmann
4572c122d8
document 'extractor.module-sources' 2023-01-30 23:32:36 +01:00
Mike Fährmann
c2bc70593e
implement ability to load external extractor classes 
- -X/--extractors
- extractor.module-sources
 2023-01-30 23:10:10 +01:00
Mike Fährmann
13a90969c7
merge #3575: [nudecollect] add 'image' and 'album' extractors 2023-01-28 16:04:47 +01:00
Mike Fährmann
abc3619feb
[lexica] add 'search' extractor (#3567) 2023-01-28 16:00:32 +01:00
Mike Fährmann
f58215705a
add '-O/--postprocessor-option' command-line option (#3565) 2023-01-26 14:59:24 +01:00
enduser420
2a5903dc16 [nudecollect] add 'image' and 'album' extractors 2023-01-26 17:25:33 +05:30
Mike Fährmann
58c008e30a
[hiperdex] update domain (#3572) 2023-01-26 12:01:16 +01:00
Mike Fährmann
caae8fefe1
merge #3541: [deviantart] add extractor for status updates 2023-01-23 13:54:51 +01:00
ClosedPort22
3b32671fbd
[deviantart] add extractor for status updates 
extract user status updates using the '/user/statuses/' endpoint
 2023-01-19 11:30:48 +08:00
enduser420
5cb263fdd2 [wikifeet/wikifeetx] add 'gallery' extractor 2023-01-16 21:08:45 +05:30
Mike Fährmann
4e86aaa4db
document 'extractor.headers' and 'extractor.ciphers' 2023-01-16 14:49:56 +01:00
Mike Fährmann
c881548a27
add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00
Mike Fährmann
512abeb4ae
[booru] add 'url' option 2023-01-13 16:32:32 +01:00
Mike Fährmann
c87bd1a752
[danbooru] extend 'metadata' option 
make it possible to specify a custom list of metadata includes
 2023-01-13 16:20:01 +01:00
Mike Fährmann
c0d7d2be35
[downloader:http] add 'validate' option 2023-01-11 15:37:40 +01:00
Mike Fährmann
4833ec323e
[imagefap] add 'folder' extractor (#3504) 2023-01-08 16:57:31 +01:00
Mike Fährmann
cb9d100bac
update Authentication support in docs 
https://github.com/mikf/gallery-dl/pull/3075#issuecomment-1374723412
 2023-01-08 16:57:31 +01:00
Mike Fährmann
362cd6991b
[pixiv] implement 'metadata-bookmark' option (#3417) 2023-01-07 23:19:43 +01:00
Mike Fährmann
0895e6afee
merge #3462: [docs] Update links and fix field typo 2023-01-07 15:22:03 +01:00
Mike Fährmann
4b9bb632f2
[docs] update links 2023-01-07 15:21:40 +01:00
Mike Fährmann
d1dd52349a
merge #3189: [tcbscans] add 'chapter' and 'manga' extractors 2023-01-06 13:42:27 +01:00
Mike Fährmann
ac78712618
merge #3496: [docs] update OAuth guide in README.rst 2023-01-06 13:36:23 +01:00
Mike Fährmann
2f31d21509
merge #3455: [twitter] apply tweet type checks before uniqueness check 2023-01-06 13:32:50 +01:00
Mike Fährmann
64435cc37e
[twitter] update docs 2023-01-06 13:32:08 +01:00
Mike Fährmann
b125fcafc5
update docs/configuration.rst 
- document metadata.indent
- add concrete mappings to object types
- fix lists for code examples
- fix typos
 2023-01-06 13:07:33 +01:00
enduser420
e8541a131d [tcbscans] add 'chapter' and 'manga' extractors 2023-01-06 16:16:31 +05:30
Mike Fährmann
3c03928d75
simplify 'docs/options.md' generation 
use the plain text output from '--help'
instead of trying to generate 'fancy' HTML with Markdown restrictions
 2023-01-06 11:21:47 +01:00
ClosedPort22
c94692f39b
[docs] document 'access-token' for Mastodon instances 2023-01-06 13:20:41 +08:00
Mike Fährmann
4fd6da474f
merge #3473: [twitter] fix crash when using 'expand' and 'syndication' 2023-01-05 14:19:47 +01:00
Mike Fährmann
818f6fa71f
merge #3453: [twitter] document #1396 and #2712 2023-01-05 14:18:21 +01:00
ClosedPort22
455481ad76
[twitter] improve documentation 
This includes:
- The automatic expansion of conversations (#2712)
- How to retrieve age-restricted content from search results (#1396)
 2023-01-05 12:14:42 +08:00
Mike Fährmann
f29ba089ff
merge #3474: [fanleaks] add 'post' and 'model' extractors 2023-01-04 17:39:17 +01:00
Mike Fährmann
6933727b45
merge #3483: [twitter] implement 'syndication=extended' 2023-01-04 17:36:17 +01:00
Mike Fährmann
9116398c1c
[pinterest] add 'domain' option (#3484) 
use input URL domain by default
 2023-01-04 17:20:14 +01:00
Mike Fährmann
f1a715dbcd
improve docs/options.md 
- use non-breaking hyphens
- don't replace '…' in examples
- don't force width for long option column
 2023-01-04 14:41:05 +01:00
Mike Fährmann
294108c90a
[pinterest] support 'All Pins' boards (#2855, #3484) 2023-01-03 19:11:20 +01:00
Mike Fährmann
13c16490bd
handle subcategory names ending with 'y' 2023-01-03 18:51:42 +01:00
Mike Fährmann
50d89d4acb
docs/options.md: use a separate table for each option group 2023-01-03 16:04:48 +01:00
Mike Fährmann
7de6b7f489
add list of command-line options as docs/options.md (#3464) 2023-01-02 19:14:16 +01:00
ClosedPort22
6853b14be3
[twitter] apply suggestions from code review 
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
 2023-01-02 21:03:01 +08:00
Mike Fährmann
4611237f8c
merge #3457: [danbooru] extract uploader metadata (if option is set) 2023-01-01 21:20:39 +01:00
Mike Fährmann
e7522482bb
merge #3463: [lynxchan] support 'bbw-chan.nl' 2023-01-01 15:50:09 +01:00
Mike Fährmann
0ed5986062
document custom 'output.mode' (#2529) 2022-12-30 17:14:42 +01:00
enduser420
5a740ef78b [fanleaks] add 'post' and 'model' extractors 2022-12-30 19:24:05 +05:30
ClosedPort22
7c8eab8d52
[twitter] implement 'syndication=extended' 
to be able to fetch extended user metadata
 2022-12-30 20:48:41 +08:00
ClosedPort22
be3286206a
[twitter] assume 'conversation_id' when using syndication 
not possible to expand replies at the momemt
 2022-12-30 13:57:37 +08:00
ClosedPort22
ce8dbb1ccc
[twitter] fix crash when using 'expand' and 'syndication' 
caused by KeyError: 'conversation_id_str'
 2022-12-30 12:45:44 +08:00
Mike Fährmann
d651d45239
implement specifying ranges in slice notation (#918, #2865) 
e.g.
- '1:101'   or ':101' or ':101:'  for files 1 to 100
- '1::2'    or '::2'              for every second file
- '1:101:5' or ':101:5'           for files 1, 6, 11, ..., 91, 96

(the second argument specifies the first index NOT included)
 2022-12-27 18:21:12 +01:00
enduser420
527bb2c4ab [lynxchan/bbw-chan] add 'thread' and 'board' extractors 2022-12-26 20:59:42 +05:30
pi_allen
64902f518e [docs] Update links and fix field typo 2022-12-26 07:46:41 +01:00
ClosedPort22
5d5ea58887
[twitter] update docs for 'quoted' & 'unique' 2022-12-25 15:59:26 +08:00
blankie
f82ee93676
[danbooru] extract uploader metadata (if metadata is set) 2022-12-25 11:00:59 +07:00
Mike Fährmann
1800bd7d14
allow '*-filter' options to be a list of expressions 2022-12-23 22:20:21 +01:00
lx30011
895b41f1ac [jschan] add generic jschan extractor 2022-12-23 00:32:52 +01:00
Mike Fährmann
73ab5d84c0
update docs/configuration.rst 2022-12-20 17:30:46 +01:00
Mike Fährmann
bee354c264
Merge pull request #3415 from enduser420/extractor/fapello 
[fapello] add 'post', 'user' and 'path' extractors
 2022-12-17 20:05:46 +01:00
ClosedPort22
dd4a4a3fa6
[e621] softcode the pagination threshold 2022-12-17 21:06:47 +08:00
enduser420
e5076ba056 [fapello] add 'post', 'user' and 'path' extractors 2022-12-16 16:53:32 +05:30
ClosedPort22
d0ad6d0e67
[e621] implement manual pagination mode 2022-12-16 16:15:36 +08:00
Mike Fährmann
1317625ec4
[webmshare] add 'video' extractor (#2410) 2022-12-14 19:59:07 +01:00
enduser420
41bf236d36
[lynxchan] add generic extractors for lynxchan imageboards (#3394) 
* [lynxchan] add generic extractors for lynxchan imageboards

includes kohlchan.net, endchan.org:wq

* [lynxchan] set pop default to empty tuple

* Apply suggestions from code review

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
 2022-12-12 13:13:39 +01:00
Mike Fährmann
3c75c3bbc4
[soundgasm] add 'user' extractor (#3384) 
based on code from PR #3388 by @enduser420
 2022-12-12 12:23:59 +01:00
Mike Fährmann
2952add4a8
[reddit] increase 'id-max' default value (#3397) 
to float("inf")
 2022-12-12 12:10:06 +01:00
Mike Fährmann
6b6f886dcf
[bunkr] update domain (#3391) 
and improve bunkr/app.bunkr handling
 2022-12-11 23:18:01 +01:00
Mike Fährmann
eb94568e1f
[soundgasm] add 'audio' extractor (#3384) 2022-12-09 23:19:07 +01:00
Mike Fährmann
dfe7b23579
support Firefox containers for --cookies-from-browser (#3346) 2022-12-09 19:43:55 +01:00
Mike Fährmann
4a3a1f4c87
[komikcast] update domain and fix extraction 2022-12-06 22:00:23 +01:00
Mike Fährmann
a42ba25ca1
[foolslide] remove 'kireicake' 
site redirects to (unclaimed) mangadex group
 2022-12-05 19:47:56 +01:00
Mike Fährmann
86f0597c95
[kissgoddess] remove module 
site does not host albums anymore
 2022-12-05 19:28:50 +01:00
Mike Fährmann
fceaee3c4f
[lolisafe] remove zz.ht 2022-12-04 16:21:13 +01:00
enduser420
54844944ab
[pixhost] add 'gallery' support (#3353) 2022-12-04 16:19:14 +01:00
enduser420
213676c785
[fapachi] add 'post' and 'user' extractors (#3347) 
* [fapachi] add 'post' and 'user' extractors

* [fapachi] add 'keyword' to test

* [fapachi] remove whitespaces
 2022-12-02 13:54:29 +01:00
Mike Fährmann
80102fa367
[downloader:http] add 'retry-codes' option (#3313) 2022-12-01 11:08:23 +01:00
ClosedPort22
b14b33f19e
Implement version-metadata option (#3201) 2022-11-27 16:09:42 +01:00
Mike Fährmann
6cb12f513b
[nitter] support quoted Tweets 
- distinguish between regular and quoted Tweets and media
- add 'quoted' option and metadata field
 2022-11-26 11:23:03 +01:00
Mike Fährmann
a41d093bb1
[nitter] add 'retweets' option (#3278) 2022-11-25 19:53:28 +01:00
Mike Fährmann
e31d12139c
[nitter] add 'videos' option (#3279) 
with the same semantics as for twitter
 2022-11-24 22:56:01 +01:00
Mike Fährmann
501d9bccfe
[artstation] add 'max-posts' option (#3270) 2022-11-23 22:00:18 +01:00
Mike Fährmann
b1ad6f2289
[artstation] add 'pro-first' option (#3273) 2022-11-23 21:45:20 +01:00
Mike Fährmann
42481aed59
[formatter] implement 'S' format specifier (#3266) 
to Sort lists
 2022-11-21 21:44:42 +01:00
Mike Fährmann
6b97dcf2e0
[postprocessor:metadata] add 'private' option 2022-11-20 15:27:36 +01:00
Mike Fährmann
08fd1ff835
[twitter] add 'avatar' and 'background' extractors (#349, #3023) 2022-11-18 23:06:22 +01:00
Mike Fährmann
6379157543
[instagram] use REST API by default 
regardless of logged in status
 2022-11-17 17:15:38 +01:00
enduser420
7897f68225
[wallhaven] update 'user' extractor (#3226) 
* [wallhaven] update 'user' extractor

* [wallhaven] update 'configuration.rst'

add 'extractor.wallhaven.include' entry

* [wallhaven] add 'wallhaven.include' in gallery-dl.conf
 2022-11-15 17:35:52 +01:00
enduser420
5a68b5cb3c
[wallhaven] add 'user' extractor (#3213) 2022-11-15 12:25:12 +01:00
Mike Fährmann
6c153750fa
[nitter] add extractors for Nitter instances (#2696) 2022-11-15 11:44:16 +01:00
Mike Fährmann
9f06e79868
implement '"user-agent": "browser"' (#2636) 2022-11-13 19:17:39 +01:00
enduser420
039d06c8f6 [mangaread] add 'chapter' and 'manga' extractors 2022-11-13 16:00:34 +05:30
Mike Fährmann
70c7fbe89a
[instagram] add 'guide' extractor (#3192) 2022-11-11 20:21:41 +01:00
Mike Fährmann
8d543b040d
update docs 
- document '"mode": "jsonl"' for metadata pp
- remove 'username' and 'password' from 'instagram'
- add 'ytdl' example (#3180)
 2022-11-09 22:17:08 +01:00
Mike Fährmann
171262c1b6
[instagram] remove login support 
broken feature that I cannot get to work anymore
 2022-11-08 21:48:31 +01:00
Mike Fährmann
895f36e53b
[postprocessor:metadata] add 'open' and 'encoding' options 2022-11-07 15:37:22 +01:00
Mike Fährmann
870e6a48a0
implement 'http-metadata' option 
or at least attempt to.
 2022-11-05 18:29:29 +01:00
Mike Fährmann
ccb80f1b8b
[uploadir] add support for 'uploadir.com' (#3162) 2022-11-05 14:25:09 +01:00
enduser420
c01cad599a
[lolisafe] add support for xbunkr (#3156) 2022-11-04 16:13:52 +01:00
Mike Fährmann
bca9f965e5
[downloader:http] add 'chunk-size' option (#3143) 
and double the previous default from 16384 (2**14) to 32768 (2**15)
 2022-11-02 16:50:26 +01:00
Mike Fährmann
c12a97bcde
[postprocessor] add 'post-after' event (#3117) 2022-10-31 14:35:48 +01:00
Mike Fährmann
58d97188b4
[mastodon] add 'bookmark' extractor (#3109) 2022-10-26 21:28:50 +02:00
Mike Fährmann
7c6af27eb8
[tumblr] add 'fallback-*' options (#2957) 
specifically 'fallback-delay' and 'fallback-retries'
and change default number of retries to 2 (down from 3)
 2022-10-26 13:59:09 +02:00
enduser420
ac6111e693
[mangasee] add support for 'mangalife' (#3086) 2022-10-21 16:46:36 +02:00
Mike Fährmann
a7d23f1484
[vichan] add generic extractors for vichan imageboards 
includes 8kun.top, smuglo.li, and wikieat.club
 2022-10-21 14:40:45 +02:00
thatfuckingbird
062ef238a6
add support for aibooru (using danbooru extractor) (#3075) 2022-10-19 11:53:59 +02:00
enduser420
0163ca86f7
[smugloli] add smugloli extractors (#3060) 2022-10-19 11:25:18 +02:00
Mike Fährmann
cf86f68864
[instagram] add 'avatar' extractor (#929, #1097, #2992) 2022-10-19 10:58:42 +02:00
Mike Fährmann
618c81afdf
[ngomik] remove module 
"Access denied"
 2022-10-19 10:47:25 +02:00
Mike Fährmann
52d1eb928d
[pixiv] extend 'metadata' option (#3057) 
make it usable for all 'pixiv' extractors
 2022-10-16 15:32:31 +02:00
Mike Fährmann
0714274f1f
[instagram] remove 'channel' extractor 2022-10-15 21:13:58 +02:00
Mike Fährmann
3b369ce3d1
[nijie] add 'followed' extractor (#3048) 2022-10-14 14:59:18 +02:00
Mike Fährmann
c4a62a48ae
[nijie] add 'feed' extractor (#3048) 2022-10-14 12:03:00 +02:00
Mike Fährmann
f1f89b2436
[tumblr] add 'offset' option 2022-10-11 10:54:23 +02:00
Mike Fährmann
15a4d7696f
[docs] update 'adjust-extensions' explanation (#3031) 2022-10-11 10:48:28 +02:00
Mike Fährmann
1696f68a68
[8chan] add 'thread' and 'board' extractors (#2938) 2022-10-11 10:47:19 +02:00
Mike Fährmann
560f7b41d8
[vk] add 'tagged' extractor (#2997) 2022-10-08 15:41:39 +02:00
Mike Fährmann
7f30a0d7a7
add 'path-extended' option (#3021) 2022-10-08 15:41:39 +02:00
Mike Fährmann
14717f3fc9
[deviantart] add 'group' option (#3018) 
disabling this option allows to better download from deleted accounts
 2022-10-08 15:41:21 +02:00
enduser420
f0321f423d
[2chen] Add 2chen.moe extractor (#2707) 
* [2chen] Add 2chen.moe extractor

* change "==" to is

* fix for "test_unique_pattern_matches"

* fix regex pattern and group matching

* fix regex again

* [2chen] add 'reply_no' and 'hash' metadata and change 'filename_fmt'

also made an entry in supportedsites.md

* [2chen] unescape 'title'

* [2chen] partition() -> rpartition()

* [2chen] extract 'date' and 'name' metadata

* [2chen] remove 'offset' argument

* [2chen] do some changes

* [2chen] do some more changes

* [2chen] unescape 'name' and 'filename'
 2022-10-04 22:18:13 +02:00
enduser420
f7ba19a1c0
[nana] add 'nana' extractors (#2967) 2022-10-04 09:23:24 +02:00
Mike Fährmann
4089bceddd
[sankaku] implement 'refresh' option (#2958) 2022-09-30 19:55:48 +02:00
Mike Fährmann
63e0924927
[pixiv] add 'series' extractor (#2964) 2022-09-27 23:24:03 +02:00