Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-25 12:12:34 +01:00
Code Issues Releases Wiki Activity
6,098 Commits 6 Branches 164 Tags 15 MiB
master
Commit Graph

701 Commits

Author SHA1 Message Date
Mike Fährmann
f3de6b7a87
[postprocessor:metadata] implement 'mode: delete' (#2640) 2022-07-19 00:57:29 +02:00
Mike Fährmann
eb68d45544
add global 'warnings' option (#2762) 2022-07-18 22:20:30 +02:00
Mike Fährmann
f225247670
[gelbooru] add support for api_key and user_id (#2767) 2022-07-18 18:46:31 +02:00
Mike Fährmann
74865adae5
implement 'format-separator' option (#2737) 
a global option, that servers as a workaround for shortcomings due to
lack of a proper format string parser
 2022-07-10 13:31:43 +02:00
bradenhilton
117eeefda0
[postprocessor:mtime] add 'value' option (#2739) 2022-07-08 20:56:01 +02:00
Mike Fährmann
1d14928bd9
[twitter] ignore previously seen Tweets (#2712) 
occurs primarily for /with_replies results when logged in
 2022-07-03 16:13:53 +02:00
Mike Fährmann
4b2a0a0eda
[twitter] implement 'strategy' option (#2712) 
to be able to better control what Tweets get used an returned
for twitter.com/USER URLs.
 2022-07-03 14:29:15 +02:00
Mike Fährmann
124b5c6410
[postprocessor:ugoira] enable 'mtime' by default (#2714) 2022-07-01 20:59:19 +02:00
Mike Fährmann
3a5d5c3a91
update default User-Agent header to Firefox 102 ESR 
snd update headers and ciphers for "browser": "firefox"
 2022-06-28 17:38:58 +02:00
Mike Fährmann
9d8e99af80
[itaku] support videos (#1842) 2022-06-20 19:47:53 +02:00
Mike Fährmann
9c8d895d19
[twitter] implement 'csrf' option (#2676) 2022-06-13 18:36:39 +02:00
Mike Fährmann
6db77d4656
[weibo] support '?tabtype=video' listings (#2601) 2022-06-12 17:55:23 +02:00
Mike Fährmann
4b78bd423f
[paheal] add 'metadata' option (#2641) 2022-06-04 16:05:49 +02:00
Mike Fährmann
57508d3bb7
[weibo] support all different 'tabtype' listings (#686, #2601) 2022-06-03 16:36:22 +02:00
HRXN
6d0aa8c0f3
[docs/configuration.rst] Fix small typo (#2649) 2022-06-01 18:31:39 +02:00
Mike Fährmann
e0ac358aa5
[gofile] fix 401 Unauthorized errors (#2632) 2022-06-01 13:02:34 +02:00
Mike Fährmann
4bf5bc2403
[weibo] support 'livephoto' entries (#2146) 2022-05-31 15:35:24 +02:00
Mike Fährmann
5b43faffed
[postprocessor:metadata] write to stdout by setting filename to "-" 
(#2624)
 2022-05-30 21:17:31 +02:00
Mike Fährmann
603af48265
implement 'output.ansi' option (#2628) 2022-05-29 19:15:25 +02:00
Mike Fährmann
915dba8345
[twitter] improve results for regular user URLs 
- continuation of 3346f58a
- use media timeline results (or tweet timeline if retweets are enabled)
  plus search results starting from the last tweet id of the first
  timeline, similar to how Twitter Media Downloader operates
- the old behavior can be forced by appending '/tweets' to a user URL,
  like with '/media' (https://twitter.com/USER/tweets)
  although there should be no need to ever do that
 2022-05-23 18:33:52 +02:00
Mike Fährmann
9df4e0f65b
[twitter] disable 'cards' by default 2022-05-21 15:39:25 +02:00
Mike Fährmann
79dce8ae68
[weasyl] implement 'metadata' option (#2610) 2022-05-20 22:32:35 +02:00
Mike Fährmann
6b6eb0b8f6
[lolisafe] implement 'domain' option (#2575) 2022-05-10 12:17:59 +02:00
Mike Fährmann
6742f3bc1e
implement --cookies-from-browser (#1606) 
most of the code is adapted from yt-dlp's implementation
and *should* work the same.
 2022-05-07 23:06:37 +02:00
Mike Fährmann
61887c895b
implement 'output.colors' options (#2532) 2022-05-02 12:41:14 +02:00
Mike Fährmann
84756982e9
[pixiv] implement 'include' option 
- split 'user' extractor and its 'avatar' and 'background' options into
  separate extractors ('artworks', 'avatar', 'background')
- avatars can now be downloaded with
  https://www.pixiv.net/en/users/ID/avatar
  as URL and will use a proper archive key; similar for backgrounds
- options for the 'user' subcategory must be moved to 'artworks' to have
  the same effect as before
 2022-05-02 09:03:35 +02:00
Mike Fährmann
d11e2191ae
[nijie] support /history_nuita.php listings (closes #2541) 2022-05-02 09:03:34 +02:00
Mike Fährmann
61afa973b7
use extended format strings for archive keys 2022-04-28 20:26:12 +02:00
Mike Fährmann
9e6ff42a9d
[pixiv] implement 'background' option (#623, #1124, #2495) 2022-04-21 13:53:02 +02:00
Mike Fährmann
97e8a15295
[deviantart] implement 'pagination' option (#2488) 2022-04-18 18:08:01 +02:00
Mike Fährmann
12bd9ba33a
[readcomiconline] add 'quality' option (#2467) 2022-04-15 18:10:37 +02:00
Mike Fährmann
24ea1398bc
[postprocessor:ugoira] do not auto-select image2 demuxer (#2492) 2022-04-12 21:47:47 +02:00
Mike Fährmann
1171911dc3
[twitter] add 'syndication' option (#2354) 
to fetch age-restricted content using Twitter's  syndication API
 2022-04-01 16:56:47 +02:00
loragja
7e545a3ae9
[gofile] add gofile.io extractor (#2364) 
* Add gofile extractor

* add gofile extractor to module list

* add support for tiny monitors and ancient python versions

* seriously, f-strings are not *that* new...

* i love flake8 :)

* add 'api-token' and 'recursive' options
* add tests
 2022-03-29 17:31:57 +02:00
Mike Fährmann
5c2d953766
[postprocessor:ugoira] auto-select demuxer by default 2022-03-26 21:10:59 +01:00
Mike Fährmann
37d584a9b2
[hitomi] update metadata extraction (fixes #2444) 
remove 'hitomi.metadata' option, as it is no longer necessary
to make additional HTTP requests to fetch all metadata.
 2022-03-26 12:46:18 +01:00
Mike Fährmann
e718dd7b32
[postprocessor:ugoira] support setting timecodes with mkvmerge 
by selecting the "mkvmerge" demuxer

(#1550)
 2022-03-26 12:41:55 +01:00
Mike Fährmann
71bba774da
respect 'output.private' in '-K/--list-keywords' output 2022-03-25 22:19:37 +01:00
Mike Fährmann
ba69fb669d
[kemonoparty] add 'duplicates' option (closes #2440) 2022-03-24 11:58:38 +01:00
Mike Fährmann
40ce505420
[postprocessor:ugoira] add 'mtime' option (#2307) 2022-03-21 10:02:29 +01:00
Mike Fährmann
9bd27b1b8d
[postprocessor:metadata] implement archive options (#2421) 
'archive', 'archive-format', and 'archive-prefix'
 2022-03-20 21:16:46 +01:00
Mike Fährmann
f8230dde43
[instagram] add 'previews' option (#2135) 2022-03-19 15:26:40 +01:00
Mike Fährmann
fc277fa45f
[seiga] require authentication with 'user_session' cookie (#2372) 
Login with username & password would now require entering a 2FA token.

see also 7b009cc893
 2022-03-11 02:10:15 +01:00
Mike Fährmann
47cf05c4ab
refactor proxy handling code (#2357) 
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
 2022-03-10 23:55:35 +01:00
Mike Fährmann
acbb0f5a57
clarify 'metadata.mtime' usage and shortcomings (#2307) 2022-03-08 21:11:06 +01:00
Mike Fährmann
bfa5e61900
[patreon] add explicit 'image_large' file type (#2257) 
to allow more control over when and if to download 'large_url' images

4fee3a0e52 forced them to be downloaded
instead of regular images, even though 'large_url' images are most likely
an upscaled version of the original.
 2022-03-06 17:07:13 +01:00
Alice
f1cab23724
[skeb] add 'sent-requests' option (#2322) (#2330) 
* Update skeb.py

* Update configuration.rst

* flake8
 2022-02-28 22:42:15 +01:00
Mike Fährmann
4385a34e05
[twitter] fix handling of 429 responses (fixes #2339) 
Twitter doesn't return a valid JSON response for 429 errors anymore.
 2022-02-28 16:42:55 +01:00
Mike Fährmann
59749555b1
[postprocessor:mtime] add 'event' option (#2307) 2022-02-22 23:27:40 +01:00
Mike Fährmann
7958995398
[postprocessor:metadata] implement 'mtime' option (#2307) 2022-02-22 23:02:13 +01:00
Mike Fährmann
254a5b26e0
[twibooru] add extractors for searches, galleries, and posts 
(#2219)
 2022-02-18 23:43:57 +01:00
Mike Fährmann
f5b2a9fcc6
add 'signals-ignore' option (#2296) 2022-02-13 23:00:50 +01:00
Mike Fährmann
f1c853c6ef
[furaffinity] add 'layout' option (#2277) 
to be able to force gallery-dl to parse according to a specific layout
in case its auto-detect fails
 2022-02-11 00:28:47 +01:00
Mike Fährmann
86fa412b47
[hitomi] add 'format' option (#2260) 
default is 'webp' since downloading original files is no longer allowed
 2022-02-03 23:32:19 +01:00
Mike Fährmann
492436f936
[twitter] add 'warnings' option (#2258) 
disable reporting any non-fatal errors by default
 2022-02-02 18:37:19 +01:00
Mike Fährmann
de754590e0
add --source-address command-line option (closes #2206) 2022-01-21 17:07:56 +01:00
Mike Fährmann
f2e8aedd74
[twitter] changes to 'cards' option 
- change default value to 'true'
- only invoke youtube-dl for cards unsupported by gallery
  when 'cards' is set to "ytdl"

"cards": true   --> only download card images
"cards": "ytdl" --> download card images and
                    use youtube_dl on otherwise unsupported cards
 2022-01-15 22:02:57 +01:00
Mike Fährmann
3c79c9b271
document extended blacklist/whitelist syntax (#2025) 
and not just in the commit message of 010d65dc
 2022-01-06 23:36:57 +01:00
Mike Fährmann
fa7d92f7a9
add docs for 'extractor.generic.enabled' 2021-12-29 22:46:17 +01:00
Mike Fährmann
64cf26eaf4
allow specifying sleep-* options as string 
either as single value or as range: "3.5", "2.1 - 5.0"
 2021-12-18 23:28:56 +01:00
Mike Fährmann
62692c6842
[exhentai] add 'source' option 
setting it to "hitomi" downloads the corresponding gallery from
hitomi.la; might be extended to other sources in the future
 2021-12-16 23:16:19 +01:00
Mike Fährmann
099ed72de7
[hitomi] disable extra 'metadata' by default 
safes one HTTP request that not needed with default filename settings
 2021-12-16 22:21:07 +01:00
Mike Fährmann
f587458a3c
[twitter] include '4096x4096' as a default image fallback 
(closes #2107, closes #1881)
 2021-12-15 23:19:30 +01:00
Mike Fährmann
f4e3cee6ac
use yt-dlp by default (#1850, #2028) 2021-11-29 18:24:26 +01:00
Mike Fährmann
f1b142e993
{kemonoparty[ change default 'files' order to attachments,file,inline 
(#1991)
 2021-11-29 04:41:30 +01:00
Alice
612850438e
[skeb] add 'thumbnails' option (#2047) (#2051) 2021-11-23 21:16:42 +01:00
Mike Fährmann
11a3d96d13
[mangadex] load additional metadata using includes[] directives 
- always provide 'artist', 'author', and 'group' metadata fields (#2049)
- remove 'metadata' option
 2021-11-22 01:16:33 +01:00
Mike Fährmann
c67756e187
[kemonoparty] add 'dms' option (#2008) 2021-11-20 23:36:16 +01:00
Mike Fährmann
9bc83af3a6
[kemonoparty] 'postfile' -> 'file' (#1991) 
to stay consistent with the existing file types for kemono
 2021-11-19 01:50:48 +01:00
Mike Fährmann
d433735750
[kemonoparty] skip duplicate files (#2032, #1991, #1899) 
Extract the SHA-256 file hash from URLs
and skip files with the same hash in the same post.

- provide a 'hash' metadata field (empty string if not available)
- remove 'patreon-skip-file' option
 2021-11-17 22:44:15 +01:00
Mike Fährmann
d4ec245554
[kemonoparty] implement a 'files' option (#1991) 
similar to 8d676151
 2021-11-17 22:43:41 +01:00
Mike Fährmann
7c50f473e4
[ytdl] document 'cmdline-args' and 'config-file' options (#1680) 2021-11-12 18:54:04 +01:00
Mike Fährmann
b315a0ecef
[redgifs] update to API v2 (#1984) 2021-11-04 21:31:20 +01:00
Mike Fährmann
f0fc3b0ba1
[kemonoparty] add 'comments' option (#1980) 2021-11-03 23:02:13 +01:00
Mike Fährmann
9156e90f1f
[twitter] add 'pinned' option 2021-10-29 22:10:58 +02:00
Mike Fährmann
06b414c9a3
[redgifs] 'gfyId' -> 'id' (#1984) 2021-10-29 02:05:39 +02:00
Mike Fährmann
02a247f4e5
[deviantart] full resolution for non-downloadable images (#293) 
Many thanks to @Ironchest337 for discovering this method
and providing a well-documented implementation.
 2021-10-24 21:11:12 +02:00
Mike Fährmann
8bb442f20d
[redgifs][gfycat] provide fallback URLs (fixes #1962) 
and extend the 'format' option
 2021-10-22 22:47:29 +02:00
Mike Fährmann
8d676151b7
[patreon] implement 'files' option (#1935) 2021-10-17 04:17:10 +02:00
Mike Fährmann
373d3e1c57
[seisoparty] implement login with username & password (#1906) 2021-10-08 22:44:31 +02:00
Mike Fährmann
d93b5474c3
[mangadex] update parameter handling for API requests 
- move common parameters into '_pagination()'
- add 'ratings' (#1908) and 'api-parameters' options
 2021-10-05 19:46:48 +02:00
Mike Fährmann
cd66c3c415
[twitter] add 'size' option (#1881) 2021-10-05 19:14:54 +02:00
Mike Fährmann
df8050b81d
[postprocessor:compare] add 'equal' option (#1592) 
Move functionality from cdd72e14 to its own option,
where it can be used with any 'action'
 2021-10-05 03:03:37 +02:00
Mike Fährmann
cdd72e1413
[postprocessor:compare] extend 'action' option (#1592) 
allow setting it to "abort", "terminate", or "exit" as with 'skip'
 2021-10-01 20:18:36 +02:00
Mike Fährmann
8c29a6e491
add example for 'url-metadata' option (closes #1884) 2021-09-28 22:59:37 +02:00
Mike Fährmann
d0761454b1
implement a download progress indicator (#1519) 2021-09-28 22:48:58 +02:00
Mike Fährmann
140a7bfdcd
update --clear-cache command in deviantart client-id instructions 2021-09-23 18:02:19 +02:00
Mike Fährmann
e69ee41f25
implement 'page-reverse' option (#1854) 2021-09-23 18:02:19 +02:00
Mike Fährmann
7b05600056
[pinterest] only mention cookies for authentication (#1856) 
regular login is broken due to CAPTCHA
 2021-09-16 16:43:05 +02:00
Mike Fährmann
c9e6693530
allow specifying a minimum/maximum for 'sleep-*' options (#1835) 
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
 2021-09-14 17:40:05 +02:00
Mike Fährmann
bd845303ad
implement a way to shorten filenames with east-asian characters 
(#1377)

Setting 'output.shorten' to "eaw" (East-Asian Width) uses a slower
algorithm that also considers characters with a width > 1.
 2021-09-13 21:38:33 +02:00
Mike Fährmann
4ec11af6a4
[kemonoparty] implement login with username & password (#1824) 2021-09-09 01:06:25 +02:00
Mike Fährmann
44f572c27f
[deviantart] implement a 'auto-unwatch' option (#1466, #1757) 2021-09-08 00:27:28 +02:00
Mike Fährmann
96fec14ef7
[deviantart] rename 'watch' option to 'auto-watch' 
(#1466, #1757)

Similar reason as in e05a96db.
'watch' is already used by the WatchExtractor class.
 2021-08-29 21:10:53 +02:00
Mike Fährmann
e75f2de9da
[deviantart] add 'comments' option (#1800) 2021-08-29 20:56:34 +02:00
Mike Fährmann
ecc8da4704
[deviantart] implement a 'watch' option (#1466, #1757) 2021-08-27 18:39:43 +02:00
Mike Fährmann
d3eab417ed
implement a 'path-strip' option 2021-08-24 23:23:12 +02:00
Mike Fährmann
d320ee6251
implement a 'fallback' option (closes #1770) 2021-08-16 01:47:59 +02:00
Mike Fährmann
c866fcba48
[twitter] fix 'logout' (#1719) 
delete 'auth_token' cookie and cookies.txt path
 2021-08-16 01:36:34 +02:00
Mike Fährmann
ed4b3c48cb
fix flake8 and other tests 2021-08-12 16:05:26 +02:00
Nyasume
fa6af46756
Added ability to download GIFs instead of mp4 from Luscious and Reactor (#1701) 2021-08-12 15:12:42 +02:00
Mike Fährmann
e5a93e113f
[twitter] extend 'replies' option (#1254) 
Allow setting 'replies to '"self"' to only download from self-replies.
 2021-08-10 22:14:00 +02:00
Mike Fährmann
cadfad4eea
[danbooru] add 'external' option (closes #1747) 2021-08-08 19:18:41 +02:00
Mike Fährmann
5b1c62bfa9
[furaffinity] add 'external' option (closes #1492) 2021-08-08 19:16:02 +02:00
Mike Fährmann
a316e44f8e
[bbc] add 'width' option (#1706) 2021-07-30 01:09:32 +02:00
Mike Fährmann
7a0da4f93f
[newgrounds] add 'format' option (closes #1729) 2021-07-29 19:11:20 +02:00
Mike Fährmann
486474800f
[kemonoparty] skip duplicated patreon files (closes #1689) 
this behavior can be disabled with the 'patreon-skip-file' option
 2021-07-23 16:38:57 +02:00
Mike Fährmann
13d4045a8a
add 'archive-prefix' option (#1711) 2021-07-20 20:21:33 +02:00
Mike Fährmann
b9783403d9
add 'url-metadata' option (#1659, #1073) 2021-07-14 03:08:49 +02:00
Mike Fährmann
e95f99882f
extend 'parent-metadata' functionality (#1687, #1651, #1364) 2021-07-14 02:53:41 +02:00
Mike Fährmann
dff0da60f9
[ytdl] add 'generic' option (#1680) 2021-07-11 23:48:18 +02:00
Mike Fährmann
36ac2197db
[ytdl] add extractor for sites supported by youtube-dl 
(#1680, #878)

Can be used by prefixing any URL with 'ytdl:',
or by setting 'extractor,ytdl.enabled' to 'true'.
 2021-07-10 20:55:47 +02:00
Mike Fährmann
8b036778e3
[kemonoparty] add 'max-posts' option (#1674) 2021-07-09 18:19:02 +02:00
Mike Fährmann
312a28e78a
[mastodon] add 'replies' option (#1669) 2021-07-07 00:59:02 +02:00
Mike Fährmann
a1f5b78039
[mastodon] add 'reblogs' option (#1669) 2021-07-06 23:27:32 +02:00
Mike Fährmann
93d356712c
[mastodon] implement 'text-posts' option (#1569) 
similar to Twitter's 'text-tweets'
 2021-07-02 22:12:41 +02:00
Mike Fährmann
963d177a68
document format string syntax 
or at least attempt to
 2021-06-29 19:35:07 +02:00
Mike Fährmann
e661607e8b
[mangadex] document 'metadata' and 'lang' options (#1535) 2021-06-20 22:44:02 +02:00
Mike Fährmann
fd00d47116
implement conditional directories (#1394) 
They work the same way as conditional filenames (84d2e640), e.g.

"directory": {
    "score >= 20": ["high score"],
    "score >= 5" : ["mid score"],
    ""           : ["{category}", "default"]
}
 2021-06-20 20:09:35 +02:00
Mike Fährmann
a751afdfb3
[twitter] change some defaults 
- 'retweets' option: true -> false
- 'quoted' option  : true -> false

  i.e. disable downloading tweets from other user's timelines by default

- search directory:
    '["{category}", "Search", "{search}"]' ->
    '["{category}", "{user[name]}"]'

  i.e. change it to the same as other twitter extractors (#1308)
 2021-06-11 21:26:11 +02:00
Mike Fährmann
84d2e64024
combine conditional filenames into filename option (#1394) 2021-06-08 18:00:06 +02:00
Mike Fährmann
07c8adbd8b
[mangadex] implement login with username & password (#1535) 2021-06-08 02:12:57 +02:00
Mike Fährmann
3cbbefd4ed
support 'filter' option for post processors (#1460) 2021-06-04 18:23:32 +02:00
Mike Fährmann
4cf40434d7
initial support for conditional filenames (#1394) 2021-06-04 16:45:32 +02:00
Mike Fährmann
f3d692a378
improve docs for danbooru and e621 passwords (#1593, #1596) 2021-06-02 23:02:23 +02:00
Mike Fährmann
c80b18a477
[weibo] extend 'retweets' option (closes #1542) 
Setting 'retweets' to "original" will use metadata from the
original posts, and not from the retweeted ones.
 2021-05-27 23:09:42 +02:00
Mike Fährmann
74d97e6c6b
[postprocessor:ugoira] add 'repeat-last-frame' option (#1550) 2021-05-26 15:13:11 +02:00
Mike Fährmann
be9547a5f3
[postprocessor:ugoira] use 'image2' demuxer (#1550) 
Ensures exact frame timecodes with no duplicate frames.

Possible issues are the duration the last frame in an Ugoira with variable
frame durations is shown and insufficient timestamp precision of the
underlying file system (e.g. FAT32, ext3; works on ext4, tmpfs, NTFS).
 2021-05-26 14:59:51 +02:00
Mike Fährmann
b8de0671e5
update description for 'instagram.include' 
- add double quotes for "reels"
- add "tagged"
 2021-05-26 14:59:34 +02:00
thatfuckingbird
e6811c7450
[pixiv] implement 'max-posts' option (#1558) 
* implement max-rank for pixiv

* rename to max-posts and make more generic
 2021-05-24 17:49:46 +02:00
Mike Fährmann
b5affc62aa
[twitter] rename 'text-only' to 'text-tweets' (#570) 2021-05-22 21:41:12 +02:00
Mike Fährmann
724ca61f36
[twitter] add 'text-only' option (#570) 2021-05-22 17:01:49 +02:00
Mike Fährmann
1eabfa5c7a
[pillowfort] implement login with username & password (#846) 2021-05-19 02:59:16 +02:00
Mike Fährmann
a7e4917ee1
[pillowfort] add 'inline' option (#846) 
to support images present in a post's 'content',
but not listed in 'media'.

also separates the file hash present at the beginning
of each 'filename' into its own field.
 2021-05-17 03:03:58 +02:00
Mike Fährmann
efa6cc8ec3
[pillowfort] add 'external' option (#846) 
for links to external Twitter posts etc.
 2021-05-17 01:46:42 +02:00
Mike Fährmann
5eeaaee01d
[pixiv] add 'metadata' option (#1551) 2021-05-14 20:30:28 +02:00
Mike Fährmann
0717456b4e
[kemonoparty] add 'metadata' option (closes #1548) 
to fetch creator names with an additional HTTP request
 2021-05-14 19:56:49 +02:00
Mike Fährmann
7ab8374385
add 'parent-skip' option (#1399) 2021-05-13 16:40:04 +02:00
Mike Fährmann
c693db5b1a
add '"skip": "terminate"' option 
Stops not only the current extractor/job,
but all parent extractors/jobs as well.
 2021-05-12 02:22:28 +02:00
Mike Fährmann
36ed1efcfb
[pixiv] rename "noop" value for 'tags' option to "original" 
(#1507)
 2021-05-07 20:41:54 +02:00
Mike Fährmann
e300da1424
add 'output.skip' option 2021-05-04 19:27:18 +02:00
Mike Fährmann
fa519f9202
[pixiv] change 'translated-tags' option (#1507) 
- rename to 'tags'
- use string-values: "japanese", "translated", "noop"
- remove duplicate entries for "translated" tags
 2021-04-29 19:30:43 +02:00
thatfuckingbird
e47952ac14
add extractors for fantia and fanbox (#1459) 
* add extractors for fantia and fanbox

* appease linter

* make docstrings unique

* [fantia] refactor post extraction

* [fantia] capitalize

* [fantia] improve regex pattern

* code style

* capitalize

* [fanbox] use BASE_PATTERN for url regexes

* [fanbox] refactor metadata and post extraction

* [fanbox] improve url base pattern

* [fanbox] accept creator page links ending with /posts

* [fanbox] more tests

* [fantia] improved pagination

* [fanbox] misc. code logic improvements

* [fantia] finish restructuring pagination code

* [fanbox] avoid making a request for each individual post when processing a creator page

* [fanbox] support embedded videos

* [fanbox] fix errors

* [fanbox] document extractor.fanbox.videos

* [fanbox] handle "article" and "entry" post types, all embeds

* [fanbox] fix downloading of embedded fanbox posts
 2021-04-25 19:39:13 +02:00
Mike Fährmann
9514cb8c12
[exhentai] update 'limits' check (#1487) 
Only use 'limits' to set a custom upper bound.
Checking if the actual maximum gets exceeded is not necessary.
 2021-04-23 23:20:45 +02:00
Renan Vedovato Traba
9322c5e43b
[exhentai] restore limit config (#1487) 
This partially reverts commit e9ec91c8
 2021-04-22 21:21:41 +02:00
thatfuckingbird
224b883ff4
[danbooru] add option for extended metadata extraction (#1458) 
* [danbooru] add option for extended metadata extraction

* appease linter

* [danbooru] update docs/configuration.rst

* [danbooru] rename extended-metadata -> metadata
 2021-04-13 23:41:30 +02:00
thatfuckingbird
dff03a6605
[booru] add an option to extract notes (only gelbooru for now) (#1457) 
* [booru] add an option to extract notes (currently implemented only for gelbooru)

* appease linter

* [gelbooru] rename "text" to "body" in note extraction

* add a code comment about reusing return value of _extended_tags
 2021-04-13 23:40:24 +02:00
Mike Fährmann
a86ffb04bb
add 'output.fallback' option 
to enable/disable fallback URLs for -g/--get-urls
 2021-04-12 02:00:41 +02:00
Mike Fährmann
d5aad999dc
[tapas] implement login with username & password (#692) 2021-03-30 01:45:28 +02:00
Mike Fährmann
e9ec91c811
[exhentai] improve image limits check 
- check if current image is the '509 Bandwidth Exceeded' notification
  (https://ehgt.org/g/509.gif or https://exhentai.org/img/509.gif)
- remove 'limits' option
 2021-03-29 19:01:13 +02:00
Mike Fährmann
2846235669
[twitter] allow specifying a custom format for user results 
(#1337)
 2021-03-21 22:26:26 +01:00
Mike Fährmann
b0438c8f99
Revert "[deviantart] extend 'extra' option" 
This reverts commit
5ad2b9c82b,
5c32a7bf58, and
83f465faca.

(#1387, #1356)
 2021-03-19 16:24:23 +01:00
Mike Fährmann
3378b39719
[twitter] implement 'users' option (#1337) 2021-03-16 00:51:05 +01:00
Mike Fährmann
df94182e11
implement 'parent-metadata' option (#1364) 
experimental, might not work as expected, etc.
 2021-03-11 01:10:34 +01:00
Mike Fährmann
83f465faca
[deviantart] refactor 'extra' (#1356) 
- change its expected type to string
- let users specify a list of sources (stash, posts) or 'all'
 2021-03-07 22:23:52 +01:00
Mike Fährmann
5c32a7bf58
[deviantart] allow selecting source for 'extra' (#1356) 
Setting 'extra' to "stash" or "deviations" will only download embedded
sta.sh content or deviations. 'true' still downloads both.
 2021-03-06 21:31:28 +01:00
Mike Fährmann
a677123abb
[instagram] recognize 'reels' as option for 'include' (#1329) 2021-03-06 18:50:19 +01:00
Mike Fährmann
7440d1f112
[pixiv] add 'translated-tags' option (closes #1354) 
(a lot more straight forward than I thought ...)
 2021-03-05 17:18:51 +01:00
Mike Fährmann
6a28b7045c
update 'docs/gallery-dl.conf' 2021-03-03 03:11:48 +01:00
Mike Fährmann
bae874f370
replace 'wait-min/-max' with 'sleep-request' 
on exhentai, idolcomplex, reactor
 2021-03-02 22:55:45 +01:00
Mike Fährmann
1a38fae785
add option to use different youtube-dl modules (fixes #1330) 
by setting the 'downloader.ytdl.module' value. For example

{
    "downloader": {
        "ytdl": {
            "module": "yt_dlp"
        }
    }
}

or '-o module=yt_dlp'
 2021-03-01 03:10:42 +01:00
Mike Fährmann
e58039358d
[mangadex] use 'api.mangadex.org' as default API server 
The caching issues seem to be gone.
(#1290, #1310)
 2021-02-28 01:31:50 +01:00
Mike Fährmann
29ea54dc41
[patreon] use '"browser": "firefox"' by default (#1117) 2021-02-27 16:26:42 +01:00
Mike Fährmann
61fbbd2dae
[exhentai] rename metadata fields to match API results (#1325) 
- gallery_id    -> gid
- gallery_token -> token
- title_jp      -> title_jpn
- visible       -> expunged
- gallery_size  -> filesize
- count         -> filecount

Also changes the function of the 'metadata' option.
It is now boolean and causes extra data fields from the API to be added
instead of completely replacing the data from HTML when activated.
 2021-02-26 18:21:04 +01:00
Mike Fährmann
5d69e437d0
[twitter] add option to download all media from a conversation 
(fixes #1319)
 2021-02-26 13:50:46 +01:00
Mike Fährmann
cf5fa75d4c
add 'browser' option (#1117) 
- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
 2021-02-26 13:41:27 +01:00
Mike Fährmann
e5735361ed
[exhentai] add 'metadata' option (#1325) 
to select between gallery metadata from 'api' or 'html'
 2021-02-23 00:23:42 +01:00
Mike Fährmann
5ad2b9c82b
[deviantart] extend 'extra' option 
also download from embedded DeviantArt posts
 2021-02-21 19:27:04 +01:00
Mike Fährmann
560277394e
[downloader:http] add 'headers' option (#1322) 2021-02-21 19:13:39 +01:00
Mike Fährmann
6b0ecbf6bc
[hentainexus] add 'orignal' option (#1322) 2021-02-21 19:06:12 +01:00
Mike Fährmann
8974f0361c
[pixiv] update (#1304) 
- remove login with username & password
- require a refresh token
- add 'oauth:pixiv' functionality

See also:
- https://github.com/upbit/pixivpy/issues/158
- https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362
 2021-02-12 18:07:16 +01:00
Mike Fährmann
cdb0b02e30
[pillowfort] add 'reblogs' option (#846) 2021-01-25 15:03:59 +01:00
Mike Fährmann
7a096c443f
[unsplash] add 'format' option (#1197) 2021-01-22 17:23:46 +01:00
Mike Fährmann
89a2bcbb2d
[furaffinity] add 'descriptions' option (#1231) 2021-01-19 19:09:29 +01:00
Mike Fährmann
36f281330a
[newgrounds] fix flash file extraction (closes #1257) 
… and add a 'flash' option to choose between flash and video formats.
 2021-01-19 17:48:14 +01:00
Mike Fährmann
c008cb5100
[pixiv] add 'related' option (#1237) 2021-01-17 22:48:32 +01:00
Mike Fährmann
01b9ccd4de
[derpibooru] use the "Everything" filter by default (#1243) 
when neither 'api-key' nor 'filter' are set
 2021-01-14 18:29:09 +01:00
Mike Fährmann
5d4494b15f
add "ascii" as a special 'path-restrict' value 2021-01-09 02:41:20 +01:00
Mike Fährmann
fa8ee6eac4
[derpibooru] add search and gallery extractors (#862) 2021-01-07 18:05:32 +01:00
Mike Fährmann
371e9ca6df
[pinterest] implement video support (closes #1189) 2020-12-21 16:09:06 +01:00
Mike Fährmann
bf629a2818
[instagram] add 'include' option (closes #1180) 
Split the functionality of the old 'user' extractor into separate
'posts' and 'highlights' extractors, which respond to virtual URLs
('/<user>/posts' and '/<user>/highlights')
 2020-12-21 02:06:28 +01:00
Mike Fährmann
465015f75a
[sankaku] reimplement login support (#1176, #1182) 2020-12-17 16:12:59 +01:00
Mike Fährmann
b2c55f0a72
[sankaku] remove login support 
The old login method for 'https://chan.sankakucomplex.com/user/login'
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
 2020-12-08 21:05:47 +01:00
Mike Fährmann
0bc492c0fa
add docs for 'event' and 'filename' options 
from 9c3568c3 and ca59bd69
 2020-11-25 12:12:41 +01:00
Mike Fährmann
9b1bd09454
change 'extension-map' default 
Replace all JPEG filename extensions with 'jpg'.
 2020-11-14 22:40:31 +01:00
Mike Fährmann
bc7b1d91bc
fix rST markup in configuration.rst 
[ci skip]
 2020-11-02 15:32:29 +01:00
Mike Fährmann
e3480bc8de
implement 'extension-map' option (#318) 2020-11-02 15:27:07 +01:00
Mike Fährmann
98a4d86a01
[sankakucomplex] extract videos and embeds (closes #308) 2020-10-30 01:21:11 +01:00
Mike Fährmann
d83b95fd28
[postprocessor:metadata] accept a string-list for 'content-format' 
(closes #1080)
 2020-10-27 20:09:58 +01:00
Mike Fährmann
5bcf28de93
add a 'extractor.modules' option 2020-10-25 03:05:10 +01:00
Mike Fährmann
1686dc1757
[twitter] support media from Cards (#1005, #937) 
Can be enabled with 'extractor.twitter.cards', but for now disabled by
default because cards can redirect to rather large videos from YouTube
or Twitch.
 2020-10-22 21:33:53 +02:00
Mike Fährmann
bac8af69e8
update configuration.rst 
- add some lines to better explain post processor usage
- syntax highlighting for JSON blocks and other smaller stuff
 2020-10-19 21:57:26 +02:00
Mike Fährmann
05d7009cc6
rename 'Authentication' entries in supportedsites.rst 
- change 'Optional' to 'Supported'
- use 'OAuth' and 'Cookies' in their own
- add link to weasyl API key option
 2020-10-19 20:16:17 +02:00
Mike Fährmann
d1c6d78477
fix rST markup in configuration.rst 2020-10-15 15:17:19 +02:00
Mike Fährmann
b8daabc3ca
[pinterest] implement login support (closes #1055) 
being logged allows access to secret/protected boards
 2020-10-15 15:14:18 +02:00
Zanny
3ec60e894a
[weasyl] api-key authentication (#1057) 
* [weasyl] support api keys

* [weasyl] document api-key authentication

* [weasyl] usernames can contain ~
 2020-10-15 15:12:09 +02:00
Mike Fährmann
35056a07d1
release version 1.15.1 2020-10-11 18:44:46 +02:00
Mike Fährmann
c874071f5a
[kissmanga] remove module 2020-10-04 22:46:41 +02:00
Mike Fährmann
430b6d6e2e
[twitter] extend 'retweets' option (closes #1026) 
Setting 'retweets' to '"original"' will use metadata from the
original retweeted Tweets, and not from the Retweet entry.
 2020-09-28 23:03:35 +02:00
Mike Fährmann
b9bdd2c564
[hentaifoundry] add support for stories (closes #734) 2020-09-27 02:27:40 +02:00
Mike Fährmann
117582d798
use definition lists instead of tables in docs/configuration.rst 
and strip some clutter from generated man pages
 2020-09-26 13:33:46 +02:00
Mike Fährmann
6514312126
[nijie] add 'include' option (closes #1018) 2020-09-25 18:18:35 +02:00
Mike Fährmann
0d43456323
[hentaifoundry] add 'include' option 2020-09-25 18:18:03 +02:00
Mike Fährmann
1e313d5b84
implement 'sleep-request' option 2020-09-20 20:28:17 +02:00
Mike Fährmann
6a5f6396d4
add clearing cache to dA client-id instructions (#996) 2020-09-16 01:24:18 +02:00
Mike Fährmann
3afd362e2e
add 'sleep-extractor' option (closes #964) 
(would have been nice if this were possible without code duplication)
 2020-09-12 21:04:47 +02:00
Mike Fährmann
3918b69677
remove 'extractor.blacklist' context manager 2020-09-11 13:17:35 +02:00
Mike Fährmann
c78aa17506
add general 'blacklist' and 'whitelist' options (#492, #844) 2020-09-11 13:17:12 +02:00
Mike Fährmann
5912727b88
support format string replacement fields in archive paths 
(closes #985)
 2020-09-10 22:09:30 +02:00
Mike Fährmann
f6fd449b59
reduce wait time growth rate from exponential to linear 
Waiting for 2**N seconds after each error grows too fast.
Simply waiting N seconds seems far more reasonable.
 2020-09-06 22:38:25 +02:00
Mike Fährmann
ac3036ef56
add 'filesize-min' and 'filesize-max' options (closes #780) 2020-09-03 18:21:04 +02:00
ArtaxIsSleeping
0e941553ec
[aryion] Add username/password support (#960) 
* Add username/password support to aryion extractor

* Update docs to match

* Fix code style
 2020-08-27 22:45:30 +02:00
Mike Fährmann
dbbbb21180
[exhentai] add ability to specify custom image limit (#940) 2020-08-17 22:29:20 +02:00
Mike Fährmann
901a897e6f
[inkbunny] mention login support 2020-07-26 23:52:52 +02:00
Mike Fährmann
b921fee24d
[inkbunny] fix submission order (#283) 
Getting detailed submission info via /api_submissions.php reordered the
input submissions and sorted them by ID. InkbunnyAPI.detail() now sorts
them back and ensures they are returned in their original order.

This commit also removes the 'metadata' option and always requests
submission descriptions.
 2020-07-26 23:12:45 +02:00
Mike Fährmann
c4ed9f4faa
[inkbunny] add 'metadata' option (#283) 2020-07-24 18:05:53 +02:00
Mike Fährmann
493cadb1e7
[inkbunny] add 'orderby' option (#283) 2020-07-24 17:50:32 +02:00
Mike Fährmann
d5fcffcced
[subscribestar] add login capabilities (#852) 2020-07-17 22:18:01 +02:00
Mike Fährmann
ecaecc4064
[exhentai] add 'domain' option (#897) 2020-07-17 22:17:46 +02:00
Mike Fährmann
cb0132e441
[khinsider] add 'format' option (closes #840) 2020-07-13 17:17:58 +02:00
Mike Fährmann
f1ddbff0b5
[aryion] add 'recursive' option (fixes #832) 
This is enabled by default and will recursively go through all
(sub)folders in an artist's gallery.

The old method of using "Latest Updates" lists can be restored by
disabling this option.
 2020-06-26 23:36:50 +02:00
Mike Fährmann
9da2bc67f8
[twitter] add option to filter media from quoted tweets (#854) 2020-06-25 18:59:25 +02:00
Mike Fährmann
5ba90f72ca
[pinterest] add support for sections (closes #835) 2020-06-16 14:41:05 +02:00
Mike Fährmann
c184cce876
update configuration.rst 
- fix anonymous links
- update description of 'extractor.twitter.videos'
- document 'extractor.redgifs.format' (#724)
 2020-06-12 18:25:17 +02:00
Mike Fährmann
41d03160ff
[deviantart] also search journals for sta.sh links (#712) 
when 'extra' is enabled
 2020-06-04 14:47:08 +02:00
Mike Fährmann
0138e9c1d8
[twitter] remove 'content' option 
The text content of each tweet is always available as 'full_text'
 2020-06-04 01:24:21 +02:00
Mike Fährmann
0f459f340b
[instagram] fix and re-enable login with username&password 
This reverts commit 3e0848a482.
(#756, #771, #797, #803)

https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
 2020-05-31 00:29:09 +02:00
Mike Fährmann
3e0848a482
[instagram] disable login with username&password (#756) 2020-05-29 23:29:40 +02:00
Mike Fährmann
da22ea8ced
use %APPDATA%\gallery-dl for config/cache on Windows 2020-05-28 02:56:38 +02:00
Mike Fährmann
b489f4de12
fix internal links in configuration.rst 2020-05-28 01:51:16 +02:00
Mike Fährmann
dfcf2a2c91
write OAuth token to cache by default (#616) 2020-05-25 22:35:45 +02:00
Mike Fährmann
ddc253cf9a
implement a 'path-replace' option (#662, #755) 2020-05-25 22:21:58 +02:00
Mike Fährmann
bc53302ad6
extend 'path-restrict' option 
Allow its value to be a JSON object / Python dict that specifies
a mapping from invalid/unwanted input characters to specific
output characters.

For example {"/": "-", "*": "+"} will transform
"foo / ***bar***" into "foo - +++bar+++"

(closes #662, #755)
 2020-05-25 22:21:56 +02:00
Mike Fährmann
e19f665a44
[danbooru] change default for 'ugoira' to 'false' 
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
 2020-05-20 19:57:28 +02:00
Mike Fährmann
dba87ca99e
[downloader:ytdl] change 'forward-cookies' default to 'false' 
There are currently no situations where forwarding gallery-dl's cookies
to youtube-dl is necessary, and it only causes problems when forcing
youtube-dl for Twitter video downloads while logged in.
 2020-05-12 20:17:08 +02:00
Mike Fährmann
be16d77073
simplify and improve documentation markup (#742) 
- use | to get a more "compact" representation
- link directly to example config files
 2020-05-12 17:00:34 +02:00
Vrihub
fe224416bf
Some documentation improvements (#742) 
* Link configuration examples in the intro, see #712

* Clarify overriding conf files by options, see #678
 2020-05-10 23:47:20 +02:00
Mike Fährmann
5e2974d699
[weibo] add 'videos' option 2020-04-30 00:00:30 +02:00
Mike Fährmann
9f638c2e01
[twitter] add 'replies' option (closes #705) 2020-04-29 23:20:06 +02:00
Mike Fährmann
5b676ea59d
[e621] document username & password support (#640) 2020-03-14 01:15:22 +01:00
Mike Fährmann
59edcdc822
[hitomi] restore metadata fields from before f33b13a 
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
 2020-03-12 23:43:41 +01:00
Mike Fährmann
a45fbc38ea
[pixiv] implement 'avatar' option (#595, #623) 2020-03-09 21:18:16 +01:00
Mike Fährmann
6e759fe2f5
reword note for 'extractor.deviantart.refresh-token' (#612) 2020-02-19 23:39:27 +01:00
Mike Fährmann
67ea80222c
link to post processor example in 'pixiv.ugoira' docs 2020-02-14 01:22:16 +01:00
Mike Fährmann
74e684e828
[twitter] change default value for 'videos' to 'true' 
Every other 'videos' option defaulted to 'true', except Twitter.
 2020-02-14 01:03:42 +01:00
Mike Fährmann
ab13f81fd7
add example to 'cookies' docs 2020-02-13 23:44:38 +01:00
Mike Fährmann
c7cf9dd111
[furaffinity] support classic layout (#284) 2020-02-12 21:39:43 +01:00
Mike Fährmann
da0d5f6092
[oauth] add 'port' option (#604) 2020-02-09 13:45:44 +01:00
Mike Fährmann
091f9a107d
[docs] add examples of how to filter manga chapters by language 
(closes #599)
 2020-02-07 23:21:44 +01:00
Mike Fährmann
dff33b260c
[reddit] add 'videos' option 2020-01-31 23:45:02 +01:00
Mike Fährmann
56f1c96168
implement 'parent-directory' option (#551) 2020-01-29 18:32:37 +01:00
Mike Fährmann
6703b8a86b
[blogger] implement video extraction (closes #587) 2020-01-24 23:37:23 +01:00
Mike Fährmann
37247dbaff
miscellaneous fixes 2020-01-19 22:53:06 +01:00
Mike Fährmann
25d5ec4ff3
[twitter] add option to extract TwitPic embeds (#579) 2020-01-18 21:31:29 +01:00
Mike Fährmann
254f7c3999
implement a post-processor module to compare file versions 
(#530)
 2020-01-18 21:08:25 +01:00
Mike Fährmann
87c8b89ddd
[postprocessor:metadata] add 'directory' option (#520) 2020-01-02 20:58:10 +01:00
Mike Fährmann
48e42e73fb
[reddit] change default value for 'comments' to '0' 2019-12-20 16:54:59 +01:00
Mike Fährmann
bf658fd84b
[vsco] implement 'videos' option 2019-12-19 18:34:19 +01:00
Mike Fährmann
95c90722ee
[instagram] implement 'videos' option (closes #521) 2019-12-19 18:34:06 +01:00
Mike Fährmann
15f9bb3d14
add option to disable pyOpenSSL usage (#508) 
(pyOpenSSL is now disabled by default)
 2019-12-08 21:21:00 +01:00
Mike Fährmann
93b43f74f4
update docs/configuration.rst 
- fix description of "skip": "enumerate"
- use line breaks for lists of multiple examples
- improve formatting of auto-generated man pages
 2019-12-06 01:19:23 +01:00
Mike Fährmann
e17907ee2a
change default value of 'cookies-update' to 'true' 2019-12-05 23:43:49 +01:00
Mike Fährmann
26d2334550
[postprocessor:metadata] rename 'format' to 'content-format' 
Just to be consistent with the other 'extension-format' option name,
and only 'format' is also still accepted.
 2019-11-30 17:27:49 +01:00
Mike Fährmann
a412531451
[postprocessor:metadata] implement 'extension-format' option 
closes #477
 2019-11-30 17:26:17 +01:00
Mike Fährmann
d45fabb79d
match user profile handling on deviantart and newgrounds 2019-11-22 23:20:21 +01:00
Mike Fährmann
dd5d2b2eac
[deviantart] add user profile extractor (#377, #419) 2019-11-07 18:29:49 +01:00
Mike Fährmann
9e88e7a344
[postprocessor:exec] improve (#421, #413) 
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
 2019-11-03 21:45:45 +01:00
Mike Fährmann
c01ff78467
[twitter] extend 'videos' option to force extraction with ytdl 
(closes #459)
 2019-11-01 22:06:07 +01:00
Mike Fährmann
a5e6252137
update deviantart client-id/-secret instructions (#451, #424) 2019-10-21 16:57:45 +02:00
Mike Fährmann
389d2d7e38
implement 'cookies-update' option (#445) 2019-10-19 15:23:55 +02:00
Mike Fährmann
1adafdd3d0
document cache file requirement for DeviantArt refresh tokens 2019-10-13 23:01:57 +02:00
Mike Fährmann
8af59a4bba
fix & update docs 
- update Requests links
- add example for --exec
- set '-dev' version
 2019-10-11 18:36:25 +02:00
Mike Fährmann
46ba173ded
[reddit] fix documentation inconsistencies (closes #429) 
- Require 'reddit.comments' to be a number and convert it to an
  integer to be extra sure
- Link to the README's OAuth section were appropriate
 2019-09-27 17:34:10 +02:00
Mike Fährmann
ee8b654464
[instagram] implement 'highlights' option (closes #329) 2019-09-21 23:38:20 +02:00
Mike Fährmann
f02a768b5c
[danbooru] add 'ugoira' option (#406) 
to choose between ZIP archives or converted video files
for Ugoira posts
 2019-08-31 21:51:52 +02:00
Mike Fährmann
c6c5cb1898
improve 'deviantart.quality' description 2019-08-30 18:41:18 +02:00
Mike Fährmann
7c09545f70
[downloader:ytdl] add 'outtmpl' option (#395) 2019-08-24 22:47:59 +02:00
Mike Fährmann
5a210991b6
Remove control characters from filesystem paths 
- add 'path-remove' option to specify the set of characters that
 should be removed
- rename 'restrict-filenames' to 'path-restrict'
- #348, #380
 2019-08-16 23:12:16 +02:00
Mike Fährmann
e661a1ce19
update links to Request's documentation 2019-08-15 13:25:06 +02:00
Mike Fährmann
8dc42bb178
implement 'enumerate' for 'extractor.skip' (#306) 
[ci skip]
 2019-08-08 18:37:54 +02:00
Mike Fährmann
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option 2019-08-08 16:54:20 +02:00
Mike Fährmann
3957d27d79
[deviantart] add 'quality' option (#369) 2019-08-03 11:40:35 +02:00
Mike Fährmann
0609afd1e4
update default cache directory ... again 
Use a 'gallery-dl' subdirectory in ~/.cache to adhere to how other
programs store their cached data, and call os.makedirs() so it also
works without an existing ~/.cache directory.
 2019-08-01 22:11:00 +02:00
Mike Fährmann
a8b60b2bd9
change default cache directory for unix systems 
Use either $XDG_CACHE_HOME or ~/.cache (if the former isn't set)
and store potentially sensitive cookies and tokens in a user's
home directory and not in the world-readable /tmp.
 2019-07-31 22:56:14 +02:00
Mark Henrick
923e1bb714 [docs] Fix inconsistency about which sites have optional authentication (#359) 
* [docs] Fix inconsistency about which sites have optional authentication

* update authentication docs
 2019-07-29 18:22:31 +02:00
Mike Fährmann
a90280f4e7
[postprocessor:zip] add 'mode' option (#355) 2019-07-29 16:51:26 +02:00
Mike Fährmann
547ea71463
[downloader.ytdl] add 'forward-cookies' option (#352) 
The "long" name is necessary because just calling it 'cookies' would
clash with how the lookup for '--cookies' is implemented.
 2019-07-24 21:19:11 +02:00
Mike Fährmann
b1bea8aaeb
add 'restrict-filenames' option (#348) 2019-07-23 17:41:24 +02:00
Mike Fährmann
21991acc49
add 'ciphers' option; update default User-Agent 2019-07-19 17:14:40 +02:00
Mike Fährmann
feb98cf196
[twitter] improve 'content' formatting; add option (#338) 
- include emoticons
- leave newlines intact
- remove pic.twitter.com/ links at the end
 2019-07-17 16:02:51 +02:00
Mike Fährmann
8d1ae9b715
[tumblr] enable date-min/-max/-format options (#337) 2019-07-17 14:36:41 +02:00
Mike Fährmann
16c582aaf9
implement 'mtime' post-processor (#332) 
This can set a file's modification time according to a UNIX timestamp
or a datetime object from its metadata.
 2019-07-14 22:39:17 +02:00
Mike Fährmann
62097284fe
add 'download' option (#220) 2019-07-14 18:48:18 +02:00
Mike Fährmann
69205df68d
allow '-1' for infinite retries (#300) 2019-06-30 23:10:47 +02:00
Mike Fährmann
f7b5c4c3e7
use values of 'retries' options correctly 
The RE-tries option now specifies exactly that: the maximum number a
failed HTTP request is re-tried. For example a value of 2 will now
correctly stop after 3 attempts: the initial one + 2 re-tries.

The maximum wait-time now also caps at 30min and increases exponentially
for both extractor.request() and downloader.http.download().
 2019-06-30 23:10:18 +02:00
Mike Fährmann
f2000a69aa
implement 'image-unique' and 'chapter-unique' options (#303) 
The default value for both is 'false', i.e. duplicate URLs are NOT
ignored.

The previous behavior was to always ignore duplicate URLs to make
'--abort-on-skip' work properly when new images where added to the
beginning of a collection while gallery-dl is running.
 2019-06-29 22:50:17 +02:00
Mike Fährmann
15e4ddf46d
implement custom logging formatter 
supports custom log message formats for each loglevel and, by
extension, custom ANSI codes and colors for errors and warnings

(#304)
 2019-06-21 20:17:58 +02:00
Mike Fährmann
db3f52881a
add 'mtime' option 2019-06-20 17:19:44 +02:00
Mike Fährmann
e05a96db5e
[deviantart] rename 'stash' to 'extra' (#302) 
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
 2019-06-10 21:05:25 +02:00
Mike Fährmann
c23bf263fe
[deviantart] rename 'external' to 'stash' (#302) 
restrict extracted URLs to ones from https://sta.sh/...
 2019-06-09 11:16:02 +02:00
Mike Fährmann
2fb85178da
[deviantart] add 'external' option (#302) 
If a description is available, this will extract URLs from the
description text and try to find Extractors for them.
 2019-06-06 18:53:50 +02:00
Mike Fährmann
d33f5a7423
[wallhaven] rewrite 
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
  IDs that can't be translated to the new ID system
- support direct links to wallpapers
 2019-05-31 14:53:02 +02:00
Mike Fährmann
f1893b2b5b
[deviantart] add 'folders' option (#276) 2019-05-30 17:28:12 +02:00
Mike Fährmann
4465a3ea68
[kissmanga][readcomiconline] add 'captcha' option (#279) 
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
 2019-05-27 22:24:48 +02:00
Mike Fährmann
9890bfdf23
[flickr] improve code and metadata 
- simplify pagination
- add more metadata and slightly change its structure
  - convert suitable values to int or list
  - move keys from ["photo"] to the base level
- proper video support (#246)
- rename method and variable names to better fit with other extractors
 2019-05-14 22:10:50 +02:00
Mike Fährmann
70be494161
[plurk] add a 'comments' options (#212) 2019-04-14 22:12:46 +02:00
Mike Fährmann
369eb66125
consistently use '*' for rst lists 2019-03-28 16:21:41 +01:00
Mike Fährmann
c7a6b0ed90
[deviantart] add 'metadata' option (#189) 2019-03-21 14:49:42 +01:00
Mike Fährmann
12482553bd
update links to youtube-dl 2019-03-13 22:03:02 +01:00
Mike Fährmann
25aaf55514
[smugmug] improve format selection (closes #183) 
- use original image if available
- support video formats
- remove user info for ImageExtractor (it is no longer possible to get
  image owner information for a single image)
 2019-03-10 15:20:35 +01:00
Mike Fährmann
ae353ed3b0
provide "extractor" and "job" keys for logging output 
This allows for stuff like "{extractor.url}" and "{extractor.category}"
in logging format strings.
Accessing 'extractor' and 'job' in any way will return "None" if those
fields aren't defined, i.e. in general logging messages.
 2019-02-14 11:09:58 +01:00
Mike Fährmann
134487ffb0
[exhentai] stop extraction if image limit is exceeded (#141) 
can be turned off with the `exhentai.limits' option
 2019-01-26 18:40:39 +01:00
Mike Fährmann
b50b30f1c9
[photobucket] download subalbums (#117) 2019-01-22 14:05:18 +01:00
Mike Fährmann
277b52101a
add 'category-transfer' option 
[ci skip]
 2019-01-19 20:28:19 +01:00
Mike Fährmann
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes (#135) 2019-01-17 21:18:12 +01:00
Mike Fährmann
1e4d351ad3
[danbooru] add authentication support (closes #151) 
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
 2019-01-09 14:19:07 +01:00
Mike Fährmann
06cbf5f9c4
implement 'chapter-reverse' option (#149) 
Setting it to `true` will start with the latest chapter instead of the
first one.
 2019-01-07 18:22:33 +01:00
Mike Fährmann
e95b24f056
[reactor] add wait-min & -max options (#148) 2019-01-07 18:04:47 +01:00
Mike Fährmann
28f9539551
[tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba
[tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
c47482b110
smaller changes, missing docs, etc. 
- make 'netrc' extractor-specific
- rename 'downloader.enable' to 'enabled'
- document 'downloader.ytdl.format'
- consistent newlines in configuration.rst
 2018-11-16 18:18:07 +01:00
Mike Fährmann
8ef84a6823
add option to enable/disable specific downloader modules 
... and write URLs with no (active) downloader to unsupported-file
 2018-11-13 18:06:36 +01:00
Mike Fährmann
2fa28a2609
update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
655549df7c
[downloader:ytdl] add several options 
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.

downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.

downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
 2018-10-20 18:26:49 +02:00
Mike Fährmann
6ed629f2b6
allow specifying number of skips before abort/exit (closes #115) 
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
 2018-10-13 17:21:55 +02:00
Mike Fährmann
d8492df51b
[deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
392a081657
update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
f8b3b00249
[twitter] add experimental 'videos' option (#99) 
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
 2018-09-30 21:52:23 +02:00
Mike Fährmann
09d2f3e5e7
[postprocessor:ugoira] improve libx264 detection 2018-09-24 13:12:54 +02:00
Mike Fährmann
55f5c87160
[postprocessor:ugoira] add 'libx264-prevent-odd' option 
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
 2018-09-21 20:01:14 +02:00
Mike Fährmann
7742cf8601
[tumblr] change 'reblogs' option (#103) 
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
 2018-09-10 15:40:25 +02:00
Mike Fährmann
a666ddd16b
[tumblr] extend 'reblogs' functionality (#103) 
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
 2018-09-07 19:13:52 +02:00
Mike Fährmann
6ecb36d88c
[postprocessor:ugoira] add 'ffmpeg-output' option 2018-08-31 17:37:35 +02:00
Mike Fährmann
10365394d7
[twitter] add support for user-timelines (closes #96) 
also adds a 'retweets' option to filter retweeted content
 2018-08-17 20:04:11 +02:00
Mike Fährmann
b164231bca
[sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
5f27cfeff6
[deviantart] remove prefer-public option 
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
 2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048
[deviantart] improve public-private token switching 
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
 2018-07-25 12:52:36 +02:00
Mike Fährmann
886d662582
[deviantart] add option to minimize refresh-token usage 
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
 2018-07-24 21:44:57 +02:00
Mike Fährmann
54a0d72dc8
[postprocessor:ugoira] improve frame rate handling 
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.

This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
 2018-07-23 21:33:20 +02:00
Mike Fährmann
ff436692bf
["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
00032b828c
[deviantart] add 'wait-min' option 2018-07-14 11:52:21 +02:00
Mike Fährmann
269dc2bbd5
[sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
1d43cbbf52
[gelbooru] tag-splitting for non-api mode 2018-07-06 15:24:19 +02:00
Mike Fährmann
4a57509392
generalize tag-splitting option (#92) 
- extend functionality to other booru sites:
  - http://behoimi.org/
  - https://konachan.com/
  - https://e621.net/
  - https://rule34.xxx/
  - https://safebooru.org/
  - https://yande.re/
 2018-07-04 12:21:16 +02:00
Mike Fährmann
87853538b4
[yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
0c1c4557dd
[postprocessor:ugoira] add option for two-pass encoding 2018-06-20 18:48:10 +02:00
Mike Fährmann
eb7a1f3b98
[pixiv] rework ugoira handling 
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
 2018-06-18 17:57:57 +02:00
Mike Fährmann
b84e71da91
add postprocessor documentation to configuration.rst 2018-06-16 15:46:41 +02:00
Mike Fährmann
12797e3b1f
update configuration.rst 
... again

- some more 'Path' references
- fixed some inconsistencies and errors
- added note about logging config for files
 2018-05-28 22:14:38 +02:00