Mike Fährmann
057668e17e
extend input-file format with per-URL config and comments
...
- see docstring of parse_inputfile() for details
- TODO: unittests, recursion (currently setting for example
{"extractor": {"key": "value"}} will override the whole "extractor"
branch instead of merging {"key": "value"} into the already existing
dictionary)
2018-02-07 21:47:27 +01:00
Mike Fährmann
5b3c34aa96
use generic chapter-extractor in more modules
2018-02-07 12:36:39 +01:00
Mike Fährmann
347baf7ac5
improve util.parse_range() performance
...
It is never going to actually matter, but using partition() instead
of split() is twice as fast.
2018-02-05 22:28:11 +01:00
Mike Fährmann
7b5ba69951
[hentaihere] ensure consistent extraction results
...
sometimes there is a random space before the next <a>
2018-02-05 15:26:25 +01:00
Mike Fährmann
377b78b3c9
[hentai2read] fix manga name extraction
2018-02-04 22:12:24 +01:00
Mike Fährmann
54c36a8a34
[subapics] add chapter- and manga-extractor ( #70 )
2018-02-04 22:02:10 +01:00
Mike Fährmann
2dd3aeeeae
[komikcast] add chapter- and manga-extractor ( #70 )
2018-02-04 22:02:10 +01:00
Mike Fährmann
7a412f5c32
implement generic manga-chapter extractor
2018-02-04 22:02:04 +01:00
Mike Fährmann
aa38eab2be
allow not-defined fields in format strings
...
... and replace them with "None", for now
2018-02-03 22:28:41 +01:00
Mike Fährmann
6a07e38366
implement extractor.add() and .add_module()
...
... as a public and non-hacky way to add (external) extractors to
gallery-dl's pool and make them available for extractor.find()
2018-02-02 00:01:41 +01:00
Mike Fährmann
c0dd922c13
add '--download-archive' cmdline option
...
… as well as a config file equivalent
2018-02-01 22:00:44 +01:00
Mike Fährmann
8c3b713362
rework DownloadJob.handle_url(); include archive functionality
...
todo:
"abort" and "exit" skip modes if download is skipped because of archive
2018-02-01 20:49:41 +01:00
Mike Fährmann
34873dbd90
set 'archive_fmt' values
...
These are going to be used to create an unique id for each image.
2018-02-01 15:30:49 +01:00
Mike Fährmann
a34cebc253
[luscious] jump to first image if cover does not link to it
2018-01-30 22:39:01 +01:00
Mike Fährmann
84a52a9256
add DownloadArchive class
2018-01-30 15:23:23 +01:00
Mike Fährmann
915807dd77
log HTTP errors as warnings
2018-01-29 21:55:46 +01:00
Mike Fährmann
db7f04dd97
emit log messages on download failure
...
and when retrying with fallback URLs
2018-01-28 18:44:10 +01:00
Mike Fährmann
d951f13e37
add config option for unsupported-URL file
...
for consistency's sake
2018-01-28 18:42:10 +01:00
Mike Fährmann
619387cbb1
update extractor unittest results
2018-01-28 18:29:05 +01:00
Mike Fährmann
364e335440
smaller adjustments and improvements
...
- requests and urllib3 version on 1 line
- close input file after reading from it
- use expand_path for unsupported-urls file
- remove unnecessary logging from options.py
2018-01-27 01:05:17 +01:00
Mike Fährmann
c9a9664a65
change --write-log behaviour
...
- log files now get truncated when opening them
(mode "w" instead of "a")
- log verbosity to file depends on -q/-v
(same as logging to stderr)
2018-01-27 00:51:40 +01:00
Mike Fährmann
97f4f15ec0
add option to write logging output to a file
...
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
2018-01-26 18:51:51 +01:00
Mike Fährmann
f94e3706a8
use logging module for error messages during downloads
2018-01-26 18:11:13 +01:00
Mike Fährmann
db91cf871c
document message identifiers
2018-01-23 21:38:30 +01:00
Mike Fährmann
0dd48d644f
update test results
...
nothing broke, but things got updated or changed
2018-01-23 21:38:29 +01:00
Mike Fährmann
1e93955170
[batoto] remove module
...
Site officially shut down on 2018.01.18
2018-01-23 21:37:32 +01:00
Mike Fährmann
27fce6f600
fix UrlJob behavior
2018-01-23 15:42:26 +01:00
Mike Fährmann
76509a6d3c
[imgur] update test results
2018-01-20 18:49:29 +01:00
Mike Fährmann
9fccd7b783
[tumblr] provide fallback URLs ( #64 )
...
Each image now produces 3 URLs:
- amazonaws.com _raw (or _1280 for older images)
- amazonaws.com _500
- media.tumblr.com (URL returned by API)
2018-01-19 23:12:15 +01:00
Mike Fährmann
b837420291
fix minor urllist issues
2018-01-19 22:54:15 +01:00
Mike Fährmann
9d69401391
initial support for multiple URLs per image
2018-01-17 22:08:19 +01:00
Mike Fährmann
6174a5c4ef
[download] adjust filename extension on filetype mismatch
...
(closes #63 )
2018-01-17 18:37:06 +01:00
Mike Fährmann
91ed147cef
[oauth] use custom key/secret values during oauth:…
2018-01-16 17:39:46 +01:00
Mike Fährmann
421a9740a3
[tumblr] add 'tumblr:' to force Tumblr extractor ( #71 )
2018-01-15 18:27:58 +01:00
Mike Fährmann
40d35c87bc
[paheal] add tag- and post-extractors ( closes #69 )
2018-01-15 16:39:05 +01:00
Mike Fährmann
cc0c2cca57
[reddit] add extractor for reddit-hosted images ( closes #68 )
2018-01-14 18:55:42 +01:00
Mike Fährmann
f10ffc0839
update extractor blacklist to also allow classes
2018-01-14 18:47:22 +01:00
Mike Fährmann
b6797032e3
release version 1.1.2
2018-01-12 15:09:18 +01:00
Mike Fährmann
35e09869d1
[mangapark] fix image URLs and use HTTPS
2018-01-12 14:59:49 +01:00
Mike Fährmann
9a049bdf51
[tumblr] add 'likes' extractor ( #65 )
2018-01-12 14:56:01 +01:00
Mike Fährmann
67d4462d26
[batoto] rudimentary Cloudflare bypass
2018-01-11 18:49:19 +01:00
Mike Fährmann
29d75fc3fa
[tumblr] add support for OAuth authentication ( #65 )
2018-01-11 14:11:37 +01:00
Mike Fährmann
4edb25346e
[slideshare] support mobile URLs ( closes #67 )
2018-01-10 14:15:00 +01:00
Mike Fährmann
e420a28bbc
fix cookie tests
2018-01-09 21:43:52 +01:00
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com
2018-01-09 17:54:37 +01:00
Mike Fährmann
75b2e84b6d
[tumblr] use s3.amazonaws.com for image URLs ( #64 )
2018-01-09 15:13:00 +01:00
Mike Fährmann
9a8e98f699
add gitter badge to README
2018-01-09 15:10:40 +01:00
Mike Fährmann
5b094328b5
[puremashiro] add chapter- and manga-extractor ( closes #66 )
...
Also adds support for region subtags in language codes (e.g. en-us)
2018-01-07 21:50:43 +01:00
Mike Fährmann
974e73bdbb
[booru] smaller code adjustments
2018-01-06 17:48:49 +01:00
Mike Fährmann
03b8a548cb
[tumblr] change reblogs default value to true ( #61 )
2018-01-06 15:52:08 +01:00
