1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-25 20:22:36 +01:00
Commit Graph

1135 Commits

Author SHA1 Message Date
Mike Fährmann
cc36f88586
rename safe_int to parse_int; move parse_* to text module 2018-04-20 14:53:21 +02:00
Mike Fährmann
ff643793bd
improve and document cloudflare bypass code 2018-04-19 21:32:10 +02:00
Mike Fährmann
728c64a3fb
[tumblr] rename 'offset' to 'num and adjust formats
Trying to somehow emulate Tumblr filenames is a bad idea ...
2018-04-15 18:58:32 +02:00
Mike Fährmann
4ffa94f634
remove 'shorten_path()' and 'shorten_filename()' 2018-04-15 18:44:13 +02:00
Mike Fährmann
27eab4e467
rewrite text tests and improve functions
- test more edge cases
- consistently return an empty string for invalid arguments
- remove the ungreedy-flag in 'remove_html()'
2018-04-15 18:13:46 +02:00
Mike Fährmann
e3f2bd4087
add tests for 'text.clean_xml()' and improve it 2018-04-14 22:07:01 +02:00
Mike Fährmann
6d8b191ea7
improve 'parse_query()' and add tests
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
  just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
  created
2018-04-13 19:21:32 +02:00
Mike Fährmann
51ea699083
add 'abort()' as function to filter expressions
calling 'abort()' in a filter aborts the current extractor run
in a cleaner way than using something like 1/0, which
causes an error message to be printed
2018-04-12 17:07:12 +02:00
Mike Fährmann
6bd857a319
[tumblr] handle rate limits / 429 errors
- wait for the hourly limit to reset
- abort upon exceeding the daily limit (it doesn't seem useful to
  potentially wait for several hours)
2018-04-12 16:25:20 +02:00
Mike Fährmann
7073ab7707
[komikcast] update regex to only match manga pages
The 'readerarea' section now includes some (shady) external
Javascript file, which got matched as well.
2018-04-11 15:48:17 +02:00
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
This reverts commit 4a26ae32df.
2018-04-09 16:08:08 +02:00
Mike Fährmann
48a83a89e9
[loveisover] remove module
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
2018-04-09 16:05:15 +02:00
Mike Fährmann
564e12ca8f
replace 'imgyt' with 'imxto'
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
2018-04-09 15:53:20 +02:00
Mike Fährmann
1b80fa82a9
[imgur] update URL pattern and tests 2018-04-08 21:06:21 +02:00
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
6b72be8ee6
[tumblr] add 'hash' keyword
'hash' is the middle part of the filename in a tumblr image URL.
For example an image with '.../tumblr_p6tgemp1NZ1wgha4yo1_250.png' as
its URL would have 'p6tgemp1NZ1wgha4yo1' as hash.
2018-04-07 15:54:30 +02:00
Mike Fährmann
ffc0c67701
release version 1.3.3 2018-04-06 15:45:45 +02:00
Mike Fährmann
d11fcf4804
smaller changes and fixes
- fix the cloudflare challenge result if the last decimal places
  are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
  (accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
2018-04-06 15:30:09 +02:00
Mike Fährmann
f6c95dccf9
[cloudflare] fix bypass procedure
Cloudflare challenges, at least for kissmanga and readcomiconline,
now use slightly different Javascript expressions.

Instead of a single value per expression, they now have a numerator
and a denominator of a fractional value, which in the end gets
truncated to 10 decimal places.
2018-04-05 20:28:04 +02:00
Mike Fährmann
759ba26fb0
[luscious] proper image order for picture albums
... and (try) to start with the first image instead of somewhere
in the middle of an album.
2018-04-05 18:12:01 +02:00
Mike Fährmann
68e9fbee16
[tumblr] check all 4 keys/secrets before using OAuth
it was possible to cause a crash by setting api-key or -secret to null.
(this commit also slightly improves the blog-cache implementation)
2018-04-05 15:42:23 +02:00
Mike Fährmann
4810d446bb
remove the obsolete safeprint() and error() functions
- safeprint() was used to print values which might have caused a
  UnicodeEncodeError, but that is no longer necessary (0381ae5)
- errors are now handled via logging output (f94e370)
2018-04-05 13:10:33 +02:00
Mike Fährmann
0381ae5318
replace error handlers for stdout and co.
Python3.5 and lower throw an UnicodeEncodeError when trying to print
not-encodable characters when not using 'utf-8' as encoding.
Setting their error handlers to 'replace' should help.
2018-04-04 17:30:42 +02:00
Mike Fährmann
f8168c693e
[tumblr] avoid calls to '/blog/.../info'
The same information returned by the 'blog/.../info' API endpoint
is also included in the result of every 'blog/.../posts' call.
2018-04-04 14:15:24 +02:00
Mike Fährmann
64d7c85b55
[exhentai] improve metadata
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
2018-04-03 18:59:53 +02:00
Mike Fährmann
64b22e0fc1
[pawoo] update URL pattern
adds support for 'https://pawoo.net/@.../media'
2018-04-02 13:00:59 +02:00
Mike Fährmann
7b562907c3
[nijie] add favorites extractor
adds support for 'https://nijie.info/user_like_illust_view.php?id=...'
2018-03-31 18:54:25 +02:00
Mike Fährmann
445db75955
[nijie] improve extraction and metadata
- add 'title' and 'description'
- split 'artist_id' into 'user_id' and 'artist_id'
  - 'user_id' is the ID of the user from which the image entry
    originates from
  - 'artist_id' is the ID of the actual image artist
- improve pagination and URL patterns
2018-03-31 18:48:41 +02:00
Mike Fährmann
a112e3f2a0
[nijie] add doujin extractor
adds support for "https://nijie.info/members_dojin.php?id=<artist_id>"
2018-03-31 18:17:41 +02:00
Mike Fährmann
f39153b6e9
[nhentai] add extractor for search results 2018-03-28 17:21:44 +02:00
Mike Fährmann
52d41c41e7
[exhentai] add extractor for favorited galleries 2018-03-27 18:58:42 +02:00
Mike Fährmann
63cc2599c4
[exhentai] add extractor for search results 2018-03-27 16:50:47 +02:00
Mike Fährmann
d1c91a1f2b
[mangadex] fix manga-page extraction 2018-03-25 17:22:12 +02:00
Mike Fährmann
299ae24996
[test] add a few downloader tests 2018-03-25 15:10:25 +02:00
Mike Fährmann
dd314279fb
[test] add unit tests for extractor module functions 2018-03-25 11:49:42 +02:00
Mike Fährmann
a993d0ea90
release version 1.3.2 2018-03-23 15:27:56 +01:00
Mike Fährmann
e7525b1b0e
[artstation] add challenge extractor (#80) 2018-03-23 15:06:09 +01:00
Mike Fährmann
3f2dd6b6f8
avoid double path-separators
(#74)
2018-03-22 10:24:59 +01:00
Mike Fährmann
b25ae30a95
reorder options in gallery-dl.conf 2018-03-22 10:05:53 +01:00
Mike Fährmann
f5c6a2d7f5
[nhentai] use API to get gallery info 2018-03-21 12:58:41 +01:00
Mike Fährmann
b2ba2b821d
[hitomi] fix image URLs and improve metadata
- use '?a.hitomi.la' as subdomain depending in gallery-id
- add 'characters', 'tags' and 'date' information
- support multiple entires per metadata-value
- rename 'num' to 'page'
2018-03-20 18:09:42 +01:00
Mike Fährmann
d75dc71394
fix travis tests 2018-03-19 18:07:19 +01:00
Mike Fährmann
8ef790de12
update .travis.yml
- restrict builds to master branch and release tags
- implement 'core' and 'results' test categories
2018-03-19 17:57:32 +01:00
Mike Fährmann
3905474805
[booru] call update_page() with correct dict (closes #82) 2018-03-19 11:33:19 +01:00
Mike Fährmann
44c267e362
[artstation] add search extractor (#80) 2018-03-17 19:04:37 +01:00
Mike Fährmann
40ca562d7b
[artstation] add album extractor (#80) 2018-03-17 17:36:31 +01:00
Mike Fährmann
7121eeae8b
check supportedsites.rst in release script 2018-03-17 15:35:38 +01:00
HRXN
241eda8c3d Update supportedsites.rst (#81)
Add ArtStation to Supported Sites.
2018-03-16 13:40:59 +01:00
Mike Fährmann
c59f9b71f1
release version 1.3.1 2018-03-16 12:29:02 +01:00
Mike Fährmann
f0e3861ec7
update configuration.rst
- move `base-directory` entry to extractor options
  (it should've been there since v1.0.0 / commit e3bfb83)
- rename "General" to "Miscellaneous" and move it down a bit, since it
  has only two options left
- mention the ability to use a list for tumblr.posts
- replace any appearance of bato.to
2018-03-16 12:00:55 +01:00