Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Code Issues Releases Wiki Activity
4,430 Commits 6 Branches 164 Tags 16 MiB
2bb937014f
Commit Graph

573 Commits

Author SHA1 Message Date
Mike Fährmann
e3055d356c
release version 1.5.1 2018-08-17 13:21:36 +02:00
Mike Fährmann
f9ded38d89
[test:results] add support for "range" options in tests 2018-08-15 21:49:44 +02:00
Mike Fährmann
c9e6ccbd7c
[test:extractor] small fixes and improvements 2018-08-15 21:49:33 +02:00
Mike Fährmann
7f4e41c989
increase timeout during extractor tests 
cloudflare's 522 response takes longer than 30 seconds
 2018-08-10 16:51:05 +02:00
Mike Fährmann
b55e39d1ee
[mangadex] improve extraction 
- cache manga API results
- add artist, author and date fields to chapter metadata
- remove Manga-/ChapterExtractor inheritance
- minor code simplifications and improvements
 2018-08-10 16:50:07 +02:00
Mike Fährmann
2a9f3341a2
[behance] fix title extraction 2018-08-08 10:48:58 +02:00
Mike Fährmann
a86f2bfc80
[pinterest] update not-found redirects 2018-08-07 12:13:19 +02:00
Mike Fährmann
7442d2940c
release version 1.5.0 2018-08-03 17:50:27 +02:00
Mike Fährmann
b040ca0718
[rule34] small unit test fixes 2018-08-03 17:28:47 +02:00
Mike Fährmann
f3793660ef
update tests 2018-08-02 14:57:28 +02:00
Mike Fährmann
42a346413b
fix "re:" prefix for keyword tests 2018-08-02 14:48:51 +02:00
Mike Fährmann
e0dd8dff5f
implement L<maxlen>/<replacement>/ format option 
The L option allows for the contents of a format field to be replaced
with <replacement> if its length is greater than <maxlen>.

Example:
{f:L5/too long/} -> "foo"      (if "f" is "foo")
                 -> "too long" (if "f" is "foobar")

(#92) (#94)
 2018-07-29 13:52:07 +02:00
Mike Fährmann
bb89a1e6d7
[mangahere] use http:// 
invalid SSL cert for quite some time now
 2018-07-26 18:11:31 +02:00
Mike Fährmann
ce34d82cb4
fix skipping tests on 5xx status codes 2018-07-19 18:47:23 +02:00
Mike Fährmann
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor 2018-07-14 09:53:16 +02:00
Mike Fährmann
0ba93650e0
[8chan] replace unit test URL 
the other thread is no longer accessible
 2018-07-14 09:53:16 +02:00
Mike Fährmann
8fe9056b16
implement string slicing for format strings 
It is now possible to slice string (or list) values of format string
replacement fields with the same syntax as in regular Python code.

"{digits}"       -> "0123456789"
"{digits[2:-2]}" -> "234567"
"{digits[:5]}"   -> "01234"

The optional third parameter (step) has been left out to simplify things.
 2018-07-14 09:53:15 +02:00
Mike Fährmann
269dc2bbd5
[sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
764331823b
release version 1.4.2 2018-07-06 16:02:40 +02:00
Mike Fährmann
2eefaa99a3
[mangapark] support .net and .com mirrors 2018-07-05 14:45:05 +02:00
Mike Fährmann
188e956c4e
[imagefap] use HTTPS + update test results 2018-06-30 19:40:46 +02:00
Mike Fährmann
a699787d01
[deviantart] update URL patterns to new format 
DeviantArt changed its URL format from
https://<name>.deviantart.com/...
to
https://www.deviantart.com/<name>/...

With this change both formats will be supported.
 2018-06-28 20:21:59 +02:00
Mike Fährmann
0055fdd714
change OAuth test server 
DNS record for oauthbin.com expired
 2018-06-28 14:32:02 +02:00
Mike Fährmann
b8c97d2295
use 'extractor.request()' for more HTTP requests 2018-06-25 23:40:59 +02:00
Mike Fährmann
7a98cc9798
[smugmug] update tests 
My test account expired and all uploaded images got deleted.
 2018-06-22 15:04:31 +02:00
Mike Fährmann
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
a9e276bc37
reset delete-flag 
Since 'PathFormat' objects are being reused, setting `delete`
to True once caused all files downloaded after to be deleted as well.
 2018-06-20 18:12:59 +02:00
Mike Fährmann
6ac403c5d3
add postprocessor config example 2018-06-08 18:31:59 +02:00
Mike Fährmann
2403c405e3
Merge branch 'postprocessor' 2018-06-08 17:43:11 +02:00
Mike Fährmann
b344f2290f
fix downloader tests 2018-06-07 22:27:36 +02:00
Mike Fährmann
a47c6136cd
[simplyhentai] avoid redirects for all-pages.json (#89) 2018-06-01 22:06:34 +02:00
Mike Fährmann
ae9a37a528
implement text.split_html() 2018-05-27 15:00:41 +02:00
Mike Fährmann
0a1863fce3
[pixiv] respect more query parameters for user URLs 
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.

Respected parameters are
    'type': illust, manga, ugoira
    'tag' : any image tag (this was already supported)
    'p'   : the page to start on

*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
 2018-05-18 15:36:30 +02:00
Mike Fährmann
7f899bd5d8
Merge branch 'master' into 1.4-dev 2018-05-14 14:50:02 +02:00
Mike Fährmann
4cea886177
[imgur] allow longer album hashes 2018-05-13 11:21:51 +02:00
Mike Fährmann
e1e23165a0
[pinterest] catch JSON decode errors 2018-05-11 17:37:27 +02:00
Mike Fährmann
6a31ada9e3
re-implement OAuth1.0 code 
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
 2018-05-10 18:47:05 +02:00
Mike Fährmann
e2157f594e
[mangadex] fix manga extraction (closes #84) 
Chapter listings for manga now use
https://mangadex.org/manga/<id>/_/chapters/2/
as URL instead of
https://mangadex.org/manga/<id>/_//2/
 2018-05-06 17:43:50 +02:00
Mike Fährmann
69a5e6ddb3
Merge branch 'master' into 1.4-dev 2018-05-04 10:19:02 +02:00
Mike Fährmann
3fe653d940
fix test_results for empty sets 
{} is an empty dict and doesn't support set operations
 2018-04-29 22:43:37 +02:00
Mike Fährmann
d96b3474e5
[puremashiro] remove module 
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
 2018-04-28 14:24:20 +02:00
Mike Fährmann
b44a296404
[gomanga] remove module 
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
 2018-04-28 14:24:21 +02:00
Mike Fährmann
2395d870dd
[pinterest] unquote board and user names, better errors 2018-04-26 16:38:12 +02:00
Mike Fährmann
55d4d23860
[pinterest] use Pinterest's "Web" API (#83) 
no access tokens, no user credentials of any kind ...
 2018-04-24 22:28:10 +02:00
Mike Fährmann
f471161920
Merge branch 'master' into 1.4-dev 2018-04-21 12:15:40 +02:00
Mike Fährmann
cc36f88586
rename safe_int to parse_int; move parse_* to text module 2018-04-20 14:53:21 +02:00
Mike Fährmann
10cc59f3b5
fix extractor names 2018-04-18 18:12:57 +02:00
Mike Fährmann
b1325d4d2c
fix extractor docstrings 2018-04-18 18:03:43 +02:00
Mike Fährmann
df7e18399e
[luscious] fix image order 2018-04-17 17:32:21 +02:00
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions 
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
 2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest (#83) 
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
 2018-04-16 20:03:28 +02:00
Mike Fährmann
dbe250f7e5
[pinterest] update access_token (#83) 2018-04-16 09:46:45 +02:00
Mike Fährmann
5c487300ee
improve 'parse_query()' and add tests 
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
  just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
  created
 2018-04-15 19:05:29 +02:00
Mike Fährmann
4ffa94f634
remove 'shorten_path()' and 'shorten_filename()' 2018-04-15 18:44:13 +02:00
Mike Fährmann
27eab4e467
rewrite text tests and improve functions 
- test more edge cases
- consistently return an empty string for invalid arguments
- remove the ungreedy-flag in 'remove_html()'
 2018-04-15 18:13:46 +02:00
Mike Fährmann
e3f2bd4087
add tests for 'text.clean_xml()' and improve it 2018-04-14 22:07:01 +02:00
Mike Fährmann
6d8b191ea7
improve 'parse_query()' and add tests 
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
  just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
  created
 2018-04-13 19:21:32 +02:00
Mike Fährmann
51ea699083
add 'abort()' as function to filter expressions 
calling 'abort()' in a filter aborts the current extractor run
in a cleaner way than using something like 1/0, which
causes an error message to be printed
 2018-04-12 17:07:12 +02:00
Mike Fährmann
48a83a89e9
[loveisover] remove module 
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
 2018-04-09 16:05:15 +02:00
Mike Fährmann
564e12ca8f
replace 'imgyt' with 'imxto' 
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
 2018-04-09 15:53:20 +02:00
Mike Fährmann
d11fcf4804
smaller changes and fixes 
- fix the cloudflare challenge result if the last decimal places
  are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
  (accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
 2018-04-06 15:30:09 +02:00
Mike Fährmann
759ba26fb0
[luscious] proper image order for picture albums 
... and (try) to start with the first image instead of somewhere
in the middle of an album.
 2018-04-05 18:12:01 +02:00
Mike Fährmann
0381ae5318
replace error handlers for stdout and co. 
Python3.5 and lower throw an UnicodeEncodeError when trying to print
not-encodable characters when not using 'utf-8' as encoding.
Setting their error handlers to 'replace' should help.
 2018-04-04 17:30:42 +02:00
Mike Fährmann
64d7c85b55
[exhentai] improve metadata 
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
 2018-04-03 18:59:53 +02:00
Mike Fährmann
a112e3f2a0
[nijie] add doujin extractor 
adds support for "https://nijie.info/members_dojin.php?id=<artist_id>"
 2018-03-31 18:17:41 +02:00
Mike Fährmann
299ae24996
[test] add a few downloader tests 2018-03-25 15:10:25 +02:00
Mike Fährmann
dd314279fb
[test] add unit tests for extractor module functions 2018-03-25 11:49:42 +02:00
Mike Fährmann
f5c6a2d7f5
[nhentai] use API to get gallery info 2018-03-21 12:58:41 +01:00
Mike Fährmann
8ef790de12
update .travis.yml 
- restrict builds to master branch and release tags
- implement 'core' and 'results' test categories
 2018-03-19 17:57:32 +01:00
Mike Fährmann
557cb94f81
[deviantart] use proper exponential backoff on API errors 
... and use separate API credentials for unit tests.
 2018-03-15 16:01:42 +01:00
Mike Fährmann
b69cc94f0e
[util] implement bencode() 2018-03-14 13:17:34 +01:00
Mike Fährmann
4d74749496
[tests] rework filters for extractor tests 
CI incompatible tests will now only be skipped if tests are run in
a CI environment.
 2018-03-13 13:11:10 +01:00
Mike Fährmann
32bbd12f08
update extractor tests 2018-03-08 18:04:34 +01:00
Mike Fährmann
749fbbfa6c
[mangadex] add chapter- and manga-extractor 2018-03-05 18:37:21 +01:00
Mike Fährmann
5008e105ee
update archive IDs 
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.

specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
 2018-03-01 18:20:50 +01:00
Mike Fährmann
2fad0b1f1b
add 'U' conversion for format strings to unquote their content 
(#74)
 2018-02-25 21:57:59 +01:00
Mike Fährmann
8f338347b6
[imagehosts] cleanup 
removed
- chronos.to      - unable to resolve hostname
- coreimg.net     - same
- imgmaid.net     - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt    - cloudflare error 522, host down
- img4ever.net    - read timeout
 2018-02-23 01:05:42 +01:00
Mike Fährmann
e1e0668ca8
add option to set default replacement field value 
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
 2018-02-23 00:59:20 +01:00
Mike Fährmann
ac3da8115e
[util] don't add text: URLs to list of downloaded URLs 2018-02-20 18:14:27 +01:00
Mike Fährmann
89440382ad
[tumblr] use separate API key for unit tests 2018-02-19 16:54:37 +01:00
Mike Fährmann
b50bdbf3d7
change config specifiers in input file format 
Instead of a dictionary/object, input file options are now specified
by a 'key=value' pair starting with '-' for options only applying to
the next URL or '-G' for Global options applying to all following URLs.

See the docstring of parse_inputfile() for details.

Example option specifiers:

- filename = "{id}.{extension}"
- extractor.pixiv.user.directory = ["Pixiv Users", "{user[id]}"]
-spaces="are_optional"
-G keywords = {"global": "option"}
 2018-02-16 03:10:41 +01:00
Mike Fährmann
be3ea4425d
test archive-id creation and uniqueness 2018-02-12 23:02:09 +01:00
Mike Fährmann
b73b8b4f50
add OAuth unittests 2018-02-12 17:07:07 +01:00
Mike Fährmann
f5f2d29f56
[nijie] fix dojin extraction 
- correctly extract artist_id
- set extension to "jpg" if it was empty and let filetype checks do
  the rest
 2018-02-09 22:06:26 +01:00
Mike Fährmann
7a412f5c32
implement generic manga-chapter extractor 2018-02-04 22:02:04 +01:00
Mike Fährmann
aa38eab2be
allow not-defined fields in format strings 
... and replace them with "None", for now
 2018-02-03 22:28:41 +01:00
Mike Fährmann
619387cbb1
update extractor unittest results 2018-01-28 18:29:05 +01:00
Mike Fährmann
f94e3706a8
use logging module for error messages during downloads 2018-01-26 18:11:13 +01:00
Mike Fährmann
0dd48d644f
update test results 
nothing broke, but things got updated or changed
 2018-01-23 21:38:29 +01:00
Mike Fährmann
1e93955170
[batoto] remove module 
Site officially shut down on 2018.01.18
 2018-01-23 21:37:32 +01:00
Mike Fährmann
f10ffc0839
update extractor blacklist to also allow classes 2018-01-14 18:47:22 +01:00
Mike Fährmann
35e09869d1
[mangapark] fix image URLs and use HTTPS 2018-01-12 14:59:49 +01:00
Mike Fährmann
4edb25346e
[slideshare] support mobile URLs (closes #67) 2018-01-10 14:15:00 +01:00
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com 2018-01-09 17:54:37 +01:00
Mike Fährmann
1a70857a12
update extractor-unittest capabilities 
- "count" can now be a string defining a comparison in the form of
  '<operator> <value>', for example: '> 12' or '!= 1'. If its value
  is not a string, it is assumed to be a concrete integer as before.

- "keyword" can now be a dictionary defining tests for individual keys.
  These tests can either be a type, a concrete value or a regex
  starting with "re:". Dictionaries can be stacked inside each other.
  Optional keys can be indicated with a "?" before its name.

  For example:
      "keyword:" {
          "image_id": int,
          "gallery_id", 123,
          "name": "re:pattern",
          "user": {
              "id": 321,
          },
          "?optional": None,
      }
 2017-12-30 19:05:37 +01:00
Mike Fährmann
28cd78aae0
[kissmanga] extend chapter-string regex (closes #58) 2017-12-24 22:53:10 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication 
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
 2017-12-18 01:16:46 +01:00
Mike Fährmann
0a9a07a6e1
[slideshare] improve metadata; flake8 
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
 2017-12-13 21:16:49 +01:00
Mike Fährmann
291369eab2
various smaller changes/additions 2017-12-06 21:45:56 +01:00
Mike Fährmann
300346ecdf
[mangazuki] remove extractors 
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
 2017-12-04 13:36:04 +01:00
Mike Fährmann
93482a1f88
implement 'util.advance()' 2017-12-03 01:38:24 +01:00
Mike Fährmann
a718c6c6cd
implement 'util.parse_bytes()' 2017-12-02 01:24:49 +01:00
Mike Fährmann
214972bc9a
[gelbooru] use manual extraction 
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875)

This also adds an extractor for image-pools.
 2017-11-29 20:48:17 +01:00
Mike Fährmann
b14de6ffc2
[tumblr] small improvements 
- don't transform inline GIF URLs
- set 'type' parameter for API calls if there is only
  one post type selected
 2017-11-24 16:51:07 +01:00
Mike Fährmann
b8cdd42cab
[senmanga] fix extraction (again) 
this is basically a re-revert of 2ace5c7
 2017-11-18 17:23:32 +01:00
Mike Fährmann
6913eeaa40
[powermanga] replace manga extractor unit test 
My Hero Academia is gone
 2017-11-15 14:01:24 +01:00
Mike Fährmann
f72318e593
[seiga] support more than 200 images 
Due to API restrictions and/or missing knowledge about and
documentation of API usage, it was only possible to retrieve the
latest 200 images of a niconico seiga user with said API.

The new approach manually visits each HTML page and gets its
information from there.
 2017-11-13 20:46:24 +01:00
Mike Fährmann
2457b71633
skip tests on 5xx status codes 2017-11-12 20:51:12 +01:00
Mike Fährmann
305da540c3
[mangahere] fix metadata extraction 2017-11-03 14:54:46 +01:00
Mike Fährmann
035ef655f1
[imagefap] update unit tests 
old gallery/image has been deleted
 2017-10-27 12:22:16 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files (#29) 
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
 2017-10-26 00:16:48 +02:00
Mike Fährmann
27c026543f
re-enable download unit tests 2017-10-25 12:55:36 +02:00
Mike Fährmann
b0353aa02d
rewrite download modules (#29) 
- use '.part' files during file-download
- implement continuation of incomplete downloads
- check if file size matches the one reported by server
 2017-10-24 12:53:03 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve (fixes #44) 
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
 2017-10-14 23:01:33 +02:00
Mike Fährmann
75d3a1f72f
[deviantart] always download original images 
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.

[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
 2017-10-07 13:07:34 +02:00
Mike Fährmann
8e6a767109
[util] restructure formatter for better exception propagation 2017-10-06 17:10:35 +02:00
Mike Fährmann
0386503c80
fix (sub)category-transfer for DownloadJob instances (#41) 
... and extend "parent" parameters to TestJob- and DataJob-classes
as well.
 2017-10-06 15:38:35 +02:00
Mike Fährmann
41adb99e9c
[pawoo] fix extraction 
- changed access_token
- use account-search instead of general search
 2017-10-02 18:33:52 +02:00
Mike Fährmann
b319f4bab3
smaller code and text changes 2017-10-01 18:23:40 +02:00
Mike Fährmann
c1f0afe4c6
add custom string formatter class 2017-09-28 17:12:39 +02:00
Mike Fährmann
85a2b2ae59
[khinsider] fix extraction 2017-09-28 11:47:26 +02:00
Mike Fährmann
8e14714c2b
[imgspice] fix extraction 2017-09-26 21:04:48 +02:00
Mike Fährmann
a85f06d2d1
[foolslide] restructure; convert suitable values to int 2017-09-24 16:57:47 +02:00
Mike Fährmann
9fc1d0c901
implement and use 'util.safe_int()' 
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
 2017-09-24 15:59:25 +02:00
Mike Fährmann
a9e7145651
[hbrowse] extract hmanga metadata & general maintenance 2017-09-20 16:25:25 +02:00
Mike Fährmann
84d4450410
[fallenangels] extract manga metadata 2017-09-15 20:51:40 +02:00
Mike Fährmann
f32b1a0292
[imgyt] fix extraction 2017-09-14 15:04:32 +02:00
Mike Fährmann
31cd5b1c1d
[luscious] detect high-load responses 2017-09-12 15:46:21 +02:00
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords' 2017-09-09 18:48:28 +02:00
Mike Fährmann
9b21d3f13c
add '--filter' command-line option 
This allows for image filtering via Python expressions by the same
metadata that is also used to build filenames (--list-keywords).

The usually shunned eval() function is used to evaluate
filter-expressions, but it seemed quite appropriate in this case and
shouldn't introduce any new security issues, as any attacker that could do
> gallery-dl --filter "delete-everything()" ...
could as well do
> python -c "delete-everything()"
 2017-09-08 17:52:00 +02:00
Mike Fährmann
31731cbefe
update unittests for util.py 2017-09-06 17:57:19 +02:00
Mike Fährmann
f98e3e8002
[luscious] fix tag extraction 2017-09-01 16:29:52 +02:00
Mike Fährmann
65997d835b
replace popular/ranking tests with older ones 
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
 2017-08-31 15:09:18 +02:00
Mike Fährmann
c0755a4d5e
[exhentai] revert login-method to its old version (#37) 
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
 2017-08-29 22:10:38 +02:00
Mike Fährmann
3ee39ffd93
[exhentai] update login procedure (#37) 
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
 2017-08-28 21:03:32 +02:00
Mike Fährmann
07214f4007
[booru] place subcategories into base classes 2017-08-26 22:27:55 +02:00
Mike Fährmann
47bcf53ec1
implement support for additional unit test result types 
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
 2017-08-25 22:01:14 +02:00
Mike Fährmann
c7ec103e15
[batoto] fix extraction of chapter URLs 2017-08-25 16:34:42 +02:00
Mike Fährmann
f7cdfd4c25
add a simplified version of 'parse_qs' 
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
 2017-08-24 20:55:58 +02:00
Mike Fährmann
d70c66c516
fix "text:" downloader 2017-08-16 12:11:47 +02:00
Mike Fährmann
9bf9d64ad8
update unittests for util.py 2017-08-13 14:31:22 +02:00
Mike Fährmann
d74a635e41
[util] update 'default' values and improve test coverage 
for 'code_to_language()' and 'language_to_code()'
 2017-08-08 19:22:04 +02:00
Mike Fährmann
abd7c559cd
[yonkouprod] remove module 
Every manga chapter on this site has been removed.
 2017-08-07 18:32:14 +02:00
Mike Fährmann
852e7acd31
[twitter] ignore "Promoted Tweets" 2017-08-06 13:43:08 +02:00
Mike Fährmann
6950708e52
[hentaicdn] use HTTPS 2017-08-02 18:31:21 +02:00
Mike Fährmann
493bd235cf
workaround for missing 'assert_called_once' method 
this method was introduced in Python 3.6, but calling it still
works (i.e. it doesn't cause the test to fail) on Python 3.3/3.4
 2017-07-26 10:33:15 +02:00
Mike Fährmann
7aa9fa796a
code cleanup and fixes 2017-07-25 14:59:41 +02:00
Mike Fährmann
f08af03845
Merge branch 'cookies' 2017-07-25 14:04:53 +02:00
Mike Fährmann
6db93a8b9e
add cookie tests 2017-07-25 14:02:53 +02:00
Mike Fährmann
d7cb3c668a
update supportedsites.rst 2017-07-24 10:50:40 +02:00
Mike Fährmann
a13eb6010f
[fallenangels] fix extraction of chapter URLs 2017-07-20 14:58:47 +02:00
Mike Fährmann
1cb1d2e0a3
[mangazuki] add chapter extractor 2017-07-19 17:20:03 +02:00
Mike Fährmann
3460dc8950
update gallery-dl.conf 2017-07-14 08:23:11 +02:00
Mike Fährmann
af9bd17b19
[deviantart] adjust default paths 
- user.deviantart.com/(gallery|favourites|journal)/ images go into
  * <user>/
  * <user>/Favourites/
  * <user>/Journal/
  (having an extra "Gallery" folder for a user's gallery-images seems
   a bit too much if these are all you want to download, which is
   probably the default use-case)

- single "deviations" (user.deviantart.com/(art|journal)/name-123) go
  into their owner's directory:
  * <user>/
  (putting them into their own directory seems weird in practice)
 2017-07-10 18:54:10 +02:00
Mike Fährmann
8c16cbe7ea
fix tests 2017-07-04 19:31:32 +02:00
Mike Fährmann
ce55ec6490
enable extractor tests without filters 
$ python test_extractors.py all
 2017-07-02 08:20:04 +02:00
Mike Fährmann
44d98e562b
[pixiv] support pixiv.me URLs (#23) 2017-06-25 20:21:01 +02:00
Mike Fährmann
e68af4febe
[flickr] add 'width-max' option (#16) 
This option allows for simple format selection by
specifying a maximum image width.
 2017-06-18 22:20:15 +02:00
Mike Fährmann
9759fe8c6b
allow 'only_matching' tests 2017-06-14 08:43:05 +02:00
Mike Fährmann
05ed95e5b0
[flickr] add search extractor 2017-06-13 08:01:32 +02:00
Mike Fährmann
c921b4f32a
code cleanup and fixing tests 2017-06-02 09:10:58 +02:00
Mike Fährmann
b6fffa9e26
[directlink] update filename format and metadata 2017-05-30 17:33:09 +02:00
Mike Fährmann
48ccee2505
[gfycat] add image extractor 2017-05-28 17:09:54 +02:00
Mike Fährmann
691c4dd709
support direct image links 2017-05-24 12:51:18 +02:00
Mike Fährmann
398506da45
update release script 2017-05-22 08:47:58 +02:00
Mike Fährmann
2974d782a3
[yomanga] remove module 
site has been shut down
 2017-05-20 11:18:44 +02:00
Mike Fährmann
a90c6acc9c
code cleanup + fixes 2017-05-18 15:18:18 +02:00
Mike Fährmann
4c88c0d496
rework the output format for --list-keywords 2017-05-15 18:30:47 +02:00
Mike Fährmann
107d29ad8a
improve handling of text:... URLs 
- don't require // after the colon
- open output files in text mode
 2017-05-12 14:10:25 +02:00
Mike Fährmann
9f1c83297f
[pinterest] allow URLs with any TLD 2017-05-08 15:08:39 +02:00
Mike Fährmann
ef90a2de2f
implement the "exit" option for the "skip" config-key 2017-05-05 15:49:58 +02:00
Mike Fährmann
fc9223c072
add '--abort-on-skip' option and ability to control skip behavior 
the 'skip' config option controls skipping behavior:
    true    - skip download if file already exist (default)
    false   - download and overwrite files even if it exists
    "abort" - abort extractor run if a download would be skipped
              (same as '--abort-on-skip')
 2017-05-03 15:26:04 +02:00
Mike Fährmann
d948ba1322
[readcomics] remove module 
- site has been unavailable for two weeks
- (#12)
 2017-05-01 11:44:12 +02:00
Mike Fährmann
244ab75cad
[kissmanga] update AES key retrieval 2017-04-21 20:36:47 +02:00
Mike Fährmann
13dc5d72bc
update some extractors to use https 2017-04-20 13:32:40 +02:00
Mike Fährmann
f4aa452bd1
update unit test results 2017-04-14 14:40:36 +02:00
Mike Fährmann
fece09d326
[fallenangels] update to new domain and site-layout 2017-04-09 11:37:21 +02:00
Mike Fährmann
cf79a47b59
update unit tests 2017-04-08 11:02:32 +02:00
Mike Fährmann
841fd50242
move code into util.py 2017-03-28 13:12:44 +02:00
Mike Fährmann
e3212dd98f
fix some smaller stuff 
- remove support for old windows config paths
- catch exception if cache-database can't be opened
- fix username/password settings for unit tests
- rename variable 'max_tries' to 'retries'
 2017-03-27 14:30:32 +02:00
Mike Fährmann
e4b3077168
improve config module 
- speed improvements, especially in the 'interpolate' function
- 'interpolate' now prioritizes base-level values if they exist
  - "username" is chosen before "extractor.<category>.username"
  - -u/--username & co can now override config-file values
 2017-03-27 11:59:27 +02:00
Mike Fährmann
f2ef49563b
fix argument order for python33 and 34 2017-03-23 16:51:42 +01:00
Mike Fährmann
0257d3e7ac
[mangamint] remove extractors - site is down 2017-03-20 13:38:54 +01:00
Mike Fährmann
66eb3d3488
fix tests 2017-03-14 08:51:58 +01:00
Mike Fährmann
872b8aed97
[4chan] update test 2017-03-03 15:50:10 +01:00
Mike Fährmann
2fa575b273
restore exception-testing to its old form 2017-02-27 23:05:08 +01:00
Mike Fährmann
40be4933b8
fix exception based tests 2017-02-26 02:06:56 +01:00
Mike Fährmann
6208d9dd79
implement '--images' and '--chapters' options 
- the former '--items' has been renamed to '--chapters'
- #6
 2017-02-23 21:51:29 +01:00
Mike Fährmann
e87e6fbc67
change some config keys 
directory_fmt     -> directory
filename_fmt      -> filename
download-original -> original
 2017-02-21 22:11:02 +01:00
Mike Fährmann
96baea0959
[hentaibox] remove extractor 2017-02-18 12:34:10 +01:00
Mike Fährmann
9a08f8a097
improved foolslide-based extractors 
- this includes dokireader, fallenangels, jaiminisbox, powermanga,
  sensescans, worldthree, yonkouprod, gomanga, yomanga
- added 'chapter_string', 'chapter_id', 'chapter_minor' and 'count'
  keywords
- changed the 'chapter' keyword to always be just a number
- changed the default directory format
 2017-02-16 23:42:30 +01:00
Mike Fährmann
bad623751f
[hentaihere] transition to https 2017-02-10 21:28:17 +01:00
Mike Fährmann
331b413c60
update extractor test script 2017-02-10 21:22:57 +01:00
Mike Fährmann
4f123b8513
code adjustments according to pep8 2017-01-30 19:40:15 +01:00
Mike Fährmann
9538a873cb
move create_test_data script 2017-01-19 01:35:59 +01:00
Mike Fährmann
434ef4d5a2
[mangafox] raise proper exception if chapter is not available 2017-01-14 22:11:47 +01:00
Mike Fährmann
cc0b4f2661
[yomanga] add chapter extractor 2017-01-13 00:03:12 +01:00
Mike Fährmann
ad4b02508f
trying to understand travis-ci unit test failures 
- added some debug output via logging module
- unit tests work on my machine (tm)
 2017-01-12 22:35:42 +01:00
Mike Fährmann
72d3ca0bf9
[deviantart] use API for user-extractor 2017-01-12 21:08:49 +01:00
Mike Fährmann
989820d719
workaround for python 3.6s behaviour with BEGIN EXCLUSIVE in sqlite 2017-01-11 00:34:32 +01:00
Mike Fährmann
00074a71d7
several changes to make travis build work 
- fixed html.unescape not being available on Python3.3
- removed inconsistent test result
- added username/password pairs for authenticating extractors
 2017-01-10 13:41:00 +01:00
Mike Fährmann
a1fb46d984
make extractor unittest discoverable 2017-01-09 12:27:20 +01:00
Mike Fährmann
2c9bc50c23
support expecting exceptions in tests 2016-12-30 01:41:17 +01:00
Mike Fährmann
4e75e923cd
sanatize extractor test output 2016-12-09 01:15:40 +01:00
Mike Fährmann
d1f90bb424
add method to re-create existing test-data 2016-12-09 00:25:52 +01:00
Mike Fährmann
57a12f5be0
fix module-names for extractor tests 2016-07-23 17:53:19 +02:00
Mike Fährmann
b0c333b799
rewrite cache module 2016-04-20 08:40:41 +02:00
Mike Fährmann
871f4c8a48
initialize cache-module before running tests 2016-03-08 18:01:35 +01:00
Mike Fährmann
b3aab5745b
dynamically create extractor testcases 2016-02-18 15:53:53 +01:00
Mike Fährmann
2b8977cb2e
script to easily create testdata 2015-12-22 01:49:25 +01:00
Mike Fährmann
565ea042c9
update HashJob to generate hashes for downloaded content 2015-12-21 22:49:04 +01:00
Mike Fährmann
5a8541afa5
remove 'unstable' tests 2015-12-15 23:45:40 +01:00
Mike Fährmann
0961ece989
make test-parameters optional 2015-12-13 03:56:29 +01:00
Mike Fährmann
5304e5beef
testing environment for extractor results 2015-12-12 15:58:07 +01:00
Mike Fährmann
eeae580781
more tests 2015-11-28 01:47:17 +01:00
Mike Fährmann
ca523b9f64
add helper method to text module 2015-11-16 03:46:43 +01:00
Mike Fährmann
506242740f
fix capitalization issues 2015-11-02 14:58:26 +01:00
Mike Fährmann
b24bb1da7d
new module for simple language code to name mapping 2015-11-02 00:18:26 +01:00
Mike Fährmann
cba4b91b14 add tests 2015-10-31 00:39:10 +01:00
Mike Fährmann
bea33ae9cb implement config.setdefault 2015-10-07 00:58:43 +02:00
Mike Fährmann
2026223ed1 change argument format for config-calls 2015-10-05 12:42:42 +02:00
Mike Fährmann
7ac106096f add tests for config-module 2015-10-03 20:24:35 +02:00
Mike Fährmann
2962bf36f6 add tests for text-module 2015-10-03 14:51:13 +02:00