Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 18:53:21 +01:00
Code Issues Releases Wiki Activity
5,383 Commits 6 Branches 164 Tags 15 MiB
225d849139
Commit Graph

827 Commits

Author SHA1 Message Date
Mike Fährmann
10cc59f3b5
fix extractor names 2018-04-18 18:12:57 +02:00
Mike Fährmann
b1325d4d2c
fix extractor docstrings 2018-04-18 18:03:43 +02:00
Mike Fährmann
df7e18399e
[luscious] fix image order 2018-04-17 17:32:21 +02:00
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions 
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
 2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest (#83) 
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
 2018-04-16 20:03:28 +02:00
Mike Fährmann
dbe250f7e5
[pinterest] update access_token (#83) 2018-04-16 09:46:45 +02:00
Mike Fährmann
5c487300ee
improve 'parse_query()' and add tests 
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
  just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
  created
 2018-04-15 19:05:29 +02:00
Mike Fährmann
4ffa94f634
remove 'shorten_path()' and 'shorten_filename()' 2018-04-15 18:44:13 +02:00
Mike Fährmann
27eab4e467
rewrite text tests and improve functions 
- test more edge cases
- consistently return an empty string for invalid arguments
- remove the ungreedy-flag in 'remove_html()'
 2018-04-15 18:13:46 +02:00
Mike Fährmann
e3f2bd4087
add tests for 'text.clean_xml()' and improve it 2018-04-14 22:07:01 +02:00
Mike Fährmann
6d8b191ea7
improve 'parse_query()' and add tests 
- another irrelevant micro-optimization !
- use urllib.parse.parse_qsl directly instead of parse_qs, which
  just packs the results of parse_qsl in a different data structure
- reduced memory requirements since no additional dict and lists are
  created
 2018-04-13 19:21:32 +02:00
Mike Fährmann
51ea699083
add 'abort()' as function to filter expressions 
calling 'abort()' in a filter aborts the current extractor run
in a cleaner way than using something like 1/0, which
causes an error message to be printed
 2018-04-12 17:07:12 +02:00
Mike Fährmann
48a83a89e9
[loveisover] remove module 
archive.loveisover.me was shut down on 2018-03-29;
https://www.archiveteam.org/index.php?title=4chan#archive.loveisover.me
 2018-04-09 16:05:15 +02:00
Mike Fährmann
564e12ca8f
replace 'imgyt' with 'imxto' 
https://img.yt/ wasn't available for a couple of days, but has now
re-emerged as https://imx.to/ with a new web-interface.
Links to older images still work (see tests).
 2018-04-09 15:53:20 +02:00
Mike Fährmann
d11fcf4804
smaller changes and fixes 
- fix the cloudflare challenge result if the last decimal places
  are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
  (accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
 2018-04-06 15:30:09 +02:00
Mike Fährmann
759ba26fb0
[luscious] proper image order for picture albums 
... and (try) to start with the first image instead of somewhere
in the middle of an album.
 2018-04-05 18:12:01 +02:00
Mike Fährmann
0381ae5318
replace error handlers for stdout and co. 
Python3.5 and lower throw an UnicodeEncodeError when trying to print
not-encodable characters when not using 'utf-8' as encoding.
Setting their error handlers to 'replace' should help.
 2018-04-04 17:30:42 +02:00
Mike Fährmann
64d7c85b55
[exhentai] improve metadata 
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
 2018-04-03 18:59:53 +02:00
Mike Fährmann
a112e3f2a0
[nijie] add doujin extractor 
adds support for "https://nijie.info/members_dojin.php?id=<artist_id>"
 2018-03-31 18:17:41 +02:00
Mike Fährmann
299ae24996
[test] add a few downloader tests 2018-03-25 15:10:25 +02:00
Mike Fährmann
dd314279fb
[test] add unit tests for extractor module functions 2018-03-25 11:49:42 +02:00
Mike Fährmann
f5c6a2d7f5
[nhentai] use API to get gallery info 2018-03-21 12:58:41 +01:00
Mike Fährmann
8ef790de12
update .travis.yml 
- restrict builds to master branch and release tags
- implement 'core' and 'results' test categories
 2018-03-19 17:57:32 +01:00
Mike Fährmann
557cb94f81
[deviantart] use proper exponential backoff on API errors 
... and use separate API credentials for unit tests.
 2018-03-15 16:01:42 +01:00
Mike Fährmann
b69cc94f0e
[util] implement bencode() 2018-03-14 13:17:34 +01:00
Mike Fährmann
4d74749496
[tests] rework filters for extractor tests 
CI incompatible tests will now only be skipped if tests are run in
a CI environment.
 2018-03-13 13:11:10 +01:00
Mike Fährmann
32bbd12f08
update extractor tests 2018-03-08 18:04:34 +01:00
Mike Fährmann
749fbbfa6c
[mangadex] add chapter- and manga-extractor 2018-03-05 18:37:21 +01:00
Mike Fährmann
5008e105ee
update archive IDs 
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.

specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
 2018-03-01 18:20:50 +01:00
Mike Fährmann
2fad0b1f1b
add 'U' conversion for format strings to unquote their content 
(#74)
 2018-02-25 21:57:59 +01:00
Mike Fährmann
8f338347b6
[imagehosts] cleanup 
removed
- chronos.to      - unable to resolve hostname
- coreimg.net     - same
- imgmaid.net     - same
- hosturimage.com - everything returns 404
- imageontime.org - redirects to some shady site
- imgupload.yt    - cloudflare error 522, host down
- img4ever.net    - read timeout
 2018-02-23 01:05:42 +01:00
Mike Fährmann
e1e0668ca8
add option to set default replacement field value 
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
 2018-02-23 00:59:20 +01:00
Mike Fährmann
ac3da8115e
[util] don't add text: URLs to list of downloaded URLs 2018-02-20 18:14:27 +01:00
Mike Fährmann
89440382ad
[tumblr] use separate API key for unit tests 2018-02-19 16:54:37 +01:00
Mike Fährmann
b50bdbf3d7
change config specifiers in input file format 
Instead of a dictionary/object, input file options are now specified
by a 'key=value' pair starting with '-' for options only applying to
the next URL or '-G' for Global options applying to all following URLs.

See the docstring of parse_inputfile() for details.

Example option specifiers:

- filename = "{id}.{extension}"
- extractor.pixiv.user.directory = ["Pixiv Users", "{user[id]}"]
-spaces="are_optional"
-G keywords = {"global": "option"}
 2018-02-16 03:10:41 +01:00
Mike Fährmann
be3ea4425d
test archive-id creation and uniqueness 2018-02-12 23:02:09 +01:00
Mike Fährmann
b73b8b4f50
add OAuth unittests 2018-02-12 17:07:07 +01:00
Mike Fährmann
f5f2d29f56
[nijie] fix dojin extraction 
- correctly extract artist_id
- set extension to "jpg" if it was empty and let filetype checks do
  the rest
 2018-02-09 22:06:26 +01:00
Mike Fährmann
7a412f5c32
implement generic manga-chapter extractor 2018-02-04 22:02:04 +01:00
Mike Fährmann
aa38eab2be
allow not-defined fields in format strings 
... and replace them with "None", for now
 2018-02-03 22:28:41 +01:00
Mike Fährmann
619387cbb1
update extractor unittest results 2018-01-28 18:29:05 +01:00
Mike Fährmann
f94e3706a8
use logging module for error messages during downloads 2018-01-26 18:11:13 +01:00
Mike Fährmann
0dd48d644f
update test results 
nothing broke, but things got updated or changed
 2018-01-23 21:38:29 +01:00
Mike Fährmann
1e93955170
[batoto] remove module 
Site officially shut down on 2018.01.18
 2018-01-23 21:37:32 +01:00
Mike Fährmann
f10ffc0839
update extractor blacklist to also allow classes 2018-01-14 18:47:22 +01:00
Mike Fährmann
35e09869d1
[mangapark] fix image URLs and use HTTPS 2018-01-12 14:59:49 +01:00
Mike Fährmann
4edb25346e
[slideshare] support mobile URLs (closes #67) 2018-01-10 14:15:00 +01:00
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com 2018-01-09 17:54:37 +01:00
Mike Fährmann
1a70857a12
update extractor-unittest capabilities 
- "count" can now be a string defining a comparison in the form of
  '<operator> <value>', for example: '> 12' or '!= 1'. If its value
  is not a string, it is assumed to be a concrete integer as before.

- "keyword" can now be a dictionary defining tests for individual keys.
  These tests can either be a type, a concrete value or a regex
  starting with "re:". Dictionaries can be stacked inside each other.
  Optional keys can be indicated with a "?" before its name.

  For example:
      "keyword:" {
          "image_id": int,
          "gallery_id", 123,
          "name": "re:pattern",
          "user": {
              "id": 321,
          },
          "?optional": None,
      }
 2017-12-30 19:05:37 +01:00
Mike Fährmann
28cd78aae0
[kissmanga] extend chapter-string regex (closes #58) 2017-12-24 22:53:10 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication 
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
 2017-12-18 01:16:46 +01:00
Mike Fährmann
0a9a07a6e1
[slideshare] improve metadata; flake8 
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
 2017-12-13 21:16:49 +01:00
Mike Fährmann
291369eab2
various smaller changes/additions 2017-12-06 21:45:56 +01:00
Mike Fährmann
300346ecdf
[mangazuki] remove extractors 
This site has been in "rebuild"-mode for a fairly long time and the
current extractor code isn't going to work for the new version either.
 2017-12-04 13:36:04 +01:00
Mike Fährmann
93482a1f88
implement 'util.advance()' 2017-12-03 01:38:24 +01:00
Mike Fährmann
a718c6c6cd
implement 'util.parse_bytes()' 2017-12-02 01:24:49 +01:00
Mike Fährmann
214972bc9a
[gelbooru] use manual extraction 
... to compensate for their disabled API.
(https://gelbooru.com/index.php?page=forum&s=view&id=3875)

This also adds an extractor for image-pools.
 2017-11-29 20:48:17 +01:00
Mike Fährmann
b14de6ffc2
[tumblr] small improvements 
- don't transform inline GIF URLs
- set 'type' parameter for API calls if there is only
  one post type selected
 2017-11-24 16:51:07 +01:00
Mike Fährmann
b8cdd42cab
[senmanga] fix extraction (again) 
this is basically a re-revert of 2ace5c7
 2017-11-18 17:23:32 +01:00
Mike Fährmann
6913eeaa40
[powermanga] replace manga extractor unit test 
My Hero Academia is gone
 2017-11-15 14:01:24 +01:00
Mike Fährmann
f72318e593
[seiga] support more than 200 images 
Due to API restrictions and/or missing knowledge about and
documentation of API usage, it was only possible to retrieve the
latest 200 images of a niconico seiga user with said API.

The new approach manually visits each HTML page and gets its
information from there.
 2017-11-13 20:46:24 +01:00
Mike Fährmann
2457b71633
skip tests on 5xx status codes 2017-11-12 20:51:12 +01:00
Mike Fährmann
305da540c3
[mangahere] fix metadata extraction 2017-11-03 14:54:46 +01:00
Mike Fährmann
035ef655f1
[imagefap] update unit tests 
old gallery/image has been deleted
 2017-10-27 12:22:16 +02:00
Mike Fährmann
caf26412dd
add option to set alternate location of .part files (#29) 
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
 2017-10-26 00:16:48 +02:00
Mike Fährmann
27c026543f
re-enable download unit tests 2017-10-25 12:55:36 +02:00
Mike Fährmann
b0353aa02d
rewrite download modules (#29) 
- use '.part' files during file-download
- implement continuation of incomplete downloads
- check if file size matches the one reported by server
 2017-10-24 12:53:03 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve (fixes #44) 
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
 2017-10-14 23:01:33 +02:00
Mike Fährmann
75d3a1f72f
[deviantart] always download original images 
Deviation-objects returned by the DeviantArt API don't always contain
the URL and metadata of the original image ([1]). Getting this
information requires an additional API call [2], which is indicated by
the 'is_downloadable' and 'download_filesize' metadata within a
deviation-object.

[1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576
[2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
 2017-10-07 13:07:34 +02:00
Mike Fährmann
8e6a767109
[util] restructure formatter for better exception propagation 2017-10-06 17:10:35 +02:00
Mike Fährmann
0386503c80
fix (sub)category-transfer for DownloadJob instances (#41) 
... and extend "parent" parameters to TestJob- and DataJob-classes
as well.
 2017-10-06 15:38:35 +02:00
Mike Fährmann
41adb99e9c
[pawoo] fix extraction 
- changed access_token
- use account-search instead of general search
 2017-10-02 18:33:52 +02:00
Mike Fährmann
b319f4bab3
smaller code and text changes 2017-10-01 18:23:40 +02:00
Mike Fährmann
c1f0afe4c6
add custom string formatter class 2017-09-28 17:12:39 +02:00
Mike Fährmann
85a2b2ae59
[khinsider] fix extraction 2017-09-28 11:47:26 +02:00
Mike Fährmann
8e14714c2b
[imgspice] fix extraction 2017-09-26 21:04:48 +02:00
Mike Fährmann
a85f06d2d1
[foolslide] restructure; convert suitable values to int 2017-09-24 16:57:47 +02:00
Mike Fährmann
9fc1d0c901
implement and use 'util.safe_int()' 
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
 2017-09-24 15:59:25 +02:00
Mike Fährmann
a9e7145651
[hbrowse] extract hmanga metadata & general maintenance 2017-09-20 16:25:25 +02:00
Mike Fährmann
84d4450410
[fallenangels] extract manga metadata 2017-09-15 20:51:40 +02:00
Mike Fährmann
f32b1a0292
[imgyt] fix extraction 2017-09-14 15:04:32 +02:00
Mike Fährmann
31cd5b1c1d
[luscious] detect high-load responses 2017-09-12 15:46:21 +02:00
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords' 2017-09-09 18:48:28 +02:00
Mike Fährmann
9b21d3f13c
add '--filter' command-line option 
This allows for image filtering via Python expressions by the same
metadata that is also used to build filenames (--list-keywords).

The usually shunned eval() function is used to evaluate
filter-expressions, but it seemed quite appropriate in this case and
shouldn't introduce any new security issues, as any attacker that could do
> gallery-dl --filter "delete-everything()" ...
could as well do
> python -c "delete-everything()"
 2017-09-08 17:52:00 +02:00
Mike Fährmann
31731cbefe
update unittests for util.py 2017-09-06 17:57:19 +02:00
Mike Fährmann
f98e3e8002
[luscious] fix tag extraction 2017-09-01 16:29:52 +02:00
Mike Fährmann
65997d835b
replace popular/ranking tests with older ones 
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
 2017-08-31 15:09:18 +02:00
Mike Fährmann
c0755a4d5e
[exhentai] revert login-method to its old version (#37) 
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
 2017-08-29 22:10:38 +02:00
Mike Fährmann
3ee39ffd93
[exhentai] update login procedure (#37) 
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
 2017-08-28 21:03:32 +02:00
Mike Fährmann
07214f4007
[booru] place subcategories into base classes 2017-08-26 22:27:55 +02:00
Mike Fährmann
47bcf53ec1
implement support for additional unit test result types 
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
 2017-08-25 22:01:14 +02:00
Mike Fährmann
c7ec103e15
[batoto] fix extraction of chapter URLs 2017-08-25 16:34:42 +02:00
Mike Fährmann
f7cdfd4c25
add a simplified version of 'parse_qs' 
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
 2017-08-24 20:55:58 +02:00
Mike Fährmann
d70c66c516
fix "text:" downloader 2017-08-16 12:11:47 +02:00
Mike Fährmann
9bf9d64ad8
update unittests for util.py 2017-08-13 14:31:22 +02:00
Mike Fährmann
d74a635e41
[util] update 'default' values and improve test coverage 
for 'code_to_language()' and 'language_to_code()'
 2017-08-08 19:22:04 +02:00
Mike Fährmann
abd7c559cd
[yonkouprod] remove module 
Every manga chapter on this site has been removed.
 2017-08-07 18:32:14 +02:00
Mike Fährmann
852e7acd31
[twitter] ignore "Promoted Tweets" 2017-08-06 13:43:08 +02:00
Mike Fährmann
6950708e52
[hentaicdn] use HTTPS 2017-08-02 18:31:21 +02:00
Mike Fährmann
493bd235cf
workaround for missing 'assert_called_once' method 
this method was introduced in Python 3.6, but calling it still
works (i.e. it doesn't cause the test to fail) on Python 3.3/3.4
 2017-07-26 10:33:15 +02:00
Mike Fährmann
7aa9fa796a
code cleanup and fixes 2017-07-25 14:59:41 +02:00
Mike Fährmann
f08af03845
Merge branch 'cookies' 2017-07-25 14:04:53 +02:00
Mike Fährmann
6db93a8b9e
add cookie tests 2017-07-25 14:02:53 +02:00
Mike Fährmann
d7cb3c668a
update supportedsites.rst 2017-07-24 10:50:40 +02:00
Mike Fährmann
a13eb6010f
[fallenangels] fix extraction of chapter URLs 2017-07-20 14:58:47 +02:00
Mike Fährmann
1cb1d2e0a3
[mangazuki] add chapter extractor 2017-07-19 17:20:03 +02:00
Mike Fährmann
3460dc8950
update gallery-dl.conf 2017-07-14 08:23:11 +02:00
Mike Fährmann
af9bd17b19
[deviantart] adjust default paths 
- user.deviantart.com/(gallery|favourites|journal)/ images go into
  * <user>/
  * <user>/Favourites/
  * <user>/Journal/
  (having an extra "Gallery" folder for a user's gallery-images seems
   a bit too much if these are all you want to download, which is
   probably the default use-case)

- single "deviations" (user.deviantart.com/(art|journal)/name-123) go
  into their owner's directory:
  * <user>/
  (putting them into their own directory seems weird in practice)
 2017-07-10 18:54:10 +02:00
Mike Fährmann
8c16cbe7ea
fix tests 2017-07-04 19:31:32 +02:00
Mike Fährmann
ce55ec6490
enable extractor tests without filters 
$ python test_extractors.py all
 2017-07-02 08:20:04 +02:00
Mike Fährmann
44d98e562b
[pixiv] support pixiv.me URLs (#23) 2017-06-25 20:21:01 +02:00
Mike Fährmann
e68af4febe
[flickr] add 'width-max' option (#16) 
This option allows for simple format selection by
specifying a maximum image width.
 2017-06-18 22:20:15 +02:00
Mike Fährmann
9759fe8c6b
allow 'only_matching' tests 2017-06-14 08:43:05 +02:00
Mike Fährmann
05ed95e5b0
[flickr] add search extractor 2017-06-13 08:01:32 +02:00
Mike Fährmann
c921b4f32a
code cleanup and fixing tests 2017-06-02 09:10:58 +02:00
Mike Fährmann
b6fffa9e26
[directlink] update filename format and metadata 2017-05-30 17:33:09 +02:00
Mike Fährmann
48ccee2505
[gfycat] add image extractor 2017-05-28 17:09:54 +02:00
Mike Fährmann
691c4dd709
support direct image links 2017-05-24 12:51:18 +02:00
Mike Fährmann
398506da45
update release script 2017-05-22 08:47:58 +02:00
Mike Fährmann
2974d782a3
[yomanga] remove module 
site has been shut down
 2017-05-20 11:18:44 +02:00
Mike Fährmann
a90c6acc9c
code cleanup + fixes 2017-05-18 15:18:18 +02:00
Mike Fährmann
4c88c0d496
rework the output format for --list-keywords 2017-05-15 18:30:47 +02:00
Mike Fährmann
107d29ad8a
improve handling of text:... URLs 
- don't require // after the colon
- open output files in text mode
 2017-05-12 14:10:25 +02:00
Mike Fährmann
9f1c83297f
[pinterest] allow URLs with any TLD 2017-05-08 15:08:39 +02:00
Mike Fährmann
ef90a2de2f
implement the "exit" option for the "skip" config-key 2017-05-05 15:49:58 +02:00
Mike Fährmann
fc9223c072
add '--abort-on-skip' option and ability to control skip behavior 
the 'skip' config option controls skipping behavior:
    true    - skip download if file already exist (default)
    false   - download and overwrite files even if it exists
    "abort" - abort extractor run if a download would be skipped
              (same as '--abort-on-skip')
 2017-05-03 15:26:04 +02:00
Mike Fährmann
d948ba1322
[readcomics] remove module 
- site has been unavailable for two weeks
- (#12)
 2017-05-01 11:44:12 +02:00
Mike Fährmann
244ab75cad
[kissmanga] update AES key retrieval 2017-04-21 20:36:47 +02:00
Mike Fährmann
13dc5d72bc
update some extractors to use https 2017-04-20 13:32:40 +02:00
Mike Fährmann
f4aa452bd1
update unit test results 2017-04-14 14:40:36 +02:00
Mike Fährmann
fece09d326
[fallenangels] update to new domain and site-layout 2017-04-09 11:37:21 +02:00
Mike Fährmann
cf79a47b59
update unit tests 2017-04-08 11:02:32 +02:00
Mike Fährmann
841fd50242
move code into util.py 2017-03-28 13:12:44 +02:00
Mike Fährmann
e3212dd98f
fix some smaller stuff 
- remove support for old windows config paths
- catch exception if cache-database can't be opened
- fix username/password settings for unit tests
- rename variable 'max_tries' to 'retries'
 2017-03-27 14:30:32 +02:00
Mike Fährmann
e4b3077168
improve config module 
- speed improvements, especially in the 'interpolate' function
- 'interpolate' now prioritizes base-level values if they exist
  - "username" is chosen before "extractor.<category>.username"
  - -u/--username & co can now override config-file values
 2017-03-27 11:59:27 +02:00
Mike Fährmann
f2ef49563b
fix argument order for python33 and 34 2017-03-23 16:51:42 +01:00
Mike Fährmann
0257d3e7ac
[mangamint] remove extractors - site is down 2017-03-20 13:38:54 +01:00
Mike Fährmann
66eb3d3488
fix tests 2017-03-14 08:51:58 +01:00
Mike Fährmann
872b8aed97
[4chan] update test 2017-03-03 15:50:10 +01:00
Mike Fährmann
2fa575b273
restore exception-testing to its old form 2017-02-27 23:05:08 +01:00
Mike Fährmann
40be4933b8
fix exception based tests 2017-02-26 02:06:56 +01:00
Mike Fährmann
6208d9dd79
implement '--images' and '--chapters' options 
- the former '--items' has been renamed to '--chapters'
- #6
 2017-02-23 21:51:29 +01:00
Mike Fährmann
e87e6fbc67
change some config keys 
directory_fmt     -> directory
filename_fmt      -> filename
download-original -> original
 2017-02-21 22:11:02 +01:00
Mike Fährmann
96baea0959
[hentaibox] remove extractor 2017-02-18 12:34:10 +01:00
Mike Fährmann
9a08f8a097
improved foolslide-based extractors 
- this includes dokireader, fallenangels, jaiminisbox, powermanga,
  sensescans, worldthree, yonkouprod, gomanga, yomanga
- added 'chapter_string', 'chapter_id', 'chapter_minor' and 'count'
  keywords
- changed the 'chapter' keyword to always be just a number
- changed the default directory format
 2017-02-16 23:42:30 +01:00
Mike Fährmann
bad623751f
[hentaihere] transition to https 2017-02-10 21:28:17 +01:00
Mike Fährmann
331b413c60
update extractor test script 2017-02-10 21:22:57 +01:00
Mike Fährmann
4f123b8513
code adjustments according to pep8 2017-01-30 19:40:15 +01:00
Mike Fährmann
9538a873cb
move create_test_data script 2017-01-19 01:35:59 +01:00
Mike Fährmann
434ef4d5a2
[mangafox] raise proper exception if chapter is not available 2017-01-14 22:11:47 +01:00
Mike Fährmann
cc0b4f2661
[yomanga] add chapter extractor 2017-01-13 00:03:12 +01:00
Mike Fährmann
ad4b02508f
trying to understand travis-ci unit test failures 
- added some debug output via logging module
- unit tests work on my machine (tm)
 2017-01-12 22:35:42 +01:00
Mike Fährmann
72d3ca0bf9
[deviantart] use API for user-extractor 2017-01-12 21:08:49 +01:00
Mike Fährmann
989820d719
workaround for python 3.6s behaviour with BEGIN EXCLUSIVE in sqlite 2017-01-11 00:34:32 +01:00
Mike Fährmann
00074a71d7
several changes to make travis build work 
- fixed html.unescape not being available on Python3.3
- removed inconsistent test result
- added username/password pairs for authenticating extractors
 2017-01-10 13:41:00 +01:00
Mike Fährmann
a1fb46d984
make extractor unittest discoverable 2017-01-09 12:27:20 +01:00
Mike Fährmann
2c9bc50c23
support expecting exceptions in tests 2016-12-30 01:41:17 +01:00
Mike Fährmann
4e75e923cd
sanatize extractor test output 2016-12-09 01:15:40 +01:00
Mike Fährmann
d1f90bb424
add method to re-create existing test-data 2016-12-09 00:25:52 +01:00
Mike Fährmann
57a12f5be0
fix module-names for extractor tests 2016-07-23 17:53:19 +02:00
Mike Fährmann
b0c333b799
rewrite cache module 2016-04-20 08:40:41 +02:00
Mike Fährmann
871f4c8a48
initialize cache-module before running tests 2016-03-08 18:01:35 +01:00
Mike Fährmann
b3aab5745b
dynamically create extractor testcases 2016-02-18 15:53:53 +01:00
Mike Fährmann
2b8977cb2e
script to easily create testdata 2015-12-22 01:49:25 +01:00
Mike Fährmann
565ea042c9
update HashJob to generate hashes for downloaded content 2015-12-21 22:49:04 +01:00
Mike Fährmann
5a8541afa5
remove 'unstable' tests 2015-12-15 23:45:40 +01:00
Mike Fährmann
0961ece989
make test-parameters optional 2015-12-13 03:56:29 +01:00
Mike Fährmann
5304e5beef
testing environment for extractor results 2015-12-12 15:58:07 +01:00
Mike Fährmann
eeae580781
more tests 2015-11-28 01:47:17 +01:00
Mike Fährmann
ca523b9f64
add helper method to text module 2015-11-16 03:46:43 +01:00
Mike Fährmann
506242740f
fix capitalization issues 2015-11-02 14:58:26 +01:00
Mike Fährmann
b24bb1da7d
new module for simple language code to name mapping 2015-11-02 00:18:26 +01:00
Mike Fährmann
cba4b91b14 add tests 2015-10-31 00:39:10 +01:00
Mike Fährmann
bea33ae9cb implement config.setdefault 2015-10-07 00:58:43 +02:00
Mike Fährmann
2026223ed1 change argument format for config-calls 2015-10-05 12:42:42 +02:00
Mike Fährmann
7ac106096f add tests for config-module 2015-10-03 20:24:35 +02:00
Mike Fährmann
2962bf36f6 add tests for text-module 2015-10-03 14:51:13 +02:00