Mike Fährmann
b319f4bab3
smaller code and text changes
2017-10-01 18:23:40 +02:00
Mike Fährmann
c1f0afe4c6
add custom string formatter class
2017-09-28 17:12:39 +02:00
Mike Fährmann
85a2b2ae59
[khinsider] fix extraction
2017-09-28 11:47:26 +02:00
Mike Fährmann
8e14714c2b
[imgspice] fix extraction
2017-09-26 21:04:48 +02:00
Mike Fährmann
a85f06d2d1
[foolslide] restructure; convert suitable values to int
2017-09-24 16:57:47 +02:00
Mike Fährmann
9fc1d0c901
implement and use 'util.safe_int()'
...
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
2017-09-24 15:59:25 +02:00
Mike Fährmann
a9e7145651
[hbrowse] extract hmanga metadata & general maintenance
2017-09-20 16:25:25 +02:00
Mike Fährmann
84d4450410
[fallenangels] extract manga metadata
2017-09-15 20:51:40 +02:00
Mike Fährmann
f32b1a0292
[imgyt] fix extraction
2017-09-14 15:04:32 +02:00
Mike Fährmann
31cd5b1c1d
[luscious] detect high-load responses
2017-09-12 15:46:21 +02:00
Mike Fährmann
81877bb5f6
add '-K' as shortcut for '--list-keywords'
2017-09-09 18:48:28 +02:00
Mike Fährmann
9b21d3f13c
add '--filter' command-line option
...
This allows for image filtering via Python expressions by the same
metadata that is also used to build filenames (--list-keywords).
The usually shunned eval() function is used to evaluate
filter-expressions, but it seemed quite appropriate in this case and
shouldn't introduce any new security issues, as any attacker that could do
> gallery-dl --filter "delete-everything()" ...
could as well do
> python -c "delete-everything()"
2017-09-08 17:52:00 +02:00
Mike Fährmann
31731cbefe
update unittests for util.py
2017-09-06 17:57:19 +02:00
Mike Fährmann
f98e3e8002
[luscious] fix tag extraction
2017-09-01 16:29:52 +02:00
Mike Fährmann
65997d835b
replace popular/ranking tests with older ones
...
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
2017-08-31 15:09:18 +02:00
Mike Fährmann
c0755a4d5e
[exhentai] revert login-method to its old version ( #37 )
...
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
2017-08-29 22:10:38 +02:00
Mike Fährmann
3ee39ffd93
[exhentai] update login procedure ( #37 )
...
This new version behaves pretty much exactly like a browser would and
caches all cookies sent to it and not just "ipb_member_id" and
"ipb_pass_hash".
2017-08-28 21:03:32 +02:00
Mike Fährmann
07214f4007
[booru] place subcategories into base classes
2017-08-26 22:27:55 +02:00
Mike Fährmann
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
2017-08-25 22:01:14 +02:00
Mike Fährmann
c7ec103e15
[batoto] fix extraction of chapter URLs
2017-08-25 16:34:42 +02:00
Mike Fährmann
f7cdfd4c25
add a simplified version of 'parse_qs'
...
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
2017-08-24 20:55:58 +02:00
Mike Fährmann
d70c66c516
fix "text:" downloader
2017-08-16 12:11:47 +02:00
Mike Fährmann
9bf9d64ad8
update unittests for util.py
2017-08-13 14:31:22 +02:00
Mike Fährmann
d74a635e41
[util] update 'default' values and improve test coverage
...
for 'code_to_language()' and 'language_to_code()'
2017-08-08 19:22:04 +02:00
Mike Fährmann
abd7c559cd
[yonkouprod] remove module
...
Every manga chapter on this site has been removed.
2017-08-07 18:32:14 +02:00
Mike Fährmann
852e7acd31
[twitter] ignore "Promoted Tweets"
2017-08-06 13:43:08 +02:00
Mike Fährmann
6950708e52
[hentaicdn] use HTTPS
2017-08-02 18:31:21 +02:00
Mike Fährmann
493bd235cf
workaround for missing 'assert_called_once' method
...
this method was introduced in Python 3.6, but calling it still
works (i.e. it doesn't cause the test to fail) on Python 3.3/3.4
2017-07-26 10:33:15 +02:00
Mike Fährmann
7aa9fa796a
code cleanup and fixes
2017-07-25 14:59:41 +02:00
Mike Fährmann
f08af03845
Merge branch 'cookies'
2017-07-25 14:04:53 +02:00
Mike Fährmann
6db93a8b9e
add cookie tests
2017-07-25 14:02:53 +02:00
Mike Fährmann
d7cb3c668a
update supportedsites.rst
2017-07-24 10:50:40 +02:00
Mike Fährmann
a13eb6010f
[fallenangels] fix extraction of chapter URLs
2017-07-20 14:58:47 +02:00
Mike Fährmann
1cb1d2e0a3
[mangazuki] add chapter extractor
2017-07-19 17:20:03 +02:00
Mike Fährmann
3460dc8950
update gallery-dl.conf
2017-07-14 08:23:11 +02:00
Mike Fährmann
af9bd17b19
[deviantart] adjust default paths
...
- user.deviantart.com/(gallery|favourites|journal)/ images go into
* <user>/
* <user>/Favourites/
* <user>/Journal/
(having an extra "Gallery" folder for a user's gallery-images seems
a bit too much if these are all you want to download, which is
probably the default use-case)
- single "deviations" (user.deviantart.com/(art|journal)/name-123) go
into their owner's directory:
* <user>/
(putting them into their own directory seems weird in practice)
2017-07-10 18:54:10 +02:00
Mike Fährmann
8c16cbe7ea
fix tests
2017-07-04 19:31:32 +02:00
Mike Fährmann
ce55ec6490
enable extractor tests without filters
...
$ python test_extractors.py all
2017-07-02 08:20:04 +02:00
Mike Fährmann
44d98e562b
[pixiv] support pixiv.me URLs ( #23 )
2017-06-25 20:21:01 +02:00
Mike Fährmann
e68af4febe
[flickr] add 'width-max' option ( #16 )
...
This option allows for simple format selection by
specifying a maximum image width.
2017-06-18 22:20:15 +02:00
Mike Fährmann
9759fe8c6b
allow 'only_matching' tests
2017-06-14 08:43:05 +02:00
Mike Fährmann
05ed95e5b0
[flickr] add search extractor
2017-06-13 08:01:32 +02:00
Mike Fährmann
c921b4f32a
code cleanup and fixing tests
2017-06-02 09:10:58 +02:00
Mike Fährmann
b6fffa9e26
[directlink] update filename format and metadata
2017-05-30 17:33:09 +02:00
Mike Fährmann
48ccee2505
[gfycat] add image extractor
2017-05-28 17:09:54 +02:00
Mike Fährmann
691c4dd709
support direct image links
2017-05-24 12:51:18 +02:00
Mike Fährmann
398506da45
update release script
2017-05-22 08:47:58 +02:00
Mike Fährmann
2974d782a3
[yomanga] remove module
...
site has been shut down
2017-05-20 11:18:44 +02:00
Mike Fährmann
a90c6acc9c
code cleanup + fixes
2017-05-18 15:18:18 +02:00
Mike Fährmann
4c88c0d496
rework the output format for --list-keywords
2017-05-15 18:30:47 +02:00
Mike Fährmann
107d29ad8a
improve handling of text:... URLs
...
- don't require // after the colon
- open output files in text mode
2017-05-12 14:10:25 +02:00
Mike Fährmann
9f1c83297f
[pinterest] allow URLs with any TLD
2017-05-08 15:08:39 +02:00
Mike Fährmann
ef90a2de2f
implement the "exit" option for the "skip" config-key
2017-05-05 15:49:58 +02:00
Mike Fährmann
fc9223c072
add '--abort-on-skip' option and ability to control skip behavior
...
the 'skip' config option controls skipping behavior:
true - skip download if file already exist (default)
false - download and overwrite files even if it exists
"abort" - abort extractor run if a download would be skipped
(same as '--abort-on-skip')
2017-05-03 15:26:04 +02:00
Mike Fährmann
d948ba1322
[readcomics] remove module
...
- site has been unavailable for two weeks
- (#12 )
2017-05-01 11:44:12 +02:00
Mike Fährmann
244ab75cad
[kissmanga] update AES key retrieval
2017-04-21 20:36:47 +02:00
Mike Fährmann
13dc5d72bc
update some extractors to use https
2017-04-20 13:32:40 +02:00
Mike Fährmann
f4aa452bd1
update unit test results
2017-04-14 14:40:36 +02:00
Mike Fährmann
fece09d326
[fallenangels] update to new domain and site-layout
2017-04-09 11:37:21 +02:00
Mike Fährmann
cf79a47b59
update unit tests
2017-04-08 11:02:32 +02:00
Mike Fährmann
841fd50242
move code into util.py
2017-03-28 13:12:44 +02:00
Mike Fährmann
e3212dd98f
fix some smaller stuff
...
- remove support for old windows config paths
- catch exception if cache-database can't be opened
- fix username/password settings for unit tests
- rename variable 'max_tries' to 'retries'
2017-03-27 14:30:32 +02:00
Mike Fährmann
e4b3077168
improve config module
...
- speed improvements, especially in the 'interpolate' function
- 'interpolate' now prioritizes base-level values if they exist
- "username" is chosen before "extractor.<category>.username"
- -u/--username & co can now override config-file values
2017-03-27 11:59:27 +02:00
Mike Fährmann
f2ef49563b
fix argument order for python33 and 34
2017-03-23 16:51:42 +01:00
Mike Fährmann
0257d3e7ac
[mangamint] remove extractors - site is down
2017-03-20 13:38:54 +01:00
Mike Fährmann
66eb3d3488
fix tests
2017-03-14 08:51:58 +01:00
Mike Fährmann
872b8aed97
[4chan] update test
2017-03-03 15:50:10 +01:00
Mike Fährmann
2fa575b273
restore exception-testing to its old form
2017-02-27 23:05:08 +01:00
Mike Fährmann
40be4933b8
fix exception based tests
2017-02-26 02:06:56 +01:00
Mike Fährmann
6208d9dd79
implement '--images' and '--chapters' options
...
- the former '--items' has been renamed to '--chapters'
- #6
2017-02-23 21:51:29 +01:00
Mike Fährmann
e87e6fbc67
change some config keys
...
directory_fmt -> directory
filename_fmt -> filename
download-original -> original
2017-02-21 22:11:02 +01:00
Mike Fährmann
96baea0959
[hentaibox] remove extractor
2017-02-18 12:34:10 +01:00
Mike Fährmann
9a08f8a097
improved foolslide-based extractors
...
- this includes dokireader, fallenangels, jaiminisbox, powermanga,
sensescans, worldthree, yonkouprod, gomanga, yomanga
- added 'chapter_string', 'chapter_id', 'chapter_minor' and 'count'
keywords
- changed the 'chapter' keyword to always be just a number
- changed the default directory format
2017-02-16 23:42:30 +01:00
Mike Fährmann
bad623751f
[hentaihere] transition to https
2017-02-10 21:28:17 +01:00
Mike Fährmann
331b413c60
update extractor test script
2017-02-10 21:22:57 +01:00
Mike Fährmann
4f123b8513
code adjustments according to pep8
2017-01-30 19:40:15 +01:00
Mike Fährmann
9538a873cb
move create_test_data script
2017-01-19 01:35:59 +01:00
Mike Fährmann
434ef4d5a2
[mangafox] raise proper exception if chapter is not available
2017-01-14 22:11:47 +01:00
Mike Fährmann
cc0b4f2661
[yomanga] add chapter extractor
2017-01-13 00:03:12 +01:00
Mike Fährmann
ad4b02508f
trying to understand travis-ci unit test failures
...
- added some debug output via logging module
- unit tests work on my machine (tm)
2017-01-12 22:35:42 +01:00
Mike Fährmann
72d3ca0bf9
[deviantart] use API for user-extractor
2017-01-12 21:08:49 +01:00
Mike Fährmann
989820d719
workaround for python 3.6s behaviour with BEGIN EXCLUSIVE in sqlite
2017-01-11 00:34:32 +01:00
Mike Fährmann
00074a71d7
several changes to make travis build work
...
- fixed html.unescape not being available on Python3.3
- removed inconsistent test result
- added username/password pairs for authenticating extractors
2017-01-10 13:41:00 +01:00
Mike Fährmann
a1fb46d984
make extractor unittest discoverable
2017-01-09 12:27:20 +01:00
Mike Fährmann
2c9bc50c23
support expecting exceptions in tests
2016-12-30 01:41:17 +01:00
Mike Fährmann
4e75e923cd
sanatize extractor test output
2016-12-09 01:15:40 +01:00
Mike Fährmann
d1f90bb424
add method to re-create existing test-data
2016-12-09 00:25:52 +01:00
Mike Fährmann
57a12f5be0
fix module-names for extractor tests
2016-07-23 17:53:19 +02:00
Mike Fährmann
b0c333b799
rewrite cache module
2016-04-20 08:40:41 +02:00
Mike Fährmann
871f4c8a48
initialize cache-module before running tests
2016-03-08 18:01:35 +01:00
Mike Fährmann
b3aab5745b
dynamically create extractor testcases
2016-02-18 15:53:53 +01:00
Mike Fährmann
2b8977cb2e
script to easily create testdata
2015-12-22 01:49:25 +01:00
Mike Fährmann
565ea042c9
update HashJob to generate hashes for downloaded content
2015-12-21 22:49:04 +01:00
Mike Fährmann
5a8541afa5
remove 'unstable' tests
2015-12-15 23:45:40 +01:00
Mike Fährmann
0961ece989
make test-parameters optional
2015-12-13 03:56:29 +01:00
Mike Fährmann
5304e5beef
testing environment for extractor results
2015-12-12 15:58:07 +01:00
Mike Fährmann
eeae580781
more tests
2015-11-28 01:47:17 +01:00
Mike Fährmann
ca523b9f64
add helper method to text module
2015-11-16 03:46:43 +01:00
Mike Fährmann
506242740f
fix capitalization issues
2015-11-02 14:58:26 +01:00
Mike Fährmann
b24bb1da7d
new module for simple language code to name mapping
2015-11-02 00:18:26 +01:00
Mike Fährmann
cba4b91b14
add tests
2015-10-31 00:39:10 +01:00
Mike Fährmann
bea33ae9cb
implement config.setdefault
2015-10-07 00:58:43 +02:00
Mike Fährmann
2026223ed1
change argument format for config-calls
2015-10-05 12:42:42 +02:00
Mike Fährmann
7ac106096f
add tests for config-module
2015-10-03 20:24:35 +02:00
Mike Fährmann
2962bf36f6
add tests for text-module
2015-10-03 14:51:13 +02:00
