Mike Fährmann
78fd63b8f0
remove 'text.clean_xml()'
...
was not used anywhere
2021-03-28 04:05:16 +02:00
Mike Fährmann
8553b218d9
replace calls to 'os.path.splitext()' with 'str.rpartition()'
...
Makes functions who used it more than twice as fast
and we can get rid of an import as well.
2021-03-28 04:01:27 +02:00
Mike Fährmann
bff71cde80
implement 'util.unique_squence()'
2021-03-02 23:11:08 +01:00
Mike Fährmann
5f1a6ff6fa
remove unneeded 'TRAVIS_SKIP' from test_results.py
2021-03-01 01:38:18 +01:00
Mike Fährmann
8821dceb79
use __import__() to dynamically load modules
2021-03-01 01:27:02 +01:00
Mike Fährmann
36bf76fa44
update 'oauth:mastodon:<instance>' code
2021-01-28 02:20:12 +01:00
Mike Fährmann
91308140ec
make 'generate_token()' compatible with Python 3.4
2021-01-14 03:48:10 +01:00
Mike Fährmann
780b6adb91
rename 'generate_csrf_token()' to just 'generate_token()'
...
and add a 'size' argument
2021-01-11 22:12:40 +01:00
Mike Fährmann
0fdaea00a3
[postprocessor:metadata] sanitize filenames
2021-01-10 00:13:20 +01:00
Mike Fährmann
aac00a2024
add 'd' conversion for format strings
...
to convert a timestamp to a formattable 'datetime' object.
For example '{created_at!d:%Y-%m-%d}'
transforms the timestamp in 'created_at' into a 'datetime' object
and then formats its content using '%Y-%m-%d' as template.
1262304000 -> datetime(2010, 1, 1) -> "2010-01-01"
2021-01-09 01:58:44 +01:00
Mike Fährmann
912eea29bc
update extractor test results
2020-12-27 17:41:08 +01:00
Mike Fährmann
1f9121fecb
release version 1.16.0
2020-12-12 23:08:25 +01:00
Mike Fährmann
b2c55f0a72
[sankaku] remove login support
...
The old login method for 'https://chan.sankakucomplex.com/user/login '
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
2020-12-08 21:05:47 +01:00
Mike Fährmann
547107307e
fix 'Metadata' messages in result tests
2020-11-24 13:34:54 +01:00
Mike Fährmann
578dcf805c
[mangapanda] don't force https://
2020-11-21 20:24:37 +01:00
Mike Fährmann
ca59bd691c
[postprocessor:metadata] add 'event' and 'filename' options
2020-11-20 22:29:11 +01:00
Mike Fährmann
9fffa9c343
rework post processor callbacks
2020-11-19 02:29:06 +01:00
Mike Fährmann
1e3dd7330e
merge SharedConfigMixin functionality into Extractor
2020-11-17 00:34:07 +01:00
Mike Fährmann
e5438b8a29
release version 1.15.3
2020-11-13 15:50:05 +01:00
Mike Fährmann
b9bfa4c675
update extractor test results
2020-11-07 02:03:22 +01:00
Mike Fährmann
c3f01dc4e6
implement 'util.unique()'
2020-10-29 23:33:41 +01:00
Mike Fährmann
d83b95fd28
[postprocessor:metadata] accept a string-list for 'content-format'
...
(closes #1080 )
2020-10-27 20:09:58 +01:00
Mike Fährmann
350b1afe1c
speed up _list_classes() after iterating over all modules once
2020-10-26 22:18:15 +01:00
Mike Fährmann
18213dc5ba
release version 1.15.2
2020-10-24 18:57:29 +02:00
Mike Fährmann
ec61696316
add 't' format string conversion ( closes #1065 )
...
to Trim whitespace from the beginning and end of strings.
Example: '{field!t}' becomes 'foo' for 'field' == " \nfoo\t\r"
2020-10-16 00:37:22 +02:00
Mike Fährmann
07432d6262
[seiga] fix flake8 and cookie test ( #1063 )
2020-10-15 15:37:58 +02:00
Mike Fährmann
b8daabc3ca
[pinterest] implement login support ( closes #1055 )
...
being logged allows access to secret/protected boards
2020-10-15 15:14:18 +02:00
kurumigi
7e0e872f4f
[seiga] Add metadata for single image downloads ( #1063 )
...
* [seiga] Support image metadata.
* [seiga] Update test data.
* [seiga] Fix cookie check.
* [test_cookies] [seiga] Fit test_cookies.py to the last commit.
2020-10-15 15:13:27 +02:00
Mike Fährmann
844793847c
update extractor test results
2020-10-11 18:15:41 +02:00
Mike Fährmann
c874071f5a
[kissmanga] remove module
2020-10-04 22:46:41 +02:00
Mike Fährmann
844502cad5
update extractor test results
2020-10-03 19:24:19 +02:00
Mike Fährmann
7cd383c0f9
update extractor test results
2020-09-20 21:54:39 +02:00
Mike Fährmann
65744a7a31
use alternative for all falsey values in format strings
...
… and not just None (#525 )
It would be better to consistently use None for all non-existent
fields and/or fields without a valid value, but this is a good
enough workaround for now.
2020-09-19 22:02:47 +02:00
Mike Fährmann
f5b7ae01c1
update extractor test results
2020-09-15 18:07:08 +02:00
Mike Fährmann
392d022b04
implement 'config.accumulate()' ( #994 )
2020-09-14 21:13:08 +02:00
Mike Fährmann
3108e85b89
[worldthree] remove extractors
...
http://www.slide.world-three.org/ hasn't been accessible for a long time.
2020-09-11 18:12:57 +02:00
Mike Fährmann
3918b69677
remove 'extractor.blacklist' context manager
2020-09-11 13:17:35 +02:00
Mike Fährmann
ac3036ef56
add 'filesize-min' and 'filesize-max' options ( closes #780 )
2020-09-03 18:21:04 +02:00
Mike Fährmann
fd0685d9b5
[postprocessor:zip] defer zip file creation ( fixes #968 )
...
don't try to create zip files on postprocessor construction,
wait until directory creation during file download,
2020-08-31 21:53:18 +02:00
Mike Fährmann
d50f3b333a
update extractor test results
2020-08-30 20:55:22 +02:00
Mike Fährmann
e33293fdd8
[hentaihand] update to new site layout
2020-08-30 00:41:03 +02:00
Mike Fährmann
69e4871005
update extractor test results
...
- sensescans: replace 404d chapters
- mangapark: replace 404d chapters
- subscribestar: update test for attached files
2020-08-28 22:32:32 +02:00
Mike Fährmann
688bd046fc
release version 1.14.4
2020-08-15 21:29:02 +02:00
Mike Fährmann
422e69f187
skip external OAuth tests ( closes #908 )
2020-07-30 19:26:09 +02:00
Mike Fährmann
8dbf827649
[bobx] remove module
2020-07-24 17:00:43 +02:00
Mike Fährmann
87202b8d74
[inkbunny] add 'user' and 'post' extractors ( #283 )
2020-07-22 22:21:30 +02:00
Mike Fährmann
2ecf1efb16
update extractor test results
...
- tumblr: remove deleted post
- jaiminisbox: replace removed manga/chapters
- smugmug: one inconsequential field got removed
2020-07-18 15:12:28 +02:00
Mike Fährmann
e62ebb4643
update CHANGELOG before building sdist and wheel packages
2020-06-27 19:45:09 +02:00
Mike Fährmann
0cac14c3bd
update extractor test results
2020-06-25 19:11:47 +02:00
Mike Fährmann
53cc498d9c
improve config lookup when there are multiple possible locations
...
This specifically applies to all Mastodon extractors and all
extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc.
Values inside those general config locations wouldn't be recognized
when a value with the same was set on the 'extractor' level.
For example 'extractor.mastodon.directory' should be used over
'extractor.directory' when both are set, but this was impossible
with the previous implementation.
(fixes #843 )
2020-06-21 00:07:10 +02:00
Mike Fährmann
d81a8e6544
[twitter] update tests
2020-06-19 23:01:02 +02:00
Mike Fährmann
37d71f6e09
strip microseconds in text.parse_datetime()
2020-06-17 21:40:16 +02:00
Mike Fährmann
6db7ed90cb
release version 1.14.1
2020-06-12 20:12:09 +02:00
Mike Fährmann
087e3184dc
use a non-twitter URL when testing snap creation
2020-06-12 18:31:14 +02:00
Mike Fährmann
7daef6ee70
update extractor test results
...
- certain posts on Instagram now return
https://static.cdninstagram.com/rsrc.php/null.jpg
for public users
- MangaDex is deploying its new MangaDex@Home network similar to
exhentai's Hentai@Home
- realbooru has a new site layout, but the underlying booru API still
works like before
2020-06-12 00:36:06 +02:00
Mike Fährmann
3bad1579ee
update extractor test results
2020-05-31 17:42:07 +02:00
Mike Fährmann
45baa13615
update extractor test results
...
- don't run Instagram tests on Travis anymore
- replace Twitter test because timeline was made private
- update Hiperdex domain to '.com' (again ...)
2020-05-28 02:18:06 +02:00
Mike Fährmann
dfcf2a2c91
write OAuth token to cache by default ( #616 )
2020-05-25 22:35:45 +02:00
Mike Fährmann
6294e2c540
add 'text.ensure_http_scheme()'
2020-05-19 22:32:53 +02:00
Mike Fährmann
ece73b5b2a
make 'path' and 'keywords' available in logging messages
...
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.
__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.
(#574 , #575 )
2020-05-18 19:04:51 +02:00
Mike Fährmann
4b606b68e4
skip OAuth tests when server is unreachable
2020-05-10 00:33:00 +02:00
Mike Fährmann
8b60bd6a91
mock 'time()' in cache tests
...
instead of calling 'sleep()' to let time advance.
This shortens the time needed to run those tests,
and ensures consistent results.
(Tests would randomly fail when using 'sleep()')
2020-05-09 23:55:14 +02:00
Mike Fährmann
8f2c1da041
skip example config tests if files are not available ( #730 )
2020-05-08 22:56:00 +02:00
Mike Fährmann
5df8f2959b
insert local directory into PYTHONPATH when running tests
2020-05-02 01:15:50 +02:00
Mike Fährmann
ff47641b13
test whether default/example config files contain valid JSON
2020-04-30 00:00:41 +02:00
Mike Fährmann
d6facdee7b
[mastodon] add tests ( #701 )
2020-04-22 21:10:34 +02:00
Mike Fährmann
fd438f0d78
update extractor test results
2020-04-11 23:00:42 +02:00
Mike Fährmann
a0f4c295c0
add optional 'utcoffset' argument to 'parse_datetime()'
2020-04-11 02:05:00 +02:00
Mike Fährmann
406449b0d6
ensure keys for mastodon instances are available during tests
...
Calls to config.clear() from other tests are removing the API
credentials set when importing mastodon.py for the first time.
2020-04-08 21:56:14 +02:00
Mike Fährmann
9e7dfc0cfc
[myportfolio] fix extraction of galleries without title
2020-04-08 21:08:05 +02:00
Mike Fährmann
3b50c4f49d
add tests for "Extractors" in oauth.py ( #670 )
2020-04-07 20:26:12 +02:00
Mike Fährmann
04bd0472de
add tests for Extractor.wait()
2020-04-07 20:24:56 +02:00
Mike Fährmann
7499d71d02
[simplyhentai] ignore certificate errors in video test
2020-03-28 21:07:30 +01:00
Mike Fährmann
4203dc0bdc
[mangapark] fix metadata extraction
2020-03-28 03:00:26 +01:00
Mike Fährmann
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor
...
There is still a nice genre overview page (https://35photo.pro/genre/ )
but the individual sub-pages don't list photos anymore
2020-03-24 02:48:25 +01:00
Mike Fährmann
32e36d8f02
[sexcom] replace tests
2020-03-17 22:47:45 +01:00
Mike Fährmann
e0b0e8d62a
release version 1.13.2
2020-03-14 20:19:36 +01:00
Mike Fährmann
a63a376ad2
[mangoxo] fix login
2020-03-08 23:01:51 +01:00
Mike Fährmann
ebc70e87ce
[e621] update to new interface / API endpoints ( closes #635 )
2020-03-06 21:12:58 +01:00
Mike Fährmann
72122eb9b3
release version 1.13.1
2020-03-01 21:17:29 +01:00
Mike Fährmann
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
2020-03-01 20:38:25 +01:00
Mike Fährmann
1d4a369ea2
update extractor test results
2020-02-27 22:15:40 +01:00
Mike Fährmann
ec85bf90de
use context managers in cache.py & add tests
2020-02-25 23:08:47 +01:00
Mike Fährmann
4e361b3008
add tests for specific datetime values
2020-02-23 16:48:30 +01:00
Mike Fährmann
90e4c645ba
[formatter] allow multiple "special" format specifiers ( #595 )
...
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
2020-02-16 21:47:08 +01:00
Mike Fährmann
219c4cc78c
[formatter] allow for numeric list and string indices
2020-02-15 22:46:22 +01:00
Mike Fährmann
7d1da614d9
[formatter] implement field name alternatives ( #525 )
...
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
2020-02-15 17:58:21 +01:00
Mike Fährmann
c7cf9dd111
[furaffinity] support classic layout ( #284 )
2020-02-12 21:39:43 +01:00
Mike Fährmann
40fe062851
[pixiv] fix user id for bookmarks API calls ( closes #596 )
2020-02-01 01:48:46 +01:00
Mike Fährmann
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
2020-01-27 22:19:41 +01:00
Mike Fährmann
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
2020-01-25 23:02:15 +01:00
Mike Fährmann
b3b5754f2d
update test_cookies.py
2020-01-21 22:46:58 +01:00
Mike Fährmann
174117f827
allow multiple hashes for content tests
2020-01-18 00:19:35 +01:00
Mike Fährmann
60a43f0264
fix downloader tests
2020-01-14 11:51:06 +01:00
Mike Fährmann
e89413da22
update test results
2020-01-12 21:41:06 +01:00
Mike Fährmann
5cac79c3d9
[erolord] remove extractor
2020-01-10 14:40:30 +01:00
Mike Fährmann
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
2020-01-05 21:21:40 +01:00
Mike Fährmann
87c8b89ddd
[postprocessor:metadata] add 'directory' option ( #520 )
2020-01-02 20:58:10 +01:00
Mike Fährmann
82f7f4172a
update test results
2020-01-01 16:05:38 +01:00
Mike Fährmann
d0920e84e9
update test results
2019-12-19 18:33:48 +01:00
Mike Fährmann
9e63804347
[patreon] make retrieving user info nonfatal ( #508 )
...
… and fall back to the included data if an error occurs.
2019-12-12 00:31:35 +01:00
Mike Fährmann
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
2019-12-08 21:21:00 +01:00
Mike Fährmann
50deab5265
[deviantart] fix URL generation from /extended_fetch results
...
(closes #505 )
2019-12-07 22:07:55 +01:00
Mike Fährmann
004812258d
[hentaifox] fix extraction
2019-12-02 22:21:45 +01:00
Mike Fährmann
a412531451
[postprocessor:metadata] implement 'extension-format' option
...
closes #477
2019-11-30 17:26:17 +01:00
Mike Fährmann
b5c964332b
improve config.py test coverage
2019-11-25 17:20:00 +01:00
Mike Fährmann
f5604492c3
update interface of config functions
2019-11-24 00:42:28 +01:00
Mike Fährmann
3fc1e12949
[postprocessor:metadata] filter private entries
...
i.e. keys starting with an underscore
2019-11-21 16:58:44 +01:00
Mike Fährmann
978cb03f81
update misc test results
...
- Livedoor now uses https:// for its image URLs
- Instagram image URLs got simplified
2019-11-20 21:45:48 +01:00
Mike Fährmann
bbbeff4c41
[downloader.http] implement file-specific HTTP headers
2019-11-19 23:50:54 +01:00
Mike Fährmann
3ece3976ae
[newgrounds] implement login support ( #394 )
2019-11-16 23:45:32 +01:00
Mike Fährmann
abfcb356fc
[flickr] support 3k, 4k, 5k, and 6k photo sizes ( closes #472 )
2019-11-10 17:52:51 +01:00
Mike Fährmann
da6789b2b0
disable unique archive id checks for some tests
...
- same image twice in a livedoor blog post
- unreliable results for related pinterest items
2019-11-10 17:04:51 +01:00
Mike Fährmann
ba083b30b2
fix snap build
...
… hopefully
2019-11-08 21:44:12 +01:00
Mike Fährmann
94a94f3b86
miscellaneous stuff
2019-11-08 20:58:53 +01:00
Mike Fährmann
9e88e7a344
[postprocessor:exec] improve ( #421 , #413 )
...
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
2019-11-03 21:45:45 +01:00
Mike Fährmann
2a3bd4e3c7
rename extractor classes starting with a digit
2019-11-02 20:42:09 +01:00
Mike Fährmann
64786363be
[4chan] simplify
...
- remove 'chan.py'
- slight adjustments to directory and filenames
2019-11-02 20:11:21 +01:00
Mike Fährmann
557e2c018b
[8chan] remove module
2019-11-02 20:06:47 +01:00
Mike Fährmann
322c2e7ed4
renaming variables
...
mostly 'keyword(s)' to 'kwdict'
2019-10-29 15:46:35 +01:00
Mike Fährmann
87a87bff7e
[simplyhentai] fix image URLs
2019-10-28 21:11:06 +01:00
Mike Fährmann
d5e3910270
adjust 'util.raises()'
2019-10-28 15:06:17 +01:00
Mike Fährmann
b23c822b23
[luscious] use GraphQL
2019-10-22 21:17:08 +02:00
Mike Fährmann
1693d97bd3
update extractor class hierarchies
...
- let the GalleryExtractor class inherit directly from Extractor
- make ChapterExtractor a subclass of GalleryExtractor
- change enumeration field names of GalleryExtractors to 'num'
2019-10-16 18:15:29 +02:00
Mike Fährmann
7ebd984e8d
[imgur] print error message if no JSON data is found ( #446 )
2019-10-16 17:45:14 +02:00
Mike Fährmann
de4e2029d1
[nsfwalbum] update test album
...
the old one is no longer available
2019-09-28 20:48:15 +02:00
Mike Fährmann
913460240d
[reddit] fix 'extractor.blacklist()' arguments
...
The second argument must support 'append()'.
2019-09-24 23:01:12 +02:00
Mike Fährmann
1848788970
update test results etc
2019-09-08 11:33:35 +02:00
Mike Fährmann
d5fbb2d9de
[tumblr] ignore audio links from Spotify etc.
2019-09-07 18:18:12 +02:00
Mike Fährmann
c6c5cb1898
improve 'deviantart.quality' description
2019-08-30 18:41:18 +02:00
Mike Fährmann
c9b97dbca3
extend post processor tests
2019-08-26 23:10:12 +02:00
Mike Fährmann
49f6d7176d
[deviantart] restore filenames ( #392 )
...
<title>_by_<user>_<id> --> <title>_by_<user>-<id>
2019-08-23 22:02:03 +02:00
Mike Fährmann
e528f3cb77
adjust postprocessor test results
...
see 2495b99
2019-08-21 23:54:08 +02:00
Mike Fährmann
cb7f149974
fix mtime datetime test
...
datetime.timestamp() uses local time for a naive datetime object
2019-08-14 21:57:42 +02:00
Mike Fährmann
23251356cb
require 'extension' data for each URL ( #382 )
2019-08-14 20:03:03 +02:00
Mike Fährmann
dd72ae7164
add postprocessor tests
2019-08-12 21:46:30 +02:00
Mike Fährmann
0bb873757a
update PathFormat class
...
- change 'has_extension' from a simple flag/bool to a field that
contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306 )
2019-08-12 21:40:37 +02:00
Mike Fährmann
748e37554c
update .travis.yml
...
- install pyOpenSSL before running tests
- simplify snap tests
2019-08-11 16:03:19 +02:00
Mike Fährmann
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option
2019-08-08 16:54:20 +02:00
Mike Fährmann
eb7da159e2
[imagebam] update URL test results
...
Image URLs are now using https://, but the website itself is still
served as http://.
2019-08-07 21:47:44 +02:00
Mike Fährmann
fa60109e97
[exhentai] don't use e-hentai.org for exhentai URLs
2019-08-02 21:10:09 +02:00
Mike Fährmann
4a0c98bfc9
miscellaneous fixes and adjustments
2019-08-01 22:09:43 +02:00
Mike Fährmann
40637556fa
[ngomik] fix extraction
2019-07-28 10:53:46 +02:00
Mike Fährmann
d9d44ad953
[tsumino] update test results
2019-07-24 21:17:23 +02:00
Mike Fährmann
b1bea8aaeb
add 'restrict-filenames' option ( #348 )
2019-07-23 17:41:24 +02:00
Mike Fährmann
b3851e01d9
release version 1.9.0
2019-07-19 21:55:25 +02:00
Mike Fährmann
12da6bd0c9
[simplyhentai] fix/improve extraction
2019-07-06 20:25:53 +02:00
Mike Fährmann
b89f0d8d3c
update extractor result tests
2019-07-01 20:02:47 +02:00
Mike Fährmann
40da44b17f
Merge branch 'v1.9.0'
2019-06-29 15:39:52 +02:00
Mike Fährmann
7a99e85943
[kissmanga] fix download URLs and file extensions
...
The current Blogspot image URLs hosted on Kissmanga end with an
"invalid" query parameter (/000.png&upx=...), which doesn't get
recognized by 'spliturl()' and 'parseurl()' as such and gets therefore
included in the 'extension' field from 'text.nameext_from_url()'.
2019-06-28 20:34:43 +02:00
Mike Fährmann
a9c89085fb
[instagram] implement login support ( #195 )
2019-06-26 23:58:47 +02:00
Mike Fährmann
b1985d6579
test default format strings during extractor result tests
...
A missing value or an invalid "syntax" for a format replacement field
will raise an exception.
2019-06-25 18:12:32 +02:00
Mike Fährmann
95b1e4c3c0
implement R<old>/<new>/ format option ( #318 )
2019-06-23 22:45:44 +02:00
Mike Fährmann
70713f0f28
fix extractor result tests
2019-06-20 18:12:36 +02:00
Mike Fährmann
ee4d7c3d89
update downloader.find() and related code
...
Instead of replacing 'https' with 'http' for every URL in
'get_downloader()', this now only happens once during downloader
initialization. Also unit tests.
2019-06-20 16:59:44 +02:00
Mike Fährmann
179d112083
[downloader] overhaul http and text modules
...
Get rid of the modular structure and simplify/specialize those modules.
2019-06-19 22:56:11 +02:00
Mike Fährmann
a77340c647
[keenspot] fix extraction for "TwoKinds"
2019-06-17 19:49:39 +02:00
Mike Fährmann
b171befa87
implement 'parse_unicode_escapes()'
2019-06-16 21:47:24 +02:00
Mike Fährmann
e05a96db5e
[deviantart] rename 'stash' to 'extra' ( #302 )
...
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
2019-06-10 21:05:25 +02:00
Mike Fährmann
7c6cb908f9
[xhamster] update test results
2019-06-07 16:28:49 +02:00
Mike Fährmann
62335b9015
[paheal] adjust test results
2019-06-05 11:42:01 +02:00
Mike Fährmann
6a34f4b0c1
skip tests on read timeouts; print list of skipped tests
2019-06-01 20:47:31 +02:00
Mike Fährmann
d33f5a7423
[wallhaven] rewrite
...
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
IDs that can't be translated to the new ID system
- support direct links to wallpapers
2019-05-31 14:53:02 +02:00
Mike Fährmann
5499934ae2
[ngomik] fix extraction
2019-05-30 20:18:36 +02:00
Mike Fährmann
2b1999476e
implement 'text.rextract()'
2019-05-28 21:03:41 +02:00
Mike Fährmann
e30ada162d
fix cookie tests
...
update _get_extractor():
- always return an Extractor instance with a _login_impl() method
- use Extractor.from_url()
2019-05-26 20:22:04 +02:00
Mike Fährmann
2316e0ed3d
fix strptime workaround from b0e85a4
...
Don't return a modified version of 'date_time' if strptime fails.
2019-05-25 23:22:26 +02:00
Mike Fährmann
6764847349
fix cookie tests
...
'cookies' is a CookieJar, not a dict,
and removing the call to '.keys()' doesn't have the same effect
2019-05-14 22:32:40 +02:00
Mike Fährmann
a5b060765d
improve code in tests
...
- use 'assertRaises' as context manager
- remove calls to .keys()
2019-05-13 11:48:20 +02:00
Mike Fährmann
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself
2019-05-09 21:53:17 +02:00
Mike Fährmann
4736912d4e
[pixiv] work around strptime limitations in Python < 3.7
...
"%z" doesn't allow a colon separator in older Python versions:
- "+0900" is OK
- "+09:00" raises an exception
2019-05-08 18:08:03 +02:00
Mike Fährmann
d09864b581
implement text.parse_datetime()
2019-05-08 15:43:59 +02:00
Mike Fährmann
5582b06ae4
fix tests with 'urllist' messages
2019-04-30 16:31:48 +02:00
Mike Fährmann
5018781898
allow type tests by name
2019-04-29 17:27:59 +02:00
Mike Fährmann
6264a46212
use 'utcfromtimestamp()'
...
'fromtimestamp()' converts its results to the local timezone and causes
problems when running tests on a different machine.
2019-04-21 16:22:53 +02:00
Mike Fährmann
d670de0344
implement 'text.parse_timestamp()'
2019-04-21 15:28:27 +02:00
Mike Fährmann
21a7e395a7
implement convenience wrapper for text.extract functionality
2019-04-19 22:30:11 +02:00
Mike Fährmann
e25ebc4bff
don't disable certificate checks anymore
...
Executables generated with PyInstaller auto-include the root certificate
file and certificate checks now work out-of-the-box.
2019-04-17 13:27:19 +02:00
Mike Fährmann
d6ddb74cde
update test results
...
- deviantart: 'index' is now an integer
- flickr: image file with lower quality
- paheal: image server name changed
- rule34: post got deleted
2019-04-12 09:59:48 +02:00
Mike Fährmann
d9b94a585d
[mangoxo] add login support ( #184 )
...
A very recent change: It is now only possible to see more
than the first 5 images of an album if you are logged in.
2019-04-10 18:55:25 +02:00
Mike Fährmann
e730fc9045
[twitter] add login support ( #214 )
2019-04-09 09:27:49 +02:00
Mike Fährmann
790f15a56f
[photobucket] use HTTPS
2019-04-03 18:30:45 +02:00
Mike Fährmann
c70b21248d
[wikiart] add extractors ( #179 )
...
for
- artists: https://www.wikiart.org/en/thomas-cole
- artist-listings: https://www.wikiart.org/en/artists-by-century/12
- artwork-listings: https://www.wikiart.org/en/paintings-by-media/grisaille
2019-04-02 17:34:57 +02:00
Mike Fährmann
0c991a3155
add convenience targets to Makefile
2019-03-29 15:35:00 +01:00
Mike Fährmann
6277a739e4
[35photo] add user-, genre-, and image-extractors ( #162 )
2019-03-18 01:11:30 +01:00
Mike Fährmann
973a720a7a
[weibo] fix unit test URL patterns
2019-03-15 15:19:39 +01:00
Mike Fährmann
6f57d44ec2
[seaotterscans] remove extractor
...
http://seaotterscans.com/ now redirects to their MangaDex profile
2019-03-13 22:02:45 +01:00
Mike Fährmann
0887fb61f4
[komikcast] update test results
2019-03-07 14:55:52 +01:00
Mike Fährmann
a881537b91
more util.py tests
2019-03-06 21:09:37 +01:00
Mike Fährmann
976ccb267f
[myportfolio] combine gallery and user extractors
...
An URL alone isn't good enough to distinguish between a gallery or a
gallery-listing, so the new extractor decides what to do based on the
page's content.
2019-03-06 19:45:01 +01:00
Mike Fährmann
9c0e2f294b
[shopify] add generic collection and product extractors ( #175 )
...
with fashionnova.com as a default domain
2019-03-05 22:33:37 +01:00
Mike Fährmann
176b7253a1
update function signature for config.load()
2019-03-01 14:13:34 +01:00
Mike Fährmann
e687a6095e
[luscious] raise exception if album is not available
2019-02-19 13:30:39 +01:00
Mike Fährmann
b09a8184ca
move TestJob into test module; test _extractor values
2019-02-17 18:18:31 +01:00
Mike Fährmann
5530871b5a
change results of text.nameext_from_url()
...
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)
Example: "https://example.org/path/filename.ext "
before:
- filename : filename.ext
- name : filename
- extension: ext
now:
- filename : filename
- extension: ext
2019-02-14 16:07:17 +01:00
Mike Fährmann
148b8f15d0
update tests for util.py
2019-02-14 11:15:19 +01:00
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor
2019-02-11 13:31:10 +01:00
Mike Fährmann
1f3422c28b
[mangahere] fix extraction
2019-02-10 22:10:53 +01:00
Mike Fährmann
84ae72b8d8
[ngomik] fix extraction
2019-02-10 14:19:08 +01:00
Mike Fährmann
9a9cd32461
implement alternative constructor for extractors
2019-02-09 14:42:25 +01:00
