Mike Fährmann
32447d0d24
[pixiv] simplify default filename format
...
(#366 )
2019-08-15 13:32:47 +02:00
Mike Fährmann
ad3ac02fbc
[pixiv] update metadata entries ( #366 )
...
- change 'num' to a simple enumerating integer
- change default filename format
- provide content of the old 'num' field as 'suffix'
- add 'filename' for ugoira
2019-08-05 22:41:56 +02:00
Mike Fährmann
64b2935d8e
[pixiv] provide 'filename' and change default filename format
...
to '{filename}.{extension}' (closes #366 )
2019-08-02 22:35:10 +02:00
Mike Fährmann
fdec59f8e2
replace extractor.request() 'expect' argument
...
with
- 'fatal': allow 4xx status codes
- 'notfound': raise NotFoundError on 404
2019-07-05 00:42:16 +02:00
Mike Fährmann
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself
2019-05-09 21:53:17 +02:00
Mike Fährmann
4736912d4e
[pixiv] work around strptime limitations in Python < 3.7
...
"%z" doesn't allow a colon separator in older Python versions:
- "+0900" is OK
- "+09:00" raises an exception
2019-05-08 18:08:03 +02:00
Mike Fährmann
80fdb11508
[pixiv] add 'date' metadata field ( closes #248 )
2019-05-08 15:43:59 +02:00
Mike Fährmann
a2af2d2965
adjust cache maxage values
2019-03-14 22:21:49 +01:00
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor
2019-02-11 13:31:10 +01:00
Mike Fährmann
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
2019-02-08 13:45:40 +01:00
Mike Fährmann
34bab080ae
rewrite URL patterns to use only 1 per extractor
2019-02-08 12:03:10 +01:00
Mike Fährmann
7471933d5f
use extractor.request for all other API calls
...
- deviantart
- pawoo
- pixiv
- reddit
2018-12-22 14:42:23 +01:00
Mike Fährmann
f6bf66f72c
[pixiv] create directory for each "work" item ( #136 )
2018-12-11 20:37:47 +01:00
Mike Fährmann
8faf03ed84
[pixiv] use refresh_token based authentication
...
The first login will still use username and password, but everything
afterwards will use the refresh_token obtained from that.
This will prevent pixiv from sending a "New login to pixiv" email every
time a new access_token is requested.
2018-10-12 22:26:27 +02:00
Mike Fährmann
f4df6c2396
[pixiv] remove 'type' and 'page' query parameter handling
...
The "new and improved" /member_illust.php and /bookmark.php listings
don't quite work with how things were.
2018-10-03 16:08:36 +02:00
Mike Fährmann
b8c97d2295
use 'extractor.request()' for more HTTP requests
2018-06-25 23:40:59 +02:00
Mike Fährmann
91340d9d27
[pixiv] fix ugoira test
2018-06-18 19:22:54 +02:00
Mike Fährmann
eb7a1f3b98
[pixiv] rework ugoira handling
...
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
2018-06-18 17:57:57 +02:00
Mike Fährmann
a62bd81e9b
[pixiv] fix filter for 'type=all'
2018-05-29 10:30:41 +02:00
Mike Fährmann
1ff626db97
[pixiv] improve bookmark extraction
...
- combine 'favorite' and 'bookmark' extractors
- it is now one extractor class, but its subcategory still
distinguishes between your own bookmarks ('bookmark') and other
user's bookmarks ('favorite') like before
- allow filtering by bookmark tags and public/private bookmarks
- fix pagination for bookmark results
2018-05-18 17:04:59 +02:00
Mike Fährmann
0a1863fce3
[pixiv] respect more query parameters for user URLs
...
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.
Respected parameters are
'type': illust, manga, ugoira
'tag' : any image tag (this was already supported)
'p' : the page to start on
*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
2018-05-18 15:36:30 +02:00
Mike Fährmann
b8e53b8c6b
[pixiv] move query parsing out of constructor
...
better exception handling, among other things
2018-05-15 13:28:08 +02:00
Mike Fährmann
909d105ae6
[pixiv] add extractor for illusts from followed users
2018-05-15 13:05:15 +02:00
Mike Fährmann
fe69d01083
[pixiv] add extractor for search results
2018-05-14 14:46:05 +02:00
Mike Fährmann
247f785af1
[pixiv] use App API
...
Transitioning to the App API breaks favorites archive IDs (there is
no longer any bookmark ID information), but the favorites API endpoint
of the public API was gone anyways ...
2018-05-14 10:56:37 +02:00
Mike Fährmann
0e3883303f
[pixiv] implement AppAPI wrapper
2018-05-08 15:42:45 +02:00
Mike Fährmann
f9884e2338
[pixiv] update URL pattern
...
add support for 'https://www.pixiv.net/user/ <id>'
2018-03-10 16:05:12 +01:00
Mike Fährmann
1d71123f91
[pixiv] update archive IDs and add metadata-fields
...
(Pixiv bookmarks actually have their own IDs, comments and tags,
independent of the bookmarked image, which makes creating an
archive ID a lot easier)
2018-03-02 16:11:53 +01:00
Mike Fährmann
367b963d37
[pixiv] fix ugoira extraction ... again ( #78 )
...
Some animations are not available for mobile devices, so we
pretend to be a desktop browser when requesting the ugoira page.
2018-02-19 16:50:12 +01:00
Mike Fährmann
b79f1f2ca7
[pixiv] fix ugoira extraction ( closes #78 )
2018-02-19 08:51:09 +01:00
Mike Fährmann
34873dbd90
set 'archive_fmt' values
...
These are going to be used to create an unique id for each image.
2018-02-01 15:30:49 +01:00
Mike Fährmann
c24e0e70a7
[pixiv] simplify main loop
2017-12-28 14:13:39 +01:00
Mike Fährmann
4d8387f93b
[pixiv] support mobile URLs ( https://touch.pixiv.net/ )
2017-10-17 16:49:42 +02:00
Mike Fährmann
b319f4bab3
smaller code and text changes
2017-10-01 18:23:40 +02:00
Mike Fährmann
ad4580800c
[pixiv] add support for more URL patterns
...
- https://www.pixiv.net/mypage.php#id=USERID
- https://www.pixiv.net/#id=USERID
2017-09-30 18:07:20 +02:00
Mike Fährmann
0dedbe759c
enable '--chapter-filter'
...
The same filter infrastructure that can be applied to image URLS now
also works for manga chapters and other delegated URLs.
TODO: actually provide any metadata (currently supported is only
deviantart and imagefap).
2017-09-12 16:19:00 +02:00
Mike Fährmann
47bcf53ec1
implement support for additional unit test result types
...
- "pattern" matches all resulting URLs against the given regex
- "count" allows to specify the amount of returned URLs
2017-08-25 22:01:14 +02:00
Mike Fährmann
f7cdfd4c25
add a simplified version of 'parse_qs'
...
This version only returns a dict of plain string to string key-value
pairs and ignores multiple values for the same query variable.
2017-08-24 20:55:58 +02:00
Mike Fährmann
e61a3a56d1
[hentai2read] fix and update keywords
...
Added the "author" keyword and changed the name of a few others to be
consistent with other manga/chapter extractors.
2017-08-22 15:01:47 +02:00
Mike Fährmann
c45770331a
use 'str.partition()'
...
The (r)partition method is always faster then split() or any other
method that has been replaced in this commit.
2017-08-21 18:29:50 +02:00
Mike Fährmann
017a72f448
[pixiv] improve input validation
2017-08-21 17:53:27 +02:00
Mike Fährmann
dcf42c5e89
[pixiv] add extractor for ranking lists
2017-08-20 20:21:52 +02:00
Mike Fährmann
7aa9fa796a
code cleanup and fixes
2017-07-25 14:59:41 +02:00
Mike Fährmann
f08af03845
Merge branch 'cookies'
2017-07-25 14:04:53 +02:00
Mike Fährmann
b37a62501b
[pixiv] unquote tags
2017-07-12 08:21:29 +02:00
Mike Fährmann
58e95a7487
share extractor and downloader sessions
...
There was never any "good" reason for the strict separation
between extractors and downloaders. This change allows for
reduced resource usage (probably unnoticeable) and less lines
of code at the "cost" of tighter coupling.
2017-06-30 19:38:14 +02:00
Mike Fährmann
44d98e562b
[pixiv] support pixiv.me URLs ( #23 )
2017-06-25 20:21:01 +02:00
Mike Fährmann
b373fe0eea
[pixiv] support shortened URLs and other variants ( #23 )
2017-06-25 17:49:24 +02:00
Mike Fährmann
d3b04076f7
add .netrc support ( #22 )
...
Use the '--netrc' cmdline option or set the 'netrc' config option
to 'true' to enable the use of .netrc authentication data.
The 'machine' names for the .netrc info are the lowercase extractor
names (or categories): batoto, exhentai, nijie, pixiv, seiga.
2017-06-24 12:17:26 +02:00
Mike Fährmann
e365f1d799
[pixiv] rewrite
...
- same functionality, better(?) code quality, easier to extend
- added test for the user-tag functionality
- removed the 'artist-id', 'artist-name' and 'artist-nick'
keywords, which can be replaced with 'user[id]', 'user[name]'
and 'user[account]' respectively
2017-06-04 16:33:36 +02:00
aiasdfd
338f79147f
[pixiv] support tag for user downloads ( #17 )
...
[pixiv] support tag for user downloads
2017-06-04 10:15:36 +02:00
Mike Fährmann
107d29ad8a
improve handling of text:... URLs
...
- don't require // after the colon
- open output files in text mode
2017-05-12 14:10:25 +02:00
Mike Fährmann
cfbf79d788
[pixiv] fix login
2017-05-05 10:38:22 +02:00
Mike Fährmann
4e8587bad4
[pixiv] add support for https://i.pximg.net URLs
2017-04-30 22:54:49 +02:00
Mike Fährmann
4b967fa189
implement and use extractor.config() method
2017-04-25 17:12:48 +02:00
Mike Fährmann
13dc5d72bc
update some extractors to use https
2017-04-20 13:32:40 +02:00
Mike Fährmann
f4aa452bd1
update unit test results
2017-04-14 14:40:36 +02:00
Mike Fährmann
1d46be545c
add login notifications
2017-03-17 09:42:59 +01:00
Mike Fährmann
94e10f249a
code adjustments according to pep8 nr2
2017-02-01 00:53:19 +01:00
Mike Fährmann
4a8d74973c
adjust login methods to a specific style
2017-01-08 17:33:25 +01:00
Mike Fährmann
7952b8d18d
add a few tests expecting exceptions
2016-12-30 01:46:42 +01:00
Mike Fährmann
19c2d4ff6f
remove explicit (sub)category keywords
2016-09-25 14:22:07 +02:00
Mike Fährmann
d7e168799d
consistent extractor naming scheme + docstrings
2016-09-12 10:34:31 +02:00
Mike Fährmann
30d10cf595
[pixiv] fix KeyError if no status is present
2016-08-28 16:49:49 +02:00
Mike Fährmann
effa1084f2
[pixiv] raise NotFoundError instead of failing
2016-08-28 16:21:51 +02:00
Mike Fährmann
85cd7c34b1
[pixiv] match direct-links to images
2016-08-25 20:10:02 +02:00
Mike Fährmann
0e20be5a0a
[exhentai] provide username/password auth
2016-07-23 17:55:46 +02:00
Mike Fährmann
3136278f79
[pixiv] add option to skip ugoiras
2016-07-20 14:19:46 +02:00
Mike Fährmann
d6c06f9efd
add authentication-exception
2016-07-14 14:57:42 +02:00
Mike Fährmann
c9b7db5af5
[pixiv] code cleanup
2016-03-07 17:01:04 +01:00
Mike Fährmann
c8e83f6560
[pixiv] cache login sessions
2016-03-06 21:00:42 +01:00
Mike Fährmann
23a572a20d
[pixiv] update testdata
2016-02-18 15:58:47 +01:00
Mike Fährmann
ba99506c72
more extractor test-cases
2015-12-14 03:00:58 +01:00
Mike Fährmann
f7c47a6018
add subcategories to extractors
2015-11-30 01:11:13 +01:00
Mike Fährmann
1497da07de
remove unused format-strings
2015-11-29 23:41:43 +01:00
Mike Fährmann
a4c880f4c5
[pixiv] add bookmark extractor
2015-11-23 03:25:08 +01:00
Mike Fährmann
daf5fcd416
[pixiv] add user-favorite extractor
2015-11-23 00:23:05 +01:00
Mike Fährmann
20a4782173
[pixiv] get rid of an unnecessry api-call
2015-11-22 03:58:31 +01:00
Mike Fährmann
2a97296e69
[pixiv] add single work/illust extractor
2015-11-22 02:21:02 +01:00
Mike Fährmann
4d56b76aa8
update all other extractors
2015-11-21 04:26:30 +01:00
Mike Fährmann
f5d5684119
[pixiv] reuse api-tokens
2015-10-07 01:09:11 +02:00
Mike Fährmann
c8e0efe1ff
[pixiv] refresh access-token every 50 minutes
2015-10-06 17:29:08 +02:00
Mike Fährmann
539faa0322
remove SequentialExtractor class
2015-10-05 17:15:31 +02:00
Mike Fährmann
3c13548f29
rewrite extractors to use config-module
2015-10-05 15:51:08 +02:00
Mike Fährmann
42b8e81a68
rewrite extractors to use text-module
2015-10-03 15:43:02 +02:00
Mike Fährmann
1998ec9b13
[pixiv] update user-agent to newest version
2015-09-07 13:48:16 +02:00
Mike Fährmann
63992363a6
[pixiv] handle timestamps at the end of urls
2015-06-28 12:36:30 +02:00
Mike Fährmann
d9d88f1448
[pixiv] fix ugoira parsing
2015-05-20 11:06:35 +02:00
Mike Fährmann
68c4306040
[pixiv] transition to pixiv public api
2015-05-14 19:08:20 +02:00
Mike Fährmann
fae1b28c48
[pixiv] fix parsing of illust_ids
2015-04-12 18:05:49 +02:00
Mike Fährmann
8f24329e30
[pixiv] fix http headers and cookies
2015-04-10 17:32:36 +02:00
Mike Fährmann
1cd25b5369
[pixiv] update to new extractor interface
2015-04-10 15:29:09 +02:00
Mike Fährmann
d13608e909
[pixiv] fix parsing of illust-ids
2015-03-27 18:09:30 +01:00
Mike Fährmann
54d40b85fb
[pixiv] load balancing for image servers
2015-01-27 21:51:48 +01:00
Mike Fährmann
deef91eddc
initial commit
2014-10-12 21:56:44 +02:00
