Mike Fährmann
921914141e
[imgbb] improve redirect handling
2020-04-20 23:36:57 +02:00
Mike Fährmann
6cc800aad4
[instagram] add 'post_id' and 'num' metadata fields ( closes #698 )
2020-04-20 22:22:29 +02:00
Mike Fährmann
a3de234e70
[hitomi] add extractor for tag searches ( closes #697 )
2020-04-20 21:55:19 +02:00
Mike Fährmann
456f6e8d05
[nozomi] move '_unpack()' method to global scope
2020-04-20 21:44:16 +02:00
Mike Fährmann
55ac408bdf
[hitomi] fix extraction of galleries without tags
2020-04-20 21:42:14 +02:00
Mike Fährmann
db6685eeae
[aryion] support downloading from folders ( fixes #694 )
2020-04-18 01:25:54 +02:00
Mike Fährmann
fa2952ac55
[furaffinity] add 'following' extractor ( #515 )
2020-04-17 22:18:39 +02:00
Mike Fährmann
9b194520db
[newgrounds] add 'following' extractor ( closes #684 )
2020-04-17 22:17:43 +02:00
Mike Fährmann
6386ee54e1
[deviantart] add extractor info to 'following' results
2020-04-16 23:20:07 +02:00
Mike Fährmann
d5273f9b0c
[hiperdex] update domain to hiperdex.net
2020-04-16 20:39:56 +02:00
Mike Fährmann
08674a91f3
[patreon] fix hash extraction from download URLs ( closes #693 )
...
The old method was assuming every URL path ends with '/1'. For URLs
where this is not the case, the segment containing the post ID was
used as file hash.
2020-04-15 23:28:57 +02:00
Mike Fährmann
a6286bb551
[hiperdex] add 'artist' extractor ( #606 )
2020-04-12 02:32:37 +02:00
Mike Fährmann
291033720a
[hiperdex] fix manga extraction
2020-04-12 02:27:13 +02:00
Mike Fährmann
dfc0557807
[vsco] fix collection extraction
2020-04-11 23:06:29 +02:00
Mike Fährmann
fd438f0d78
update extractor test results
2020-04-11 23:00:42 +02:00
Mike Fährmann
bae1e8ed12
[deviantart] fix JPEG quality replacement pattern
...
'q_\d+' would sometimes also replace something in the 'token' query
parameter, invalidating the URL.
2020-04-11 02:37:06 +02:00
Mike Fährmann
cf4cef3d63
[aryion] adjust 'date' to UTC time
2020-04-11 02:08:05 +02:00
Mike Fährmann
6c531be294
[aryion] fix malformed 'last-modified' headers ( #390 )
2020-04-10 23:08:52 +02:00
Mike Fährmann
dc65f7d8dc
[aryion] use generic download URLs ( #390 )
...
i.e. /g4/data.php?id=…
- get filename & extension from Content-Disposition header
- handle all downloadable file types (docx, swf, etc)
2020-04-10 22:08:45 +02:00
Mike Fährmann
96b78bcf04
[aryion] include path in default directory format ( #390 )
2020-04-10 21:58:46 +02:00
Mike Fährmann
6143050980
[aryion] add gallery and post extractors ( #390 , #673 )
2020-04-08 21:52:51 +02:00
Mike Fährmann
9e7dfc0cfc
[myportfolio] fix extraction of galleries without title
2020-04-08 21:08:05 +02:00
Mike Fährmann
88fca0a172
[mastodon] update OAuth credentials for pawoo.net ( #665 )
2020-04-06 00:50:30 +02:00
Mike Fährmann
4ae8a25567
[mastodon] use 'combine_dict()' to combine extractor info dicts
2020-04-05 21:45:00 +02:00
Mike Fährmann
220c06b86e
[mastodon] handle rate limits
2020-04-05 21:44:00 +02:00
Mike Fährmann
d02f7c1118
improve Extractor.wait()
...
- allow 'until' to be a datetime object
- do "time calculations" with UTC timestamps
- set a default 'reason'
2020-04-05 21:23:05 +02:00
Mike Fährmann
5d7404ab58
[oauth] use the new name for 'DeviantartAPI' ( fixes #670 )
2020-04-04 20:34:47 +02:00
Mike Fährmann
762c758af4
[hiperdex] fix extraction
2020-04-03 21:25:25 +02:00
Mike Fährmann
f9a590f92b
[deviantart] apply HTTP request limits in more places
...
"Request blocked" can also happen on sta.sh and for *any* HTTP
request directed at deviantart.com
2020-04-03 21:21:59 +02:00
Mike Fährmann
2587296deb
[mastodon] add access tokens for mastodon.social and baraag.net
...
(closes #665 )
2020-04-02 22:34:32 +02:00
Mike Fährmann
ff7c0b7eff
[deviantart] handle "Request blocked" errors ( #655 )
...
- add a 2 second wait time between requests to deviantart.com
- catch 403 "Request blocked" errors and wait for 3 minutes until
retrying
2020-04-02 22:14:02 +02:00
Mike Fährmann
c874684f05
[deviantart] retrieve *all* download URLs through OAuth API
...
'/extended_fetch' as well as Deviation webpages now again contain
Deviation UUIDs needed to grab Deviation info through the OAuth API,
meaning cookies are no longer necessary to grab original files.
The only instance were cookies are still needed are scraps marked as
"mature", since those entries are hidden for public users.
(#655 , #657 , #660 )
2020-04-02 22:10:33 +02:00
Mike Fährmann
5c27b25a8f
[deviantart] improve sta.sh extraction
...
Extract all sta.sh items in a single extractor run.
Don't spawn a new StashExtractor for each individual sta.sh item to
preserve the current requests.Session and its opened TCP connections.
2020-04-01 03:17:25 +02:00
Mike Fährmann
e2fc4eaa6f
[deviantart] detect stash folders ( fixes #659 )
2020-04-01 01:59:03 +02:00
Mike Fährmann
c034159701
[piczel] fix extraction for single images
2020-03-31 22:47:23 +02:00
Mike Fährmann
699036ea0c
[weibo] accept status URLs with non-numeric IDs ( #664 )
2020-03-31 22:46:50 +02:00
Mike Fährmann
fe96f99e4b
[hentainexus] reduce line length (flake8) & update test
2020-03-31 22:08:43 +02:00
墨焓
6f81cac8fa
Add metadata to hentainexus: circle, event, title_conventional. ( #661 )
2020-03-31 21:59:02 +02:00
Mike Fährmann
6f911aeb1c
[deviantart] add error message for cloudFront blocks ( #655 )
2020-03-28 21:18:04 +01:00
Mike Fährmann
7499d71d02
[simplyhentai] ignore certificate errors in video test
2020-03-28 21:07:30 +01:00
Mike Fährmann
4203dc0bdc
[mangapark] fix metadata extraction
2020-03-28 03:00:26 +01:00
Mike Fährmann
1b82d36ab2
[deviantart] handle decode errors for extended_fetch results ( #655 )
...
This isn't going to solve the underlying problem, but it should at
least provide the server response when those errors happen.
2020-03-24 20:56:41 +01:00
Mike Fährmann
09f2271528
[35photo] add 'tag' extractor
2020-03-24 02:49:00 +01:00
Mike Fährmann
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor
...
There is still a nice genre overview page (https://35photo.pro/genre/ )
but the individual sub-pages don't list photos anymore
2020-03-24 02:48:25 +01:00
Mike Fährmann
fb846c9ee5
[instagram] reduce line lengths and make flake8 happy
2020-03-23 22:56:43 +01:00
Mike Fährmann
ad2efa8509
[e621] derive from Danbooru extractors ( #651 )
...
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
"tags": "id:<[ID]", avoiding infinite loops with certain
post orders
- bump User-Agent version
2020-03-22 21:08:45 +01:00
Mike Fährmann
9b39e1cd7e
[e621] fix bug in API rate limiting ( #651 )
2020-03-22 14:01:23 +01:00
Mike Fährmann
b607d0ad7f
[twitter] fix typo in 'x-twitter-auth-type' header ( #625 )
2020-03-21 23:11:39 +01:00
Mike Fährmann
2c3b9e1450
[nozomi] support multiple images per post ( #646 )
...
This changes the default filename format as well as archive IDs,
since those assumed that each post would only have one image.
2020-03-19 21:07:31 +01:00
Mike Fährmann
c606d0c854
[instagram] update pattern for user profile URLs
...
Allow for query parameters and fragments,
for example https://www.instagram.com/instagram/?hl=en
2020-03-18 22:24:20 +01:00
Mike Fährmann
2530db3f4d
[mangadex] transform 'date' timestamps to datetime objects
2020-03-18 02:19:19 +01:00
Mike Fährmann
ae2a33243b
[newgrounds] catch general Exceptions
2020-03-18 02:17:43 +01:00
Mike Fährmann
32e36d8f02
[sexcom] replace tests
2020-03-17 22:47:45 +01:00
Mike Fährmann
33b42dc847
[nozomi] sort search results ( fixes #646 )
2020-03-17 22:28:23 +01:00
Mike Fährmann
eaa60a438b
[piczel] fix extraction
...
- manually filter by folder_id
- extract data for single posts from embedded JSON, since the
'/api/gallery/image/<id>' endpoint is no longer available
2020-03-17 17:12:28 +01:00
Mike Fährmann
5bcc7184c9
[danbooru][e621] increase page limits
2020-03-17 15:53:28 +01:00
Mike Fährmann
90d15e3682
[instagram] use 'itertools.chain()'
2020-03-17 15:52:44 +01:00
Leonardo Taccari
160328d21c
[instagram] Add support for user's saved medias ( #644 )
...
* [instagram] Gracefully handle possible 'HttpErrorPage' in _extract_page()
`HttpErrorPage' is returned in shared_data at least when not authenticated or
when trying to fetch other users saved medias
(i.e. `instagram.com/<user>/saved/').
Gracefully handle it by returning nothing.
* [instagram] Add support for user's saved medias
(Please note that this need the user to be authenticated and they can
only see their saved media (not other users ones).)
Close #643 .
* [instagram] Bump copyright year
2020-03-16 21:09:14 +01:00
Mike Fährmann
d3482ace7f
[furaffinity] extract more metadata
...
- views
- favorites
- comments
- rating
- fa_category (since 'category' is already in use)
- theme
- species
- gender
- width
- height
2020-03-13 23:56:55 +01:00
Mike Fährmann
fdd2dd5136
[kabeuchi] add 'user' extractor ( closes #561 )
2020-03-13 16:45:42 +01:00
Mike Fährmann
59edcdc822
[hitomi] restore metadata fields from before f33b13a
...
... and add a 'metadata' option to disable
visiting the gallery page and extracting data from it
if this is not needed.
2020-03-12 23:43:41 +01:00
Mike Fährmann
2d5703c493
[twitter] use a simpler data structure to store cookies in cache
...
Use a dict with name-value pairs instead of an entire
RequestsCookieJar object.
2020-03-12 22:02:12 +01:00
Mike Fährmann
87d4f83597
[newgrounds] make post extraction nonfatal
2020-03-10 01:49:59 +01:00
Mike Fährmann
823fbeaae6
[newgrounds] add 'favorite' extractor ( #394 )
2020-03-10 01:07:09 +01:00
Mike Fährmann
a45fbc38ea
[pixiv] implement 'avatar' option ( #595 , #623 )
2020-03-09 21:18:16 +01:00
Mike Fährmann
a63a376ad2
[mangoxo] fix login
2020-03-08 23:01:51 +01:00
Mike Fährmann
ebc70e87ce
[e621] update to new interface / API endpoints ( closes #635 )
2020-03-06 21:12:58 +01:00
Mike Fährmann
d1cf7ccdb3
[instagram] add 'post_shortcode' metadata field ( #525 )
2020-03-06 15:20:32 +01:00
Mike Fährmann
32df8d06fe
[twitter] add 'bookmark' extractor ( closes #625 )
2020-03-06 01:20:04 +01:00
Mike Fährmann
3fb41c34c8
[bcy] reduce requests to '/item/detail/<id>' ( #613 )
...
The former implementation would try to use the embedded data from
'/item/detail/' pages for every post, even if that wasn't really
necessary.
This commit also fixes some issues with posts only visible to
logged in users.
2020-03-04 01:37:51 +01:00
Mike Fährmann
f33b13aacf
[hitomi] simplify metadata extraction
...
Use the data from https://ltn.hitomi.la/galleries/ <id>.js for both
image URLs and metadata and ignore any gallery or reader pages.
This removes 'artist', 'characters', 'group', and 'parody' metadata
fields since this information is, as for now, only available in
gallery pages.
2020-03-04 01:22:32 +01:00
Mike Fährmann
ce5e2a58fe
[imgbb] update test results
...
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
2020-03-01 20:38:25 +01:00
Mike Fährmann
f117e32910
[danbooru] restore 'popular' functionality
2020-02-29 23:37:53 +01:00
Mike Fährmann
39b48d665b
[hiperdex] use proper name for 'chapter_minor'
2020-02-29 00:18:54 +01:00
Mike Fährmann
8fbbaa54ff
[bcy] fix partial image URLs ( #613 )
...
Images from new posts can have incomplete/partial URLs (1)
without any filename extension when fetching their data from
'/apiv3/user/selfPosts', so now all data gets taken from
'/item/detail/ID' pages.
It is currently unknown how to get the non-watermarked original version
of these images, or if that is possible at all. (2)
Images with a watermark will have their 'filter' metadata field set to
"watermark". For original images this field is an empty string "".
Enabling the 'noop' option will, in addition to the watermarked version,
yield the the '~noop.image' filter version (3),
where 'filter' is set to "noop".
(1) "https://img-bcy-qn.pstatp.com/banciyuan/3ccdff22479c4060aadc86718209b281 "
(2) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~tplv-banciyuan-logo-v3:wqnpnLLlhZLlpKfprZTnjotfCuWNiuasoeWFgyAtIEFDR-eIseWlveiAheekvuWMug==.image "
(3) "https://p1-bcy.byteimg.com/img/banciyuan/3ccdff22479c4060aadc86718209b281~noop.image "
2020-02-28 22:57:10 +01:00
Mike Fährmann
86c00f9e66
[danbooru] move extractor logic from booru.py
2020-02-28 22:53:45 +01:00
Mike Fährmann
1d4a369ea2
update extractor test results
2020-02-27 22:15:40 +01:00
Mike Fährmann
7625912b31
[piczel] improve and update
...
- fix tag names
- fix a bug in _pagination()
- parse datetime in 'created_at' as 'date'
- rewrite main loop
- replace user profile test
2020-02-27 22:13:12 +01:00
Mike Fährmann
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
2020-02-25 22:55:11 +01:00
Mike Fährmann
2a4f227e08
warn about expired cookies
2020-02-25 00:34:42 +01:00
Mike Fährmann
4e361b3008
add tests for specific datetime values
2020-02-23 16:48:30 +01:00
Mike Fährmann
80ecb99089
[hitomi] fix extraction
2020-02-22 22:07:21 +01:00
Mike Fährmann
247c9e1416
[vsco] update gallery URL pattern
2020-02-22 21:39:31 +01:00
Mike Fährmann
19ae6f3fc4
update test results
...
- twitter:
Don't test the whole kwdict, only the actual content, since the
keyword hash changes whenever that user changes his display name.
- khinsider:
Download host changed
2020-02-22 03:25:32 +01:00
Mike Fährmann
cc5079c844
[hiperdex] add chapter and manga extractors ( closes #606 )
2020-02-22 03:09:29 +01:00
Mike Fährmann
64bdec8430
[deviantart] check availability of intermediary URLs ( fixes #609 )
2020-02-21 03:10:53 +01:00
Mike Fährmann
5607dd3646
[hitomi] follow multiple redirects
2020-02-20 18:22:13 +01:00
Mike Fährmann
765b2a0527
[hentaihand] add extractors ( closes #605 )
2020-02-19 21:55:47 +01:00
Mike Fährmann
d94215d119
[tumblr] replace '-' with ' ' in tag searches ( fixes #611 )
...
To search for tags with actual minus signs in them
(there shouldn't be too many,) manually replace those
with url-encoded minus characters ('-' -> '%2d')
before inputting them into gallery-dl:
https://s679874.tumblr.com/tagged/tag-with-minus
->
https://s679874.tumblr.com/tagged/tag%2dwith%2dminus
2020-02-17 23:29:13 +01:00
Mike Fährmann
e6cd49e78b
update extractor test results
2020-02-16 21:48:46 +01:00
Mike Fährmann
5d9437b398
[vsco] skip "invalid" entities
2020-02-15 23:49:44 +01:00
Mike Fährmann
650f2b6d58
[furaffinity] accept sfw.furaffinity.net URLs ( closes #608 )
...
Just as an alias for regular URLs with no extra content filtering.
2020-02-15 22:47:12 +01:00
Mike Fährmann
74e684e828
[twitter] change default value for 'videos' to 'true'
...
Every other 'videos' option defaulted to 'true', except Twitter.
2020-02-14 01:03:42 +01:00
Mike Fährmann
c7cf9dd111
[furaffinity] support classic layout ( #284 )
2020-02-12 21:39:43 +01:00
Mike Fährmann
138135c190
[furaffinity] add extractors ( #284 )
2020-02-11 19:51:24 +01:00
Mike Fährmann
b9c574bd1d
[patreon] log skipped files ( #590 )
2020-02-11 19:01:07 +01:00
Mike Fährmann
80ea9104b8
[8kun] adjust URL pattern
2020-02-11 19:00:13 +01:00
Mike Fährmann
ce26070231
[pixiv] reduce calls to '/user/detail'
2020-02-09 13:54:58 +01:00
Mike Fährmann
da0d5f6092
[oauth] add 'port' option ( #604 )
2020-02-09 13:45:44 +01:00
Mike Fährmann
719b63d0ca
[bcy] add user and post extractors ( #592 )
2020-02-09 02:37:14 +01:00
Mike Fährmann
6426e3efc7
[khinsider] fix and improve metadata extraction
2020-02-07 18:20:38 +01:00
Mike Fährmann
b7eb6cecbb
[pixiv] handle tags at the end of new bookmark URLs
2020-02-06 23:42:13 +01:00
Mike Fährmann
109f6c8685
[patreon] filter duplicate files per post ( #590 )
2020-02-05 23:38:24 +01:00
Mike Fährmann
b38cf59711
[sexcom] fix image URLs & parse 'date' fields
2020-02-04 22:52:00 +01:00
Mike Fährmann
1f4c9c5f9d
[8kun] add thread and board extractors ( closes #582 )
2020-02-04 22:50:31 +01:00
Mike Fährmann
facc5daa6d
[twitter] force old login page layout ( fixes #584 , fixes #598 )
2020-02-02 17:24:53 +01:00
Mike Fährmann
d1de7dc296
[hitomi] implement workaround for "broken" redirects
...
Some galleries redirect to a new "version" with different gallery id.
This new version might not be available any more, but the /reader/
page for the original gallery id can still work.
2020-02-02 17:24:23 +01:00
Mike Fährmann
40fe062851
[pixiv] fix user id for bookmarks API calls ( closes #596 )
2020-02-01 01:48:46 +01:00
Mike Fährmann
91aaaf1a9e
[pixiv] add 'rating' metadata field ( #595 )
...
A human-friendlier representation of 'x_restrict'
2020-02-01 01:36:06 +01:00
Mike Fährmann
dff33b260c
[reddit] add 'videos' option
2020-01-31 23:45:02 +01:00
Mike Fährmann
2ad43618cc
[piczel] fix extraction
2020-01-31 15:46:21 +01:00
Mike Fährmann
cf7a67d67f
[yaplog] remove module
...
Yaplog! ended its service on 2020-01-31
2020-01-31 12:56:54 +01:00
Mike Fährmann
e0dd073ce0
[twitter] replace embedded tweet test
...
the old one was deleted
2020-01-31 12:51:55 +01:00
Mike Fährmann
ec36df4851
[deviantart] fix video extraction from 'extended_fetch' results
...
DeviantArt is now serving videos from wixmp servers (1), instead of
the former film00.deviantart.com (2), even though those URLS are still
functional.
They seem to also have re-encoded those videos. The 10 MB 1080p video
from (2) is now only available in 720p at ~20 MB (with a higher
bitrate, but still …). Other videos are still available in 1080p, but
not this one for some reason.
(Changing the '720p' in (1) to '1080p' doesn't work.)
(1) https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/9feaa2c9-1baf-4fc2-84f7-f3384b34cefe/d5gxnb5-282a2e9a-b552-40ff-8542-b3c5eed823f5.720p.a837d7cec12c41be8ca2ee53152cea3a.mp4
(2) https://film00.deviantart.net/4c1d/v/mp4/2012/279/d/1/_video____brushes_i_use_in_paint_tool_sai_by_chi_u-d5gxnb5.mp4
2020-01-30 18:02:21 +01:00
Mike Fährmann
48be2266ed
[deviantart] better error message for 'extended_fetch' ( #585 )
2020-01-30 15:25:33 +01:00
Mike Fährmann
71851a6241
[pixiv] update URLs of followed users to the new format
2020-01-30 15:17:42 +01:00
Mike Fährmann
d086f30b42
[reddit] restore archive keys for i.redd.it images
2020-01-29 22:12:55 +01:00
Mike Fährmann
56f1c96168
implement 'parent-directory' option ( #551 )
2020-01-29 18:32:37 +01:00
Mike Fährmann
ae07f92f7e
[reddit] rewrite extractor logic ( closes #551 )
...
Handle images and videos hosted on Reddit "natively",
allowing them to use reddit-specific metadata to build directory
and file names.
2020-01-29 17:57:25 +01:00
Mike Fährmann
2852691d78
[paheal] replace test URL
...
searching for 'k-on' doesn't yield any results anymore
2020-01-27 22:19:41 +01:00
Mike Fährmann
2a9be48511
improve util.load/save_cookiestxt() and add tests
...
- take a file object as argument instead of an filename
- accept whitespace before comments (" # comment")
- map expiration "0" to None and not the number 0
2020-01-25 23:02:15 +01:00
Mike Fährmann
e35c2ea1a6
[weibo] use youtube-dl to download from m3u8 manifests
2020-01-24 23:39:34 +01:00
Mike Fährmann
6703b8a86b
[blogger] implement video extraction ( closes #587 )
2020-01-24 23:37:23 +01:00
Mike Fährmann
c1a6862863
implement functions to load/save cookies.txt files ( closes #586 )
...
The methods of the standard libraries' MozillaCookieJar have
several shortcomings (#HttpOnly_ cookies, 0 expiration timestamps, etc.)
and require construction of an ultimately pointless CookieJar object.
2020-01-21 21:59:36 +01:00
Mike Fährmann
25d5ec4ff3
[twitter] add option to extract TwitPic embeds ( #579 )
2020-01-18 21:31:29 +01:00
Mike Fährmann
32d7195d08
[pinterest] improve detection of invalid pin.it links
2020-01-18 21:06:44 +01:00
Mike Fährmann
174117f827
allow multiple hashes for content tests
2020-01-18 00:19:35 +01:00
Alice
f498a9057f
[twitter] Fix stop before real end ( #573 )
...
* [twitter] Fix stop before real end
Fix for https://github.com/mikf/gallery-dl/issues/544 . Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.
* [twitter] Fix stop before real end (update)
2020-01-14 12:24:30 +01:00
Mike Fährmann
8bb32ee188
[hitomi] fix image URLs
2020-01-14 12:04:48 +01:00
Mike Fährmann
bd5ce9855c
allow GalleryExtractors to set URL-independent extensions
2020-01-14 11:53:32 +01:00
Mike Fährmann
af42c75152
[mangadex] revert domain to 'mangadex.org'
2020-01-14 09:49:57 +01:00
Mike Fährmann
e89413da22
update test results
2020-01-12 21:41:06 +01:00
Mike Fährmann
33a6e0ac6e
[hentaifoundry] extract more metadata ( closes #565 )
2020-01-11 23:22:50 +01:00
Mike Fährmann
5cac79c3d9
[erolord] remove extractor
2020-01-10 14:40:30 +01:00
Mike Fährmann
b9cbf932b4
[pixiv] update URL patterns ( fixes #568 )
...
Pixiv now uses new URLs for
- user profiles and illustration listings:
- https://www.pixiv.net/en/users/ <ID>
- https://www.pixiv.net/en/users/ <ID>/artworks
- bookmarks:
- https://www.pixiv.net/en/users/ <ID>/bookmarks/artworks
2020-01-10 13:38:09 +01:00
Mike Fährmann
988cc2ec23
[mangadex] change domain to mangadex.cc ( closes #559 )
2020-01-05 21:21:40 +01:00
Mike Fährmann
f8e137d6b4
[deviantart] show warning about private deviations only once
...
… per call to '_pagination()'
2020-01-05 17:43:05 +01:00
Mike Fährmann
939fec8ecd
[deviantart] match new search/popular URLs ( closes #538 )
2020-01-05 17:32:06 +01:00
Mike Fährmann
09cc88b715
[deviantart] match '/favourites/all' URLs ( closes #555 )
2020-01-05 13:59:35 +01:00
Mike Fährmann
3811fd8a25
fix time formatting for Python 3.4 and 3.5
...
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
2020-01-05 00:47:10 +01:00
Mike Fährmann
43ab9572b4
[twitter] handle API rate limits ( #526 )
2020-01-04 23:46:29 +01:00
Mike Fährmann
569747a78d
implement extractor.wait()
2020-01-04 23:42:07 +01:00
Mike Fährmann
5532e9c158
[twitter] handle quoted tweets ( #526 )
...
… and categorize them as retweets
2020-01-04 21:26:55 +01:00
Mike Fährmann
0b4cb8e57a
[mangahere] send 'isAdult' cookie ( fixes #556 )
2020-01-04 21:25:35 +01:00
Mike Fährmann
1afb91363c
[imagefap] generalize URL patterns and add tests ( #552 )
2020-01-02 14:26:18 +01:00
Xope Totec
f701e9f33a
Handle beta.imagefap.com URLs ( #552 )
2020-01-02 14:22:00 +01:00
Mike Fährmann
ce54b8c04c
let extractors opt-out of cookie option usage
...
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
2020-01-01 21:12:37 +01:00
Mike Fährmann
5ad92fc196
[newgrounds] fix tags metadata extraction
2020-01-01 16:06:58 +01:00
Mike Fährmann
82f7f4172a
update test results
2020-01-01 16:05:38 +01:00
Mike Fährmann
1f2a69f3c5
add '_extractor' information to redirect results
2019-12-29 23:37:34 +01:00
Mike Fährmann
a27f43dad1
[pixiv] wait and retry after rate limit error ( closes #535 )
2019-12-28 22:06:58 +01:00
Mike Fährmann
6b373cb7e2
[exhentai] restrict default directory name length ( #545 )
2019-12-27 21:29:00 +01:00
Mike Fährmann
b347bf68c7
[deviantart] add extractor for followed users ( #515 )
2019-12-27 21:27:39 +01:00
Mike Fährmann
c0f391a4e2
[pixiv] support listing followed users ( #515 )
2019-12-26 23:42:42 +01:00
Mike Fährmann
896896a490
[twitter] fix URLs forwarded to youtube-dl ( closes #540 )
...
Since commit 3bba763
2019-12-25 17:28:55 +01:00
Mike Fährmann
1e2713b895
[artstation] fix search result pagination ( closes #537 )
2019-12-25 17:26:37 +01:00
Mike Fährmann
bf3df3d0b0
[directlink] send Referer headers ( closes #536 )
2019-12-25 17:17:07 +01:00
Mike Fährmann
9be7ff600e
[imagetwist] replace test image
...
the old one expired, it seems
2019-12-22 22:13:13 +01:00
Mike Fährmann
66905b1664
[foolslide] add fallback for chapter data extraction
2019-12-21 20:44:03 +01:00
Mike Fährmann
48e42e73fb
[reddit] change default value for 'comments' to '0'
2019-12-20 16:54:59 +01:00
Mike Fährmann
9c0928457a
[reddit] fix errors with 't1_…' submissions
2019-12-20 16:49:44 +01:00
Mike Fährmann
bf658fd84b
[vsco] implement 'videos' option
2019-12-19 18:34:19 +01:00
Mike Fährmann
95c90722ee
[instagram] implement 'videos' option ( closes #521 )
2019-12-19 18:34:06 +01:00
Mike Fährmann
d0920e84e9
update test results
2019-12-19 18:33:48 +01:00
Mike Fährmann
8c11e81c9f
Merge commit '63e6993716db8d8bedfb7b0d445c7161493046b6'
2019-12-16 17:25:23 +01:00
Mike Fährmann
63e6993716
merge 'bypost' functionality into metadata postprocessor
2019-12-16 17:19:23 +01:00
Mike Fährmann
31a29835ff
[realbooru] simplify extractors and update tests ( #514 )
2019-12-16 16:06:50 +01:00
The Oddball
9a4ce20b8e
[realbooru] Add Realbooru extractor ( #514 )
2019-12-16 15:49:30 +01:00
Mike Fährmann
72b8fbfbad
[instagram] make post-page extraction nonfatal
2019-12-15 22:56:27 +01:00
Mike Fährmann
922b8a9595
[weibo] raise NotFoundError for unavailable/deleted statuses
2019-12-14 22:10:02 +01:00
Mike Fährmann
0cd157300e
[patreon] fix regex pattern for posts
...
The previous one would match the first number in the URL slug as
post ID, which would fail for posts with numbers in their title.
2019-12-14 22:08:49 +01:00
Mike Fährmann
fe19e233f3
[xvideos] improve
...
- derive from GalleryExtractor
- match '…-channels' URLs
- "better" metadata structure
2019-12-13 19:03:56 +01:00
Mike Fährmann
d3e44e899d
raise NotFoundErrors for 404 responses in GalleryExtractors
2019-12-13 18:42:04 +01:00
Mike Fährmann
a4dd8b3dab
improve _check_cookies()
...
Only loop over all cookies once instead of calling
cookiejar._find() for each cookie name.
2019-12-13 15:51:20 +01:00
Mike Fährmann
76e60d10a6
[patreon] raise proper exception if creator/post doesn't exist
2019-12-12 01:14:32 +01:00
Mike Fährmann
9e63804347
[patreon] make retrieving user info nonfatal ( #508 )
...
… and fall back to the included data if an error occurs.
2019-12-12 00:31:35 +01:00
Mike Fährmann
964dc57286
[vsco] improve image resolutions
...
https://im.vsco.co/ URLs redirect to the appropriate CDN server
and occasionally insert a '/1200x1600/' into the image path,
limiting image dimensions.
This commit constructs redirect targets out of the given
im,vsco.co URLs without sending extra HTTP requests
and without any "builtin" resolution restrictions.
2019-12-12 00:27:45 +01:00
Mike Fährmann
0629fe8fa4
[vsco] fix user profile extraction … again
...
Given the pattern from last time, collections will also change
in due time and use cursor-based pagination.
2019-12-11 23:57:56 +01:00
Mike Fährmann
ab17ea9632
[deviantart] only print warning if 'original' is enabled
2019-12-11 21:58:43 +01:00
Mike Fährmann
2188db6284
[gelbooru] fix non-API tag extraction
2019-12-10 21:31:55 +01:00
Mike Fährmann
c4702ec9b6
simplify some logging calls
2019-12-10 21:30:08 +01:00
Gio
c0b9ad678d
Separate metadata from handle_url into handle_metadata, commenting
2019-12-09 16:02:15 -06:00
Mike Fährmann
c9ef1b21c3
[patreon] get partial user info without /api/user/<id> ( #507 )
...
It's a lot less data, but doesn't invoke any additional
HTTP requests with potential Cloudflare CAPTCHAs.
2019-12-09 20:38:05 +01:00
Mike Fährmann
0ab9bb1721
[4chan] add extractor for entire boards ( closes #510 )
2019-12-09 20:36:05 +01:00
Gio
cfc70a97ab
Added an additional channel for downloading the metadata of an entire post or gallery.
2019-12-09 00:56:27 -06:00
Mike Fährmann
15f9bb3d14
add option to disable pyOpenSSL usage ( #508 )
...
(pyOpenSSL is now disabled by default)
2019-12-08 21:21:00 +01:00
Mike Fährmann
c8e99e3b3b
[deviantart] fix crash on missing "token" field ( #505 )
2019-12-07 23:38:36 +01:00
Mike Fährmann
6ed2c7823c
[deviantart] disable original downloads if no cookies set
...
For 'deviation' and 'scraps' extractors only, since original file
downloads for those two will always fail with a 404 Not Found
when not logged in.
2019-12-07 22:39:30 +01:00
Mike Fährmann
50deab5265
[deviantart] fix URL generation from /extended_fetch results
...
(closes #505 )
2019-12-07 22:07:55 +01:00
Mike Fährmann
1f209da4c0
[pixiv] match new search URLs ( closes #507 )
2019-12-06 21:38:38 +01:00
Mike Fährmann
e17907ee2a
change default value of 'cookies-update' to 'true'
2019-12-05 23:43:49 +01:00
Mike Fährmann
07dafad26d
[twitter] attempt to fix infinite loops ( #499 )
...
(Hopefully this doesn't break anything else)
2019-12-03 22:55:29 +01:00
Mike Fährmann
71acbdabf4
[2chan] fix metadata extraction
2019-12-03 17:01:11 +01:00
Mike Fährmann
c0a1241648
[livedoor] force https:// for image URLs
2019-12-03 16:14:05 +01:00
Mike Fährmann
6e23c0da09
[imgur] add extractor for subreddit links ( closes #500 )
2019-12-02 23:44:13 +01:00
Mike Fährmann
372ffe95ee
[oauth] adjust Flickr redirect URI ( fixes #503 )
...
Flickr now automatically forces https:// for all redirect URIs.
2019-12-02 22:24:20 +01:00
Mike Fährmann
004812258d
[hentaifox] fix extraction
2019-12-02 22:21:45 +01:00
Mike Fährmann
e2710702d4
fix Cloudflare bypss
2019-12-01 01:07:24 +01:00
Mike Fährmann
8759403f37
[plurk] add delay between comment requests
2019-12-01 01:03:31 +01:00
Mike Fährmann
a28552fd19
update test results
...
- hbrowse: one tag got removed
- mangoxo: gallery changed owner
- photobucket: ?, but photo still downloads
2019-11-30 23:59:32 +01:00
Mike Fährmann
dcaa3d01bd
[imagefap] adapt to new image URL format
2019-11-30 23:48:02 +01:00
Mike Fährmann
e62c209ca0
[nijie] fix 'date' parsing
2019-11-30 23:08:21 +01:00
Mike Fährmann
3bba763ab9
[twitter] improve
...
- update metadata structure
- combine all user… entries into their own dict
- let 'user' always specify the Timeline owner
- add 'author' entry that specifies the original Tweet author
- create directories per post (closes #491 )
- fix username issues with /i/web/ URLs
2019-11-30 22:30:37 +01:00
Mike Fährmann
db35c3b581
[directlink] separate filenames from paths
...
With this, all default filename formats specify an '{extension}'
and PathFormat.set_extension() reliably works for all files.
2019-11-28 23:50:00 +01:00
Mike Fährmann
41a3169c67
[foolfuuka] use '{extension}' in default filename format
2019-11-28 23:12:48 +01:00
Mike Fährmann
e9aed62c91
[imgur] unescape image titles
2019-11-28 22:13:24 +01:00
Mike Fährmann
2c332edaad
[plurk] fix comment pagination
2019-11-27 19:39:56 +01:00
Mike Fährmann
a3fa45bbb1
[behance] get images from 'media_collection' modules
2019-11-27 01:04:33 +01:00
Mike Fährmann
359c3bc1c5
[deviantart] revert to getting download URLs from OAuth API
...
This commit (partially) reverts 27b5b2494eb7c6a437e78#488 )
TODO:
- "native" login with --username and --password
- better handling of internally stored cookies
2019-11-26 23:29:46 +01:00
Mike Fährmann
42b9633c7e
update test results
2019-11-26 23:27:15 +01:00
Mike Fährmann
b28bd1c73e
[bobx] set generated session cookie ( closes #482 )
...
This reverts commit 490831f
2019-11-25 20:04:11 +01:00
Mike Fährmann
ae09f87602
improve SharedConfigMixin config lookups
2019-11-25 18:31:38 +01:00
Mike Fährmann
f5604492c3
update interface of config functions
2019-11-24 00:42:28 +01:00
Mike Fährmann
4ca883c66f
[smugmug] replace test for custom URLs
...
The old one (http://www.creativedogportraits.com/ ) is empty and/or
no longer handled by SmugMug.
2019-11-22 23:25:55 +01:00
Mike Fährmann
d45fabb79d
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
Mike Fährmann
ea80dadd09
[deviantart] restore archive keys
...
Commit 9fdc5e7
2019-11-21 17:00:08 +01:00
Mike Fährmann
ea094692c8
[vsco] fix collection extraction ( #480 )
2019-11-20 22:06:23 +01:00
Mike Fährmann
490831f84a
[bobx] "fix" image download URLs
...
Access to original images got restricted to (paid) members only.
All that's publicly accessible now are essentially preview pictures.
2019-11-20 21:59:37 +01:00
Mike Fährmann
978cb03f81
update misc test results
...
- Livedoor now uses https:// for its image URLs
- Instagram image URLs got simplified
2019-11-20 21:45:48 +01:00
Mike Fährmann
fca87974fe
[sexcom] fix video downloads by sending specific Referer headers
2019-11-19 23:52:34 +01:00
Mike Fährmann
edc080468d
[instagram] make 'video_url' fields optional ( fixes #479 )
...
[ci skip]
2019-11-19 11:18:43 +01:00
Mike Fährmann
9fdc5e74cb
[deviantart] ensure consistent username capitalization ( #455 )
...
The 'username' field was capitalized in a very inconsistent manner:
Either all lowercase, or as given by the input URL, or with the
"original" capitalization, depending on the extractor used among
other things.
Now usernames use their original capitalization for all extractors.
('UserName' instead of 'username' or 'uSeRnAmE')
2019-11-18 22:09:58 +01:00
Mike Fährmann
b1f0609de5
[newgrounds] rewrite ( #394 )
...
- restructure extractor hierarchy
- extract more metadata
- extract videos without youtube-dl
- be more resilient to errors
TODO:
- favorites
- games, but that might be near impossible for non-flash titles
2019-11-18 21:13:33 +01:00
Mike Fährmann
3ece3976ae
[newgrounds] implement login support ( #394 )
2019-11-16 23:45:32 +01:00
Mike Fährmann
3a07c06865
[newgrounds] update
...
- create directory per post
- rename variables and methods
2019-11-14 23:17:14 +01:00
Mike Fährmann
5513b66eb0
[vsco] fix user profile extraction
2019-11-12 23:36:48 +01:00
Mike Fährmann
abfcb356fc
[flickr] support 3k, 4k, 5k, and 6k photo sizes ( closes #472 )
2019-11-10 17:52:51 +01:00
Mike Fährmann
521fcd2eb9
[imgbb] fix error in galleries without user info ( closes #471 )
2019-11-10 17:10:51 +01:00
Mike Fährmann
8061263d4c
[imgbb] improve pagination logic
...
- avoid unnecessary API calls for small or empty galleries
- combine duplicate code
2019-11-10 17:07:27 +01:00
Mike Fährmann
da6789b2b0
disable unique archive id checks for some tests
...
- same image twice in a livedoor blog post
- unreliable results for related pinterest items
2019-11-10 17:04:51 +01:00
Mike Fährmann
b0197098e6
[imgur] get title from webpage if missing in API response
...
(closes #467 )
2019-11-07 21:10:04 +01:00
Mike Fährmann
dd5d2b2eac
[deviantart] add user profile extractor ( #377 , #419 )
2019-11-07 18:29:49 +01:00
Mike Fährmann
a437e78620
[deviantart] minimize cookie usage during scraps extraction
...
(#445 )
2019-11-05 21:55:13 +01:00
Mike Fährmann
1a197d2195
store the original cookiejar as Extractor._cookiejar
2019-11-05 21:53:22 +01:00
Mike Fährmann
de83ae4576
make 'method' argument of Extractor.request keyword-only
2019-11-05 17:28:09 +01:00
Mike Fährmann
4325695d74
[luscious] expand GraphQL queries
2019-11-04 21:17:22 +01:00
Mike Fährmann
94dbdbf506
[nijie] change default filename format
...
… to be consistent with Pixiv filenames
2019-11-04 20:47:38 +01:00
Mike Fährmann
c18fadc221
[instagram] extract videos without youtube-dl ( #391 )
2019-11-03 14:02:56 +01:00
Mike Fährmann
f15eedb634
[sexcom] set Referer header for file downloads ( closes #464 )
2019-11-03 13:27:58 +01:00
Mike Fährmann
2a3bd4e3c7
rename extractor classes starting with a digit
2019-11-02 20:42:09 +01:00
Mike Fährmann
b3b9da6d74
[photobucket] replace test URL
...
The other user deleted all of is images.
2019-11-02 20:17:08 +01:00
Mike Fährmann
64786363be
[4chan] simplify
...
- remove 'chan.py'
- slight adjustments to directory and filenames
2019-11-02 20:11:21 +01:00
Mike Fährmann
557e2c018b
[8chan] remove module
2019-11-02 20:06:47 +01:00
Mike Fährmann
e14782a948
[instagram] simplify graphql extraction for post pages
2019-11-01 22:08:25 +01:00
Mike Fährmann
c01ff78467
[twitter] extend 'videos' option to force extraction with ytdl
...
(closes #459 )
2019-11-01 22:06:07 +01:00
Mike Fährmann
f8ac67ce50
[hitomi] extend URL pattern + follow redirects
2019-11-01 21:40:10 +01:00
Mike Fährmann
e877ca97c3
[naver] adjust directory names and metadata structure
2019-10-31 16:53:48 +01:00
Mike Fährmann
702f2fbd1f
[issuu] add publication and user extractors ( #413 )
2019-10-31 16:52:57 +01:00
Mike Fährmann
8361d874d7
[hitomi] fix extraction
2019-10-29 16:23:20 +01:00
Mike Fährmann
5fa6ff04dd
[instagram] extract '__additionalDataLoaded' ( #391 )
...
The '_sharedData' of Post pages is missing its 'graphql' part for
logged in users. This data is now included in the parameters of a
function call to '__additionalDataLoaded(...)'
And, of course, video extraction with youtube-dl broke because of
this change as well.
2019-10-29 16:00:31 +01:00
