286 Commits

Author	SHA1	Message	Date
Mike Fährmann	2fb85178da	[deviantart] add 'external' option (#302) ... If a description is available, this will extract URLs from the description text and try to find Extractors for them.	2019-06-06 18:53:50 +02:00
Mike Fährmann	096009367b	[xhamster] add gallery & user extractor (#281)	2019-06-05 11:11:51 +02:00
Mike Fährmann	d33f5a7423	[wallhaven] rewrite ... - use API - remove login support, add 'api-key' option - remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric IDs that can't be translated to the new ID system - support direct links to wallpapers	2019-05-31 14:53:02 +02:00
Mike Fährmann	f1893b2b5b	[deviantart] add 'folders' option (#276)	2019-05-30 17:28:12 +02:00
Mike Fährmann	c849574def	[keenspot] add comic extractor (#223) ... Doesn't work for - http://brawlinthefamily.keenspot.com/ - http://flipside.keenspot.com/ - http://lastblood.keenspot.com/ - http://mysticrevolution.keenspot.com/ - http://porcelain.keenspot.com/ - http://twokinds.keenspot.com/ yet, because of custom layouts.	2019-05-28 21:34:38 +02:00
Mike Fährmann	bcd1801aa8	[sankakucomplex] add 'tag' extractor (#258)	2019-05-27 23:57:44 +02:00
Mike Fährmann	74c2415138	[sankakucomplex] move article extractor to its own module (#258)	2019-05-27 23:49:23 +02:00
Mike Fährmann	4465a3ea68	[kissmanga][readcomiconline] add 'captcha' option (#279) ... to configure how to handle CAPTCHA page redirects: - either interactively wait for the user to solve the CAPTCHA - or raise StopExtraction like before	2019-05-27 22:24:48 +02:00
Mike Fährmann	1e3e15c4f3	[sankaku] add article extractor (#258)	2019-05-26 17:42:36 +02:00
Mike Fährmann	e82cadac61	[patreon] add extractors (#226)	2019-05-17 10:47:22 +02:00
Mike Fährmann	4891f4a328	[hentainexus] add search extractor (#256)	2019-05-16 23:55:30 +02:00
Mike Fährmann	9890bfdf23	[flickr] improve code and metadata ... - simplify pagination - add more metadata and slightly change its structure - convert suitable values to int or list - move keys from ["photo"] to the base level - proper video support (#246) - rename method and variable names to better fit with other extractors	2019-05-14 22:10:50 +02:00
Mike Fährmann	ba8eb1ffec	[hentainexus] add gallery extractor (#256)	2019-05-12 23:59:41 +02:00
Mike Fährmann	0318c610dc	[sexcom] add extractor for search results (#147)	2019-04-24 22:10:01 +02:00
Mike Fährmann	a247c94c34	[sexcom] add pin and board extractors (#147)	2019-04-24 22:09:19 +02:00
Mike Fährmann	70be494161	[plurk] add a 'comments' options (#212)	2019-04-14 22:12:46 +02:00
Mike Fährmann	0b2ff406f6	[plurk] add timeline- and post-extractors (#212)	2019-04-14 21:48:38 +02:00
Mike Fährmann	e7cd5510d5	[pixnet] add extractors (closes #177) ... for: - users/blogs: http://albertayu773.pixnet.net/ - folders: https://albertayu773.pixnet.net/album/folder/1405768 - sets : https://albertayu773.pixnet.net/album/set/15078995 - photos : https://albertayu773.pixnet.net/album/photo/159443828	2019-04-11 19:27:02 +02:00
Mike Fährmann	d9b94a585d	[mangoxo] add login support (#184) ... A very recent change: It is now only possible to see more than the first 5 images of an album if you are logged in.	2019-04-10 18:55:25 +02:00
Mike Fährmann	e730fc9045	[twitter] add login support (#214)	2019-04-09 09:27:49 +02:00
Mike Fährmann	35919a9bb8	[livedoor] add blog- and post-extractors (#190)	2019-04-06 16:27:48 +02:00
Mike Fährmann	2ff043edfa	[yaplog] add user- and post-extractors (#190)	2019-04-04 17:56:56 +02:00
Mike Fährmann	790f15a56f	[photobucket] use HTTPS	2019-04-03 18:30:45 +02:00
Mike Fährmann	6da665f32e	[mangoxo] add album- and channel-extractors (closes #184)	2019-04-03 07:55:51 +02:00
Mike Fährmann	c70b21248d	[wikiart] add extractors (#179) ... for - artists: https://www.wikiart.org/en/thomas-cole - artist-listings: https://www.wikiart.org/en/artists-by-century/12 - artwork-listings: https://www.wikiart.org/en/paintings-by-media/grisaille	2019-04-02 17:34:57 +02:00
Mike Fährmann	bc2020e86c	release version 1.8.1	2019-03-29 17:37:11 +01:00
Mike Fährmann	369eb66125	consistently use '*' for rst lists	2019-03-28 16:21:41 +01:00
Mike Fährmann	c7a6b0ed90	[deviantart] add 'metadata' option (#189)	2019-03-21 14:49:42 +01:00
Mike Fährmann	8d96a8ce4c	[500px] add user-, gallery-, and image-extractors (#185)	2019-03-20 17:32:36 +01:00
Mike Fährmann	6277a739e4	[35photo] add user-, genre-, and image-extractors (#162)	2019-03-18 01:11:30 +01:00
Mike Fährmann	12482553bd	update links to youtube-dl	2019-03-13 22:03:02 +01:00
Mike Fährmann	6f57d44ec2	[seaotterscans] remove extractor ... http://seaotterscans.com/ now redirects to their MangaDex profile	2019-03-13 22:02:45 +01:00
Mike Fährmann	25aaf55514	[smugmug] improve format selection (closes #183) ... - use original image if available - support video formats - remove user info for ImageExtractor (it is no longer possible to get image owner information for a single image)	2019-03-10 15:20:35 +01:00
Mike Fährmann	976ccb267f	[myportfolio] combine gallery and user extractors ... An URL alone isn't good enough to distinguish between a gallery or a gallery-listing, so the new extractor decides what to do based on the page's content.	2019-03-06 19:45:01 +01:00
Mike Fährmann	9c0e2f294b	[shopify] add generic collection and product extractors (#175) ... with fashionnova.com as a default domain	2019-03-05 22:33:37 +01:00
Mike Fährmann	13e0f2a78f	[deviantart] add 'scraps' extractor (closes #168)	2019-03-01 14:13:34 +01:00
Mike Fährmann	280531c8ff	[pururin] add gallery extractor (closes #174)	2019-02-25 14:54:57 +01:00
Mike Fährmann	3159dd79d5	[seiga] use HTTPS	2019-02-21 22:51:11 +01:00
Mike Fährmann	ea6fbfdd39	simplify build_supportedsites.py	2019-02-21 14:29:19 +01:00
Mike Fährmann	22d3a2fcc8	[artstation] add extractor for artwork listings (#80) ... like https://www.artstation.com/artwork?sorting=latest or https://www.artstation.com/artwork?sorting=picks	2019-02-18 12:45:44 +01:00
Mike Fährmann	937a802b49	[dynastyscans] add extractors for images and image searches ... (closes #163)	2019-02-18 12:25:52 +01:00
Mike Fährmann	19860655a3	[weibo] add 'user' and 'status' extractors	2019-02-17 18:18:31 +01:00
Mike Fährmann	ae353ed3b0	provide "extractor" and "job" keys for logging output ... This allows for stuff like "{extractor.url}" and "{extractor.category}" in logging format strings. Accessing 'extractor' and 'job' in any way will return "None" if those fields aren't defined, i.e. in general logging messages.	2019-02-14 11:09:58 +01:00
Mike Fährmann	53c2fd4664	add mastodon/foolslide/foolfuuka examples to example config	2019-02-05 16:17:25 +01:00
Mike Fährmann	d36ec51e5a	[tsumino] add extractor for search results (#161)	2019-02-02 14:56:46 +01:00
Mike Fährmann	bfbbac4495	[tsumino] add login capabilities (#161)	2019-01-30 17:58:48 +01:00
Mike Fährmann	6126615698	update URLs for supportedsites.rst	2019-01-30 16:18:22 +01:00
Mike Fährmann	80a75a1ecf	[tsumino] add gallery extractor (#161)	2019-01-29 17:28:48 +01:00
Mike Fährmann	0c32dc5858	[hentaifox] add extractor for search results (#160)	2019-01-28 22:38:32 +01:00
Mike Fährmann	0156189468	[hentaifox] add chapter extractor (#160)	2019-01-28 18:00:32 +01:00
Mike Fährmann	e4171d6baf	[luscious] add login capabilities (closes #159)	2019-01-28 17:14:15 +01:00
Mike Fährmann	134487ffb0	[exhentai] stop extraction if image limit is exceeded (#141) ... can be turned off with the `exhentai.limits' option	2019-01-26 18:40:39 +01:00
Mike Fährmann	b50b30f1c9	[photobucket] download subalbums (#117)	2019-01-22 14:05:18 +01:00
Mike Fährmann	277b52101a	add 'category-transfer' option ... [ci skip]	2019-01-19 20:28:19 +01:00
Mike Fährmann	4b441c162e	release version 1.6.3	2019-01-18 12:31:04 +01:00
Mike Fährmann	8aba2bdebf	[postprocessor:metadata] add 'tags' and 'custom' modes (#135)	2019-01-17 21:18:12 +01:00
Mike Fährmann	e53cdfd6a8	update build_supportedsites.py	2019-01-09 14:58:35 +01:00
Mike Fährmann	1e4d351ad3	[danbooru] add authentication support (closes #151) ... ... via HTTP Basic Auth with username and "password". The password value in this case is not the account password itself, but the"api_key" found in your user profile.	2019-01-09 14:19:07 +01:00
Mike Fährmann	06cbf5f9c4	implement 'chapter-reverse' option (#149) ... Setting it to `true` will start with the latest chapter instead of the first one.	2019-01-07 18:22:33 +01:00
Mike Fährmann	e95b24f056	[reactor] add wait-min & -max options (#148)	2019-01-07 18:04:47 +01:00
Mike Fährmann	a36f52a730	[joyreactor] add extractor for search results (#148)	2019-01-03 16:25:56 +01:00
Mike Fährmann	6c71e9cf5d	[deviantart] add separate 'sta.sh' extractor (#113) ... - supports multiple stashed deviations per page - explicitly mentions sta.sh support on supportedsites.rst	2018-12-26 18:56:57 +01:00
Mike Fährmann	28f9539551	[tumblr] change default values for post types and inline media	2018-12-26 18:55:59 +01:00
Mike Fährmann	5be95034ba	[tumblr] add option to download avatars (#137)	2018-12-26 14:29:30 +01:00
Mike Fährmann	be52069cbc	update CHANGELOG and docs/supportedsites	2018-11-28 14:53:27 +01:00
Mike Fährmann	c47482b110	smaller changes, missing docs, etc. ... - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst	2018-11-16 18:18:07 +01:00
Mike Fährmann	8ef84a6823	add option to enable/disable specific downloader modules ... ... and write URLs with no (active) downloader to unsupported-file	2018-11-13 18:06:36 +01:00
Mike Fährmann	2fa28a2609	update default user-agent string (closes #122)	2018-11-11 10:07:10 +01:00
Mike Fährmann	655549df7c	[downloader:ytdl] add several options ... The "default" downloader options (rate, retries, timeout, verify) are mapped to corresponding youtube-dl options. downloader.ytdl.logging tells the downloader to pass youtube-dl's output to a Logger object. downloader.ytdl.raw-options allows to pass arbitrary options to the YoutubeDL constructor.	2018-10-20 18:26:49 +02:00
Mike Fährmann	6ed629f2b6	allow specifying number of skips before abort/exit (closes #115) ... In addition to 'abort' and 'exit', it is now possible to specify 'abort:N' and 'exit:N' (where N is any integer) as value for 'skip' to abort/exit after consecutively skipping N downloads.	2018-10-13 17:21:55 +02:00
Mike Fährmann	d8492df51b	[deviantart] extend functionality of 'original' option	2018-10-11 22:11:49 +02:00
Mike Fährmann	392a081657	update 'docs/configuration.rst'	2018-10-08 23:09:01 +02:00
Mike Fährmann	f8b3b00249	[twitter] add experimental 'videos' option (#99) ... Enabling this option will detect videos in tweets and output them as "unsupported" URLs, so that these can then be downloaded with youtube-dl There are a lot of improvements to be made to the current implementation, but it works and does what it is supposed to, even if inefficient as can be ...	2018-09-30 21:52:23 +02:00
Mike Fährmann	2cf3f53839	[yuki] add thread extractor (closes #111)	2018-09-28 12:47:32 +02:00
Mike Fährmann	17611bfec0	update build_supportedsites.py script	2018-09-28 12:43:19 +02:00
Mike Fährmann	09d2f3e5e7	[postprocessor:ugoira] improve libx264 detection	2018-09-24 13:12:54 +02:00
Mike Fährmann	c402cc4047	[hentaifoundry] add 'popular' and 'recent' extractors ... for "Popular Pictures" and "Recent Pictures" listings	2018-09-24 13:11:18 +02:00
Mike Fährmann	a5fc311dfa	[hentaifoundry] add 'favorite' extractor	2018-09-22 21:23:29 +02:00
Mike Fährmann	55f5c87160	[postprocessor:ugoira] add 'libx264-prevent-odd' option ... A rather crude workaround for "width/height not divisible by 2" errors when using libx264.	2018-09-21 20:01:14 +02:00
Mike Fährmann	006f75b538	[hentaifoundry] rewrite + more metadata ... - extract width, height, artist per image - improve pattern regex - better extensibility for other listings	2018-09-21 11:23:51 +02:00
Mike Fährmann	6ea9a78588	[wallhaven] add login capabilities ... Being logged in is required to access NSFW wallpapers.	2018-09-19 21:04:01 +02:00
Mike Fährmann	c9290d8212	[wallhaven] add wallpaper and search extractors ... todo: - login support to gain access to NSFW wallpapers - extractors for tag-, similar-, latest-listings - skip() support	2018-09-17 21:26:13 +02:00
Mike Fährmann	22ab509a70	[bobx] rename "model" to "idol" extractor	2018-09-14 18:11:36 +02:00
Mike Fährmann	69fd61ea86	[bobx] add gallery and model extractors	2018-09-13 20:13:12 +02:00
Mike Fährmann	7742cf8601	[tumblr] change 'reblogs' option (#103) ... - rename "deleted" to "same-blog" - change test for deleted original post to test if original post owner has the same UUID (full blog name) as the one being downloaded from - add 'blog[uuid]' metadata to allow comparison with 'reblogged_from_uuid'	2018-09-10 15:40:25 +02:00
Mike Fährmann	f1695567e8	adjust values in template config file ... (#104)	2018-09-09 14:10:55 +02:00
Mike Fährmann	a666ddd16b	[tumblr] extend 'reblogs' functionality (#103) ... Setting 'reblogs' to "deleted" will check if the parent post of a reblog has been deleted and download its media content if that is the case, otherwise it will be skipped. This is a rather costly operation (1 API request per reblogged post) and should therefore be used with care.	2018-09-07 19:13:52 +02:00
Mike Fährmann	c9b8e6aefc	[reddit] fix submission-ID parsing (#104) ... Uppercase characters caused a ValueError exception	2018-09-07 18:27:54 +02:00
Mike Fährmann	3ecea4cf36	[hentaicafe] add chapter and manga extractors (#101)	2018-09-05 21:08:40 +02:00
Mike Fährmann	75862715ac	[behance] add user extractor	2018-08-31 17:42:09 +02:00
Mike Fährmann	6ecb36d88c	[postprocessor:ugoira] add 'ffmpeg-output' option	2018-08-31 17:37:35 +02:00
Mike Fährmann	ab2bfaeb46	[ngomik] add replacement for 'subapics' ... http://subapics.com/ got discontinued and replaced by http://ngomik.in/. ngomik.in is still displaying a link to the "old site" showing a big "Account Suspended" sign.	2018-08-23 15:29:53 +02:00
Mike Fährmann	e9dd2eff1d	[twitter] add extractor for media-tweet timelines (#96) ... For example "https://twitter.com/PicturesEarth/media". They are different from normal timelines in that they do not contain any (re)tweets from other users and feature all media the user ever posted, including responses to other tweets.	2018-08-19 20:46:12 +02:00
Mike Fährmann	9b1c39032c	[twitter] changes and improvements ... - rename User- to TimelineExtractor - rename 'userid' to 'user_id' to conform to the other ..._id values - adjust archive_fmt to deal with retweets - emulate browser behavior for API calls	2018-08-18 23:04:45 +02:00
Mike Fährmann	10365394d7	[twitter] add support for user-timelines (closes #96) ... also adds a 'retweets' option to filter retweeted content	2018-08-17 20:04:11 +02:00
Mike Fährmann	63fa0b2006	[pinterest] add extractors for related pins ... Related pins can not be accessed by adding a "#related" fragment to the end of a Pinterest URL, for example: - https://www.pinterest.com/pin/858146903966145189/#related - https://www.pinterest.com/g1952849/test-/#related There are no explicit real URLs for related pins, using an option to enable them results in "clunky" code, and a custom "related:<URL>" scheme doesn't feel right either.	2018-08-15 21:49:45 +02:00
Mike Fährmann	3c90df6635	[piczel] add user, folder and image extractors	2018-08-08 10:53:01 +02:00
Mike Fährmann	b164231bca	[sankaku] increase default values for 'wait-min/-max'	2018-08-03 17:06:51 +02:00
Mike Fährmann	df082e923c	[behance] add gallery extractor (#95)	2018-08-01 21:46:55 +02:00
Mike Fährmann	5f27cfeff6	[deviantart] remove prefer-public option ... All API requests now always use a public token and only switch to a private token for pagination results if `refresh-token` is set and less deviations than requested were returned.	2018-07-26 19:43:46 +02:00