73 Commits

Author	SHA1	Message	Date
Mike Fährmann	54c36a8a34	[subapics] add chapter- and manga-extractor (#70)	2018-02-04 22:02:10 +01:00
Mike Fährmann	2dd3aeeeae	[komikcast] add chapter- and manga-extractor (#70)	2018-02-04 22:02:10 +01:00
Mike Fährmann	d951f13e37	add config option for unsupported-URL file ... for consistency's sake	2018-01-28 18:42:10 +01:00
Mike Fährmann	97f4f15ec0	add option to write logging output to a file ... - '--write-log FILE' as cmdline argument - 'output.logfile' as config file option	2018-01-26 18:51:51 +01:00
Mike Fährmann	1e93955170	[batoto] remove module ... Site officially shut down on 2018.01.18	2018-01-23 21:37:32 +01:00
Mike Fährmann	40d35c87bc	[paheal] add tag- and post-extractors (closes #69)	2018-01-15 16:39:05 +01:00
Mike Fährmann	cc0c2cca57	[reddit] add extractor for reddit-hosted images (closes #68)	2018-01-14 18:55:42 +01:00
Mike Fährmann	35e09869d1	[mangapark] fix image URLs and use HTTPS	2018-01-12 14:59:49 +01:00
Mike Fährmann	29d75fc3fa	[tumblr] add support for OAuth authentication (#65)	2018-01-11 14:11:37 +01:00
Mike Fährmann	b33efc99a4	[idolcomplex] add support for idol.sankakucomplex.com	2018-01-09 17:54:37 +01:00
Mike Fährmann	5b094328b5	[puremashiro] add chapter- and manga-extractor (closes #66) ... Also adds support for region subtags in language codes (e.g. en-us)	2018-01-07 21:50:43 +01:00
Mike Fährmann	03b8a548cb	[tumblr] change reblogs default value to true (#61)	2018-01-06 15:52:08 +01:00
Mike Fährmann	d235f68f59	[tumblr] add option to filter reblogged posts (#61) ... Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.	2018-01-05 13:05:57 +01:00
Mike Fährmann	d0886f411e	[gelbooru] re-enable API use (closes #56) ... Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.	2017-12-21 21:42:40 +01:00
Mike Fährmann	fc7d165c97	[deviantart] add support for OAuth2 authentication ... Some user galleries [*] require you to be either logged in or authenticated via OAuth2 to access their deviations. [*] e.g. https://polinaegorussia.deviantart.com/gallery/ -------------- known issue: A deviantart 'refresh_token' can only be used once and gets updated whenever it is used to request a new 'access_token', so storing its initial value in a config file and reusing it again and again is not possible.	2017-12-18 01:16:46 +01:00
Mike Fährmann	0a9a07a6e1	[slideshare] improve metadata; flake8 ... - added 'views' and 'published' keywords - fixed longer titles and descriptions	2017-12-13 21:16:49 +01:00
Mike Fährmann	19a6ae57b2	[sankaku] add pool extractor	2017-12-12 19:45:10 +01:00
Mike Fährmann	4fb6803fa6	add option to sleep before each download	2017-12-04 17:33:10 +01:00
Mike Fährmann	300346ecdf	[mangazuki] remove extractors ... This site has been in "rebuild"-mode for a fairly long time and the current extractor code isn't going to work for the new version either.	2017-12-04 13:36:04 +01:00
Mike Fährmann	8f518e03f8	add options to set maximum download rate ... - -r/--limit-rate as cmdline option - downloader.http.rate as config option This implementation very roughly uses the idea of the token bucket algorithm [1] and mostly uses Wget's approach [2] as inspiration. [1] https://en.wikipedia.org/wiki/Token_bucket [2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111	2017-12-02 01:47:26 +01:00
Mike Fährmann	214972bc9a	[gelbooru] use manual extraction ... ... to compensate for their disabled API. (https://gelbooru.com/index.php?page=forum&s=view&id=3875) This also adds an extractor for image-pools.	2017-11-29 20:48:17 +01:00
Mike Fährmann	12de658937	[tumblr] add options to control extraction behavior (#48) ... - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links	2017-11-23 15:32:54 +01:00
Mike Fährmann	e6814aebe2	add 'extractor.*.user-agent' config option	2017-11-15 14:01:33 +01:00
Mike Fährmann	2d0cfb33e1	[xvideos] add user profile extractor (#45)	2017-11-02 17:28:35 +01:00
Mike Fährmann	707b15b586	create missing directories for 'part-directory' ... also some code improvements regarding downloader config values	2017-10-27 12:22:45 +02:00
Mike Fährmann	caf26412dd	add option to set alternate location of .part files (#29) ... Note: The path set for 'downloader.*.part-directory' needs to point to an already existing directory.	2017-10-26 00:16:48 +02:00
Mike Fährmann	963670d73b	add options to control usage of .part files (#29) ... - '--no-part' command line option to disable them - 'downloader.http.part' and 'downloader.text.part' config options Disabling .part files restores the behaviour of the old downloader implementation.	2017-10-24 23:33:44 +02:00
Mike Fährmann	5fa42336a2	[sankaku] add warning for unauthenticated users ... also improve URL pattern and add missing options to default config file	2017-10-16 21:21:08 +02:00
Mike Fährmann	6af921a952	[sankaku] rewrite/improve (fixes #44) ... - add wait-time between HTTP requests similar to exhentai - add 'wait-min' and 'wait-max' options - increase retry-count for HTTP requests to 10 - implement user authentication (non-authenticated users can only view images up to page 25) - implement 'skip()' functionality (only works up to page 50) - implement image-retrieval for pages >= 51 - fix issue with multiple tags	2017-10-14 23:01:33 +02:00
Mike Fährmann	abfd2924a9	update configuration.rst ... - add instructions on how to register your own applications - combine descriptions of a few config keys	2017-10-12 23:37:28 +02:00
Mike Fährmann	f3fbaa5c3e	[reddit] allow users to override the API User-Agent ... Only overriding the Client-ID is not enough if you want to follow Reddit's API access rules [1]. [1] https://github.com/reddit/reddit/wiki/API#rules	2017-10-10 17:29:46 +02:00
Mike Fährmann	2ef3c35c98	smaller textual changes ... - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"	2017-10-09 23:23:19 +02:00
Mike Fährmann	75d3a1f72f	[deviantart] always download original images ... Deviation-objects returned by the DeviantArt API don't always contain the URL and metadata of the original image ([1]). Getting this information requires an additional API call [2], which is indicated by the 'is_downloadable' and 'download_filesize' metadata within a deviation-object. [1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576 [2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd	2017-10-07 13:07:34 +02:00
Mike Fährmann	cebf800a7f	[foolfuuka] add support for more sites (#18) ... - https://arch.b4k.co - https://archive.whatisthisimnotgoodwithcomputers.com - https://archive.yeet.net Notes: - The name "whatisthisimnotgoodwithcomputers" is way too long ... - archive.yeet.net is out of date and also blocked by 4chan servers - newest threads are 2 weeks old - using "https://archive.yeet.net" as Referer header results in "403 Forbidden" when accessing 4chan	2017-09-16 21:36:16 +02:00
Mike Fährmann	470bbe9d8c	fix smaller stuff ... - change filename option in example config file - adapt default filename format for mangafox - remove unnecessary newline [skip ci]	2017-09-11 17:07:29 +02:00
Mike Fährmann	81877bb5f6	add '-K' as shortcut for '--list-keywords'	2017-09-09 18:48:28 +02:00
Mike Fährmann	54c0715135	allow users to set their own API access_tokens/client_ids	2017-09-09 17:50:19 +02:00
Mike Fährmann	49c7e70c10	[acidimg] add image extractor	2017-09-09 15:19:18 +02:00
Mike Fährmann	b8862ff15e	add 'downloader.http.verify' option ... (also: change the default 'timeout' from None to 30)	2017-08-31 15:21:08 +02:00
Mike Fährmann	18e6ed1c7e	[booru] add extractors for "Popular" images	2017-08-24 21:29:22 +02:00
Mike Fährmann	4ea82ea556	[warosu] add thread extractor	2017-08-18 19:54:07 +02:00
Mike Fährmann	3c9f190757	extend output of --list-keywords	2017-08-10 17:36:21 +02:00
Mike Fährmann	abd7c559cd	[yonkouprod] remove module ... Every manga chapter on this site has been removed.	2017-08-07 18:32:14 +02:00
Mike Fährmann	f08af03845	Merge branch 'cookies'	2017-07-25 14:04:53 +02:00
Mike Fährmann	d7cb3c668a	update supportedsites.rst	2017-07-24 10:50:40 +02:00
Mike Fährmann	00a23fe208	update configuration.rst	2017-07-21 18:32:56 +02:00
Mike Fährmann	4a60f6068a	[mangazuki] add manga extractor	2017-07-20 16:02:09 +02:00
Mike Fährmann	1cb1d2e0a3	[mangazuki] add chapter extractor	2017-07-19 17:20:03 +02:00
Mike Fährmann	74a2b5af06	update supportedsites.rst	2017-07-15 15:01:30 +02:00
Mike Fährmann	3460dc8950	update gallery-dl.conf	2017-07-14 08:23:11 +02:00