584 Commits

Author	SHA1	Message	Date
Mike Fährmann	97f4f15ec0	add option to write logging output to a file ... - '--write-log FILE' as cmdline argument - 'output.logfile' as config file option	2018-01-26 18:51:51 +01:00
Mike Fährmann	03b8a548cb	[tumblr] change reblogs default value to true (#61)	2018-01-06 15:52:08 +01:00
Mike Fährmann	d235f68f59	[tumblr] add option to filter reblogged posts (#61) ... Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.	2018-01-05 13:05:57 +01:00
Mike Fährmann	d0886f411e	[gelbooru] re-enable API use (closes #56) ... Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.	2017-12-21 21:42:40 +01:00
Mike Fährmann	fc7d165c97	[deviantart] add support for OAuth2 authentication ... Some user galleries [*] require you to be either logged in or authenticated via OAuth2 to access their deviations. [*] e.g. https://polinaegorussia.deviantart.com/gallery/ -------------- known issue: A deviantart 'refresh_token' can only be used once and gets updated whenever it is used to request a new 'access_token', so storing its initial value in a config file and reusing it again and again is not possible.	2017-12-18 01:16:46 +01:00
Mike Fährmann	4fb6803fa6	add option to sleep before each download	2017-12-04 17:33:10 +01:00
Mike Fährmann	8f518e03f8	add options to set maximum download rate ... - -r/--limit-rate as cmdline option - downloader.http.rate as config option This implementation very roughly uses the idea of the token bucket algorithm [1] and mostly uses Wget's approach [2] as inspiration. [1] https://en.wikipedia.org/wiki/Token_bucket [2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111	2017-12-02 01:47:26 +01:00
Mike Fährmann	12de658937	[tumblr] add options to control extraction behavior (#48) ... - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links	2017-11-23 15:32:54 +01:00
Mike Fährmann	e6814aebe2	add 'extractor.*.user-agent' config option	2017-11-15 14:01:33 +01:00
Mike Fährmann	707b15b586	create missing directories for 'part-directory' ... also some code improvements regarding downloader config values	2017-10-27 12:22:45 +02:00
Mike Fährmann	caf26412dd	add option to set alternate location of .part files (#29) ... Note: The path set for 'downloader.*.part-directory' needs to point to an already existing directory.	2017-10-26 00:16:48 +02:00
Mike Fährmann	963670d73b	add options to control usage of .part files (#29) ... - '--no-part' command line option to disable them - 'downloader.http.part' and 'downloader.text.part' config options Disabling .part files restores the behaviour of the old downloader implementation.	2017-10-24 23:33:44 +02:00
Mike Fährmann	6af921a952	[sankaku] rewrite/improve (fixes #44) ... - add wait-time between HTTP requests similar to exhentai - add 'wait-min' and 'wait-max' options - increase retry-count for HTTP requests to 10 - implement user authentication (non-authenticated users can only view images up to page 25) - implement 'skip()' functionality (only works up to page 50) - implement image-retrieval for pages >= 51 - fix issue with multiple tags	2017-10-14 23:01:33 +02:00
Mike Fährmann	abfd2924a9	update configuration.rst ... - add instructions on how to register your own applications - combine descriptions of a few config keys	2017-10-12 23:37:28 +02:00
Mike Fährmann	f3fbaa5c3e	[reddit] allow users to override the API User-Agent ... Only overriding the Client-ID is not enough if you want to follow Reddit's API access rules [1]. [1] https://github.com/reddit/reddit/wiki/API#rules	2017-10-10 17:29:46 +02:00
Mike Fährmann	2ef3c35c98	smaller textual changes ... - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"	2017-10-09 23:23:19 +02:00
Mike Fährmann	75d3a1f72f	[deviantart] always download original images ... Deviation-objects returned by the DeviantArt API don't always contain the URL and metadata of the original image ([1]). Getting this information requires an additional API call [2], which is indicated by the 'is_downloadable' and 'download_filesize' metadata within a deviation-object. [1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576 [2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd	2017-10-07 13:07:34 +02:00
Mike Fährmann	81877bb5f6	add '-K' as shortcut for '--list-keywords'	2017-09-09 18:48:28 +02:00
Mike Fährmann	54c0715135	allow users to set their own API access_tokens/client_ids	2017-09-09 17:50:19 +02:00
Mike Fährmann	b8862ff15e	add 'downloader.http.verify' option ... (also: change the default 'timeout' from None to 30)	2017-08-31 15:21:08 +02:00
Mike Fährmann	3c9f190757	extend output of --list-keywords	2017-08-10 17:36:21 +02:00
Mike Fährmann	f08af03845	Merge branch 'cookies'	2017-07-25 14:04:53 +02:00
Mike Fährmann	00a23fe208	update configuration.rst	2017-07-21 18:32:56 +02:00
Mike Fährmann	9be8f7e106	[deviantart] add "extractor.deviantart.flat" option ... Setting this to 'false' downloads images into individual subdirectories for each gallery-folder or favourite-collection, otherwise it is just creating a flat list of images.	2017-07-12 17:05:31 +02:00
Mike Fährmann	9edbd6ff01	update configuration.rst	2017-07-04 19:34:34 +02:00
Mike Fährmann	06c4cae05b	extend the output of '--list-extractors' ... It now includes category and subcategory values for each extractor class.	2017-06-28 18:51:47 +02:00
H R X N	b757687606	Update configuration.rst (#24) ... Fixed minor typo.	2017-06-28 13:55:39 +02:00
Mike Fährmann	598e3a8d88	update configuration.rst	2017-06-27 18:56:24 +02:00
Mike Fährmann	86561d7c8b	update configuration.rst	2017-06-27 17:44:02 +02:00
Mike Fährmann	d3b04076f7	add .netrc support (#22) ... Use the '--netrc' cmdline option or set the 'netrc' config option to 'true' to enable the use of .netrc authentication data. The 'machine' names for the .netrc info are the lowercase extractor names (or categories): batoto, exhentai, nijie, pixiv, seiga.	2017-06-24 12:17:26 +02:00
Mike Fährmann	fbe8c519e2	update README ... - mention and link to docs/configuration.rst - add OAuth section	2017-06-23 16:14:51 +02:00
Mike Fährmann	ccb7973b57	update configuration.rst	2017-06-20 16:20:28 +02:00
H R X N	b206b831ee	Update configuration.rst ... Fix heading.	2017-06-19 14:12:43 +02:00
Mike Fährmann	48b444e276	add a list of all configuration options ... (the format might be subject to change; putting everything in one single table is even worse)	2017-06-18 22:16:26 +02:00