1316 Commits

Author	SHA1	Message	Date
Mike Fährmann	c43f02245f	update configuration.rst ... - fix default values for 'log' and 'unsupportedfile' [ci skip]	2018-05-27 17:12:57 +02:00
Mike Fährmann	dacda69c9e	update configuration.rst ... - document logging options - add a section for "custom types" [ci skip]	2018-05-27 16:50:35 +02:00
Mike Fährmann	55b0913412	[simplyhentai] add gallery extractor (#89)	2018-05-27 15:25:04 +02:00
Mike Fährmann	53f36176fd	update configuration.rst ... - update the API Tokens & IDs section - mention redirect URIs for deviantart - include api-secret for tumblr - add instructions for smugmug - [ci skip]	2018-05-26 11:26:50 +02:00
Mike Fährmann	ecdc3475b8	[pixhost] support .to TLDs	2018-05-23 18:32:34 +02:00
Mike Fährmann	909d105ae6	[pixiv] add extractor for illusts from followed users	2018-05-15 13:05:15 +02:00
Mike Fährmann	f3483a2b7c	[smugmug] add OAuth support	2018-05-10 18:58:05 +02:00
Mike Fährmann	69a5e6ddb3	Merge branch 'master' into 1.4-dev	2018-05-04 10:19:02 +02:00
Mike Fährmann	82c50fa609	release version 1.3.5	2018-05-04 10:03:20 +02:00
Mike Fährmann	8bf3cdd82b	implement logging options ... Standard logging to stderr, logfiles, and unsupported URL files (which are now handled through the logging module) can now be configured by setting their respective option keys (log, logfile, unsupportedfile) to a dict and specifying the following options; - format: format string for logging messages available keys: see [1] default: "[{name}][{levelname}] {message}" - format-date: format string for {asctime} fields in logging messages available keys: see [2] default: "%Y-%m-%d %H:%M:%S" - level: the lowercase levelname until which the logger should activate; available levels are debug, info, warning, error, exception default: "info" - path: path of the file to be written to - mode: 'mode' argument when opening the specified file can be either "w" to truncate the file or "a" to append to it (see [3]) If 'output.log', '.logfile', or '.unsupportedfile' is a string, it will be interpreted, as it has been, as the filepath (or as format string for .log) [1] https://docs.python.org/3/library/logging.html#logrecord-attributes [2] https://docs.python.org/3/library/time.html#time.strftime [3] https://docs.python.org/3/library/functions.html#open	2018-05-01 17:54:52 +02:00
Mike Fährmann	16e014baaa	[smugmug] added image and album extractor ... just some initial code that still requires a lot of work ... TODO: - folders - old-style albums (which are nearly all of them ...) - images from users - OAuth It could also happen that the API credentials used will become invalid whenever my 14 day trial period ends (7 days remaining), but that would just require users to supply their own.	2018-04-29 21:27:25 +02:00
Mike Fährmann	d96b3474e5	[puremashiro] remove module ... site has been unreachable for a couple of weeks and now the DNS record is gone as well	2018-04-28 14:24:20 +02:00
Mike Fährmann	b44a296404	[gomanga] remove module ... site has been unreachable for a couple of weeks and the cloudflare status page shows host errors	2018-04-28 14:24:21 +02:00
Mike Fährmann	0f1e07f627	[pinterest] scrap OAuth implementation; code improvements ... OAuth authentication isn't needed anymore and other tools like Postman are better suited for this job anyway.	2018-04-25 16:04:30 +02:00
Mike Fährmann	d10579edb5	[pinterest] improve PinterestAPI code; remove OAuth mentions ... on another note: access_tokens have been set to only allow for 10 requests per hour (from 200 yesterday)	2018-04-17 17:12:42 +02:00
Mike Fährmann	4bd182c107	[pinterest] implement oauth:pinterest (#83) ... Pinterest access tokens are rate limited at 200 requests per hour (or maybe per 2 or 3 hours?) so having just one access token for all users isn't going to work in the long run.	2018-04-16 20:03:28 +02:00
Mike Fährmann	dd49127408	[spectrumnexus] remove module ... Site stopped hosting manga scans (http://view.thespectrum.net/)	2018-04-16 09:45:07 +02:00
Mike Fährmann	a1fa4b43b0	Revert "[tumblr] add option to sort photosets by upload order" ... This reverts commit 4a26ae32df.	2018-04-09 16:08:08 +02:00
Mike Fährmann	564e12ca8f	replace 'imgyt' with 'imxto' ... https://img.yt/ wasn't available for a couple of days, but has now re-emerged as https://imx.to/ with a new web-interface. Links to older images still work (see tests).	2018-04-09 15:53:20 +02:00
Mike Fährmann	4a26ae32df	[tumblr] add option to sort photosets by upload order	2018-04-07 15:57:55 +02:00
Mike Fährmann	7b562907c3	[nijie] add favorites extractor ... adds support for 'https://nijie.info/user_like_illust_view.php?id=...'	2018-03-31 18:54:25 +02:00
Mike Fährmann	f39153b6e9	[nhentai] add extractor for search results	2018-03-28 17:21:44 +02:00
Mike Fährmann	52d41c41e7	[exhentai] add extractor for favorited galleries	2018-03-27 18:58:42 +02:00
Mike Fährmann	a993d0ea90	release version 1.3.2	2018-03-23 15:27:56 +01:00
Mike Fährmann	e7525b1b0e	[artstation] add challenge extractor (#80)	2018-03-23 15:06:09 +01:00
Mike Fährmann	b25ae30a95	reorder options in gallery-dl.conf	2018-03-22 10:05:53 +01:00
Mike Fährmann	40ca562d7b	[artstation] add album extractor (#80)	2018-03-17 17:36:31 +01:00
HRXN	241eda8c3d	Update supportedsites.rst (#81) ... Add ArtStation to Supported Sites.	2018-03-16 13:40:59 +01:00
Mike Fährmann	f0e3861ec7	update configuration.rst ... - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to	2018-03-16 12:00:55 +01:00
Mike Fährmann	723cc66bb1	[artstation] add user-, image- and likes-extractors	2018-03-14 14:05:14 +01:00
Mike Fährmann	d6ef52897c	[imgchili] remove module ... All previously hosted images yield a 404 and the main page is just a logo.	2018-03-13 11:02:58 +01:00
Mike Fährmann	7847ab1d5a	[imagehosts] remove even more dead sites ... All removed sites either - reject all incoming connections or - display a message from their domain registrar	2018-03-12 21:25:13 +01:00
Mike Fährmann	74c8f83056	update gallery-dl.conf and add gallery-dl-example.conf ... - gallery-dl.conf contains all config options set to their default values, with the exception of 'filename', 'directory', and 'cache.file' - gallery-dl-example.conf contains a potential real-world configuration file and demonstrates some more involved options	2018-03-12 18:15:30 +01:00
Mike Fährmann	1400868f53	[mangadex] general improvements ... - support >100 chapter entries per manga - custom archive ID format - detect non-existing chapters	2018-03-06 14:15:15 +01:00
Mike Fährmann	749fbbfa6c	[mangadex] add chapter- and manga-extractor	2018-03-05 18:37:21 +01:00
Mike Fährmann	6e38cf5aab	[mangareader] use 'https://' ... The site now redirects from http://mangareader.net/ to https://mangareader.net/	2018-03-02 17:19:17 +01:00
Mike Fährmann	8cdce21dcb	make archive keys user-configurable	2018-02-25 21:57:01 +01:00
Mike Fährmann	8f338347b6	[imagehosts] cleanup ... removed - chronos.to - unable to resolve hostname - coreimg.net - same - imgmaid.net - same - hosturimage.com - everything returns 404 - imageontime.org - redirects to some shady site - imgupload.yt - cloudflare error 522, host down - img4ever.net - read timeout	2018-02-23 01:05:42 +01:00
Mike Fährmann	e1e0668ca8	add option to set default replacement field value ... Missing or undefined keywords will now be replaced with the value set for 'keywords-default'. The default is Python's 'None', which is equivalent to setting this option to JSON's 'null'.	2018-02-23 00:59:20 +01:00
Mike Fährmann	8704d850bf	add explicit proxy support (#76) ... - '--proxy' as command-line argument - 'extractor.*.proxy' as config option	2018-02-19 18:45:06 +01:00
Mike Fährmann	3cec533c28	Merge branch 'archive'	2018-02-12 18:07:58 +01:00
Mike Fährmann	7f7c16ae37	add option to specify additional key-value pairs	2018-02-08 23:10:58 +01:00
Mike Fährmann	54c36a8a34	[subapics] add chapter- and manga-extractor (#70)	2018-02-04 22:02:10 +01:00
Mike Fährmann	2dd3aeeeae	[komikcast] add chapter- and manga-extractor (#70)	2018-02-04 22:02:10 +01:00
Mike Fährmann	c0dd922c13	add '--download-archive' cmdline option ... … as well as a config file equivalent	2018-02-01 22:00:44 +01:00
Mike Fährmann	d951f13e37	add config option for unsupported-URL file ... for consistency's sake	2018-01-28 18:42:10 +01:00
Mike Fährmann	97f4f15ec0	add option to write logging output to a file ... - '--write-log FILE' as cmdline argument - 'output.logfile' as config file option	2018-01-26 18:51:51 +01:00
Mike Fährmann	1e93955170	[batoto] remove module ... Site officially shut down on 2018.01.18	2018-01-23 21:37:32 +01:00
Mike Fährmann	40d35c87bc	[paheal] add tag- and post-extractors (closes #69)	2018-01-15 16:39:05 +01:00
Mike Fährmann	cc0c2cca57	[reddit] add extractor for reddit-hosted images (closes #68)	2018-01-14 18:55:42 +01:00
Mike Fährmann	35e09869d1	[mangapark] fix image URLs and use HTTPS	2018-01-12 14:59:49 +01:00
Mike Fährmann	29d75fc3fa	[tumblr] add support for OAuth authentication (#65)	2018-01-11 14:11:37 +01:00
Mike Fährmann	b33efc99a4	[idolcomplex] add support for idol.sankakucomplex.com	2018-01-09 17:54:37 +01:00
Mike Fährmann	5b094328b5	[puremashiro] add chapter- and manga-extractor (closes #66) ... Also adds support for region subtags in language codes (e.g. en-us)	2018-01-07 21:50:43 +01:00
Mike Fährmann	03b8a548cb	[tumblr] change reblogs default value to true (#61)	2018-01-06 15:52:08 +01:00
Mike Fährmann	d235f68f59	[tumblr] add option to filter reblogged posts (#61) ... Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.	2018-01-05 13:05:57 +01:00
Mike Fährmann	d0886f411e	[gelbooru] re-enable API use (closes #56) ... Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.	2017-12-21 21:42:40 +01:00
Mike Fährmann	fc7d165c97	[deviantart] add support for OAuth2 authentication ... Some user galleries [*] require you to be either logged in or authenticated via OAuth2 to access their deviations. [*] e.g. https://polinaegorussia.deviantart.com/gallery/ -------------- known issue: A deviantart 'refresh_token' can only be used once and gets updated whenever it is used to request a new 'access_token', so storing its initial value in a config file and reusing it again and again is not possible.	2017-12-18 01:16:46 +01:00
Mike Fährmann	0a9a07a6e1	[slideshare] improve metadata; flake8 ... - added 'views' and 'published' keywords - fixed longer titles and descriptions	2017-12-13 21:16:49 +01:00
Mike Fährmann	19a6ae57b2	[sankaku] add pool extractor	2017-12-12 19:45:10 +01:00
Mike Fährmann	4fb6803fa6	add option to sleep before each download	2017-12-04 17:33:10 +01:00
Mike Fährmann	300346ecdf	[mangazuki] remove extractors ... This site has been in "rebuild"-mode for a fairly long time and the current extractor code isn't going to work for the new version either.	2017-12-04 13:36:04 +01:00
Mike Fährmann	8f518e03f8	add options to set maximum download rate ... - -r/--limit-rate as cmdline option - downloader.http.rate as config option This implementation very roughly uses the idea of the token bucket algorithm [1] and mostly uses Wget's approach [2] as inspiration. [1] https://en.wikipedia.org/wiki/Token_bucket [2] http://git.savannah.gnu.org/cgit/wget.git/tree/src/retr.c?h=v1.19.2&id=ba6b44f6745b14dce414761a8e4b35d31b176bba#n111	2017-12-02 01:47:26 +01:00
Mike Fährmann	214972bc9a	[gelbooru] use manual extraction ... ... to compensate for their disabled API. (https://gelbooru.com/index.php?page=forum&s=view&id=3875) This also adds an extractor for image-pools.	2017-11-29 20:48:17 +01:00
Mike Fährmann	12de658937	[tumblr] add options to control extraction behavior (#48) ... - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links	2017-11-23 15:32:54 +01:00
Mike Fährmann	e6814aebe2	add 'extractor.*.user-agent' config option	2017-11-15 14:01:33 +01:00
Mike Fährmann	2d0cfb33e1	[xvideos] add user profile extractor (#45)	2017-11-02 17:28:35 +01:00
Mike Fährmann	707b15b586	create missing directories for 'part-directory' ... also some code improvements regarding downloader config values	2017-10-27 12:22:45 +02:00
Mike Fährmann	caf26412dd	add option to set alternate location of .part files (#29) ... Note: The path set for 'downloader.*.part-directory' needs to point to an already existing directory.	2017-10-26 00:16:48 +02:00
Mike Fährmann	963670d73b	add options to control usage of .part files (#29) ... - '--no-part' command line option to disable them - 'downloader.http.part' and 'downloader.text.part' config options Disabling .part files restores the behaviour of the old downloader implementation.	2017-10-24 23:33:44 +02:00
Mike Fährmann	5fa42336a2	[sankaku] add warning for unauthenticated users ... also improve URL pattern and add missing options to default config file	2017-10-16 21:21:08 +02:00
Mike Fährmann	6af921a952	[sankaku] rewrite/improve (fixes #44) ... - add wait-time between HTTP requests similar to exhentai - add 'wait-min' and 'wait-max' options - increase retry-count for HTTP requests to 10 - implement user authentication (non-authenticated users can only view images up to page 25) - implement 'skip()' functionality (only works up to page 50) - implement image-retrieval for pages >= 51 - fix issue with multiple tags	2017-10-14 23:01:33 +02:00
Mike Fährmann	abfd2924a9	update configuration.rst ... - add instructions on how to register your own applications - combine descriptions of a few config keys	2017-10-12 23:37:28 +02:00
Mike Fährmann	f3fbaa5c3e	[reddit] allow users to override the API User-Agent ... Only overriding the Client-ID is not enough if you want to follow Reddit's API access rules [1]. [1] https://github.com/reddit/reddit/wiki/API#rules	2017-10-10 17:29:46 +02:00
Mike Fährmann	2ef3c35c98	smaller textual changes ... - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"	2017-10-09 23:23:19 +02:00
Mike Fährmann	75d3a1f72f	[deviantart] always download original images ... Deviation-objects returned by the DeviantArt API don't always contain the URL and metadata of the original image ([1]). Getting this information requires an additional API call [2], which is indicated by the 'is_downloadable' and 'download_filesize' metadata within a deviation-object. [1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576 [2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd	2017-10-07 13:07:34 +02:00
Mike Fährmann	cebf800a7f	[foolfuuka] add support for more sites (#18) ... - https://arch.b4k.co - https://archive.whatisthisimnotgoodwithcomputers.com - https://archive.yeet.net Notes: - The name "whatisthisimnotgoodwithcomputers" is way too long ... - archive.yeet.net is out of date and also blocked by 4chan servers - newest threads are 2 weeks old - using "https://archive.yeet.net" as Referer header results in "403 Forbidden" when accessing 4chan	2017-09-16 21:36:16 +02:00
Mike Fährmann	470bbe9d8c	fix smaller stuff ... - change filename option in example config file - adapt default filename format for mangafox - remove unnecessary newline [skip ci]	2017-09-11 17:07:29 +02:00
Mike Fährmann	81877bb5f6	add '-K' as shortcut for '--list-keywords'	2017-09-09 18:48:28 +02:00
Mike Fährmann	54c0715135	allow users to set their own API access_tokens/client_ids	2017-09-09 17:50:19 +02:00
Mike Fährmann	49c7e70c10	[acidimg] add image extractor	2017-09-09 15:19:18 +02:00
Mike Fährmann	b8862ff15e	add 'downloader.http.verify' option ... (also: change the default 'timeout' from None to 30)	2017-08-31 15:21:08 +02:00
Mike Fährmann	18e6ed1c7e	[booru] add extractors for "Popular" images	2017-08-24 21:29:22 +02:00
Mike Fährmann	4ea82ea556	[warosu] add thread extractor	2017-08-18 19:54:07 +02:00
Mike Fährmann	3c9f190757	extend output of --list-keywords	2017-08-10 17:36:21 +02:00
Mike Fährmann	abd7c559cd	[yonkouprod] remove module ... Every manga chapter on this site has been removed.	2017-08-07 18:32:14 +02:00
Mike Fährmann	f08af03845	Merge branch 'cookies'	2017-07-25 14:04:53 +02:00
Mike Fährmann	d7cb3c668a	update supportedsites.rst	2017-07-24 10:50:40 +02:00
Mike Fährmann	00a23fe208	update configuration.rst	2017-07-21 18:32:56 +02:00
Mike Fährmann	4a60f6068a	[mangazuki] add manga extractor	2017-07-20 16:02:09 +02:00
Mike Fährmann	1cb1d2e0a3	[mangazuki] add chapter extractor	2017-07-19 17:20:03 +02:00
Mike Fährmann	74a2b5af06	update supportedsites.rst	2017-07-15 15:01:30 +02:00
Mike Fährmann	3460dc8950	update gallery-dl.conf	2017-07-14 08:23:11 +02:00
Mike Fährmann	9be8f7e106	[deviantart] add "extractor.deviantart.flat" option ... Setting this to 'false' downloads images into individual subdirectories for each gallery-folder or favourite-collection, otherwise it is just creating a flat list of images.	2017-07-12 17:05:31 +02:00
Mike Fährmann	9edbd6ff01	update configuration.rst	2017-07-04 19:34:34 +02:00
Mike Fährmann	06c4cae05b	extend the output of '--list-extractors' ... It now includes category and subcategory values for each extractor class.	2017-06-28 18:51:47 +02:00
Mike Fährmann	1dac76fd1c	update extractor docstrings	2017-06-28 17:39:07 +02:00
H R X N	b757687606	Update configuration.rst (#24) ... Fixed minor typo.	2017-06-28 13:55:39 +02:00
Mike Fährmann	598e3a8d88	update configuration.rst	2017-06-27 18:56:24 +02:00
Mike Fährmann	86561d7c8b	update configuration.rst	2017-06-27 17:44:02 +02:00
Mike Fährmann	d3b04076f7	add .netrc support (#22) ... Use the '--netrc' cmdline option or set the 'netrc' config option to 'true' to enable the use of .netrc authentication data. The 'machine' names for the .netrc info are the lowercase extractor names (or categories): batoto, exhentai, nijie, pixiv, seiga.	2017-06-24 12:17:26 +02:00
Mike Fährmann	fbe8c519e2	update README ... - mention and link to docs/configuration.rst - add OAuth section	2017-06-23 16:14:51 +02:00
Mike Fährmann	ccb7973b57	update configuration.rst	2017-06-20 16:20:28 +02:00
H R X N	b206b831ee	Update configuration.rst ... Fix heading.	2017-06-19 14:12:43 +02:00
Mike Fährmann	48b444e276	add a list of all configuration options ... (the format might be subject to change; putting everything in one single table is even worse)	2017-06-18 22:16:26 +02:00
Mike Fährmann	fb1904dd59	add 'Authentication' column to supportedsites.rst	2017-06-15 21:06:20 +02:00
Mike Fährmann	5f55c854b9	[flickr] replace getPublic... API call with regular ones	2017-06-12 16:37:06 +02:00
Mike Fährmann	d5a70f2580	add simple progress indicator for multiple URLs (#19) ... The output can be configured via the 'output.progress' config value. Possible values: - true: Show the default progress indicator "[{current}/{total}] {url}" (default) - false: Never show the progress indicator - <string>: Show the progress indicator using this as a custom format string(1). Possible replacement keys are: - current: current URL index - total : total number of URLs - url : current URL (1) https://docs.python.org/3/library/string.html#formatstrings	2017-06-09 20:12:15 +02:00
Mike Fährmann	090e11b35d	[reddit] enable user authentication with OAuth2 (#15) ... Call '$ gallery-dl oauth:reddit' to get a refresh_token for your account.	2017-06-08 16:17:13 +02:00
Mike Fährmann	4e80e0c884	[flickr] add user extractor (#16)	2017-06-02 17:15:05 +02:00
Mike Fährmann	c921b4f32a	code cleanup and fixing tests	2017-06-02 09:10:58 +02:00
Mike Fährmann	67791e1b36	[imgur] improve and add image extractor	2017-05-26 22:30:09 +02:00
Mike Fährmann	832a4a8ee9	[fallenangels] add manga extractor	2017-05-21 10:37:38 +02:00
Mike Fährmann	2974d782a3	[yomanga] remove module ... site has been shut down	2017-05-20 11:18:44 +02:00
Mike Fährmann	677c8ced11	[deviantart] add "journal" extractor ... (#14)	2017-05-10 17:21:33 +02:00
Mike Fährmann	1e416e5227	update README	2017-05-09 15:43:55 +02:00