Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-25 04:02:32 +01:00
Code Issues Releases Wiki Activity
699062b91f
gallery-dl/docs/configuration.rst

2218 lines
65 KiB
ReStructuredText
Raw Normal View History

add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
Configuration
#############
simplify and improve documentation markup (#742) - use | to get a more "compact" representation - link directly to example config files
2020-05-11 00:56:41 +02:00
| Configuration files for *gallery-dl* use a JSON-based file format.
| For a (more or less) complete example with options set to their default values,
see `gallery-dl.conf <gallery-dl.conf>`__.
| For a configuration file example with more involved settings and options,
see `gallery-dl-example.conf <gallery-dl-example.conf>`__.
Some documentation improvements (#742) * Link configuration examples in the intro, see #712 * Clarify overriding conf files by options, see #678
2020-05-10 23:47:20 +02:00
This file lists all available configuration options and their descriptions.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
Contents
========
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
1) `Extractor Options`_
2) `Extractor-specific Options`_
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
3) `Downloader Options`_
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
4) `Output Options`_
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
5) `Postprocessor Options`_
6) `Miscellaneous Options`_
7) `API Tokens & IDs`_
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
Extractor Options
=================
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
extend the output of '--list-extractors' It now includes category and subcategory values for each extractor class.
2017-06-28 18:51:47 +02:00
Each extractor is identified by its ``category`` and ``subcategory``.
The ``category`` is the lowercase site name without any spaces or special
characters, which is usually just the module name
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
(``pixiv``, ``danbooru``, ...).
extend the output of '--list-extractors' It now includes category and subcategory values for each extractor class.
2017-06-28 18:51:47 +02:00
The ``subcategory`` is a lowercase word describing the general functionality
of that extractor (``user``, ``favorite``, ``manga``, ...).
update configuration.rst
2017-06-27 18:56:24 +02:00
update configuration.rst
2017-06-27 17:44:02 +02:00
Each one of the following options can be specified on multiple levels of the
configuration tree:
================== =====
Base level: ``extractor.<option-name>``
Category level: ``extractor.<category>.<option-name>``
Subcategory level: ``extractor.<category>.<subcategory>.<option-name>``
================== =====
A value in a "deeper" level hereby overrides a value of the same name on a
lower level. Setting the ``extractor.pixiv.filename`` value, for example, lets
you specify a general filename pattern for all the different pixiv extractors.
Using the ``extractor.pixiv.user.filename`` value lets you override this
general pattern specifically for ``PixivUserExtractor`` instances.
extend the output of '--list-extractors' It now includes category and subcategory values for each extractor class.
2017-06-28 18:51:47 +02:00
The ``category`` and ``subcategory`` of all extractors are included in the
output of ``gallery-dl --list-extractors``. For a specific URL these values
add '-K' as shortcut for '--list-keywords'
2017-09-09 18:48:28 +02:00
can also be determined by using the ``-K``/``--list-keywords`` command-line
option (see the example below).
Update configuration.rst (#24) Fixed minor typo.
2017-06-28 13:55:39 +02:00
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
extractor.*.filename
--------------------
=========== =====
Type ``string``
Example ``"{manga}_c{chapter}_{page:>03}.{extension}"``
Description A `format string`_ to build the resulting filename
for a downloaded file.
update configuration.rst
2017-06-27 17:44:02 +02:00
The available replacement keys depend on the extractor used. A list
of keys for a specific one can be acquired by calling *gallery-dl*
add '-K' as shortcut for '--list-keywords'
2017-09-09 18:48:28 +02:00
with the ``-K``/``--list-keywords`` command-line option.
update configuration.rst
2017-06-27 17:44:02 +02:00
For example:
.. code::
add '-K' as shortcut for '--list-keywords'
2017-09-09 18:48:28 +02:00
$ gallery-dl -K http://seiga.nicovideo.jp/seiga/im5977527
update configuration.rst
2017-06-27 17:44:02 +02:00
Keywords for directory names:
-----------------------------
category
seiga
subcategory
image
Keywords for filenames:
-----------------------
category
seiga
extension
None
image-id
5977527
subcategory
image
[postprocessor:ugoira] add 'libx264-prevent-odd' option A rather crude workaround for "width/height not divisible by 2" errors when using libx264.
2018-09-21 19:52:45 +02:00
Note: Even if the value of the ``extension`` key is missing or
update configuration.rst
2017-06-27 17:44:02 +02:00
``None``, it will filled in later when the file download is
starting. This key is therefore always available to provide
a valid filename extension.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
extractor.*.directory
---------------------
=========== =====
Type ``list`` of ``strings``
Example ``["{category}", "{manga}", "c{chapter} - {title}"]``
Description A list of `format strings`_ for the resulting target directory.
update configuration.rst
2017-06-27 17:44:02 +02:00
Each individual string in such a list represents a single path
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
segment, which will be joined together and appended to the
update configuration.rst
2017-06-27 17:44:02 +02:00
base-directory_ to form the complete target directory path.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
extractor.*.base-directory
--------------------------
=========== =====
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Type |Path|_
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Default ``"./gallery-dl/"``
Description Directory path used as the base for all download destinations.
=========== =====
implement 'parent-directory' option (#551)
2020-01-29 18:32:37 +01:00
extractor.*.parent-directory
----------------------------
=========== =====
Type ``bool``
Default ``false``
Description Use an extractor's current target directory as
`base-directory <extractor.*.base-directory_>`__
for any spawned child extractors.
=========== =====
Remove control characters from filesystem paths - add 'path-remove' option to specify the set of characters that should be removed - rename 'restrict-filenames' to 'path-restrict' - #348, #380
2019-08-16 21:13:49 +02:00
extractor.*.path-restrict
-------------------------
add 'restrict-filenames' option (#348)
2019-07-23 17:36:07 +02:00
=========== =====
extend 'path-restrict' option Allow its value to be a JSON object / Python dict that specifies a mapping from invalid/unwanted input characters to specific output characters. For example {"/": "-", "*": "+"} will transform "foo / ***bar***" into "foo - +++bar+++" (closes #662, #755)
2020-05-22 01:14:10 +02:00
Type ``string`` or ``object``
add 'restrict-filenames' option (#348)
2019-07-23 17:36:07 +02:00
Default ``"auto"``
extend 'path-restrict' option Allow its value to be a JSON object / Python dict that specifies a mapping from invalid/unwanted input characters to specific output characters. For example {"/": "-", "*": "+"} will transform "foo / ***bar***" into "foo - +++bar+++" (closes #662, #755)
2020-05-22 01:14:10 +02:00
Example | ``"/!? (){}"``
| ``{" ": "_", "/": "-", "|": "-", ":": "-", "*": "+"}``
implement a 'path-replace' option (#662, #755)
2020-05-24 17:35:25 +02:00
Description | A string of characters to be replaced with the value of
fix internal links in configuration.rst
2020-05-28 01:51:16 +02:00
`path-replace <extractor.*.path-replace_>`__
implement a 'path-replace' option (#662, #755)
2020-05-24 17:35:25 +02:00
| or an object mapping invalid/unwanted characters to their replacements
| for generated path segment names.
add 'restrict-filenames' option (#348)
2019-07-23 17:36:07 +02:00
Special values:
* ``"auto"``: Use characters from ``"unix"`` or ``"windows"``
depending on the local operating system
* ``"unix"``: ``"/"``
Remove control characters from filesystem paths - add 'path-remove' option to specify the set of characters that should be removed - rename 'restrict-filenames' to 'path-restrict' - #348, #380
2019-08-16 21:13:49 +02:00
* ``"windows"``: ``"\\\\|/<>:\"?*"``
extend 'path-restrict' option Allow its value to be a JSON object / Python dict that specifies a mapping from invalid/unwanted input characters to specific output characters. For example {"/": "-", "*": "+"} will transform "foo / ***bar***" into "foo - +++bar+++" (closes #662, #755)
2020-05-22 01:14:10 +02:00
Note: In a string with 2 or more characters, ``[]^-\`` need to be
Remove control characters from filesystem paths - add 'path-remove' option to specify the set of characters that should be removed - rename 'restrict-filenames' to 'path-restrict' - #348, #380
2019-08-16 21:13:49 +02:00
escaped with backslashes, e.g. ``"\\[\\]"``
=========== =====
implement a 'path-replace' option (#662, #755)
2020-05-24 17:35:25 +02:00
extractor.*.path-replace
------------------------
=========== =====
Type ``string``
Default ``"_"``
Description The replacement character(s) for
fix internal links in configuration.rst
2020-05-28 01:51:16 +02:00
`path-restrict <extractor.*.path-restrict_>`__
implement a 'path-replace' option (#662, #755)
2020-05-24 17:35:25 +02:00
=========== =====
Remove control characters from filesystem paths - add 'path-remove' option to specify the set of characters that should be removed - rename 'restrict-filenames' to 'path-restrict' - #348, #380
2019-08-16 21:13:49 +02:00
extractor.*.path-remove
-----------------------
=========== =====
Type ``string``
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
Default ``"\u0000-\u001f\u007f"`` (ASCII control characters)
Remove control characters from filesystem paths - add 'path-remove' option to specify the set of characters that should be removed - rename 'restrict-filenames' to 'path-restrict' - #348, #380
2019-08-16 21:13:49 +02:00
Description Set of characters to remove from generated path names.
extend 'path-restrict' option Allow its value to be a JSON object / Python dict that specifies a mapping from invalid/unwanted input characters to specific output characters. For example {"/": "-", "*": "+"} will transform "foo / ***bar***" into "foo - +++bar+++" (closes #662, #755)
2020-05-22 01:14:10 +02:00
Note: In a string with 2 or more characters, ``[]^-\`` need to be
Remove control characters from filesystem paths - add 'path-remove' option to specify the set of characters that should be removed - rename 'restrict-filenames' to 'path-restrict' - #348, #380
2019-08-16 21:13:49 +02:00
escaped with backslashes, e.g. ``"\\[\\]"``
add 'restrict-filenames' option (#348)
2019-07-23 17:36:07 +02:00
=========== =====
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
extractor.*.skip
----------------
=========== =====
Type ``bool`` or ``string``
Default ``true``
implement 'enumerate' for 'extractor.skip' (#306) [ci skip]
2019-08-08 18:34:31 +02:00
Description Controls the behavior when downloading files that have been
downloaded before, i.e. a file with the same filename already
update configuration.rst - fix anonymous links - update description of 'extractor.twitter.videos' - document 'extractor.redgifs.format' (#724)
2020-06-12 18:25:17 +02:00
exists or its ID is in a `download archive <extractor.*.archive_>`__.
implement 'enumerate' for 'extractor.skip' (#306) [ci skip]
2019-08-08 18:34:31 +02:00
allow specifying number of skips before abort/exit (closes #115) In addition to 'abort' and 'exit', it is now possible to specify 'abort:N' and 'exit:N' (where N is any integer) as value for 'skip' to abort/exit after consecutively skipping N downloads.
2018-10-13 17:21:55 +02:00
* ``true``: Skip downloads
* ``false``: Overwrite already existing files
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
* ``"abort"``: Abort the current extractor run
allow specifying number of skips before abort/exit (closes #115) In addition to 'abort' and 'exit', it is now possible to specify 'abort:N' and 'exit:N' (where N is any integer) as value for 'skip' to abort/exit after consecutively skipping N downloads.
2018-10-13 17:21:55 +02:00
* ``"abort:N"``: Skip downloads and abort extractor run
after ``N`` consecutive skips
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
* ``"exit"``: Exit the program altogether
allow specifying number of skips before abort/exit (closes #115) In addition to 'abort' and 'exit', it is now possible to specify 'abort:N' and 'exit:N' (where N is any integer) as value for 'skip' to abort/exit after consecutively skipping N downloads.
2018-10-13 17:21:55 +02:00
* ``"exit:N"``: Skip downloads and exit the program
after ``N`` consecutive skips
implement 'enumerate' for 'extractor.skip' (#306) [ci skip]
2019-08-08 18:34:31 +02:00
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
* ``"enumerate"``: Add an enumeration index to the beginning of the
filename extension (``file.1.ext``, ``file.2.ext``, etc.)
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
add option to sleep before each download
2017-12-04 17:06:17 +01:00
extractor.*.sleep
[deviantart] add support for OAuth2 authentication Some user galleries [*] require you to be either logged in or authenticated via OAuth2 to access their deviations. [*] e.g. https://polinaegorussia.deviantart.com/gallery/ -------------- known issue: A deviantart 'refresh_token' can only be used once and gets updated whenever it is used to request a new 'access_token', so storing its initial value in a config file and reusing it again and again is not possible.
2017-12-18 00:12:08 +01:00
-----------------
add option to sleep before each download
2017-12-04 17:06:17 +01:00
=========== =====
Type ``float``
Default ``0``
Description Number of seconds to sleep before each download.
=========== =====
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
extractor.*.username & .password
--------------------------------
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
Type ``string``
Default ``null``
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
Description The username and password to use when attempting to log in to
another site.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
[e621] document username & password support (#640)
2020-03-14 01:13:14 +01:00
Specifying a username and password is required for
* ``pixiv``
* ``nijie``
* ``seiga``
and optional for
* ``danbooru``
* ``e621``
* ``exhentai``
* ``idolcomplex``
[instagram] fix and re-enable login with username&password This reverts commit 3e0848a482463415e40b3f12534da33f8a81d5be. (#756, #771, #797, #803) https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
2020-05-30 18:07:45 +02:00
* ``instagram``
[e621] document username & password support (#640)
2020-03-14 01:13:14 +01:00
* ``luscious``
* ``sankaku``
* ``tsumino``
* ``twitter``
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
These values can also be set via the ``-u/--username`` and
``-p/--password`` command-line options or by using a |.netrc|_ file.
update configuration.rst
2017-06-27 17:44:02 +02:00
(see Authentication_)
[danbooru] add authentication support (closes #151) ... via HTTP Basic Auth with username and "password". The password value in this case is not the account password itself, but the"api_key" found in your user profile.
2019-01-09 14:08:06 +01:00
[e621] document username & password support (#640)
2020-03-14 01:13:14 +01:00
Note: The password values for ``danbooru`` and ``e621`` should be
the API keys found in your user profile, not your actual account
password.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
extractor.*.netrc
-----------------
=========== =====
Type ``bool``
Default ``false``
Description Enable the use of |.netrc|_ authentication data.
=========== =====
update configuration.rst
2017-07-21 18:32:56 +02:00
extractor.*.cookies
-------------------
=========== =====
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
Type |Path|_ or ``object``
update configuration.rst
2017-07-21 18:32:56 +02:00
Default ``null``
add example to 'cookies' docs
2020-02-13 23:44:02 +01:00
Description Source to read additional cookies from. Either as
* the |Path|_ to a Mozilla/Netscape format cookies.txt file or
* a JSON ``object`` specifying cookies as a name-to-value mapping
Example:
.. code::
{
"cookie-name": "cookie-value",
"sessionid" : "14313336321%3AsabDFvuASDnlpb%3A31",
"isAdult" : "1"
}
update configuration.rst
2017-07-21 18:32:56 +02:00
=========== =====
implement 'cookies-update' option (#445)
2019-10-18 21:31:33 +02:00
extractor.*.cookies-update
--------------------------
=========== =====
Type ``bool``
change default value of 'cookies-update' to 'true'
2019-12-05 23:41:25 +01:00
Default ``true``
add example to 'cookies' docs
2020-02-13 23:44:02 +01:00
Description If `extractor.*.cookies`_ specifies the |Path|_ to a cookies.txt
file and it can be opened and parsed without errors,
update its contents with cookies received during data extraction.
implement 'cookies-update' option (#445)
2019-10-18 21:31:33 +02:00
=========== =====
add explicit proxy support (#76) - '--proxy' as command-line argument - 'extractor.*.proxy' as config option
2018-02-19 18:24:56 +01:00
extractor.*.proxy
-----------------
=========== =====
Type ``string`` or ``object``
Default ``null``
Description Proxy (or proxies) to be used for remote connections.
* If this is a ``string``, it is the proxy URL for all
outgoing requests.
* If this is an ``object``, it is a scheme-to-proxy mapping to
specify different proxy URLs for each scheme.
It is also possible to set a proxy for a specific host by using
``scheme://host`` as key.
See `Requests' proxy documentation`_ for more details.
Example:
.. code::
{
"http": "http://10.10.1.10:3128",
"https": "http://10.10.1.10:1080",
"http://10.20.1.128": "http://10.10.1.10:5323"
}
[postprocessor:ugoira] add 'libx264-prevent-odd' option A rather crude workaround for "width/height not divisible by 2" errors when using libx264.
2018-09-21 19:52:45 +02:00
Note: All proxy URLs should include a scheme,
add explicit proxy support (#76) - '--proxy' as command-line argument - 'extractor.*.proxy' as config option
2018-02-19 18:24:56 +01:00
otherwise ``http://`` is assumed.
=========== =====
add 'extractor.*.user-agent' config option
2017-11-15 13:54:40 +01:00
extractor.*.user-agent
----------------------
=========== =====
Type ``string``
add 'ciphers' option; update default User-Agent
2019-07-18 20:06:21 +02:00
Default ``"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"``
add 'extractor.*.user-agent' config option
2017-11-15 13:54:40 +01:00
Description User-Agent header value to be used for HTTP requests.
[postprocessor:ugoira] add 'libx264-prevent-odd' option A rather crude workaround for "width/height not divisible by 2" errors when using libx264.
2018-09-21 19:52:45 +02:00
Note: This option has no effect on `pixiv` and
add 'extractor.*.user-agent' config option
2017-11-15 13:54:40 +01:00
`readcomiconline` extractors, as these need specific values to
function correctly.
=========== =====
add option to specify additional key-value pairs
2018-02-08 23:10:58 +01:00
extractor.*.keywords
--------------------
=========== =====
Type ``object``
add explicit proxy support (#76) - '--proxy' as command-line argument - 'extractor.*.proxy' as config option
2018-02-19 18:24:56 +01:00
Example ``{"type": "Pixel Art", "type_id": 123}``
add option to specify additional key-value pairs
2018-02-08 23:10:58 +01:00
Description Additional key-value pairs to be added to each metadata dictionary.
=========== =====
add 'extractor.*.user-agent' config option
2017-11-15 13:54:40 +01:00
add option to set default replacement field value Missing or undefined keywords will now be replaced with the value set for 'keywords-default'. The default is Python's 'None', which is equivalent to setting this option to JSON's 'null'.
2018-02-21 23:18:21 +01:00
extractor.*.keywords-default
----------------------------
=========== =====
Type any
Default ``"None"``
Description Default value used for missing or undefined keyword names in
format strings.
=========== =====
add 'category-transfer' option [ci skip]
2019-01-19 20:28:19 +01:00
extractor.*.category-transfer
-----------------------------
=========== =====
Type ``bool``
Default Extractor-specific
Description Transfer an extractor's (sub)category values to all child
extractors spawned by it, to let them inherit their parent's
config options.
=========== =====
add '--download-archive' cmdline option … as well as a config file equivalent
2018-02-01 22:00:44 +01:00
extractor.*.archive
-------------------
=========== =====
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
Type |Path|_
add '--download-archive' cmdline option … as well as a config file equivalent
2018-02-01 22:00:44 +01:00
Default ``null``
Description File to store IDs of downloaded files in. Downloads of files
already recorded in this archive file will be skipped_.
The resulting archive file is not a plain text file but an SQLite3
make archive keys user-configurable
2018-02-24 21:21:59 +01:00
database, as either lookup operations are significantly faster or
memory requirements are significantly lower when the
add '--download-archive' cmdline option … as well as a config file equivalent
2018-02-01 22:00:44 +01:00
amount of stored IDs gets reasonably large.
=========== =====
make archive keys user-configurable
2018-02-24 21:21:59 +01:00
extractor.*.archive-format
--------------------------
=========== =====
Type ``string``
Example ``"{id}_{offset}"``
Description An alternative `format string`_ to build archive IDs with.
=========== =====
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
extractor.*.postprocessors
--------------------------
=========== =====
Type ``list`` of |Postprocessor Configuration|_ objects
Example .. code::
[
{"name": "zip", "compression": "zip"},
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
{"name": "exec", "command": ["/home/foobar/script", "{category}", "{image_id}"]}
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
]
Description A list of post-processors to be applied to each downloaded file
in the same order as they are specified.
=========== =====
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
extractor.*.retries
-------------------
=========== =====
Type ``integer``
use values of 'retries' options correctly The RE-tries option now specifies exactly that: the maximum number a failed HTTP request is re-tried. For example a value of 2 will now correctly stop after 3 attempts: the initial one + 2 re-tries. The maximum wait-time now also caps at 30min and increases exponentially for both extractor.request() and downloader.http.download().
2019-06-30 21:27:28 +02:00
Default ``4``
allow '-1' for infinite retries (#300)
2019-06-30 22:55:31 +02:00
Description Maximum number of times a failed HTTP request is retried before
giving up or ``-1`` for infinite retries.
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
=========== =====
extractor.*.timeout
-------------------
=========== =====
Type ``float`` or ``null``
Default ``30``
Description Amount of time (in seconds) to wait for a successful connection
and response from a remote server.
This value gets internally used as the |timeout|_ parameter for the
|requests.request()|_ method.
=========== =====
extractor.*.verify
------------------
=========== =====
Type ``bool`` or ``string``
Default ``true``
Description Controls whether to verify SSL/TLS certificates for HTTPS requests.
If this is a ``string``, it must be the path to a CA bundle to use
instead of the default certificates.
This value gets internally used as the |verify|_ parameter for the
|requests.request()|_ method.
=========== =====
add 'download' option (#220)
2019-07-13 21:49:26 +02:00
extractor.*.download
--------------------
=========== =====
Type ``bool``
Default ``true``
Description Controls whether to download media files.
Setting this to ``false`` won't download any files, but all other
functions (postprocessors_, `download archive`_, etc.)
will be executed as normal.
=========== =====
.. _postprocessors: `extractor.*.postprocessors`_
.. _download archive: `extractor.*.archive`_
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
extractor.*.image-range
-----------------------
=========== =====
Type ``string``
Example | ``"10-20"``,
| ``"-5, 10, 30-50, 100-"``
Description Index-range(s) specifying which images to download.
Note: The index of the first image is ``1``.
=========== =====
extractor.*.chapter-range
-------------------------
=========== =====
Type ``string``
[docs] add examples of how to filter manga chapters by language (closes #599)
2020-02-07 23:21:44 +01:00
Description Like `image-range <extractor.*.image-range_>`__,
but applies to delegated URLs like manga-chapters, etc.
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
=========== =====
extractor.*.image-filter
------------------------
=========== =====
Type ``string``
Example | ``"width >= 1200 and width/height > 1.2"``,
| ``"re.search(r'foo(bar)+', description)"``
Description | Python expression controlling which images to download.
| Files for which the expression evaluates to ``False``
are ignored.
| Available keys are the filename-specific ones listed
by ``-K`` or ``-j``.
=========== =====
extractor.*.chapter-filter
--------------------------
=========== =====
Type ``string``
[docs] add examples of how to filter manga chapters by language (closes #599)
2020-02-07 23:21:44 +01:00
Example | ``"lang == 'en'"``
| ``"language == 'French' and 10 <= chapter < 20"``
Description Like `image-filter <extractor.*.image-filter_>`__,
but applies to delegated URLs like manga-chapters, etc.
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
implement 'image-unique' and 'chapter-unique' options (#303) The default value for both is 'false', i.e. duplicate URLs are NOT ignored. The previous behavior was to always ignore duplicate URLs to make '--abort-on-skip' work properly when new images where added to the beginning of a collection while gallery-dl is running.
2019-06-29 22:48:59 +02:00
extractor.*.image-unique
------------------------
=========== =====
Type ``bool``
Default ``false``
Description Ignore image URLs that have been encountered before during the
current extractor run.
=========== =====
extractor.*.chapter-unique
--------------------------
=========== =====
Type ``bool``
Default ``false``
update configuration.rst - fix anonymous links - update description of 'extractor.twitter.videos' - document 'extractor.redgifs.format' (#724)
2020-06-12 18:25:17 +02:00
Description Like `image-unique <extractor.*.image-unique_>`__,
but applies to delegated URLs like manga-chapters, etc.
implement 'image-unique' and 'chapter-unique' options (#303) The default value for both is 'false', i.e. duplicate URLs are NOT ignored. The previous behavior was to always ignore duplicate URLs to make '--abort-on-skip' work properly when new images where added to the beginning of a collection while gallery-dl is running.
2019-06-29 22:48:59 +02:00
=========== =====
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
extractor.*.date-format
----------------------------
=========== =====
Type ``string``
Default ``"%Y-%m-%dT%H:%M:%S"``
Description Format string used to parse ``string`` values of
`date-min` and `date-max`.
See |strptime|_ for a list of formatting directives.
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
Extractor-specific Options
==========================
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
[artstation] add user-, image- and likes-extractors
2018-03-14 14:03:53 +01:00
extractor.artstation.external
-----------------------------
=========== =====
Type ``bool``
Default ``false``
Description Try to follow external URLs of embedded players.
=========== =====
[blogger] implement video extraction (closes #587)
2020-01-23 22:42:56 +01:00
extractor.blogger.videos
------------------------
=========== =====
Type ``bool``
Default ``true``
Description Download embedded videos hosted on https://www.blogger.com/
=========== =====
[danbooru] add 'ugoira' option (#406) to choose between ZIP archives or converted video files for Ugoira posts
2019-08-31 21:46:49 +02:00
extractor.danbooru.ugoira
-------------------------
=========== =====
Type ``bool``
[danbooru] change default for 'ugoira' to 'false' Downloading the pre-rendered versions should be a better default than .zip files with individual frames.
2020-05-20 19:55:13 +02:00
Default ``false``
[danbooru] add 'ugoira' option (#406) to choose between ZIP archives or converted video files for Ugoira posts
2019-08-31 21:46:49 +02:00
Description Controls the download target for Ugoira posts.
* ``true``: Original ZIP archives
* ``false``: Converted video files
=========== =====
[deviantart] rename 'stash' to 'extra' (#302) 'stash' is already used as a name for the StashExtractor and therefore expected to be a dictionary.
2019-06-10 21:05:25 +02:00
extractor.deviantart.extra
--------------------------
=========== =====
Type ``bool``
Default ``false``
[deviantart] also search journals for sta.sh links (#712) when 'extra' is enabled
2020-06-04 14:47:08 +02:00
Description Download extra Sta.sh resources from
description texts and journals.
[deviantart] rename 'stash' to 'extra' (#302) 'stash' is already used as a name for the StashExtractor and therefore expected to be a dictionary.
2019-06-10 21:05:25 +02:00
Note: Enabling this option also enables deviantart.metadata_.
=========== =====
[deviantart] add "extractor.deviantart.flat" option Setting this to 'false' downloads images into individual subdirectories for each gallery-folder or favourite-collection, otherwise it is just creating a flat list of images.
2017-07-12 17:05:31 +02:00
extractor.deviantart.flat
-------------------------
=========== =====
Type ``bool``
Default ``true``
Description Select the directory structure created by the Gallery- and
Favorite-Extractors.
* ``true``: Use a flat directory structure.
* ``false``: Collect a list of all gallery-folders or
extend output of --list-keywords
2017-08-10 17:36:21 +02:00
favorites-collections and transfer any further work to other
[deviantart] add "extractor.deviantart.flat" option Setting this to 'false' downloads images into individual subdirectories for each gallery-folder or favourite-collection, otherwise it is just creating a flat list of images.
2017-07-12 17:05:31 +02:00
extractors (``folder`` or ``collection``), which will then
create individual subdirectories for each of them.
=========== =====
[deviantart] add 'folders' option (#276)
2019-05-29 23:50:05 +02:00
extractor.deviantart.folders
----------------------------
=========== =====
Type ``bool``
Default ``false``
Description Provide a ``folders`` metadata field that contains the names of all
folders a deviation is present in.
Note: Gathering this information requires a lot of API calls.
Use with caution.
=========== =====
[deviantart] add user profile extractor (#377, #419)
2019-11-06 23:57:12 +01:00
extractor.deviantart.include
----------------------------
=========== =====
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
Type ``string`` or ``list`` of ``strings``
Default ``"gallery"``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example ``"favorite,journal,scraps"`` or ``["favorite", "journal", "scraps"]``
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
Description A (comma-separated) list of subcategories to include
when processing a user profile.
[deviantart] add user profile extractor (#377, #419)
2019-11-06 23:57:12 +01:00
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
Possible values are
``"gallery"``, ``"scraps"``, ``"journal"``, ``"favorite"``.
[deviantart] add user profile extractor (#377, #419)
2019-11-06 23:57:12 +01:00
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
You can use ``"all"`` instead of listing all values separately.
[deviantart] add user profile extractor (#377, #419)
2019-11-06 23:57:12 +01:00
=========== =====
["deviantart] add 'journals' option
2018-07-16 18:14:41 +02:00
extractor.deviantart.journals
-----------------------------
=========== =====
Type ``string``
Default ``"html"``
Description Selects the output format of journal entries.
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* ``"html"``: HTML with (roughly) the same layout as on DeviantArt.
* ``"text"``: Plain text with image references and HTML tags removed.
* ``"none"``: Don't download journals.
["deviantart] add 'journals' option
2018-07-16 18:14:41 +02:00
=========== =====
smaller textual changes - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"
2017-10-09 23:20:17 +02:00
extractor.deviantart.mature
---------------------------
[deviantart] always download original images Deviation-objects returned by the DeviantArt API don't always contain the URL and metadata of the original image ([1]). Getting this information requires an additional API call [2], which is indicated by the 'is_downloadable' and 'download_filesize' metadata within a deviation-object. [1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576 [2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
2017-10-07 13:07:34 +02:00
=========== =====
Type ``bool``
Default ``true``
smaller textual changes - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"
2017-10-09 23:20:17 +02:00
Description Enable mature content.
[deviantart] always download original images Deviation-objects returned by the DeviantArt API don't always contain the URL and metadata of the original image ([1]). Getting this information requires an additional API call [2], which is indicated by the 'is_downloadable' and 'download_filesize' metadata within a deviation-object. [1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576 [2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
2017-10-07 13:07:34 +02:00
smaller textual changes - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"
2017-10-09 23:20:17 +02:00
This option simply sets the |mature_content|_ parameter for API
calls to either ``"true"`` or ``"false"`` and does not do any other
form of content filtering.
[deviantart] always download original images Deviation-objects returned by the DeviantArt API don't always contain the URL and metadata of the original image ([1]). Getting this information requires an additional API call [2], which is indicated by the 'is_downloadable' and 'download_filesize' metadata within a deviation-object. [1] https://myria-moon.deviantart.com/art/Aime-Moi-part-en-vadrouille-261986576 [2] https://www.deviantart.com/developers/http/v1/20160316/deviation_download/bed6982b88949bdb08b52cd6763fcafd
2017-10-07 13:07:34 +02:00
=========== =====
[deviantart] add 'metadata' option (#189)
2019-03-21 14:46:47 +01:00
extractor.deviantart.metadata
-----------------------------
=========== =====
Type ``bool``
Default ``false``
Description Request extended metadata for deviation objects to additionally
provide ``description``, ``tags``, ``license`` and ``is_watching``
fields.
=========== =====
smaller textual changes - swapped doc for deviantart.mature and .original - updated gallery-dl.conf - "transferred" -> "delegated"
2017-10-09 23:20:17 +02:00
extractor.deviantart.original
-----------------------------
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
[deviantart] extend functionality of 'original' option
2018-10-10 21:19:45 +02:00
Type ``bool`` or ``string``
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
Default ``true``
[deviantart] extend functionality of 'original' option
2018-10-10 21:19:45 +02:00
Description Download original files if available.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
[deviantart] extend functionality of 'original' option
2018-10-10 21:19:45 +02:00
Setting this option to ``"images"`` only downloads original
files if they are images and falls back to preview versions for
everything else (archives, etc.).
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
[deviantart] add 'quality' option (#369)
2019-08-02 23:29:38 +02:00
extractor.deviantart.quality
----------------------------
=========== =====
Type ``integer``
Default ``100``
improve 'deviantart.quality' description
2019-08-30 17:57:46 +02:00
Description JPEG quality level of newer images for which
an original file download is not available.
[deviantart] add 'quality' option (#369)
2019-08-02 23:29:38 +02:00
=========== =====
[deviantart] improve public-private token switching - rename option to `prefer-public` - now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
extractor.deviantart.refresh-token
----------------------------------
=========== =====
Type ``string``
Default ``null``
document cache file requirement for DeviantArt refresh tokens
2019-10-13 23:01:57 +02:00
Description The ``refresh-token`` value you get from
[reddit] fix documentation inconsistencies (closes #429) - Require 'reddit.comments' to be a number and convert it to an integer to be extra sure - Link to the README's OAuth section were appropriate
2019-09-27 17:34:10 +02:00
`linking your DeviantArt account to gallery-dl <OAuth_>`__.
[deviantart] improve public-private token switching - rename option to `prefer-public` - now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
document cache file requirement for DeviantArt refresh tokens
2019-10-13 23:01:57 +02:00
Using a ``refresh-token`` allows you to access private or otherwise
[deviantart] improve public-private token switching - rename option to `prefer-public` - now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
not publicly available deviations.
document cache file requirement for DeviantArt refresh tokens
2019-10-13 23:01:57 +02:00
reword note for 'extractor.deviantart.refresh-token' (#612)
2020-02-19 23:39:27 +01:00
Note: The ``refresh-token`` becomes invalid
`after 3 months <https://www.deviantart.com/developers/authentication#refresh>`__
or whenever your `cache file <cache.file_>`__ is deleted or cleared.
[deviantart] improve public-private token switching - rename option to `prefer-public` - now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
=========== =====
[deviantart] add 'wait-min' option
2018-07-14 11:52:21 +02:00
extractor.deviantart.wait-min
-----------------------------
=========== =====
[deviantart] improve public-private token switching - rename option to `prefer-public` - now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
Type ``integer``
[deviantart] add 'wait-min' option
2018-07-14 11:52:21 +02:00
Default ``0``
["deviantart] add 'journals' option
2018-07-16 18:14:41 +02:00
Description Minimum wait time in seconds before API requests.
[deviantart] add 'wait-min' option
2018-07-14 11:52:21 +02:00
Note: This value will internally be rounded up
to the next power of 2.
=========== =====
[exhentai] stop extraction if image limit is exceeded (#141) can be turned off with the `exhentai.limits' option
2019-01-26 18:40:39 +01:00
extractor.exhentai.limits
-------------------------
=========== =====
Type ``bool``
Default ``true``
Description Check image download limits
and stop extraction when they are exceeded.
=========== =====
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
extractor.exhentai.original
---------------------------
=========== =====
Type ``bool``
Default ``true``
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Description Download full-sized original images if available.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
extractor.exhentai.wait-min & .wait-max
---------------------------------------
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
Type ``float``
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
Default ``3.0`` and ``6.0``
Description Minimum and maximum wait time in seconds between each image
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
add 'downloader.http.verify' option (also: change the default 'timeout' from None to 30)
2017-08-31 15:21:08 +02:00
ExHentai detects and blocks automated downloaders.
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
*gallery-dl* waits a randomly selected number of
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
seconds between ``wait-min`` and ``wait-max`` after
each image to prevent getting blocked.
=========== =====
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
extractor.flickr.access-token & .access-token-secret
----------------------------------------------------
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
Type ``string``
Default ``null``
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
Description The ``access_token`` and ``access_token_secret`` values you get
[reddit] fix documentation inconsistencies (closes #429) - Require 'reddit.comments' to be a number and convert it to an integer to be extra sure - Link to the README's OAuth section were appropriate
2019-09-27 17:34:10 +02:00
from `linking your Flickr account to gallery-dl <OAuth_>`__.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
[flickr] improve code and metadata - simplify pagination - add more metadata and slightly change its structure - convert suitable values to int or list - move keys from ["photo"] to the base level - proper video support (#246) - rename method and variable names to better fit with other extractors
2019-05-14 18:12:02 +02:00
extractor.flickr.videos
-----------------------
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
Type ``bool``
[flickr] improve code and metadata - simplify pagination - add more metadata and slightly change its structure - convert suitable values to int or list - move keys from ["photo"] to the base level - proper video support (#246) - rename method and variable names to better fit with other extractors
2019-05-14 18:12:02 +02:00
Default ``true``
Description Extract and download videos.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
update configuration.rst
2017-06-20 16:20:28 +02:00
extractor.flickr.size-max
--------------------------
=========== =====
Type ``integer`` or ``string``
Default ``null``
Description Sets the maximum allowed size for downloaded images.
* If this is an ``integer``, it specifies the maximum image dimension
(width and height) in pixels.
* If this is a ``string``, it should be one of Flickr's format specifiers
update configuration.rst
2017-06-27 18:56:24 +02:00
(``"Original"``, ``"Large"``, ... or ``"o"``, ``"k"``, ``"h"``,
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
``"l"``, ...) to use as an upper limit.
update configuration.rst
2017-06-20 16:20:28 +02:00
=========== =====
[gelbooru] re-enable API use (closes #56) Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00
[furaffinity] support classic layout (#284)
2020-02-12 21:39:43 +01:00
extractor.furaffinity.include
-----------------------------
=========== =====
Type ``string`` or ``list`` of ``strings``
Default ``"gallery"``
Example ``"scraps,favorite"`` or ``["scraps", "favorite"]``
Description A (comma-separated) list of subcategories to include
when processing a user profile.
Possible values are
``"gallery"``, ``"scraps"``, ``"favorite"``.
You can use ``"all"`` instead of listing all values separately.
=========== =====
[gelbooru] re-enable API use (closes #56) Gelbooru's API allows access to all images and is not restricted to the first 20000. This also adds an option to select between API use and manual information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00
extractor.gelbooru.api
----------------------
=========== =====
Type ``bool``
Default ``true``
Description Enable use of Gelbooru's API.
Set this value to `false` if the API has been disabled to switch
to manual information extraction.
=========== =====
update configuration.rst
2017-06-20 16:20:28 +02:00
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
extractor.gfycat.format
-----------------------
=========== =====
Type ``string``
Default ``"mp4"``
Description The name of the preferred animation format, which can be one of
``"mp4"``, ``"webm"``, ``"gif"``, ``"webp"`` or ``"mjpg"``.
If the selected format is not available, ``"mp4"``, ``"webm"``
and ``"gif"`` (in that order) will be tried instead, until an
available format is found.
=========== =====
[hitomi] restore metadata fields from before f33b13a ... and add a 'metadata' option to disable visiting the gallery page and extracting data from it if this is not needed.
2020-03-12 23:39:29 +01:00
extractor.hitomi.metadata
-------------------------
=========== =====
Type ``bool``
Default ``true``
Description Try to extract
``artist``, ``group``, ``parody``, and ``characters``
metadata.
=========== =====
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
extractor.imgur.mp4
-------------------
=========== =====
Type ``bool`` or ``string``
Default ``true``
add 'downloader.http.verify' option (also: change the default 'timeout' from None to 30)
2017-08-31 15:21:08 +02:00
Description Controls whether to choose the GIF or MP4 version of an animation.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
update configuration.rst
2017-06-20 16:20:28 +02:00
* ``true``: Follow Imgur's advice and choose MP4 if the
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
``prefer_video`` flag in an image's metadata is set.
update configuration.rst
2017-06-20 16:20:28 +02:00
* ``false``: Always choose GIF.
* ``"always"``: Always choose MP4.
=========== =====
[instagram] implement 'highlights' option (closes #329)
2019-09-21 23:38:20 +02:00
extractor.instagram.highlights
------------------------------
=========== =====
Type ``bool``
Default ``false``
Description Include *Story Highlights* when downloading a user profile.
(requires authentication)
=========== =====
[instagram] implement 'videos' option (closes #521)
2019-12-19 17:15:41 +01:00
extractor.instagram.videos
--------------------------
=========== =====
Type ``bool``
Default ``true``
Description Download video files.
=========== =====
[kissmanga][readcomiconline] add 'captcha' option (#279) to configure how to handle CAPTCHA page redirects: - either interactively wait for the user to solve the CAPTCHA - or raise StopExtraction like before
2019-05-27 22:24:48 +02:00
extractor.kissmanga.captcha
---------------------------
=========== =====
Type ``string``
Default ``"stop"``
Description Controls how to handle redirects to CAPTCHA pages.
* ``"stop``: Stop the current extractor run.
* ``"wait``: Ask the user to solve the CAPTCHA and wait.
=========== =====
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
extractor.newgrounds.include
----------------------------
=========== =====
Type ``string`` or ``list`` of ``strings``
Default ``"art"``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example ``"movies,audio"`` or ``["movies", "audio"]``
match user profile handling on deviantart and newgrounds
2019-11-22 23:20:21 +01:00
Description A (comma-separated) list of subcategories to include
when processing a user profile.
Possible values are
``"art"``, ``"audio"``, ``"movies"``.
You can use ``"all"`` instead of listing all values separately.
=========== =====
update configuration.rst
2017-06-20 16:20:28 +02:00
extractor.oauth.browser
-----------------------
=========== =====
Type ``bool``
Default ``true``
update configuration.rst
2017-06-27 18:56:24 +02:00
Description Controls how a user is directed to an OAuth authorization site.
update configuration.rst
2017-06-20 16:20:28 +02:00
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
* ``true``: Use Python's |webbrowser.open()|_ method to automatically
update configuration.rst
2017-06-20 16:20:28 +02:00
open the URL in the user's browser.
* ``false``: Ask the user to copy & paste an URL from the terminal.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
write OAuth token to cache by default (#616)
2020-05-25 22:19:58 +02:00
extractor.oauth.cache
---------------------
=========== =====
Type ``bool``
Default ``true``
Description Store tokens received during OAuth authorizations
in `cache <cache.file_>`__.
=========== =====
[oauth] add 'port' option (#604)
2020-02-09 13:45:44 +01:00
extractor.oauth.port
--------------------
=========== =====
Type ``integer``
Default ``6414``
Description Port number to listen on during OAuth authorization.
Note: All redirects will go to http://localhost:6414/, regardless
of the port specified here. You'll have to manually adjust the
port number in your browser's address bar when using a different
port than the default.
=========== =====
[photobucket] download subalbums (#117)
2019-01-21 19:55:05 +01:00
extractor.photobucket.subalbums
-------------------------------
=========== =====
Type ``bool``
Default ``true``
Description Download subalbums.
=========== =====
[pinterest] add support for sections (closes #835)
2020-06-16 14:41:05 +02:00
extractor.pinterest.sections
----------------------------
=========== =====
Type ``bool``
Default ``true``
Description Include pins from board sections.
=========== =====
[pixiv] implement 'avatar' option (#595, #623)
2020-03-09 21:17:16 +01:00
extractor.pixiv.user.avatar
---------------------------
=========== =====
Type ``bool``
Default ``false``
Description Download user avatars.
=========== =====
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
extractor.pixiv.ugoira
----------------------
=========== =====
Type ``bool``
Default ``true``
Description Download Pixiv's Ugoira animations or ignore them.
link to post processor example in 'pixiv.ugoira' docs
2020-02-14 01:22:16 +01:00
These animations come as a ``.zip`` file containing all
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
animation frames in JPEG format.
link to post processor example in 'pixiv.ugoira' docs
2020-02-14 01:22:16 +01:00
Use an `ugoira`_ post processor to convert them
to watchable videos. (Example__)
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
update configuration.rst - fix anonymous links - update description of 'extractor.twitter.videos' - document 'extractor.redgifs.format' (#724)
2020-06-12 18:25:17 +02:00
.. __: https://github.com/mikf/gallery-dl/blob/v1.12.3/docs/gallery-dl-example.conf#L9-L14
link to post processor example in 'pixiv.ugoira' docs
2020-02-14 01:22:16 +01:00
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
[plurk] add a 'comments' options (#212)
2019-04-14 21:52:23 +02:00
extractor.plurk.comments
------------------------
=========== =====
Type ``bool``
Default ``false``
Description Also search Plurk comments for URLs.
=========== =====
[reactor] add wait-min & -max options (#148)
2019-01-07 18:04:16 +01:00
extractor.reactor.wait-min & .wait-max
--------------------------------------
=========== =====
Type ``float``
Default ``3.0`` and ``6.0``
Description Minimum and maximum wait time in seconds between HTTP requests
during the extraction process.
=========== =====
[kissmanga][readcomiconline] add 'captcha' option (#279) to configure how to handle CAPTCHA page redirects: - either interactively wait for the user to solve the CAPTCHA - or raise StopExtraction like before
2019-05-27 22:24:48 +02:00
extractor.readcomiconline.captcha
---------------------------------
=========== =====
Type ``string``
Default ``"stop"``
Description Controls how to handle redirects to CAPTCHA pages.
* ``"stop``: Stop the current extractor run.
* ``"wait``: Ask the user to solve the CAPTCHA and wait.
=========== =====
update configuration.rst
2017-07-04 19:34:34 +02:00
extractor.recursive.blacklist
-----------------------------
=========== =====
Type ``list`` of ``strings``
Default ``["directlink", "oauth", "recursive", "test"]``
Description A list of extractor categories which should be ignored when using
the ``recursive`` extractor.
=========== =====
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
extractor.reddit.comments
-------------------------
=========== =====
[reddit] fix documentation inconsistencies (closes #429) - Require 'reddit.comments' to be a number and convert it to an integer to be extra sure - Link to the README's OAuth section were appropriate
2019-09-27 17:34:10 +02:00
Type ``integer``
[reddit] change default value for 'comments' to '0'
2019-12-20 16:54:59 +01:00
Default ``0``
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
Description The value of the ``limit`` parameter when loading
a submission and its comments.
This number (roughly) specifies the total amount of comments
being retrieved with the first API call.
Reddit's internal default and maximum values for this parameter
appear to be 200 and 500 respectively.
[reddit] fix documentation inconsistencies (closes #429) - Require 'reddit.comments' to be a number and convert it to an integer to be extra sure - Link to the README's OAuth section were appropriate
2019-09-27 17:34:10 +02:00
The value ``0`` ignores all comments and significantly reduces the
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
time required when scanning a subreddit.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
extractor.reddit.morecomments
add .netrc support (#22) Use the '--netrc' cmdline option or set the 'netrc' config option to 'true' to enable the use of .netrc authentication data. The 'machine' names for the .netrc info are the lowercase extractor names (or categories): batoto, exhentai, nijie, pixiv, seiga.
2017-06-24 12:17:26 +02:00
-----------------------------
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
=========== =====
Type ``bool``
Default ``false``
Description Retrieve additional comments by resolving the ``more`` comment
update configuration.rst
2017-06-27 17:44:02 +02:00
stubs in the base comment tree.
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
This requires 1 additional API call for every 100 extra comments.
=========== =====
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
extractor.reddit.date-min & .date-max
-------------------------------------
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
Type |Date|_
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
Default ``0`` and ``253402210800`` (timestamp of |datetime.max|_)
Description Ignore all submissions posted before/after this date.
update configuration.rst
2017-07-04 19:34:34 +02:00
=========== =====
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
extractor.reddit.id-min & .id-max
---------------------------------
update configuration.rst
2017-07-04 19:34:34 +02:00
=========== =====
Type ``string``
Example ``"6kmzv2"``
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
Description Ignore all submissions posted before/after the submission with
this ID.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
extractor.reddit.recursion
--------------------------
=========== =====
Type ``integer``
Default ``0``
Description Reddit extractors can recursively visit other submissions
linked to in the initial set of submissions.
This value sets the maximum recursion depth.
Special values:
* ``0``: Recursion is disabled
* ``-1``: Infinite recursion (don't do this)
=========== =====
extractor.reddit.refresh-token
------------------------------
=========== =====
Type ``string``
Default ``null``
document cache file requirement for DeviantArt refresh tokens
2019-10-13 23:01:57 +02:00
Description The ``refresh-token`` value you get from
[reddit] fix documentation inconsistencies (closes #429) - Require 'reddit.comments' to be a number and convert it to an integer to be extra sure - Link to the README's OAuth section were appropriate
2019-09-27 17:34:10 +02:00
`linking your Reddit account to gallery-dl <OAuth_>`__.
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
document cache file requirement for DeviantArt refresh tokens
2019-10-13 23:01:57 +02:00
Using a ``refresh-token`` allows you to access private or otherwise
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
not publicly available subreddits, given that your account is
authorized to do so,
but requests to the reddit API are going to be rate limited
at 600 requests every 10 minutes/600 seconds.
add a list of all configuration options (the format might be subject to change; putting everything in one single table is even worse)
2017-06-18 22:16:26 +02:00
=========== =====
[reddit] add 'videos' option
2020-01-31 23:45:02 +01:00
extractor.reddit.videos
-----------------------
=========== =====
Type ``bool`` or ``string``
Default ``true``
Description Control video download behavior.
* ``true``: Download videos and use `youtube-dl`_ to handle
HLS and DASH manifests
* ``"ytdl"``: Download videos and let `youtube-dl`_ handle all of
video extraction and download
* ``false``: Ignore videos
=========== =====
update configuration.rst - fix anonymous links - update description of 'extractor.twitter.videos' - document 'extractor.redgifs.format' (#724)
2020-06-12 18:25:17 +02:00
extractor.redgifs.format
------------------------
=========== =====
Type ``string``
Default ``"mp4"``
Description The name of the preferred format, which can be one of
``"mp4"``, ``"webm"``, ``"gif"``, ``"webp"``, ``"mobile"``,
or ``"mini"``.
If the selected format is not available, ``"mp4"``, ``"webm"``
and ``"gif"`` (in that order) will be tried instead, until an
available format is found.
=========== =====
[sankaku] rewrite/improve (fixes #44) - add wait-time between HTTP requests similar to exhentai - add 'wait-min' and 'wait-max' options - increase retry-count for HTTP requests to 10 - implement user authentication (non-authenticated users can only view images up to page 25) - implement 'skip()' functionality (only works up to page 50) - implement image-retrieval for pages >= 51 - fix issue with multiple tags
2017-10-14 23:01:33 +02:00
extractor.sankaku.wait-min & .wait-max
--------------------------------------
=========== =====
Type ``float``
[sankaku] increase default values for 'wait-min/-max'
2018-08-03 17:06:51 +02:00
Default ``3.0`` and ``6.0``
[sankaku] rewrite/improve (fixes #44) - add wait-time between HTTP requests similar to exhentai - add 'wait-min' and 'wait-max' options - increase retry-count for HTTP requests to 10 - implement user authentication (non-authenticated users can only view images up to page 25) - implement 'skip()' functionality (only works up to page 50) - implement image-retrieval for pages >= 51 - fix issue with multiple tags
2017-10-14 23:01:33 +02:00
Description Minimum and maximum wait time in seconds between each image
Sankaku Channel responds with ``429 Too Many Requests`` if it
receives too many HTTP requests in a certain amount of time.
Waiting a few seconds between each request tries to prevent that.
=========== =====
[smugmug] improve format selection (closes #183) - use original image if available - support video formats - remove user info for ImageExtractor (it is no longer possible to get image owner information for a single image)
2019-03-10 15:20:35 +01:00
extractor.smugmug.videos
------------------------
=========== =====
Type ``bool``
Default ``true``
Description Download video files.
=========== =====
[tumblr] add option to download avatars (#137)
2018-12-26 14:29:30 +01:00
extractor.tumblr.avatar
-----------------------
=========== =====
Type ``bool``
Default ``false``
Description Download blog avatars.
=========== =====
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
extractor.tumblr.date-min & .date-max
-------------------------------------
=========== =====
Type |Date|_
Default ``0`` and ``null``
Description Ignore all posts published before/after this date.
=========== =====
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
extractor.tumblr.external
-------------------------
=========== =====
Type ``bool``
Default ``false``
Description Follow external URLs (e.g. from "Link" posts) and try to extract
images from them.
=========== =====
extractor.tumblr.inline
-----------------------
=========== =====
Type ``bool``
[tumblr] change default values for post types and inline media
2018-12-26 14:35:59 +01:00
Default ``true``
[tumblr] extend 'reblogs' functionality (#103) Setting 'reblogs' to "deleted" will check if the parent post of a reblog has been deleted and download its media content if that is the case, otherwise it will be skipped. This is a rather costly operation (1 API request per reblogged post) and should therefore be used with care.
2018-09-07 18:32:45 +02:00
Description Search posts for inline images and videos.
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
=========== =====
[tumblr] add option to filter reblogged posts (#61) Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:00:25 +01:00
extractor.tumblr.reblogs
------------------------
=========== =====
[tumblr] extend 'reblogs' functionality (#103) Setting 'reblogs' to "deleted" will check if the parent post of a reblog has been deleted and download its media content if that is the case, otherwise it will be skipped. This is a rather costly operation (1 API request per reblogged post) and should therefore be used with care.
2018-09-07 18:32:45 +02:00
Type ``bool`` or ``string``
[tumblr] change `reblogs` default value to `true` (#61)
2018-01-06 15:52:08 +01:00
Default ``true``
[tumblr] extend 'reblogs' functionality (#103) Setting 'reblogs' to "deleted" will check if the parent post of a reblog has been deleted and download its media content if that is the case, otherwise it will be skipped. This is a rather costly operation (1 API request per reblogged post) and should therefore be used with care.
2018-09-07 18:32:45 +02:00
Description * ``true``: Extract media from reblogged posts
* ``false``: Skip reblogged posts
[tumblr] change 'reblogs' option (#103) - rename "deleted" to "same-blog" - change test for deleted original post to test if original post owner has the same UUID (full blog name) as the one being downloaded from - add 'blog[uuid]' metadata to allow comparison with 'reblogged_from_uuid'
2018-09-10 15:40:25 +02:00
* ``"same-blog"``: Skip reblogged posts unless the original post
[postprocessor:ugoira] add 'libx264-prevent-odd' option A rather crude workaround for "width/height not divisible by 2" errors when using libx264.
2018-09-21 19:52:45 +02:00
is from the same blog
[tumblr] add option to filter reblogged posts (#61) Reblogs are ignored by default, but can be included by setting 'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:00:25 +01:00
=========== =====
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
extractor.tumblr.posts
----------------------
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Type ``string`` or ``list`` of ``strings``
[tumblr] change default values for post types and inline media
2018-12-26 14:35:59 +01:00
Default ``"all"``
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Example ``"video,audio,link"`` or ``["video", "audio", "link"]``
Description A (comma-separated) list of post types to extract images, etc. from.
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
Possible types are ``text``, ``quote``, ``link``, ``answer``,
``video``, ``audio``, ``photo``, ``chat``.
You can use ``"all"`` instead of listing all types separately.
=========== =====
[yandere] add option to split tags by type (#92)
2018-06-29 19:38:53 +02:00
[twitter] add option to filter media from quoted tweets (#854)
2020-06-24 21:13:16 +02:00
extractor.twitter.quoted
------------------------
=========== =====
Type ``bool``
Default ``true``
Description Fetch media from quoted Tweets.
=========== =====
[twitter] add 'replies' option (closes #705)
2020-04-29 23:11:24 +02:00
extractor.twitter.replies
-------------------------
=========== =====
Type ``bool``
Default ``true``
[twitter] add option to filter media from quoted tweets (#854)
2020-06-24 21:13:16 +02:00
Description Fetch media from replies to other Tweets.
[twitter] add 'replies' option (closes #705)
2020-04-29 23:11:24 +02:00
=========== =====
[twitter] add support for user-timelines (closes #96) also adds a 'retweets' option to filter retweeted content
2018-08-17 20:04:11 +02:00
extractor.twitter.retweets
--------------------------
=========== =====
Type ``bool``
Default ``true``
[twitter] add option to filter media from quoted tweets (#854)
2020-06-24 21:13:16 +02:00
Description Fetch media from Retweets.
[twitter] add support for user-timelines (closes #96) also adds a 'retweets' option to filter retweeted content
2018-08-17 20:04:11 +02:00
=========== =====
[twitter] add option to extract TwitPic embeds (#579)
2020-01-18 21:26:46 +01:00
extractor.twitter.twitpic
-------------------------
=========== =====
Type ``bool``
Default ``false``
Description Extract `TwitPic <https://twitpic.com/>`__ embeds.
=========== =====
[twitter] add experimental 'videos' option (#99) Enabling this option will detect videos in tweets and output them as "unsupported" URLs, so that these can then be downloaded with youtube-dl There are a lot of improvements to be made to the current implementation, but it works and does what it is supposed to, even if inefficient as can be ...
2018-09-30 18:41:39 +02:00
extractor.twitter.videos
------------------------
=========== =====
[twitter] extend 'videos' option to force extraction with ytdl (closes #459)
2019-11-01 22:06:07 +01:00
Type ``bool`` or ``string``
[twitter] change default value for 'videos' to 'true' Every other 'videos' option defaulted to 'true', except Twitter.
2020-02-14 01:03:42 +01:00
Default ``true``
[twitter] extend 'videos' option to force extraction with ytdl (closes #459)
2019-11-01 22:06:07 +01:00
Description Control video download behavior.
update configuration.rst - fix anonymous links - update description of 'extractor.twitter.videos' - document 'extractor.redgifs.format' (#724)
2020-06-12 18:25:17 +02:00
* ``true``: Download videos
* ``"ytdl"``: Download videos using `youtube-dl`_
[twitter] extend 'videos' option to force extraction with ytdl (closes #459)
2019-11-01 22:06:07 +01:00
* ``false``: Skip video Tweets
[twitter] add experimental 'videos' option (#99) Enabling this option will detect videos in tweets and output them as "unsupported" URLs, so that these can then be downloaded with youtube-dl There are a lot of improvements to be made to the current implementation, but it works and does what it is supposed to, even if inefficient as can be ...
2018-09-30 18:41:39 +02:00
=========== =====
[vsco] implement 'videos' option
2019-12-19 17:20:51 +01:00
extractor.vsco.videos
---------------------
=========== =====
Type ``bool``
Default ``true``
Description Download video files.
=========== =====
[wallhaven] rewrite - use API - remove login support, add 'api-key' option - remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric IDs that can't be translated to the new ID system - support direct links to wallpapers
2019-05-30 23:11:36 +02:00
extractor.wallhaven.api-key
---------------------------
=========== =====
Type ``string``
Default ``null``
Description Your `API Key <https://wallhaven.cc/settings/account>`__ to use
your account's browsing settings and default filters when searching.
See https://wallhaven.cc/help/api for more information.
=========== =====
[weibo] add 'videos' option
2020-04-29 23:27:29 +02:00
extractor.weibo.retweets
------------------------
=========== =====
Type ``bool``
Default ``true``
Description Extract media from retweeted posts.
=========== =====
extractor.weibo.videos
----------------------
=========== =====
Type ``bool``
Default ``true``
Description Download video files.
=========== =====
[sankaku] add 'tags' option (#94)
2018-07-13 16:20:14 +02:00
extractor.[booru].tags
[yandere] add option to split tags by type (#92)
2018-06-29 19:38:53 +02:00
----------------------
=========== =====
Type ``bool``
Default ``false``
generalize tag-splitting option (#92) - extend functionality to other booru sites: - http://behoimi.org/ - https://konachan.com/ - https://e621.net/ - https://rule34.xxx/ - https://safebooru.org/ - https://yande.re/
2018-07-01 22:28:52 +02:00
Description Categorize tags by their respective types
and provide them as ``tags_<type>`` metadata fields.
[yandere] add option to split tags by type (#92)
2018-06-29 19:38:53 +02:00
Note: This requires 1 additional HTTP request for each post.
=========== =====
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
implement 'chapter-reverse' option (#149) Setting it to `true` will start with the latest chapter instead of the first one.
2019-01-07 18:22:33 +01:00
extractor.[manga-extractor].chapter-reverse
-------------------------------------------
=========== =====
Type ``bool``
Default ``false``
Description Reverse the order of chapter URLs extracted from manga pages.
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* ``true``: Start with the latest chapter
* ``false``: Start with the first chapter
implement 'chapter-reverse' option (#149) Setting it to `true` will start with the latest chapter instead of the first one.
2019-01-07 18:22:33 +01:00
=========== =====
[sankaku] rewrite/improve (fixes #44) - add wait-time between HTTP requests similar to exhentai - add 'wait-min' and 'wait-max' options - increase retry-count for HTTP requests to 10 - implement user authentication (non-authenticated users can only view images up to page 25) - implement 'skip()' functionality (only works up to page 50) - implement image-retrieval for pages >= 51 - fix issue with multiple tags
2017-10-14 23:01:33 +02:00
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Downloader Options
==================
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.*.enabled
--------------------
=========== =====
Type ``bool``
Default ``true``
Description Enable/Disable this downloader module.
=========== =====
add 'mtime' option
2019-06-20 17:19:44 +02:00
downloader.*.mtime
------------------
=========== =====
Type ``bool``
Default ``true``
Description Use |Last-Modified|_ HTTP response headers
to set file modification times.
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.*.part
-----------------
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
Type ``bool``
Default ``true``
Description Controls the use of ``.part`` files during file downloads.
* ``true``: Write downloaded data into ``.part`` files and rename
them upon download completion. This mode additionally supports
resuming incomplete downloads.
* ``false``: Do not use ``.part`` files and write data directly
into the actual output files.
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.*.part-directory
---------------------------
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Type |Path|_
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Default ``null``
Description Alternate location for ``.part`` files.
Missing directories will be created as needed.
If this value is ``null``, ``.part`` files are going to be stored
alongside the actual output files.
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.*.rate
-----------------
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
Type ``string``
Default ``null``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example ``"32000"``, ``"500k"``, ``"2.5M"``
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Description Maximum download rate in bytes per second.
Possible values are valid integer or floating-point numbers
optionally followed by one of ``k``, ``m``. ``g``, ``t`` or ``p``.
These suffixes are case-insensitive.
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.*.retries
--------------------
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
Type ``integer``
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
Default `extractor.*.retries`_
allow '-1' for infinite retries (#300)
2019-06-30 22:55:31 +02:00
Description Maximum number of retries during file downloads
or ``-1`` for infinite retries.
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.*.timeout
--------------------
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
Type ``float`` or ``null``
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
Default `extractor.*.timeout`_
Description Connection timeout during file downloads.
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.*.verify
-------------------
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
Type ``bool`` or ``string``
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
Default `extractor.*.verify`_
Description Certificate validation during file downloads.
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
[downloader:http] add 'adjust-extensions' option
2019-08-07 22:52:29 +02:00
downloader.http.adjust-extensions
---------------------------------
=========== =====
Type ``bool``
Default ``true``
Description Check the file headers of ``jpg``, ``png``, and ``gif`` files
and adjust their filename extensions if they do not match.
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
downloader.ytdl.format
----------------------
add option to enable/disable specific downloader modules ... and write URLs with no (active) downloader to unsupported-file
2018-11-13 18:06:36 +01:00
=========== =====
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
Type ``string``
Default youtube-dl's default, currently ``"bestvideo+bestaudio/best"``
Description Video `format selection
update links to youtube-dl
2019-03-12 21:13:33 +01:00
<https://github.com/ytdl-org/youtube-dl#format-selection>`__
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
directly passed to youtube-dl.
add option to enable/disable specific downloader modules ... and write URLs with no (active) downloader to unsupported-file
2018-11-13 18:06:36 +01:00
=========== =====
[downloader.ytdl] add 'forward-cookies' option (#352) The "long" name is necessary because just calling it 'cookies' would clash with how the lookup for '--cookies' is implemented.
2019-07-24 21:19:11 +02:00
downloader.ytdl.forward-cookies
-------------------------------
=========== =====
Type ``bool``
[downloader:ytdl] change 'forward-cookies' default to 'false' There are currently no situations where forwarding gallery-dl's cookies to youtube-dl is necessary, and it only causes problems when forcing youtube-dl for Twitter video downloads while logged in.
2020-05-12 20:13:04 +02:00
Default ``false``
[downloader.ytdl] add 'forward-cookies' option (#352) The "long" name is necessary because just calling it 'cookies' would clash with how the lookup for '--cookies' is implemented.
2019-07-24 21:19:11 +02:00
Description Forward cookies to youtube-dl.
=========== =====
[downloader:ytdl] add several options The "default" downloader options (rate, retries, timeout, verify) are mapped to corresponding youtube-dl options. downloader.ytdl.logging tells the downloader to pass youtube-dl's output to a Logger object. downloader.ytdl.raw-options allows to pass arbitrary options to the YoutubeDL constructor.
2018-10-19 22:10:59 +02:00
downloader.ytdl.logging
-----------------------
=========== =====
Type ``bool``
Default ``true``
Description | Route youtube-dl's output through gallery-dl's logging system.
| Otherwise youtube-dl will write its output directly to stdout/stderr.
Note: Set ``quiet`` and ``no_warnings`` in
`downloader.ytdl.raw-options`_ to ``true`` to suppress all output.
=========== =====
[downloader:ytdl] add 'outtmpl' option (#395)
2019-08-24 22:39:37 +02:00
downloader.ytdl.outtmpl
-----------------------
=========== =====
Type ``string``
Default ``null``
Description The `Output Template <https://github.com/ytdl-org/youtube-dl#output-template>`__
used to generate filenames for files downloaded with youtube-dl.
Special values:
* ``null``: generate filenames with `extractor.*.filename`_
* ``"default"``: use youtube-dl's default, currently ``"%(title)s-%(id)s.%(ext)s"``
Note: An output template other than ``null`` might
cause unexpected results in combination with other options
(e.g. ``"skip": "enumerate"``)
=========== =====
[downloader:ytdl] add several options The "default" downloader options (rate, retries, timeout, verify) are mapped to corresponding youtube-dl options. downloader.ytdl.logging tells the downloader to pass youtube-dl's output to a Logger object. downloader.ytdl.raw-options allows to pass arbitrary options to the YoutubeDL constructor.
2018-10-19 22:10:59 +02:00
downloader.ytdl.raw-options
---------------------------
=========== =====
Type ``object``
Example .. code::
{
"quiet": true,
"writesubtitles": true,
"merge_output_format": "mkv"
}
Description | Additional options passed directly to the ``YoutubeDL`` constructor.
| All available options can be found in `youtube-dl's docstrings
update links to youtube-dl
2019-03-12 21:13:33 +01:00
<https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L138-L318>`__.
[downloader:ytdl] add several options The "default" downloader options (rate, retries, timeout, verify) are mapped to corresponding youtube-dl options. downloader.ytdl.logging tells the downloader to pass youtube-dl's output to a Logger object. downloader.ytdl.raw-options allows to pass arbitrary options to the YoutubeDL constructor.
2018-10-19 22:10:59 +02:00
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Output Options
==============
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
output.mode
-----------
=========== =====
Type ``string``
Default ``"auto"``
Description Controls the output string format and status indicators.
* ``"null"``: No output
* ``"pipe"``: Suitable for piping to other processes or files
* ``"terminal"``: Suitable for the standard Windows console
* ``"color"``: Suitable for terminals that understand ANSI escape codes and colors
* ``"auto"``: Automatically choose the best suitable output mode
=========== =====
output.shorten
--------------
=========== =====
Type ``bool``
Default ``true``
Description Controls whether the output strings should be shortened to fit
on one console line.
=========== =====
output.progress
---------------
=========== =====
Type ``bool`` or ``string``
Default ``true``
Description Controls the progress indicator when *gallery-dl* is run with
multiple URLs as arguments.
* ``true``: Show the default progress indicator
(``"[{current}/{total}] {url}"``)
* ``false``: Do not show any progress indicator
* Any ``string``: Show the progress indicator using this
as a custom `format string`_. Possible replacement keys are
``current``, ``total`` and ``url``.
=========== =====
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
output.log
----------
=========== =====
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Type ``string`` or |Logging Configuration|_
update configuration.rst - fix default values for 'log' and 'unsupportedfile' [ci skip]
2018-05-27 17:08:22 +02:00
Default ``"[{name}][{levelname}] {message}"``
Description Configuration for standard logging output to stderr.
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
If this is a simple ``string``, it specifies
the format string for logging messages.
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
output.logfile
--------------
=========== =====
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Type |Path|_ or |Logging Configuration|_
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Default ``null``
Description File to write logging output to.
=========== =====
output.unsupportedfile
----------------------
=========== =====
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Type |Path|_ or |Logging Configuration|_
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Default ``null``
Description File to write external URLs unsupported by *gallery-dl* to.
update configuration.rst - fix default values for 'log' and 'unsupportedfile' [ci skip]
2018-05-27 17:08:22 +02:00
The default format string here is ``"{message}"``.
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
=========== =====
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
output.num-to-str
-----------------
=========== =====
Type ``bool``
Default ``false``
Description Convert numeric values (``integer`` or ``float``) to ``string``
before outputting them as JSON.
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
Postprocessor Options
=====================
classify
--------
Categorize files by filename extension
classify.mapping
----------------
=========== =====
Type ``object``
Default .. code::
{
"Pictures" : ["jpg", "jpeg", "png", "gif", "bmp", "svg", "webp"],
"Video" : ["flv", "ogv", "avi", "mp4", "mpg", "mpeg", "3gp", "mkv", "webm", "vob", "wmv"],
"Music" : ["mp3", "aac", "flac", "ogg", "wma", "m4a", "wav"],
"Archives" : ["zip", "rar", "7z", "tar", "gz", "bz2"]
}
Description A mapping from directory names to filename extensions that should
be stored in them.
Files with an extension not listed will be ignored and stored
in their default location.
=========== =====
implement a post-processor module to compare file versions (#530)
2020-01-18 21:08:25 +01:00
compare
-------
| Compare versions of the same file and replace/enumerate them on mismatch
miscellaneous fixes
2020-01-19 22:53:06 +01:00
| (requires `downloader.*.part`_ = ``true`` and `extractor.*.skip`_ = ``false``)
implement a post-processor module to compare file versions (#530)
2020-01-18 21:08:25 +01:00
compare.action
--------------
=========== =====
Type ``string``
Default ``"replace"``
Description The action to take when files do not compare as equal.
* ``"replace"``: Replace/Overwrite the old version with the new one
* ``"enumerate"``: Add an enumeration index to the filename of the new
version like `skip = "enumerate" <extractor.*.skip_>`__
=========== =====
compare.shallow
---------------
=========== =====
Type ``bool``
Default ``false``
Description Only compare file sizes. Do not read and compare their content.
=========== =====
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
exec
----
Execute external commands.
exec.async
----------
=========== =====
Type ``bool``
Default ``false``
Description Controls whether to wait for a subprocess to finish
or to let it run asynchronously.
=========== =====
exec.command
------------
=========== =====
[postprocessor:exec] improve (#421, #413) - add 'final' option - include job status in pp finalization - improve and extend documentation
2019-11-03 21:45:45 +01:00
Type ``string`` or ``list`` of ``strings``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example | ``"convert {} {}.png && rm {}"``,
| ``["echo", "{user[account]}", "{id}"]``
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
Description The command to run.
[postprocessor:exec] improve (#421, #413) - add 'final' option - include job status in pp finalization - improve and extend documentation
2019-11-03 21:45:45 +01:00
* If this is a ``string``, it will be executed using the system's
shell, e.g. ``/bin/sh``. Any ``{}`` will be replaced
with the full path of a file or target directory, depending on
`exec.final`_
* If this is a ``list``, the first element specifies the program
name and any further elements its arguments.
Each element of this list is treated as a `format string`_ using
the files' metadata as well as ``{_path}``, ``{_directory}``,
and ``{_filename}``.
=========== =====
exec.final
----------
=========== =====
Type ``bool``
Default ``false``
Description Controls whether to execute `exec.command`_ for each
downloaded file or only once after all files
have been downloaded successfully.
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
=========== =====
[postprocessor:metadata] add 'tags' and 'custom' modes (#135)
2019-01-17 21:18:12 +01:00
metadata
--------
Write image metadata to separate files
metadata.mode
-------------
=========== =====
Type ``string``
Default ``"json"``
Description Select how to write metadata.
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* ``"json"``: all metadata using `json.dump()
[postprocessor:metadata] add 'tags' and 'custom' modes (#135)
2019-01-17 21:18:12 +01:00
<https://docs.python.org/3/library/json.html#json.dump>`_
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* ``"tags"``: ``tags`` separated by newlines
[postprocessor:metadata] rename 'format' to 'content-format' Just to be consistent with the other 'extension-format' option name, and only 'format' is also still accepted.
2019-11-30 17:27:49 +01:00
* ``"custom"``: result of applying `metadata.content-format`_
to a file's metadata dictionary
[postprocessor:metadata] add 'tags' and 'custom' modes (#135)
2019-01-17 21:18:12 +01:00
=========== =====
[postprocessor:metadata] add 'directory' option (#520)
2020-01-02 20:58:10 +01:00
metadata.directory
------------------
=========== =====
Type ``string``
Default ``"."``
Example ``"metadata"``
Description Directory where metadata files are stored in relative to the
current target location for file downloads.
=========== =====
[postprocessor:metadata] add 'tags' and 'custom' modes (#135)
2019-01-17 21:18:12 +01:00
metadata.extension
------------------
=========== =====
Type ``string``
Default ``"json"`` or ``"txt"``
[postprocessor:metadata] implement 'extension-format' option closes #477
2019-11-29 23:12:22 +01:00
Description Filename extension for metadata files that will be appended to the
original file names.
=========== =====
metadata.extension-format
-------------------------
=========== =====
Type ``string``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example | ``"{extension}.json"``,
| ``"json"``
[postprocessor:metadata] implement 'extension-format' option closes #477
2019-11-29 23:12:22 +01:00
Description Custom format string to build filename extensions for metadata
files with, which will replace the original filename extensions.
Note: `metadata.extension`_ is ignored if this option is set.
[postprocessor:metadata] add 'tags' and 'custom' modes (#135)
2019-01-17 21:18:12 +01:00
=========== =====
[postprocessor:metadata] rename 'format' to 'content-format' Just to be consistent with the other 'extension-format' option name, and only 'format' is also still accepted.
2019-11-30 17:27:49 +01:00
metadata.content-format
-----------------------
[postprocessor:metadata] add 'tags' and 'custom' modes (#135)
2019-01-17 21:18:12 +01:00
=========== =====
Type ``string``
Example ``"tags:\n\n{tags:J\n}\n"``
[postprocessor:metadata] implement 'extension-format' option closes #477
2019-11-29 23:12:22 +01:00
Description Custom format string to build the content of metadata files with.
[postprocessor:metadata] add 'tags' and 'custom' modes (#135)
2019-01-17 21:18:12 +01:00
Note: Only applies for ``"mode": "custom"``.
=========== =====
implement 'mtime' post-processor (#332) This can set a file's modification time according to a UNIX timestamp or a datetime object from its metadata.
2019-07-14 22:37:28 +02:00
mtime
-----
Set file modification time according to its metadata
mtime.key
---------
=========== =====
Type ``string``
Default ``"date"``
Description Name of the metadata field whose value should be used.
This value must either be a UNIX timestamp or a
|datetime|_ object.
=========== =====
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
ugoira
------
[postprocessor:ugoira] pass '-f' if not present
2018-06-21 13:16:18 +02:00
Convert Pixiv Ugoira to WebM using `FFmpeg <https://www.ffmpeg.org/>`__.
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
ugoira.extension
----------------
=========== =====
Type ``string``
Default ``"webm"``
Description Filename extension for the resulting video files.
=========== =====
ugoira.ffmpeg-args
------------------
=========== =====
Type ``list`` of ``strings``
Default ``null``
[postprocessor:ugoira] pass '-f' if not present
2018-06-21 13:16:18 +02:00
Example ``["-c:v", "libvpx-vp9", "-an", "-b:v", "2M"]``
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
Description Additional FFmpeg command-line arguments.
=========== =====
ugoira.ffmpeg-location
----------------------
=========== =====
Type |Path|_
Default ``"ffmpeg"``
Description Location of the ``ffmpeg`` (or ``avconv``) executable to use.
=========== =====
[postprocessor:ugoira] add 'ffmpeg-output' option
2018-08-29 15:58:01 +02:00
ugoira.ffmpeg-output
--------------------
=========== =====
Type ``bool``
Default ``true``
Description Show FFmpeg output.
=========== =====
[postprocessor:ugoira] add option for two-pass encoding
2018-06-20 18:48:10 +02:00
ugoira.ffmpeg-twopass
---------------------
=========== =====
Type ``bool``
[postprocessor:ugoira] add 'ffmpeg-output' option
2018-08-29 15:58:01 +02:00
Default ``false``
[postprocessor:ugoira] add option for two-pass encoding
2018-06-20 18:48:10 +02:00
Description Enable Two-Pass encoding.
=========== =====
[postprocessor:ugoira] improve frame rate handling By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped frames if the source requires a higher frame rate than that. This commit adds a `framerate` option (default "auto"), which allows to automatically assign a (more or less) fitting frame rate based on delays between ugoira frames and avoids dropped frames.
2018-07-20 22:06:48 +02:00
ugoira.framerate
----------------
=========== =====
Type ``string``
Default ``"auto"``
Description Controls the frame rate argument (``-r``) for FFmpeg
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* ``"auto"``: Automatically assign a fitting frame rate
[postprocessor:ugoira] improve frame rate handling By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped frames if the source requires a higher frame rate than that. This commit adds a `framerate` option (default "auto"), which allows to automatically assign a (more or less) fitting frame rate based on delays between ugoira frames and avoids dropped frames.
2018-07-20 22:06:48 +02:00
based on delays between frames.
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* any other ``string``: Use this value as argument for ``-r``.
* ``null`` or an empty ``string``: Don't set an explicit frame rate.
[postprocessor:ugoira] improve frame rate handling By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped frames if the source requires a higher frame rate than that. This commit adds a `framerate` option (default "auto"), which allows to automatically assign a (more or less) fitting frame rate based on delays between ugoira frames and avoids dropped frames.
2018-07-20 22:06:48 +02:00
=========== =====
[pixiv] rework ugoira handling Frame information now gets attached to the ZIP file's keyword dict instead of being written to a separate text file.
2018-06-18 17:25:52 +02:00
ugoira.keep-files
-----------------
=========== =====
Type ``bool``
Default ``false``
[postprocessor:ugoira] pass '-f' if not present
2018-06-21 13:16:18 +02:00
Description Keep ZIP archives after conversion.
[pixiv] rework ugoira handling Frame information now gets attached to the ZIP file's keyword dict instead of being written to a separate text file.
2018-06-18 17:25:52 +02:00
=========== =====
[postprocessor:ugoira] add 'libx264-prevent-odd' option A rather crude workaround for "width/height not divisible by 2" errors when using libx264.
2018-09-21 19:52:45 +02:00
ugoira.libx264-prevent-odd
--------------------------
=========== =====
Type ``bool``
Default ``true``
Description Prevent ``"width/height not divisible by 2"`` errors
[postprocessor:ugoira] improve libx264 detection
2018-09-23 22:26:35 +02:00
when using ``libx264`` or ``libx265`` encoders
[postprocessor:ugoira] add 'libx264-prevent-odd' option A rather crude workaround for "width/height not divisible by 2" errors when using libx264.
2018-09-21 19:52:45 +02:00
by applying a simple cropping filter. See this `Stack Overflow
thread <https://stackoverflow.com/questions/20847674>`__
for more information.
[postprocessor:ugoira] improve libx264 detection
2018-09-23 22:26:35 +02:00
This option, when ``libx264/5`` is used, automatically
adds ``["-vf", "crop=iw-mod(iw\\,2):ih-mod(ih\\,2)"]``
[postprocessor:ugoira] add 'libx264-prevent-odd' option A rather crude workaround for "width/height not divisible by 2" errors when using libx264.
2018-09-21 19:52:45 +02:00
to the list of FFmpeg command-line arguments
to reduce an odd width/height by 1 pixel and make them even.
=========== =====
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
zip
---
Store files in a ZIP archive.
zip.compression
---------------
=========== =====
Type ``string``
Default ``"store"``
Description Compression method to use when writing the archive.
Possible values are ``"store"``, ``"zip"``, ``"bzip2"``, ``"lzma"``.
=========== =====
zip.extension
-------------
=========== =====
Type ``string``
Default ``"zip"``
Description Filename extension for the created ZIP archive.
=========== =====
zip.keep-files
--------------
=========== =====
Type ``bool``
Default ``false``
[postprocessor:ugoira] pass '-f' if not present
2018-06-21 13:16:18 +02:00
Description Keep the actual files after writing them to a ZIP archive.
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
=========== =====
[postprocessor:zip] add 'mode' option (#355)
2019-07-28 18:13:18 +02:00
zip.mode
--------
=========== =====
Type ``string``
Default ``"default"``
Description * ``"default"``: Write the central directory file header
once after everything is done or an exception is raised.
* ``"safe"``: Update the central directory file header
each time a file is stored in a ZIP archive.
This greatly reduces the chance a ZIP archive gets corrupted in
case the Python interpreter gets shut down unexpectedly
(power outage, SIGKILL) but is also a lot slower.
=========== =====
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Miscellaneous Options
=====================
cache.file
----------
=========== =====
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
Type |Path|_
use %APPDATA%\gallery-dl for config/cache on Windows
2020-05-28 02:56:38 +02:00
Default * (``%APPDATA%`` or ``"~"``) + ``"/gallery-dl/cache.sqlite3"`` on Windows
update default cache directory ... again Use a 'gallery-dl' subdirectory in ~/.cache to adhere to how other programs store their cached data, and call os.makedirs() so it also works without an existing ~/.cache directory.
2019-08-01 21:47:13 +02:00
* (``$XDG_CACHE_HOME`` or ``"~/.cache"``) + ``"/gallery-dl/cache.sqlite3"`` on all other platforms
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
Description Path of the SQLite3 database used to cache login sessions,
cookies and API tokens across `gallery-dl` invocations.
Set this option to ``null`` or an invalid path to disable
this cache.
=========== =====
add 'ciphers' option; update default User-Agent
2019-07-18 20:06:21 +02:00
ciphers
-------
=========== =====
Type ``bool`` or ``string``
Default ``true``
Description * ``true``: Update urllib3's default cipher list
* ``false``: Leave the default cipher list as is
* Any ``string``: Replace urllib3's default ciphers with these
(See `SSLContext.set_ciphers() <https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_ciphers>`__
for details)
=========== =====
add option to disable pyOpenSSL usage (#508) (pyOpenSSL is now disabled by default)
2019-12-08 21:21:00 +01:00
pyopenssl
---------
=========== =====
Type ``bool``
Default ``false``
Description Use `pyOpenSSL <https://www.pyopenssl.org/en/stable/>`__-backed
SSL-support.
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
allow users to set their own API access_tokens/client_ids
2017-09-09 17:31:42 +02:00
API Tokens & IDs
================
smaller changes, missing docs, etc. - make 'netrc' extractor-specific - rename 'downloader.enable' to 'enabled' - document 'downloader.ytdl.format' - consistent newlines in configuration.rst
2018-11-16 18:02:24 +01:00
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
All configuration keys listed in this section have fully functional default
values embedded into *gallery-dl* itself, but if things unexpectedly break
or you want to use your own personal client credentials, you can follow these
instructions to get an alternative set of API tokens and IDs.
update configuration.rst - update the API Tokens & IDs section - mention redirect URIs for deviantart - include api-secret for tumblr - add instructions for smugmug - [ci skip]
2018-05-26 11:26:50 +02:00
allow users to set their own API access_tokens/client_ids
2017-09-09 17:31:42 +02:00
extractor.deviantart.client-id & .client-secret
-----------------------------------------------
=========== =====
Type ``string``
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
How To * login and visit DeviantArt's
[pinterest] implement `oauth:pinterest` (#83) Pinterest access tokens are rate limited at 200 requests per hour (or maybe per 2 or 3 hours?) so having just one access token for all users isn't going to work in the long run.
2018-04-16 19:43:27 +02:00
`Applications & Keys <https://www.deviantart.com/developers/apps>`__
section
update deviantart client-id/-secret instructions (#451, #424)
2019-10-20 15:10:55 +02:00
* click "Register Application"
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* scroll to "OAuth2 Redirect URI Whitelist (Required)"
update configuration.rst - update the API Tokens & IDs section - mention redirect URIs for deviantart - include api-secret for tumblr - add instructions for smugmug - [ci skip]
2018-05-26 11:26:50 +02:00
and enter "https://mikf.github.io/gallery-dl/oauth-redirect.html"
update deviantart client-id/-secret instructions (#451, #424)
2019-10-20 15:10:55 +02:00
* scroll to the bottom and agree to the API License Agreement.
Submission Policy, and Terms of Service.
* click "Save"
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* copy ``client_id`` and ``client_secret`` of your new
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
application and put them in your configuration file
update deviantart client-id/-secret instructions (#451, #424)
2019-10-20 15:10:55 +02:00
as ``"client-id"`` and ``"client-secret"``
* get a new `refresh-token <extractor.deviantart.refresh-token_>`__
if necessary
allow users to set their own API access_tokens/client_ids
2017-09-09 17:31:42 +02:00
=========== =====
extractor.flickr.api-key & .api-secret
--------------------------------------
=========== =====
Type ``string``
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
How To * login and `Create an App <https://www.flickr.com/services/apps/create/apply/>`__
[pinterest] implement `oauth:pinterest` (#83) Pinterest access tokens are rate limited at 200 requests per hour (or maybe per 2 or 3 hours?) so having just one access token for all users isn't going to work in the long run.
2018-04-16 19:43:27 +02:00
in Flickr's `App Garden <https://www.flickr.com/services/>`__
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* click "APPLY FOR A NON-COMMERCIAL KEY"
* fill out the form with a random name and description
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
and click "SUBMIT"
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* copy ``Key`` and ``Secret`` and put them in your configuration
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
file
allow users to set their own API access_tokens/client_ids
2017-09-09 17:31:42 +02:00
=========== =====
extractor.pawoo.access-token
----------------------------
=========== =====
Type ``string``
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
How To
allow users to set their own API access_tokens/client_ids
2017-09-09 17:31:42 +02:00
=========== =====
[reddit] allow users to override the API User-Agent Only overriding the Client-ID is not enough if you want to follow Reddit's API access rules [1]. [1] https://github.com/reddit/reddit/wiki/API#rules
2017-10-10 17:29:46 +02:00
extractor.reddit.client-id & .user-agent
----------------------------------------
allow users to set their own API access_tokens/client_ids
2017-09-09 17:31:42 +02:00
=========== =====
Type ``string``
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
How To * login and visit the `apps <https://www.reddit.com/prefs/apps/>`__
[pinterest] implement `oauth:pinterest` (#83) Pinterest access tokens are rate limited at 200 requests per hour (or maybe per 2 or 3 hours?) so having just one access token for all users isn't going to work in the long run.
2018-04-16 19:43:27 +02:00
section of your account's preferences
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* click the "are you a developer? create an app..." button
* fill out the form, choose "installed app", preferably set
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
"http://localhost:6414/" as "redirect uri" and finally click
"create app"
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* copy the client id (third line, under your application's name and
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
"installed app") and put it in your configuration file
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* use "``Python:<application name>:v1.0 (by /u/<username>)``" as
update configuration.rst - add instructions on how to register your own applications - combine descriptions of a few config keys
2017-10-12 23:37:28 +02:00
user-agent and replace ``<application name>`` and ``<username>``
[pinterest] implement `oauth:pinterest` (#83) Pinterest access tokens are rate limited at 200 requests per hour (or maybe per 2 or 3 hours?) so having just one access token for all users isn't going to work in the long run.
2018-04-16 19:43:27 +02:00
accordingly (see Reddit's
`API access rules <https://github.com/reddit/reddit/wiki/API>`__)
allow users to set their own API access_tokens/client_ids
2017-09-09 17:31:42 +02:00
=========== =====
update configuration.rst - update the API Tokens & IDs section - mention redirect URIs for deviantart - include api-secret for tumblr - add instructions for smugmug - [ci skip]
2018-05-26 11:26:50 +02:00
extractor.smugmug.api-key & .api-secret
---------------------------------------
=========== =====
Type ``string``
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
How To * login and `Apply for an API Key <https://api.smugmug.com/api/developer/apply>`__
* use a random name and description,
update configuration.rst - update the API Tokens & IDs section - mention redirect URIs for deviantart - include api-secret for tumblr - add instructions for smugmug - [ci skip]
2018-05-26 11:26:50 +02:00
set "Type" to "Application", "Platform" to "All",
and "Use" to "Non-Commercial"
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* fill out the two checkboxes at the bottom and click "Apply"
* copy ``API Key`` and ``API Secret``
update configuration.rst - update the API Tokens & IDs section - mention redirect URIs for deviantart - include api-secret for tumblr - add instructions for smugmug - [ci skip]
2018-05-26 11:26:50 +02:00
and put them in your configuration file
=========== =====
extractor.tumblr.api-key & .api-secret
--------------------------------------
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
=========== =====
Type ``string``
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
How To * login and visit Tumblr's
[pinterest] implement `oauth:pinterest` (#83) Pinterest access tokens are rate limited at 200 requests per hour (or maybe per 2 or 3 hours?) so having just one access token for all users isn't going to work in the long run.
2018-04-16 19:43:27 +02:00
`Applications <https://www.tumblr.com/oauth/apps>`__ section
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* click "Register application"
* fill out the form: use a random name and description, set
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
https://example.org/ as "Application Website" and "Default
callback URL"
consistently use '*' for rst lists
2019-03-28 16:12:50 +01:00
* solve Google's "I'm not a robot" challenge and click "Register"
* click "Show secret key" (below "OAuth Consumer Key")
* copy your ``OAuth Consumer Key`` and ``Secret Key``
update configuration.rst - update the API Tokens & IDs section - mention redirect URIs for deviantart - include api-secret for tumblr - add instructions for smugmug - [ci skip]
2018-05-26 11:26:50 +02:00
and put them in your configuration file
[tumblr] add options to control extraction behavior (#48) - posts : list of post-types to inspect - inline : scan post bodies for inline images - external: follow external links
2017-11-22 23:09:08 +01:00
=========== =====
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
Custom Types
============
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
Date
----
=========== =====
Type ``string`` or ``integer``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example | ``"2019-01-01T00:00:00"``,
| ``"2019"`` with ``"%Y"`` as `date-format`_,
| ``1546297200``
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
Description A |Date|_ value represents a specific point in time.
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
* If given as ``string``, it is parsed according to `date-format`_.
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
* If given as ``integer``, it is interpreted as UTC timestamp.
=========== =====
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
Path
----
=========== =====
Type ``string`` or ``list`` of ``strings``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example | ``"file.ext"``,
| ``"~/path/to/file.ext"``,
| ``"$HOME/path/to/file.ext"``,
| ``["$HOME", "path", "to", "file.ext"]``
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Description A |Path|_ is a ``string`` representing the location of a file
or directory.
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
Simple `tilde expansion <https://docs.python.org/3/library/os.path.html#os.path.expanduser>`__
and `environment variable expansion <https://docs.python.org/3/library/os.path.html#os.path.expandvars>`__
is supported.
In Windows environments, backslashes (``"\"``) can, in addition to
forward slashes (``"/"``), be used as path separators.
Because backslashes are JSON's escape character,
they themselves have to be escaped.
The path ``C:\path\to\file.ext`` has therefore to be written as
``"C:\\path\\to\\file.ext"`` if you want to use backslashes.
=========== =====
Logging Configuration
---------------------
=========== =====
Type ``object``
update docs/configuration.rst - fix description of "skip": "enumerate" - use line breaks for lists of multiple examples - improve formatting of auto-generated man pages
2019-12-05 23:43:57 +01:00
Example .. code::
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
{
"format": "{asctime} {name}: {message}",
"format-date": "%H:%M:%S",
"path": "~/log.txt",
"encoding": "ascii"
}
implement custom logging formatter supports custom log message formats for each loglevel and, by extension, custom ANSI codes and colors for errors and warnings (#304)
2019-06-21 19:56:09 +02:00
{
"level": "debug",
"format": {
"debug" : "debug: {message}",
"info" : "[{name}] {message}",
"warning": "Warning: {message}",
"error" : "ERROR: {message}"
}
}
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
Description Extended logging output configuration.
* format
implement custom logging formatter supports custom log message formats for each loglevel and, by extension, custom ANSI codes and colors for errors and warnings (#304)
2019-06-21 19:56:09 +02:00
* General format string for logging messages
or a dictionary with format strings for each loglevel.
provide "extractor" and "job" keys for logging output This allows for stuff like "{extractor.url}" and "{extractor.category}" in logging format strings. Accessing 'extractor' and 'job' in any way will return "None" if those fields aren't defined, i.e. in general logging messages.
2019-02-13 17:39:43 +01:00
In addition to the default
`LogRecord attributes <https://docs.python.org/3/library/logging.html#logrecord-attributes>`__,
it is also possible to access the current
`extractor <https://github.com/mikf/gallery-dl/blob/2e516a1e3e09cb8a9e36a8f6f7e41ce8d4402f5a/gallery_dl/extractor/common.py#L24>`__
and `job <https://github.com/mikf/gallery-dl/blob/2e516a1e3e09cb8a9e36a8f6f7e41ce8d4402f5a/gallery_dl/job.py#L19>`__
objects as well as their attributes
(e.g. ``"{extractor.url}"``)
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
* Default: ``"[{name}][{levelname}] {message}"``
* format-date
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
* Format string for ``{asctime}`` fields in logging messages
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
(see `strftime() directives <https://docs.python.org/3/library/time.html#time.strftime>`__)
* Default: ``"%Y-%m-%d %H:%M:%S"``
* level
* Minimum logging message level
(one of ``"debug"``, ``"info"``, ``"warning"``, ``"error"``, ``"exception"``)
* Default: ``"info"``
* path
* |Path|_ to the output file
* mode
* Mode in which the file is opened;
use ``"w"`` to truncate or ``"a"`` to append
(see `open() <https://docs.python.org/3/library/functions.html#open>`__)
* Default: ``"w"``
* encoding
* File encoding
* Default: ``"utf-8"``
update configuration.rst ... again - some more 'Path' references - fixed some inconsistencies and errors - added note about logging config for files
2018-05-28 22:14:38 +02:00
Note: path, mode and encoding are only applied when configuring
logging output to a file.
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
=========== =====
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
Postprocessor Configuration
---------------------------
=========== =====
Type ``object``
Example .. code::
{
"name": "zip",
"compression": "store",
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
"extension": "cbz",
"whitelist": ["mangadex", "exhentai", "nhentai"]
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
}
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
Description An object with the ``name`` of a post-processor and its options.
See `Postprocessor Options`_ for a list of all available
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
post-processors and their respective options.
update 'docs/configuration.rst'
2018-10-08 23:08:11 +02:00
You can also set a ``whitelist`` or ``blacklist`` to
only enable or disable a post-processor for the specified
extractor categories.
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
=========== =====
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
add .netrc support (#22) Use the '--netrc' cmdline option or set the 'netrc' config option to 'true' to enable the use of .netrc authentication data. The 'machine' names for the .netrc info are the lowercase extractor names (or categories): batoto, exhentai, nijie, pixiv, seiga.
2017-06-24 12:17:26 +02:00
.. |.netrc| replace:: ``.netrc``
add 'downloader.http.verify' option (also: change the default 'timeout' from None to 30)
2017-08-31 15:21:08 +02:00
.. |requests.request()| replace:: ``requests.request()``
.. |timeout| replace:: ``timeout``
.. |verify| replace:: ``verify``
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
.. |mature_content| replace:: ``mature_content``
.. |webbrowser.open()| replace:: ``webbrowser.open()``
implement 'mtime' post-processor (#332) This can set a file's modification time according to a UNIX timestamp or a datetime object from its metadata.
2019-07-14 22:37:28 +02:00
.. |datetime| replace:: ``datetime``
update configuration.rst
2017-07-04 19:34:34 +02:00
.. |datetime.max| replace:: ``datetime.max``
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
.. |Date| replace:: ``Date``
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
.. |Path| replace:: ``Path``
add 'mtime' option
2019-06-20 17:19:44 +02:00
.. |Last-Modified| replace:: ``Last-Modified``
update configuration.rst - document logging options - add a section for "custom types" [ci skip]
2018-05-27 16:48:54 +02:00
.. |Logging Configuration| replace:: ``Logging Configuration``
add postprocessor documentation to configuration.rst
2018-06-16 15:43:24 +02:00
.. |Postprocessor Configuration| replace:: ``Postprocessor Configuration``
update configuration.rst
2017-07-04 19:34:34 +02:00
.. |strptime| replace:: strftime() and strptime() Behavior
update configuration.rst - move `base-directory` entry to extractor options (it should've been there since v1.0.0 / commit e3bfb83) - rename "General" to "Miscellaneous" and move it down a bit, since it has only two options left - mention the ability to use a list for tumblr.posts - replace any appearance of bato.to
2018-03-16 11:49:49 +01:00
.. _base-directory: `extractor.*.base-directory`_
add '--download-archive' cmdline option … as well as a config file equivalent
2018-02-01 22:00:44 +01:00
.. _skipped: `extractor.*.skip`_
[tumblr] enable date-min/-max/-format options (#337)
2019-07-16 23:08:27 +02:00
.. _date-format: `extractor.*.date-format`_
[deviantart] add 'external' option (#302) If a description is available, this will extract URLs from the description text and try to find Extractors for them.
2019-06-06 18:53:50 +02:00
.. _deviantart.metadata: extractor.deviantart.metadata_
update README - mention and link to docs/configuration.rst - add OAuth section
2017-06-23 16:14:51 +02:00
fix & update docs - update Requests links - add example for --exec - set '-dev' version
2019-10-11 18:19:39 +02:00
.. _.netrc: https://stackoverflow.com/tags/.netrc/info
.. _Last-Modified: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.29
.. _datetime: https://docs.python.org/3/library/datetime.html#datetime-objects
.. _datetime.max: https://docs.python.org/3/library/datetime.html#datetime.datetime.max
.. _format string: https://docs.python.org/3/library/string.html#formatstrings
.. _format strings: https://docs.python.org/3/library/string.html#formatstrings
.. _strptime: https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior
.. _webbrowser.open(): https://docs.python.org/3/library/webbrowser.html
.. _mature_content: https://www.deviantart.com/developers/http/v1/20160316/object/deviation
.. _Authentication: https://github.com/mikf/gallery-dl#authentication
.. _OAuth: https://github.com/mikf/gallery-dl#oauth
.. _youtube-dl: https://github.com/ytdl-org/youtube-dl
.. _requests.request(): https://requests.readthedocs.io/en/master/api/#requests.request
.. _timeout: https://requests.readthedocs.io/en/master/user/advanced/#timeouts
.. _verify: https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification
.. _`Requests' proxy documentation`: https://requests.readthedocs.io/en/master/user/advanced/#proxies
Reference in New Issue Copy Permalink