Mike Fährmann
89066844f4
add 'config_instance' method
...
to allow for a more streamlined access to BaseExtractor instance options
2024-01-18 03:20:36 +01:00
Mike Fährmann
57fc6fcf83
replace '24*3600' with '86400'
...
and generalize cache maxage values
2023-12-18 23:57:22 +01:00
Mike Fährmann
e256434c9e
use custom HTTPBasicAuth class
...
to support LazyPrompt as password
and to generate the Authorization header only once
instead of for every request
2023-12-07 01:40:31 +01:00
Mike Fährmann
17e710c4bf
[oauth] warn when cache is enabled but not writeable ( #4771 )
2023-11-13 16:24:54 +01:00
Mike Fährmann
a453335a9f
remove test results in extractor modules
...
and add generic example URLs
2023-09-11 16:30:55 +02:00
Mike Fährmann
a383eca7f6
decouple extractor initialization
...
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().
This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
2023-07-25 22:16:16 +02:00
Mike Fährmann
076380e079
remove '*' indicating keyword-only arguments
...
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
2023-05-02 22:23:33 +02:00
Mike Fährmann
a05120412a
[oauth] catch exception from 'webbrowser.get()' ( #3947 )
...
It raises an exception instead of returning None
when no runnable browser is available.
2023-04-23 15:00:09 +02:00
Mike Fährmann
b8a702929d
[oauth] import extractor modules on demand
2023-03-02 18:39:54 +01:00
Mike Fährmann
dd88740ec7
replace remaining instances of base64 with binascii
2023-03-02 18:25:47 +01:00
Mike Fährmann
75570ad3f1
[oauth] remove stray 'exit()' ( #3628 )
...
- bug from 70ce45d9
2023-02-13 15:58:34 +01:00
Mike Fährmann
70ce45d965
[oauth] use default name for browsers without 'name' attribute
...
(#3645 )
Seem to only be an issue for MacOSXOSAScript before Python 3.11.
d12bec6993
2023-02-11 14:48:59 +01:00
Mike Fährmann
568112dfbb
[oauth] improve output
...
- show which api key / client id gets used (#3518 )
- show in which browser authorization URLs gets opened in
2023-01-20 12:33:26 +01:00
Mike Fährmann
df91ebb945
[oauth] simplify OAuth 1.0a init
2023-01-19 18:34:54 +01:00
Martin Juhasz
9ea521bec5
[oauth] add host config option ( #2806 )
2022-08-14 17:08:01 +02:00
Mike Fährmann
688d6553b4
replace calls to print() with stdout_write() ( #2529 )
2022-05-19 17:09:24 +02:00
Mike Fährmann
e5f6af6e32
[oauth:pixiv] add note about 'code' expiring in 30 seconds ( #2306 )
2022-02-19 23:47:30 +01:00
Mike Fährmann
4f5f9ed1e5
[oauth] fix typo
...
this has been here since February ...
(8974f036
2021-09-08 20:14:39 +02:00
Mike Fährmann
96fec14ef7
[deviantart] rename 'watch' option to 'auto-watch'
...
(#1466 , #1757 )
Similar reason as in e05a96db
2021-08-29 21:10:53 +02:00
Mike Fährmann
4718f9c5dd
[oauth] use defaults when config values are set to None/null
...
(fixes #1778 )
2021-08-21 22:46:01 +02:00
Mike Fährmann
04f4f9badb
[oauth] prevent exceptions when reporting errors ( #1603 )
2021-06-05 14:28:26 +02:00
Mike Fährmann
fbfcbcbf57
Merge branch '1.17.0'
2021-02-15 02:32:33 +01:00
Mike Fährmann
8974f0361c
[pixiv] update ( #1304 )
...
- remove login with username & password
- require a refresh token
- add 'oauth:pixiv' functionality
See also:
- https://github.com/upbit/pixivpy/issues/158
- https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362
2021-02-12 18:07:16 +01:00
Mike Fährmann
1d145a6186
[mastodon] use cache for OAuth tokens ( #616 )
2021-01-31 01:38:23 +01:00
Mike Fährmann
36bf76fa44
update 'oauth:mastodon:<instance>' code
2021-01-28 02:20:12 +01:00
Mike Fährmann
968d3e8465
remove '&' from URL patterns
...
'/?&#' -> '/?#' and '?&#' -> '?#'
According to https://www.ietf.org/rfc/rfc3986.txt , URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
2020-10-22 23:31:25 +02:00
Mike Fährmann
bdc6c8f074
improve message for 'oauth:deviantart' etc ( closes #989 )
2020-09-29 21:25:24 +02:00
Mike Fährmann
864f4220d9
update output of 'oauth:…' ( #616 )
2020-05-31 17:41:40 +02:00
Mike Fährmann
dfcf2a2c91
write OAuth token to cache by default ( #616 )
2020-05-25 22:35:45 +02:00
Mike Fährmann
c8787647ed
add global WINDOWS bool
2020-05-19 22:32:53 +02:00
Mike Fährmann
5d7404ab58
[oauth] use the new name for 'DeviantartAPI' ( fixes #670 )
2020-04-04 20:34:47 +02:00
Mike Fährmann
913b8333cc
write DeviantArt refresh-tokens to cache ( #616 )
...
Writing the token is currently disabled by default and must be
enabled with 'extractor.oauth.cache'.
'extractor.deviantart.refresh-token' must be set to '"cache"'
to use the cached token.
2020-02-25 22:55:11 +01:00
Mike Fährmann
da0d5f6092
[oauth] add 'port' option ( #604 )
2020-02-09 13:45:44 +01:00
Mike Fährmann
372ffe95ee
[oauth] adjust Flickr redirect URI ( fixes #503 )
...
Flickr now automatically forces https:// for all redirect URIs.
2019-12-02 22:24:20 +01:00
Mike Fährmann
f5604492c3
update interface of config functions
2019-11-24 00:42:28 +01:00
Mike Fährmann
4409d00141
embed error messages in StopExtraction exceptions
2019-10-28 16:39:49 +01:00
Mike Fährmann
803d8f814e
[oauth] update scope for reddit tokens ( #428 )
...
'/user/<username>/...' requires the 'history' scope to be accessible
(https://www.reddit.com/dev/api/#GET_user_{username}_{where} )
2019-09-27 17:38:55 +02:00
Mike Fährmann
a2af2d2965
adjust cache maxage values
2019-03-14 22:21:49 +01:00
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor
2019-02-11 13:31:10 +01:00
Mike Fährmann
6284731107
simplify extractor constants
...
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
2019-02-08 13:45:40 +01:00
Mike Fährmann
b8fed34548
add generalized extractors for Mastodon instances ( #144 )
...
Extractors for Mastodon instances can now be dynamically generated,
based on the instance names in the 'extractor.mastodon.*' config path.
Example:
{
"extractor": {
"mastodon": {
"pawoo.net": { ... },
"mastodon.xyz": { ... },
"tabletop.social": { ... },
...
}
}
}
Each entry requires an 'access-token' value, which can be generated with
'gallery-dl oauth:mastodon:<instance URL>'.
An 'access-token' (as well as a 'client-id' and 'client-secret') for
pawoo.net is always available, but can be overwritten as necessary.
2019-01-19 14:28:59 +01:00
Mike Fährmann
f3483a2b7c
[smugmug] add OAuth support
2018-05-10 18:58:05 +02:00
Mike Fährmann
6a31ada9e3
re-implement OAuth1.0 code
...
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
2018-05-10 18:47:05 +02:00
Mike Fährmann
0f1e07f627
[pinterest] scrap OAuth implementation; code improvements
...
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
2018-04-25 16:04:30 +02:00
Mike Fährmann
80bead739d
[oauth] require custom client-* values for pinterest
2018-04-20 15:31:05 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest ( #83 )
...
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
2018-04-16 20:03:28 +02:00
Mike Fährmann
91ed147cef
[oauth] use custom key/secret values during oauth:…
2018-01-16 17:39:46 +01:00
Mike Fährmann
29d75fc3fa
[tumblr] add support for OAuth authentication ( #65 )
2018-01-11 14:11:37 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication
...
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.
[*] e.g. https://polinaegorussia.deviantart.com/gallery/
--------------
known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
2017-12-18 01:16:46 +01:00
Mike Fährmann
b4c438c9ad
[oauth] add the 'extractor.oauth.browser' option
...
enables/disables the use of webbrowser.open() during OAuth authorization
2017-06-20 16:06:14 +02:00
