1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 19:22:32 +01:00
Commit Graph

85 Commits

Author SHA1 Message Date
Mike Fährmann
d94215d119
[tumblr] replace '-' with ' ' in tag searches (fixes #611)
To search for tags with actual minus signs in them
(there shouldn't be too many,) manually replace those
with url-encoded minus characters ('-' -> '%2d')
before inputting them into gallery-dl:

https://s679874.tumblr.com/tagged/tag-with-minus
 ->
https://s679874.tumblr.com/tagged/tag%2dwith%2dminus
2020-02-17 23:29:13 +01:00
Mike Fährmann
3811fd8a25
fix time formatting for Python 3.4 and 3.5
'datetime.time.isoformat()' only has an optional 'timespec' argument
since Python 3.6.
2020-01-05 00:47:10 +01:00
Mike Fährmann
569747a78d
implement extractor.wait() 2020-01-04 23:42:07 +01:00
Mike Fährmann
ce54b8c04c
let extractors opt-out of cookie option usage
useful to avoid sending unnecessary cookies when all authentication
is done through OAuth tokens
2020-01-01 21:12:37 +01:00
Mike Fährmann
c4702ec9b6
simplify some logging calls 2019-12-10 21:30:08 +01:00
Mike Fährmann
4409d00141
embed error messages in StopExtraction exceptions 2019-10-28 16:39:49 +01:00
Mike Fährmann
d5fbb2d9de
[tumblr] ignore audio links from Spotify etc. 2019-09-07 18:18:12 +02:00
Mike Fährmann
1133b7fcbd
[smugmug] update unit tests
The account used for tests before has been deleted.
2019-07-19 17:16:24 +02:00
Mike Fährmann
8d1ae9b715
[tumblr] enable date-min/-max/-format options (#337) 2019-07-17 14:36:41 +02:00
Mike Fährmann
208202b962
[tumblr] improve error handling (#297)
In some cases Tumblr's API responds with an HTML document.
Trying to decode it as JSON would raise an uncaught exception.
2019-06-04 14:02:17 +02:00
Mike Fährmann
add7e693d0
[tumblr] provide parsed 'date' metadata (#232) 2019-04-29 17:30:42 +02:00
Mike Fährmann
fb14f80d62
[tumblr] fix avatar URLs for non-OAuth1.0 calls (closes #193) 2019-03-17 11:07:22 +01:00
Mike Fährmann
d0059cab79
[tumblr] check for null URLs (closes #165) 2019-02-19 13:49:55 +01:00
Mike Fährmann
5530871b5a
change results of text.nameext_from_url()
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)

Example: "https://example.org/path/filename.ext"

before:
- filename : filename.ext
- name     : filename
- extension: ext

now:
- filename : filename
- extension: ext
2019-02-14 16:07:17 +01:00
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor 2019-02-11 13:31:10 +01:00
Mike Fährmann
6284731107
simplify extractor constants
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
2019-02-08 13:45:40 +01:00
Mike Fährmann
0afa913de4
[tumblr] add tests for hidden and private blogs (#145)
Hidden / dashboard-only blogs are pretty straightforward and "only"
require a valid 'access-token' and 'access-token-secret' for the given
'api-key' and 'api-secret', so that signed OAuth1.0 requests are possible.

Private / password protected blogs on the other hand are a bit
cumbersome. In addition to a valid 'access-token' and
'access-token-secret', they also require the account belonging to those
tokens to be a member of the blog itself. Knowing the password and
entering it in the website isn't enough to access a blog through the
API. Following a private blog is also impossible, so that option can't
work either.
2019-01-03 16:12:24 +01:00
Mike Fährmann
2f4f60de33
[tumblr] add tests for each post type 2018-12-27 22:41:42 +01:00
Mike Fährmann
28f9539551
[tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba
[tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
2e5f82e59e
[tumblr] don't follow 'external' Tumblr URLs (#139) 2018-12-22 14:05:43 +01:00
Mike Fährmann
049a9575c4
[tumblr] fix inline extraction #2
Using only the "comment" field isn't enough ...

[ci skip]
2018-12-11 21:57:20 +01:00
Mike Fährmann
b7a9f6cc49
[tumblr] improve inline extraction (#137) 2018-12-11 20:02:48 +01:00
HRXN
e80ee77d71 tumblr.py: update regex for video (#133)
There seems to be another sub-domain for videos, apparently..
Not just
`vt(.media).tumblr`
`vtt(media).tumblr`
But also
`ve(.media).tumblr`
2018-12-09 09:07:46 +01:00
Mike Fährmann
9a98b6769d
use extractor.request for API calls (#130)
... at least for OAuth1.0 based APIs (flickr, smugmug, tumblr)
2018-12-04 21:29:06 +01:00
Mike Fährmann
ad2cefda6b
[tumblr] in case of exception use filename as 'hash' (#129)
While a filename might not be a real 'hash', or comparable to what
tumbler usually provides, it is still better than an empty string.
At least as long as "alternatives" in format strings aren't implemented.
2018-12-04 19:15:23 +01:00
Mike Fährmann
95636418ad
[tumblr] catch exception for 'hash' extraction (fixes #129) 2018-12-02 19:48:09 +01:00
Mike Fährmann
7742cf8601
[tumblr] change 'reblogs' option (#103)
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
2018-09-10 15:40:25 +02:00
Mike Fährmann
d4d95d3154
[tumblr] improve rewrite rules for video URLs 2018-09-09 14:09:47 +02:00
Mike Fährmann
a666ddd16b
[tumblr] extend 'reblogs' functionality (#103)
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
2018-09-07 19:13:52 +02:00
Mike Fährmann
b4eca2633e
[tumblr] support /archive URLs 2018-09-06 11:09:13 +02:00
Mike Fährmann
aa1de70da0
[tumblr] recognize inline videos (#102) 2018-09-06 10:37:40 +02:00
Mike Fährmann
5b8a314de7
[tumblr] replace inline URLs with higher quality ones (#98) 2018-08-25 18:43:51 +02:00
Mike Fährmann
a74591b84b
[tumblr] remove "original image" functionality
Accessing higher/original quality images on
https://s3.amazonaws.com/data.tumblr.com and http://data.tumblr.com
is no longer possible and any HTTP request results in 403 Forbidden.

A few images can still be accessed through https//a.tumblr.com [1][2],
but not as "_raw", just "_1280", and that might also be "fixed" in
the near future.

[1] https://a.tumblr.com/tumblr_kzjlfiTnfe1qz4rgho1_1280.jpg
[2] https://a.tumblr.com/ee589c6345f29d2d5935cecb49b0a705/tumblr_oztu02dIHp1wgha4yo1_1280.png
2018-08-14 11:51:17 +02:00
Mike Fährmann
1c1e086d01
use common base class for OAuth1.0 based API interfaces 2018-05-10 21:57:45 +02:00
Mike Fährmann
6a31ada9e3
re-implement OAuth1.0 code
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
2018-05-10 18:47:05 +02:00
Mike Fährmann
69a5e6ddb3
Merge branch 'master' into 1.4-dev 2018-05-04 10:19:02 +02:00
Mike Fährmann
8b79eaafea
[tumblr] log actual time of rate limit resets
... instead of the amount of seconds until a reset
2018-04-25 16:13:03 +02:00
Mike Fährmann
f471161920
Merge branch 'master' into 1.4-dev 2018-04-21 12:15:40 +02:00
Mike Fährmann
b1325d4d2c
fix extractor docstrings 2018-04-18 18:03:43 +02:00
Mike Fährmann
728c64a3fb
[tumblr] rename 'offset' to 'num and adjust formats
Trying to somehow emulate Tumblr filenames is a bad idea ...
2018-04-15 18:58:32 +02:00
Mike Fährmann
6bd857a319
[tumblr] handle rate limits / 429 errors
- wait for the hourly limit to reset
- abort upon exceeding the daily limit (it doesn't seem useful to
  potentially wait for several hours)
2018-04-12 16:25:20 +02:00
Mike Fährmann
a1fa4b43b0
Revert "[tumblr] add option to sort photosets by upload order"
This reverts commit 4a26ae32df.
2018-04-09 16:08:08 +02:00
Mike Fährmann
4a26ae32df
[tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
6b72be8ee6
[tumblr] add 'hash' keyword
'hash' is the middle part of the filename in a tumblr image URL.
For example an image with '.../tumblr_p6tgemp1NZ1wgha4yo1_250.png' as
its URL would have 'p6tgemp1NZ1wgha4yo1' as hash.
2018-04-07 15:54:30 +02:00
Mike Fährmann
68e9fbee16
[tumblr] check all 4 keys/secrets before using OAuth
it was possible to cause a crash by setting api-key or -secret to null.
(this commit also slightly improves the blog-cache implementation)
2018-04-05 15:42:23 +02:00
Mike Fährmann
f8168c693e
[tumblr] avoid calls to '/blog/.../info'
The same information returned by the 'blog/.../info' API endpoint
is also included in the result of every 'blog/.../posts' call.
2018-04-04 14:15:24 +02:00
Mike Fährmann
858fdbdb22
[tumblr] improve 'inline' extraction
'quote' posts store their HTML content in the 'source' field
2018-03-02 06:59:44 +01:00
Mike Fährmann
5008e105ee
update archive IDs
... to behave in a more straightforward way when dealing with
bookmarks/favourites/etc.

specific IDs are now grouped by their owner, album-id, ... to
allow for duplicates when it would be expected.
2018-03-01 18:20:50 +01:00
Mike Fährmann
3cec533c28
Merge branch 'archive' 2018-02-12 18:07:58 +01:00