1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Commit Graph

1903 Commits

Author SHA1 Message Date
Mike Fährmann
20fd2d8450
[flickr] skip unavailable images/videos (fixes #398) 2019-08-27 23:26:49 +02:00
Mike Fährmann
c9b97dbca3
extend post processor tests 2019-08-26 23:10:12 +02:00
Mike Fährmann
60c8e090da
[postprocessor:zip] fix archive names (closes #397)
Remove the trailing path separator introduced in 3284c62 before
adding the archive's filename extension.

[ci skip]
2019-08-24 23:14:26 +02:00
Mike Fährmann
7c09545f70
[downloader:ytdl] add 'outtmpl' option (#395) 2019-08-24 22:47:59 +02:00
Mike Fährmann
5cc7be2536
[piczel] update and improve
- use proper pagination (fixes #396)
- update API host and endpoints
- "fix" double slash // in image URLs
2019-08-24 20:37:33 +02:00
Mike Fährmann
0c1c7abb4d
release version 1.10.2 2019-08-23 22:10:54 +02:00
Mike Fährmann
49f6d7176d
[deviantart] restore filenames (#392)
<title>_by_<user>_<id> --> <title>_by_<user>-<id>
2019-08-23 22:02:03 +02:00
Mike Fährmann
63daa68d67
[deviantart] improvements (#392)
- consistent 'filename' entries, at least as far as possible
  - GIFs and SWFs don't have a <title>_by_<artist>_<id> anywhere in
    their metadata
  - Generating <id> (from 'deviationid'?) might be something that needs
    to be figured out, so we can build those filenames ourselves
- better code structure etc.
- tests for videos, archives, and flash animations
2019-08-23 12:27:19 +02:00
Mike Fährmann
d1db5180ab
[simplyhentai] fix extraction; remove image+video extractors 2019-08-22 23:56:41 +02:00
Mike Fährmann
30d6e284b0
[deviantart] use NAPI for artworks and scraps (#392)
TODO:
- journal downloads
- test for all media types
2019-08-21 23:56:06 +02:00
Mike Fährmann
e528f3cb77
adjust postprocessor test results
see 2495b99
2019-08-21 23:54:08 +02:00
Mike Fährmann
7d6af936c5
[imgur] simplify gallery extraction 2019-08-20 20:00:43 +02:00
Mike Fährmann
3284c62f22
ensure PathFormat.directory ends with a path separator
... plus some other small optimizations
2019-08-20 00:25:13 +02:00
Mike Fährmann
ebabc5caf1
[downloader:http] treat 416 without downloaded data as error
Downloading https://pbs.twimg.com/media/EB2cGUYX4AI2Vuu.jpg:orig (NSFW)
sometimes returns a 416 status code, even though no 'Range' header was
sent and no data was downloaded prior.
This code usually means a file has already been downloaded completely
and the download method indicates success, but in this case it causes
an exception down the pipeline since no file was created.
2019-08-20 00:15:17 +02:00
Mike Fährmann
2495b99347
[postprocessor:classify] improve path generation (fixes #138)
It still doesn't work for converted ugoira animations thanks to how
those files are handled, but everything else, including files with
unknown or changing file extension, now works as it should.
2019-08-19 23:05:28 +02:00
Mike Fährmann
e77a656437
optimize directory path generation
- use str.join() instead of os.path.join()
  (less "features", but 10x as fast)
- cache directory formatters
- detect and optimize field access for 1-element format strings
2019-08-19 15:56:20 +02:00
Mike Fährmann
51d10783fc
[patreon] include image info in API results (#383) 2019-08-18 23:28:47 +02:00
Mike Fährmann
7a5e78741c
[booru] build directory path for each file (#385) 2019-08-18 23:28:33 +02:00
Mike Fährmann
b1728f512d
[patreon] support multi image posts and post URLs (#383) 2019-08-17 23:24:46 +02:00
Mike Fährmann
454bf1ebf9
preserve enumeration index after 'set_extension()' (#306) 2019-08-16 23:12:33 +02:00
Mike Fährmann
f5039b897f
replace DownloadArchive.check() with __contains__()
Interestingly enough, 'a in obj' is slightly faster than
'obj.check(a)' and is also nicer to look at, I think.
2019-08-16 23:12:32 +02:00
Mike Fährmann
5a210991b6
Remove control characters from filesystem paths
- add 'path-remove' option to specify the set of characters that
 should be removed
- rename 'restrict-filenames' to 'path-restrict'
- #348, #380
2019-08-16 23:12:16 +02:00
Mike Fährmann
c50d60a53d
[reactor] fix image URLs 2019-08-16 14:07:22 +02:00
Mike Fährmann
32447d0d24
[pixiv] simplify default filename format
(#366)
2019-08-15 13:32:47 +02:00
Mike Fährmann
5f8621b29d
improve output of active post processor modules 2019-08-15 13:31:04 +02:00
Mike Fährmann
2cbbc3dec4
add a 'whitelist' to '--ugoira-conv' (#382) 2019-08-15 13:27:57 +02:00
Mike Fährmann
e661a1ce19
update links to Request's documentation 2019-08-15 13:25:06 +02:00
Mike Fährmann
cb7f149974
fix mtime datetime test
datetime.timestamp() uses local time for a naive datetime object
2019-08-14 21:57:42 +02:00
Mike Fährmann
829b1ccf04
[imgur] distinguish album and gallery URLs (#380)
A gallery can be either an album or a single image.
2019-08-14 21:40:14 +02:00
Mike Fährmann
23251356cb
require 'extension' data for each URL (#382) 2019-08-14 20:03:03 +02:00
Mike Fährmann
a67413d64f
[xhamster] use input URL domain
Don't rewrite all URLs as 'https://xhamster.com/...'
2019-08-14 00:21:30 +02:00
Mike Fährmann
dd72ae7164
add postprocessor tests 2019-08-12 21:46:30 +02:00
Mike Fährmann
0bb873757a
update PathFormat class
- change 'has_extension' from a simple flag/bool to a field that
  contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306)
2019-08-12 21:40:37 +02:00
Mike Fährmann
423f68f585
[deviantart] fix scraps extraction (closes #376) 2019-08-11 16:06:15 +02:00
Mike Fährmann
748e37554c
update .travis.yml
- install pyOpenSSL before running tests
- simplify snap tests
2019-08-11 16:03:19 +02:00
leezhi0513
42bb0cdb9b Update gallery-dl.conf (#378) 2019-08-11 14:44:50 +02:00
Mike Fährmann
3bf20ffb70
[instagram] add support for story highlights 2019-08-10 14:34:22 +02:00
Mike Fährmann
a732e9c430
[instagram] update query hashes and headers 2019-08-10 14:13:08 +02:00
Mike Fährmann
2ccf6a9e35
[instagram] make extractor tests happy (#373) 2019-08-08 18:50:26 +02:00
Mike Fährmann
8dc42bb178
implement 'enumerate' for 'extractor.skip' (#306)
[ci skip]
2019-08-08 18:37:54 +02:00
Leonardo Taccari
bc5eaf7746 [instagram] Add support for IGTV (#373)
Add support for IGTV profile (instagram.com/<username>/channel/)
and IGTV medias (instagram.com/tv/<short_id>).
2019-08-08 18:33:13 +02:00
Mike Fährmann
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option 2019-08-08 16:54:20 +02:00
Mike Fährmann
eb7da159e2
[imagebam] update URL test results
Image URLs are now using https://, but the website itself is still
served as http://.
2019-08-07 21:47:44 +02:00
Mike Fährmann
189acbeac9
[imgbb] add extractor for individual images (closes #363) 2019-08-05 22:52:08 +02:00
Mike Fährmann
ad3ac02fbc
[pixiv] update metadata entries (#366)
- change 'num' to a simple enumerating integer
- change default filename format
- provide content of the old 'num' field as 'suffix'
- add 'filename' for ugoira
2019-08-05 22:41:56 +02:00
Mike Fährmann
1ff4c4ec03
[adultempire] consistent artist order 2019-08-05 22:06:11 +02:00
Leonardo Taccari
2df050e627 [instagram] Add support for stories (#371)
* [instagram] Add support for stories

Add support for Instagram user's stories
(https://www.instagram.com/stories/<username>/).

First the shared_data in instagram.com/stories/<username> is fetched in
order to retrieve the user_id that is then passed to fetch the stories
via the corresponding graphql query.

Please note that fetching stories is supported only when authentication
is enabled and the corresponding <username> is followed.

* [instagram] Add an only-matching test for stories

* [instagram] Simplify InstagramExtractor.items() and _extract_stories()

Simplify handling of typename in InstagramExtractor.items() and multi-line
string in _extract_stories().  NFCI.
2019-08-05 22:04:34 +02:00
Mike Fährmann
f4bc75e854
fix rate limit handling for OAuth APIs (#368) 2019-08-03 13:43:00 +02:00
Mike Fährmann
f687052daf
have e-hentai and exhentai on supportedsites.rst (#365) 2019-08-03 11:42:28 +02:00
Mike Fährmann
3957d27d79
[deviantart] add 'quality' option (#369) 2019-08-03 11:40:35 +02:00