1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-26 12:42:29 +01:00
Commit Graph

1828 Commits

Author SHA1 Message Date
Mike Fährmann
0f02e85961
[reactor] use "/full/" URLs (closes #210)
Putting a "/full/" in image URLs potentially gives higher resolution
and better quality.
2019-03-30 22:14:57 +01:00
Mike Fährmann
17c11393f5
[weibo] allow user-ids in status URLs 2019-03-30 18:38:58 +01:00
Mike Fährmann
ec88ff1562
[flickr] relax unit test results
Images are now randomly served from the 'live.staticflickr.com' domain
instead of the "old" 'farmN.staticflickr.com' one, making it impossible
to use static 'url' and 'keyword' hashes as results.

Image quality doesn't appear to be effected by which image-server is
used. Files from 'farmN' and 'live' are the same.
2019-03-30 18:31:59 +01:00
Mike Fährmann
bc2020e86c
release version 1.8.1 2019-03-29 17:37:11 +01:00
Mike Fährmann
00d604cafb
[luscious] fix SearchExtractor URL-pattern 2019-03-29 15:58:08 +01:00
Mike Fährmann
0c991a3155
add convenience targets to Makefile 2019-03-29 15:35:00 +01:00
Mike Fährmann
1384ebf907
[luscious] fix metadata extraction
- remove 'artist', 'language', and 'lang' fields
- replace 'section' with 'genre'
- provide 'tags' as list
- use GalleryExtractor as base class
2019-03-29 13:06:02 +01:00
林博仁(Buo-ren Lin)
c3a75a0c40 fixup! Snap packaging improvements (#207) (#208)
The build failed due to missing `requests` build dependency, this patch
drops the unused component to build to avoid the problem.

The manpages are still built for the upcoming read-manual workaround.

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-29 13:05:26 +01:00
林博仁(Buo-ren Lin)
81d4d49234 Snap packaging improvements (#207)
* fixup! snap: Support official config paths via *-files confinement interfaces (#197)

* FIXME no longer applied
* Obsoleted HOME environment variable assignment

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>

* snap: Migrate to selective-checkout

The selective-pull stage snap is superseded by selective-checkout, prefer the new one.

Refer-to: Selective-checkout: Check out the tagged release revision if it isn't promoted to the stable channel <https://forum.snapcraft.io/t/the-selective-pull-scriptlet-stage-snap-workaround/10389>
Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>

* snap: Support bash completion

Refer-to: Scriptlets <https://docs.snapcraft.io/scriptlets/4892>
Refer-to: Tab completion for snaps <https://docs.snapcraft.io/tab-completion-for-snaps/2261>
Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>

* snap: Implement interface connection warning in the launcher

This patch ensures that the user will be acknowledge the missing
connection to the `removable-media` interface.

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-29 09:55:43 +01:00
林博仁(Buo-ren Lin)
c689bc2971 Ignore generated manpages and bash completion data (#206)
Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-29 09:55:27 +01:00
Mike Fährmann
5398bfbd69
[exhentai] fix search and favorite extraction
removes basically all metadata, but that can be compensated for with the
right search query. writing "parsers" for all 4 possible views that have
been introduced in the latest changes is too much of a hassle ...
2019-03-28 16:22:02 +01:00
Mike Fährmann
369eb66125
consistently use '*' for rst lists 2019-03-28 16:21:41 +01:00
Mike Fährmann
089923e3dd
parse configuration.rst to build gallery-dl.conf.5 (#150)
… a man-page containing all of gallery-dl's configuration file options.

This implementation relies on Python dicts preserving their insertion
order. Python 3.4 and 3.5 need to use OrderedDict or they produce
randomly ordered man-page sections.

The man-page formatting is a bit rough around the edges, but it works
for the most part. The only real "problem" are inline-links, but it's
better if they are left in there.
2019-03-28 16:20:52 +01:00
Mike Fährmann
5476404a5c
update and fix Cloudflare bypass 2019-03-25 22:53:36 +01:00
Mike Fährmann
0df4edc20a
skip missing data_files in setup.py (#204)
[ci skip]
2019-03-24 18:05:54 +01:00
Leonardo Taccari
790b1336a6 [instagram] Add support for hashtags
Add support for hashtags (TagPage-s), i.e. explore/tags/<tag> URLs.

This also introduce a get_metadata() method in order to append
possible further metadata per-(sub)extractor.

Refactor and generalize _extract_profilepage() to _extract_page()
in order to be reused by _extract_profilepage() and _extract_tagpage()
simply by passing the type of page (`ProfilePage' or `TagPage') and picking up
the respective fields in shared data.
2019-03-24 14:05:34 +01:00
Mike Fährmann
114b8eecc5
[downloader;ytdl] utilize '_ytdl_index' metadata fields 2019-03-24 11:27:20 +01:00
Mike Fährmann
a9bdd0f153
[instagram] fix syntax for Python 3.4
Python 3.4 doesn't like '**common' in dict literals.
This also makes '_ytdl_index' zero-based.
2019-03-24 11:25:42 +01:00
Mike Fährmann
e5f44a5160
add Makefile; include manpage&completion in setup.py (#150) 2019-03-24 11:03:02 +01:00
Mike Fährmann
eacebf41e4
fix typo in README 2019-03-24 11:03:02 +01:00
Mike Fährmann
e47a24afc7
script to generate a simple man page (#150) 2019-03-24 11:03:01 +01:00
Leonardo Taccari
1e38f65996 [instagram] Add support for GraphSidecar media types (#201)
* [instagram] Add support for GraphSidecar media types

Refactor _extract_postpage() to always return a list of medias.

Fetch common keywords and gracefully handle GraphSidecar media type
by extracting each single media and adding `sidecar_media_id' and
`sidecar_shortcode' keywords to indicate the parent of sidecar
childrens.

While here join the copyright comment lines in a single one.

Closes #178.

* [instagram] Use `yield from' instead of `for ... yield' (thanks @mikf)!

* [instagram] Adjust filename for GraphSidecar medias

Add a possible leading `media_id' of the sidecar for GraphSidecar
media.

Thanks to @mikf for the suggestion!

* [instagram] Add extra metadata for youtube-dl in GraphSidecar childrens

GraphSidecar children ytdl: URLs when consumed by youtube-dl
redirects to the URL of their parent.  In GraphSidecar-s with
multiple GraphVideo-s this leads to downloading the same video
multiple times.

Add a `_ytdl_index' field to indicate the index of the youtube-dl
playlist corresponding the children of the sidecar.

This will be used by the `ytdl' downloader.
2019-03-24 11:02:32 +01:00
Mike Fährmann
e7d0d98c88
improve FFmpeg arguments for --ugoira-conv 2019-03-23 09:50:39 +01:00
Mike Fährmann
b0f88c2ab5
script to generate a simple bash completion file (#150) 2019-03-23 09:50:39 +01:00
Mike Fährmann
6ba67b0537
[hypnohub] add extractors (closes #196) 2019-03-23 09:50:39 +01:00
Mike Fährmann
fe27154a10
[komikcast] fix extraction
... again
2019-03-23 09:50:39 +01:00
林博仁(Buo-ren Lin)
b21c71021b snap: Support downloading video using youtube-dl (#199)
This patch adds a copy of the youtube-dl package to the snap to enable the video downloading feature.

Tested with the Twitter extractor.

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-23 09:50:08 +01:00
Mike Fährmann
5ec55ec4fc
[deviantart] improve URLs for non-downloadable deviations 2019-03-21 15:37:22 +01:00
Mike Fährmann
c7a6b0ed90
[deviantart] add 'metadata' option (#189) 2019-03-21 14:49:42 +01:00
林博仁(Buo-ren Lin)
eb3522601d snap: Support official config paths via *-files confinement interfaces (#197)
Refer-to: The personal-files interface - doc - snapcraft.io <https://forum.snapcraft.io/t/the-personal-files-interface/9357>
Refer-to: The system-files interface - doc - snapcraft.io <https://forum.snapcraft.io/t/the-system-files-interface/9358>
Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-21 13:49:13 +01:00
Mike Fährmann
8d96a8ce4c
[500px] add user-, gallery-, and image-extractors (#185) 2019-03-20 17:32:36 +01:00
Mike Fährmann
d0f88c35be
[komikcast] fix extraction 2019-03-18 11:12:19 +01:00
Mike Fährmann
6277a739e4
[35photo] add user-, genre-, and image-extractors (#162) 2019-03-18 01:11:30 +01:00
Mike Fährmann
fb14f80d62
[tumblr] fix avatar URLs for non-OAuth1.0 calls (closes #193) 2019-03-17 11:07:22 +01:00
Mike Fährmann
8c20443839
release version 1.8.0 2019-03-15 15:27:11 +01:00
Mike Fährmann
973a720a7a
[weibo] fix unit test URL patterns 2019-03-15 15:19:39 +01:00
Mike Fährmann
a2af2d2965
adjust cache maxage values 2019-03-14 22:21:49 +01:00
Mike Fährmann
f612284d24
cache cfclearance cookies 2019-03-14 16:14:29 +01:00
Mike Fährmann
34ea0d6a10
rewrite cache module
less complexity, better performance,
but some duplicate code here and there
2019-03-14 15:55:48 +01:00
Mike Fährmann
12482553bd
update links to youtube-dl 2019-03-13 22:03:02 +01:00
Mike Fährmann
591a07f20c
small code changes and cleanups 2019-03-13 22:03:02 +01:00
Mike Fährmann
6f57d44ec2
[seaotterscans] remove extractor
http://seaotterscans.com/ now redirects to their MangaDex profile
2019-03-13 22:02:45 +01:00
林博仁(Buo-ren Lin)
8aaa4edbcd Implement selective pulling for building tagged release snaps (#187)
This patch re-implements the pull step that will only build
development snapshots snaps if the latest tagged release has been
promoted to the stable channel.  This ensures that there's always
a revision of the stable release snap available in the edge channel
for the publisher to promote to stable as currently the build
infrastructure only supports build on code push (but not new tagged
releases) at this time.

The implementation is inspired from the snapcrafters adopted snaps workflow,
documented in the following forum topic:

    The automatic build and pubish process of snaps owned by the Snapcrafters - doc - snapcraft.io
    https://forum.snapcraft.io/t/autopublishing-of-snapcrafters-organizations-snaps-how/7954/2

Refer the following forum topic for a possible solution proposal of this problem:

    Proposal: Allow overriding the source-tag property for an one-time build in the build infrastructure - snapcraft - snapcraft.io
    https://forum.snapcraft.io/t/proposal-allow-overriding-the-source-tag-property-for-an-one-time-build-in-the-build-infrastructure/10303

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-13 21:58:46 +01:00
林博仁(Buo-ren Lin)
d398b1de40 snap: Use SNAPCRAFT_ARCH_TRIPLET for pulseaudio library workaround (#188)
Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-13 21:52:35 +01:00
Mike Fährmann
6dae6bee37
automatically detect and bypass cloudflare challenge pages
TODO: cache and re-apply cfclearance cookies
2019-03-10 15:31:33 +01:00
Mike Fährmann
25aaf55514
[smugmug] improve format selection (closes #183)
- use original image if available
- support video formats
- remove user info for ImageExtractor (it is no longer possible to get
  image owner information for a single image)
2019-03-10 15:20:35 +01:00
Mike Fährmann
7c1cb923a4
[myportfolio] replace unit test
the old gallery got removed
2019-03-10 15:06:16 +01:00
Mike Fährmann
fffbfd3dce
[imgspice] fix extraction 2019-03-09 20:29:23 +01:00
Mike Fährmann
4ca4631bad
simplify auto-disabling certificate verification
if no certificate bundle is found
2019-03-08 16:34:01 +01:00
Mike Fährmann
09d872a2b1
generalize extractor creation code 2019-03-07 22:55:26 +01:00