1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 18:53:21 +01:00
Commit Graph

1695 Commits

Author SHA1 Message Date
Mike Fährmann
fb14f80d62
[tumblr] fix avatar URLs for non-OAuth1.0 calls (closes #193) 2019-03-17 11:07:22 +01:00
Mike Fährmann
8c20443839
release version 1.8.0 2019-03-15 15:27:11 +01:00
Mike Fährmann
973a720a7a
[weibo] fix unit test URL patterns 2019-03-15 15:19:39 +01:00
Mike Fährmann
a2af2d2965
adjust cache maxage values 2019-03-14 22:21:49 +01:00
Mike Fährmann
f612284d24
cache cfclearance cookies 2019-03-14 16:14:29 +01:00
Mike Fährmann
34ea0d6a10
rewrite cache module
less complexity, better performance,
but some duplicate code here and there
2019-03-14 15:55:48 +01:00
Mike Fährmann
12482553bd
update links to youtube-dl 2019-03-13 22:03:02 +01:00
Mike Fährmann
591a07f20c
small code changes and cleanups 2019-03-13 22:03:02 +01:00
Mike Fährmann
6f57d44ec2
[seaotterscans] remove extractor
http://seaotterscans.com/ now redirects to their MangaDex profile
2019-03-13 22:02:45 +01:00
林博仁(Buo-ren Lin)
8aaa4edbcd Implement selective pulling for building tagged release snaps (#187)
This patch re-implements the pull step that will only build
development snapshots snaps if the latest tagged release has been
promoted to the stable channel.  This ensures that there's always
a revision of the stable release snap available in the edge channel
for the publisher to promote to stable as currently the build
infrastructure only supports build on code push (but not new tagged
releases) at this time.

The implementation is inspired from the snapcrafters adopted snaps workflow,
documented in the following forum topic:

    The automatic build and pubish process of snaps owned by the Snapcrafters - doc - snapcraft.io
    https://forum.snapcraft.io/t/autopublishing-of-snapcrafters-organizations-snaps-how/7954/2

Refer the following forum topic for a possible solution proposal of this problem:

    Proposal: Allow overriding the source-tag property for an one-time build in the build infrastructure - snapcraft - snapcraft.io
    https://forum.snapcraft.io/t/proposal-allow-overriding-the-source-tag-property-for-an-one-time-build-in-the-build-infrastructure/10303

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-13 21:58:46 +01:00
林博仁(Buo-ren Lin)
d398b1de40 snap: Use SNAPCRAFT_ARCH_TRIPLET for pulseaudio library workaround (#188)
Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-13 21:52:35 +01:00
Mike Fährmann
6dae6bee37
automatically detect and bypass cloudflare challenge pages
TODO: cache and re-apply cfclearance cookies
2019-03-10 15:31:33 +01:00
Mike Fährmann
25aaf55514
[smugmug] improve format selection (closes #183)
- use original image if available
- support video formats
- remove user info for ImageExtractor (it is no longer possible to get
  image owner information for a single image)
2019-03-10 15:20:35 +01:00
Mike Fährmann
7c1cb923a4
[myportfolio] replace unit test
the old gallery got removed
2019-03-10 15:06:16 +01:00
Mike Fährmann
fffbfd3dce
[imgspice] fix extraction 2019-03-09 20:29:23 +01:00
Mike Fährmann
4ca4631bad
simplify auto-disabling certificate verification
if no certificate bundle is found
2019-03-08 16:34:01 +01:00
Mike Fährmann
09d872a2b1
generalize extractor creation code 2019-03-07 22:55:26 +01:00
Mike Fährmann
8dc6be246b
[shopify] add custom retry logic for 430 status codes (#175) 2019-03-07 15:31:15 +01:00
Mike Fährmann
0887fb61f4
[komikcast] update test results 2019-03-07 14:55:52 +01:00
Mike Fährmann
a881537b91
more util.py tests 2019-03-06 21:09:37 +01:00
Mike Fährmann
976ccb267f
[myportfolio] combine gallery and user extractors
An URL alone isn't good enough to distinguish between a gallery or a
gallery-listing, so the new extractor decides what to do based on the
page's content.
2019-03-06 19:45:01 +01:00
Mike Fährmann
efd104e45e
[instagram] reject more non-user URLs (#180) 2019-03-06 10:26:01 +01:00
HRXN
56e0e92e0d [shopify] cosmetic changes in shopify.py (#181)
Glanced over the commits, randomly spotted some minor things.
2019-03-06 09:16:27 +01:00
Mike Fährmann
23baecb29e
fix 'CONVERSIONS' variable name 2019-03-05 22:50:56 +01:00
Mike Fährmann
9c0e2f294b
[shopify] add generic collection and product extractors (#175)
with fashionnova.com  as a default domain
2019-03-05 22:33:37 +01:00
Mike Fährmann
105097ddcf
add 'S' conversion options for format string fields
Same as 's' (convert to string), but has a better, human-readable
conversion for lists.
2019-03-04 21:13:34 +01:00
Mike Fährmann
1578013efc
remove unused default config path 2019-03-04 20:53:58 +01:00
Mike Fährmann
26c4365baa
adjust metadata types for GalleryExtractors 2019-03-02 14:53:04 +01:00
Mike Fährmann
13e0f2a78f
[deviantart] add 'scraps' extractor (closes #168) 2019-03-01 14:13:34 +01:00
Mike Fährmann
3ea11f5d5e
[nhentai] rewrite
- use GalleryExtractor as base class
- extract a lot more metadata (artist, tags, etc.)
2019-03-01 14:13:34 +01:00
Mike Fährmann
176b7253a1
update function signature for config.load() 2019-03-01 14:13:34 +01:00
Mike Fährmann
3595cd582f
use GalleryExtractor as common base class 2019-03-01 14:13:16 +01:00
林博仁(Buo-ren Lin)
77551bf01b Implement snap packaging, snaps are universal linux packages (#170)
This patch implements the necessary details to package gallery-dl as a snap that can be run on a broad range of supporting GNU/Linux distributions[1].

[1] https://snapcraft.io/

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2019-03-01 14:12:41 +01:00
Mike Fährmann
a138d5873d
[hentaifoundry] improve/fix extraction
- Sometimes an ad interfered when trying to get a download URL
- Resolving "www.hentai-foundry.com" yields an invalid(?) IPv6 address
  (2607:5300:60:ca9e:feed:dead:beef:1) and urllib3 only tries to connect
  to the IPv4 variant after a rather long wait time
2019-02-25 16:16:09 +01:00
Mike Fährmann
280531c8ff
[pururin] add gallery extractor (closes #174) 2019-02-25 14:54:57 +01:00
Mike Fährmann
3159dd79d5
[seiga] use HTTPS 2019-02-21 22:51:11 +01:00
Mike Fährmann
ea6fbfdd39
simplify build_supportedsites.py 2019-02-21 14:29:19 +01:00
Mike Fährmann
ee2ad76620
update and improve create_test_data.py script 2019-02-20 15:48:31 +01:00
Mike Fährmann
f6734142ee
[komikcast] remove 'width' and 'height' info 2019-02-19 15:12:40 +01:00
Mike Fährmann
d0059cab79
[tumblr] check for null URLs (closes #165) 2019-02-19 13:49:55 +01:00
Mike Fährmann
e687a6095e
[luscious] raise exception if album is not available 2019-02-19 13:30:39 +01:00
Mike Fährmann
22d3a2fcc8
[artstation] add extractor for artwork listings (#80)
like https://www.artstation.com/artwork?sorting=latest
or   https://www.artstation.com/artwork?sorting=picks
2019-02-18 12:45:44 +01:00
Mike Fährmann
937a802b49
[dynastyscans] add extractors for images and image searches
(closes #163)
2019-02-18 12:25:52 +01:00
Mike Fährmann
b09a8184ca
move TestJob into test module; test _extractor values 2019-02-17 18:18:31 +01:00
Mike Fährmann
19860655a3
[weibo] add 'user' and 'status' extractors 2019-02-17 18:18:31 +01:00
Mike Fährmann
f8782c05f2
[paheal] rename "tags" to "search_tags"
to better match field names of other booru extractors
2019-02-17 18:18:09 +01:00
Mike Fährmann
c7b8421333
[deviantart] don't match 'www' as a potential username 2019-02-15 16:38:29 +01:00
Mike Fährmann
5530871b5a
change results of text.nameext_from_url()
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)

Example: "https://example.org/path/filename.ext"

before:
- filename : filename.ext
- name     : filename
- extension: ext

now:
- filename : filename
- extension: ext
2019-02-14 16:07:17 +01:00
Mike Fährmann
148b8f15d0
update tests for util.py 2019-02-14 11:15:19 +01:00
Mike Fährmann
ae353ed3b0
provide "extractor" and "job" keys for logging output
This allows for stuff like "{extractor.url}" and "{extractor.category}"
in logging format strings.
Accessing 'extractor' and 'job' in any way will return "None" if those
fields aren't defined, i.e. in general logging messages.
2019-02-14 11:09:58 +01:00