Mike Fährmann
2f3a021d72
[hentaicafe] restore functionality
2019-02-05 10:22:52 +01:00
Mike Fährmann
347398f692
fix various tests
2019-02-04 14:40:21 +01:00
Mike Fährmann
00dc37ccbf
replace AsynchronousMixin Extractor with a Mixin
2019-02-04 14:21:19 +01:00
Mike Fährmann
4d656a81ca
replace SharedConfigExtractor class with a Mixin
2019-02-04 13:46:02 +01:00
Mike Fährmann
ccb95d0ba4
[mastodon] changes/improvements based on foolfuuka/-slide
2019-02-04 13:13:58 +01:00
Mike Fährmann
12ff750111
[foolfuuka] smaller code changes and updates
2019-02-04 12:55:33 +01:00
Mike Fährmann
e1bf3b225e
[foolslide] dynamically generate extractor classes
2019-02-04 12:54:07 +01:00
Mike Fährmann
58a9eede38
[foolfuuka] dynamically generate extractor classes
2019-02-03 17:09:45 +01:00
Mike Fährmann
22d7a783d5
update extraction result tests
2019-02-02 15:37:54 +01:00
Mike Fährmann
197d0e99a4
[tsumino] more useful error message ( #161 )
...
if Tsumino suspects a non-human user and refuses to send gallery pages
2019-02-02 14:57:51 +01:00
Mike Fährmann
d36ec51e5a
[tsumino] add extractor for search results ( #161 )
2019-02-02 14:56:46 +01:00
Mike Fährmann
1c1367ec5b
[behance] fix empty docstring
2019-02-02 14:41:05 +01:00
Mike Fährmann
373cb07b28
update .travis.yml and run_tests.sh
...
- add python3.8 and pypy3 builds
- remove deprecated 'sudo: true' and 'sudo: false'
- enable builds for 'test-...' branches
2019-01-31 15:58:52 +01:00
Mike Fährmann
45e529ab91
[behance] fix extraction
...
HTML structure for gallery pages changed quite a bit, so it is now using
the embedded JSON data. This changes a lot of metadata field names, but
'gallery_id', 'title', and 'user' are still provided for backwards
compatibility.
The internal API endpoint for user galleries also changed its data
structure, but nothing too major.
2019-01-31 14:33:23 +01:00
Mike Fährmann
e1d3e9a926
add 'ext_from_url' to text.py
2019-01-31 12:23:25 +01:00
Mike Fährmann
bfbbac4495
[tsumino] add login capabilities ( #161 )
2019-01-30 17:58:48 +01:00
Mike Fährmann
dd358b4564
improve cookie handling during logins
2019-01-30 17:09:32 +01:00
Mike Fährmann
6126615698
update URLs for supportedsites.rst
2019-01-30 16:18:22 +01:00
Mike Fährmann
80a75a1ecf
[tsumino] add gallery extractor ( #161 )
2019-01-29 17:28:48 +01:00
Mike Fährmann
2d2953a5bf
add 'text.parse_float()' + cleanup in text.py
2019-01-29 16:46:21 +01:00
Mike Fährmann
0c32dc5858
[hentaifox] add extractor for search results ( #160 )
2019-01-28 22:38:32 +01:00
Mike Fährmann
580947bfce
[hentaifox] rename Chapter- to GalleryExtractor ( #160 )
2019-01-28 21:49:26 +01:00
Mike Fährmann
8095f5f81a
[mangapark] fix manga title extraction
2019-01-28 18:04:42 +01:00
Mike Fährmann
0156189468
[hentaifox] add chapter extractor ( #160 )
2019-01-28 18:00:32 +01:00
Mike Fährmann
e4171d6baf
[luscious] add login capabilities ( closes #159 )
2019-01-28 17:14:15 +01:00
Mike Fährmann
4f49fdf065
[mastodon] various improvements and fixes ( #144 )
...
- allow instances to specify their own 'category'
- improve config lookup:
- first look into extractor.<category>.*
- and afterwards look into extractor.mastodon.<instance>.*
- add a default entry for pawoo.net in a way that actually works
- add an 'instance' keyword and turn 'tags' into a usable list
2019-01-27 20:43:33 +01:00
Mike Fährmann
3f608a84b7
[photobucket] don't crash if JSON data is missing
2019-01-27 20:37:22 +01:00
Mike Fährmann
134487ffb0
[exhentai] stop extraction if image limit is exceeded ( #141 )
...
can be turned off with the `exhentai.limits' option
2019-01-26 18:40:39 +01:00
Mike Fährmann
e868fb4393
[exhentai] improve gallery extraction
...
- match image page URLs and extract galleries from that point onward
- add a few more metadata entries: 'parent', 'visible', 'cost'
2019-01-26 18:23:25 +01:00
Mike Fährmann
a50e9faf0e
[newgrounds] recognize direct links
2019-01-25 16:35:12 +01:00
Mike Fährmann
9fba48fbd7
[postprocessor:metadata] add '--write-tags' flag ( #135 )
2019-01-23 21:56:03 +01:00
Mike Fährmann
c5559fa07d
[photobucket] improve subalbum extraction ( #117 )
...
The former implementation would produce a complete list of all subalbums
for each (sub)album extraction. This would for example result in a
level 2 subalbum getting "extracted" twice: once through the root-album
(level 0) and once through its parent album on level 1.
In the current implementation only the next level of subalbums are
returned, which themselves will handle their next level in a recursive
fashion.
2019-01-22 21:44:05 +01:00
Mike Fährmann
ecad69100a
[photobucket] add 'image' extractor ( #117 )
2019-01-22 17:24:43 +01:00
Mike Fährmann
b50b30f1c9
[photobucket] download subalbums ( #117 )
2019-01-22 14:05:18 +01:00
Mike Fährmann
d19bac71be
[photobucket] add 'album' extractor ( #117 )
2019-01-20 16:19:13 +01:00
Mike Fährmann
78b5f29a00
[sankaku] unescape tags
2019-01-20 16:18:13 +01:00
Mike Fährmann
277b52101a
add 'category-transfer' option
...
[ci skip]
2019-01-19 20:28:19 +01:00
Mike Fährmann
9b8ac12eed
[behance] enable 'categorytransfer' for collections ( #157 )
2019-01-19 20:02:20 +01:00
Mike Fährmann
217a0687ef
[behance] add 'collection' extractor ( closes #157 )
2019-01-19 18:11:20 +01:00
Mike Fährmann
b8fed34548
add generalized extractors for Mastodon instances ( #144 )
...
Extractors for Mastodon instances can now be dynamically generated,
based on the instance names in the 'extractor.mastodon.*' config path.
Example:
{
"extractor": {
"mastodon": {
"pawoo.net": { ... },
"mastodon.xyz": { ... },
"tabletop.social": { ... },
...
}
}
}
Each entry requires an 'access-token' value, which can be generated with
'gallery-dl oauth:mastodon:<instance URL>'.
An 'access-token' (as well as a 'client-id' and 'client-secret') for
pawoo.net is always available, but can be overwritten as necessary.
2019-01-19 14:28:59 +01:00
Mike Fährmann
4b441c162e
release version 1.6.3
2019-01-18 12:31:04 +01:00
Mike Fährmann
66460337f1
[mangapark] fix extraction
2019-01-17 21:24:53 +01:00
Mike Fährmann
8aba2bdebf
[postprocessor:metadata] add 'tags' and 'custom' modes ( #135 )
2019-01-17 21:18:12 +01:00
Mike Fährmann
79c01ec7ae
implement J<separator>/ format option
...
J joins list elements by calling <separator>.join(list):
Example:
{f:J - /} -> "a - b - c" (if "f" is ["a", "b", "c"])
2019-01-17 17:01:58 +01:00
Mike Fährmann
2ffc105887
[exhentai] extract tag metadata
2019-01-15 18:08:17 +01:00
Mike Fährmann
0fb98d1d79
[hbrowse] extract tag metadata
2019-01-15 18:08:10 +01:00
Mike Fährmann
9bbbadd93a
[hbrowse] use HTTPS
2019-01-15 18:07:39 +01:00
Mike Fährmann
2fbf072723
[newgrounds] ensure consistent tag order
...
... plus some code restructuring
2019-01-14 16:14:19 +01:00
Mike Fährmann
d7a4739cf6
[hbrowse] print error message if site is down
...
... instead of crashing with a meaningless exception
2019-01-14 15:44:23 +01:00
Mike Fährmann
98c6520384
[pinterest] update root URL of API calls
2019-01-14 15:22:04 +01:00
