Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-23 03:02:50 +01:00
Code Issues Releases Wiki Activity
4,897 Commits 6 Branches 164 Tags 16 MiB
b92645cd37
Commit Graph

621 Commits

Author SHA1 Message Date
Mike Fährmann
0cac14c3bd
update extractor test results 2020-06-25 19:11:47 +02:00
Mike Fährmann
53cc498d9c
improve config lookup when there are multiple possible locations 
This specifically applies to all Mastodon extractors and all
extractors with a 'basecategory', i.e. 'booru', 'foolslide', etc.

Values inside those general config locations wouldn't be recognized
when a value with the same was set on the 'extractor' level.

For example 'extractor.mastodon.directory' should be used over
'extractor.directory' when both are set, but this was impossible
with the previous implementation.

(fixes #843)
 2020-06-21 00:07:10 +02:00
Mike Fährmann
d81a8e6544
[twitter] update tests 2020-06-19 23:01:02 +02:00
Mike Fährmann
37d71f6e09
strip microseconds in text.parse_datetime() 2020-06-17 21:40:16 +02:00
Mike Fährmann
6db7ed90cb
release version 1.14.1 2020-06-12 20:12:09 +02:00
Mike Fährmann
087e3184dc
use a non-twitter URL when testing snap creation 2020-06-12 18:31:14 +02:00
Mike Fährmann
7daef6ee70
update extractor test results 
- certain posts on Instagram now return
  https://static.cdninstagram.com/rsrc.php/null.jpg
  for public users
- MangaDex is deploying its new MangaDex@Home network similar to
  exhentai's Hentai@Home
- realbooru has a new site layout, but the underlying booru API still
  works like before
 2020-06-12 00:36:06 +02:00
Mike Fährmann
3bad1579ee
update extractor test results 2020-05-31 17:42:07 +02:00
Mike Fährmann
45baa13615
update extractor test results 
- don't run Instagram tests on Travis anymore
- replace Twitter test because timeline was made private
- update Hiperdex domain to '.com' (again ...)
 2020-05-28 02:18:06 +02:00
Mike Fährmann
dfcf2a2c91
write OAuth token to cache by default (#616) 2020-05-25 22:35:45 +02:00
Mike Fährmann
6294e2c540
add 'text.ensure_http_scheme()' 2020-05-19 22:32:53 +02:00
Mike Fährmann
ece73b5b2a
make 'path' and 'keywords' available in logging messages 
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.

__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.

(#574, #575)
 2020-05-18 19:04:51 +02:00
Mike Fährmann
4b606b68e4
skip OAuth tests when server is unreachable 2020-05-10 00:33:00 +02:00
Mike Fährmann
8b60bd6a91
mock 'time()' in cache tests 
instead of calling 'sleep()' to let time advance.

This shortens the time needed to run those tests,
and ensures consistent results.
(Tests would randomly fail when using 'sleep()')
 2020-05-09 23:55:14 +02:00
Mike Fährmann
8f2c1da041
skip example config tests if files are not available (#730) 2020-05-08 22:56:00 +02:00
Mike Fährmann
5df8f2959b
insert local directory into PYTHONPATH when running tests 2020-05-02 01:15:50 +02:00
Mike Fährmann
ff47641b13
test whether default/example config files contain valid JSON 2020-04-30 00:00:41 +02:00
Mike Fährmann
d6facdee7b
[mastodon] add tests (#701) 2020-04-22 21:10:34 +02:00
Mike Fährmann
fd438f0d78
update extractor test results 2020-04-11 23:00:42 +02:00
Mike Fährmann
a0f4c295c0
add optional 'utcoffset' argument to 'parse_datetime()' 2020-04-11 02:05:00 +02:00
Mike Fährmann
406449b0d6
ensure keys for mastodon instances are available during tests 
Calls to config.clear() from other tests are removing the API
credentials set when importing mastodon.py for the first time.
 2020-04-08 21:56:14 +02:00
Mike Fährmann
9e7dfc0cfc
[myportfolio] fix extraction of galleries without title 2020-04-08 21:08:05 +02:00
Mike Fährmann
3b50c4f49d
add tests for "Extractors" in oauth.py (#670) 2020-04-07 20:26:12 +02:00
Mike Fährmann
04bd0472de
add tests for Extractor.wait() 2020-04-07 20:24:56 +02:00
Mike Fährmann
7499d71d02
[simplyhentai] ignore certificate errors in video test 2020-03-28 21:07:30 +01:00
Mike Fährmann
4203dc0bdc
[mangapark] fix metadata extraction 2020-03-28 03:00:26 +01:00
Mike Fährmann
77fda8190c
[35photo] simplify/remove tests for the 'genre' extractor 
There is still a nice genre overview page (https://35photo.pro/genre/)
but the individual sub-pages don't list photos anymore
 2020-03-24 02:48:25 +01:00
Mike Fährmann
32e36d8f02
[sexcom] replace tests 2020-03-17 22:47:45 +01:00
Mike Fährmann
e0b0e8d62a
release version 1.13.2 2020-03-14 20:19:36 +01:00
Mike Fährmann
a63a376ad2
[mangoxo] fix login 2020-03-08 23:01:51 +01:00
Mike Fährmann
ebc70e87ce
[e621] update to new interface / API endpoints (closes #635) 2020-03-06 21:12:58 +01:00
Mike Fährmann
72122eb9b3
release version 1.13.1 2020-03-01 21:17:29 +01:00
Mike Fährmann
ce5e2a58fe
[imgbb] update test results 
Image server domain changed from
https://image.ibb.co/ to https://i.ibb.co/
 2020-03-01 20:38:25 +01:00
Mike Fährmann
1d4a369ea2
update extractor test results 2020-02-27 22:15:40 +01:00
Mike Fährmann
ec85bf90de
use context managers in cache.py & add tests 2020-02-25 23:08:47 +01:00
Mike Fährmann
4e361b3008
add tests for specific datetime values 2020-02-23 16:48:30 +01:00
Mike Fährmann
90e4c645ba
[formatter] allow multiple "special" format specifiers (#595) 
It is now, for example, possible to specify multiple replacement
operations per format replacement field: {name:Ra/b/Rc/d/}
 2020-02-16 21:47:08 +01:00
Mike Fährmann
219c4cc78c
[formatter] allow for numeric list and string indices 2020-02-15 22:46:22 +01:00
Mike Fährmann
7d1da614d9
[formatter] implement field name alternatives (#525) 
The format string '{a|b|c}' will now try to use the value from 'a' and
fall back to 'b' and 'c' if accessing a field raises an exception or
if its value is None.
 2020-02-15 17:58:21 +01:00
Mike Fährmann
c7cf9dd111
[furaffinity] support classic layout (#284) 2020-02-12 21:39:43 +01:00
Mike Fährmann
40fe062851
[pixiv] fix user id for bookmarks API calls (closes #596) 2020-02-01 01:48:46 +01:00
Mike Fährmann
2852691d78
[paheal] replace test URL 
searching for 'k-on' doesn't yield any results anymore
 2020-01-27 22:19:41 +01:00
Mike Fährmann
2a9be48511
improve util.load/save_cookiestxt() and add tests 
- take a file object as argument instead of an filename
- accept whitespace before comments ("   # comment")
- map expiration "0" to None and not the number 0
 2020-01-25 23:02:15 +01:00
Mike Fährmann
b3b5754f2d
update test_cookies.py 2020-01-21 22:46:58 +01:00
Mike Fährmann
174117f827
allow multiple hashes for content tests 2020-01-18 00:19:35 +01:00
Mike Fährmann
60a43f0264
fix downloader tests 2020-01-14 11:51:06 +01:00
Mike Fährmann
e89413da22
update test results 2020-01-12 21:41:06 +01:00
Mike Fährmann
5cac79c3d9
[erolord] remove extractor 2020-01-10 14:40:30 +01:00
Mike Fährmann
988cc2ec23
[mangadex] change domain to mangadex.cc (closes #559) 2020-01-05 21:21:40 +01:00
Mike Fährmann
87c8b89ddd
[postprocessor:metadata] add 'directory' option (#520) 2020-01-02 20:58:10 +01:00
Mike Fährmann
82f7f4172a
update test results 2020-01-01 16:05:38 +01:00
Mike Fährmann
d0920e84e9
update test results 2019-12-19 18:33:48 +01:00
Mike Fährmann
9e63804347
[patreon] make retrieving user info nonfatal (#508) 
… and fall back to the included data if an error occurs.
 2019-12-12 00:31:35 +01:00
Mike Fährmann
15f9bb3d14
add option to disable pyOpenSSL usage (#508) 
(pyOpenSSL is now disabled by default)
 2019-12-08 21:21:00 +01:00
Mike Fährmann
50deab5265
[deviantart] fix URL generation from /extended_fetch results 
(closes #505)
 2019-12-07 22:07:55 +01:00
Mike Fährmann
004812258d
[hentaifox] fix extraction 2019-12-02 22:21:45 +01:00
Mike Fährmann
a412531451
[postprocessor:metadata] implement 'extension-format' option 
closes #477
 2019-11-30 17:26:17 +01:00
Mike Fährmann
b5c964332b
improve config.py test coverage 2019-11-25 17:20:00 +01:00
Mike Fährmann
f5604492c3
update interface of config functions 2019-11-24 00:42:28 +01:00
Mike Fährmann
3fc1e12949
[postprocessor:metadata] filter private entries 
i.e. keys starting with an underscore
 2019-11-21 16:58:44 +01:00
Mike Fährmann
978cb03f81
update misc test results 
- Livedoor now uses https:// for its image URLs
- Instagram image URLs got simplified
 2019-11-20 21:45:48 +01:00
Mike Fährmann
bbbeff4c41
[downloader.http] implement file-specific HTTP headers 2019-11-19 23:50:54 +01:00
Mike Fährmann
3ece3976ae
[newgrounds] implement login support (#394) 2019-11-16 23:45:32 +01:00
Mike Fährmann
abfcb356fc
[flickr] support 3k, 4k, 5k, and 6k photo sizes (closes #472) 2019-11-10 17:52:51 +01:00
Mike Fährmann
da6789b2b0
disable unique archive id checks for some tests 
- same image twice in a livedoor blog post
- unreliable results for related pinterest items
 2019-11-10 17:04:51 +01:00
Mike Fährmann
ba083b30b2
fix snap build 
… hopefully
 2019-11-08 21:44:12 +01:00
Mike Fährmann
94a94f3b86
miscellaneous stuff 2019-11-08 20:58:53 +01:00
Mike Fährmann
9e88e7a344
[postprocessor:exec] improve (#421, #413) 
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
 2019-11-03 21:45:45 +01:00
Mike Fährmann
2a3bd4e3c7
rename extractor classes starting with a digit 2019-11-02 20:42:09 +01:00
Mike Fährmann
64786363be
[4chan] simplify 
- remove 'chan.py'
- slight adjustments to directory and filenames
 2019-11-02 20:11:21 +01:00
Mike Fährmann
557e2c018b
[8chan] remove module 2019-11-02 20:06:47 +01:00
Mike Fährmann
322c2e7ed4
renaming variables 
mostly 'keyword(s)' to 'kwdict'
 2019-10-29 15:46:35 +01:00
Mike Fährmann
87a87bff7e
[simplyhentai] fix image URLs 2019-10-28 21:11:06 +01:00
Mike Fährmann
d5e3910270
adjust 'util.raises()' 2019-10-28 15:06:17 +01:00
Mike Fährmann
b23c822b23
[luscious] use GraphQL 2019-10-22 21:17:08 +02:00
Mike Fährmann
1693d97bd3
update extractor class hierarchies 
- let the GalleryExtractor class inherit directly from Extractor
- make ChapterExtractor a subclass of GalleryExtractor
- change enumeration field names of GalleryExtractors to 'num'
 2019-10-16 18:15:29 +02:00
Mike Fährmann
7ebd984e8d
[imgur] print error message if no JSON data is found (#446) 2019-10-16 17:45:14 +02:00
Mike Fährmann
de4e2029d1
[nsfwalbum] update test album 
the old one is no longer available
 2019-09-28 20:48:15 +02:00
Mike Fährmann
913460240d
[reddit] fix 'extractor.blacklist()' arguments 
The second argument must support 'append()'.
 2019-09-24 23:01:12 +02:00
Mike Fährmann
1848788970
update test results etc 2019-09-08 11:33:35 +02:00
Mike Fährmann
d5fbb2d9de
[tumblr] ignore audio links from Spotify etc. 2019-09-07 18:18:12 +02:00
Mike Fährmann
c6c5cb1898
improve 'deviantart.quality' description 2019-08-30 18:41:18 +02:00
Mike Fährmann
c9b97dbca3
extend post processor tests 2019-08-26 23:10:12 +02:00
Mike Fährmann
49f6d7176d
[deviantart] restore filenames (#392) 
<title>_by_<user>_<id> --> <title>_by_<user>-<id>
 2019-08-23 22:02:03 +02:00
Mike Fährmann
e528f3cb77
adjust postprocessor test results 
see 2495b99
 2019-08-21 23:54:08 +02:00
Mike Fährmann
cb7f149974
fix mtime datetime test 
datetime.timestamp() uses local time for a naive datetime object
 2019-08-14 21:57:42 +02:00
Mike Fährmann
23251356cb
require 'extension' data for each URL (#382) 2019-08-14 20:03:03 +02:00
Mike Fährmann
dd72ae7164
add postprocessor tests 2019-08-12 21:46:30 +02:00
Mike Fährmann
0bb873757a
update PathFormat class 
- change 'has_extension' from a simple flag/bool to a field that
  contains the original filename extension
- rename 'keywords' to 'kwdict' and some other stuff as well
- inline 'adjust_path()'
- put enumeration index before filename extension (#306)
 2019-08-12 21:40:37 +02:00
Mike Fährmann
748e37554c
update .travis.yml 
- install pyOpenSSL before running tests
- simplify snap tests
 2019-08-11 16:03:19 +02:00
Mike Fährmann
b7fb93e2b2
[downloader:http] add 'adjust-extensions' option 2019-08-08 16:54:20 +02:00
Mike Fährmann
eb7da159e2
[imagebam] update URL test results 
Image URLs are now using https://, but the website itself is still
served as http://.
 2019-08-07 21:47:44 +02:00
Mike Fährmann
fa60109e97
[exhentai] don't use e-hentai.org for exhentai URLs 2019-08-02 21:10:09 +02:00
Mike Fährmann
4a0c98bfc9
miscellaneous fixes and adjustments 2019-08-01 22:09:43 +02:00
Mike Fährmann
40637556fa
[ngomik] fix extraction 2019-07-28 10:53:46 +02:00
Mike Fährmann
d9d44ad953
[tsumino] update test results 2019-07-24 21:17:23 +02:00
Mike Fährmann
b1bea8aaeb
add 'restrict-filenames' option (#348) 2019-07-23 17:41:24 +02:00
Mike Fährmann
b3851e01d9
release version 1.9.0 2019-07-19 21:55:25 +02:00
Mike Fährmann
12da6bd0c9
[simplyhentai] fix/improve extraction 2019-07-06 20:25:53 +02:00
Mike Fährmann
b89f0d8d3c
update extractor result tests 2019-07-01 20:02:47 +02:00
Mike Fährmann
40da44b17f
Merge branch 'v1.9.0' 2019-06-29 15:39:52 +02:00
Mike Fährmann
7a99e85943
[kissmanga] fix download URLs and file extensions 
The current Blogspot image URLs hosted on Kissmanga end with an
"invalid" query parameter (/000.png&upx=...), which doesn't get
recognized by 'spliturl()' and 'parseurl()' as such and gets therefore
included in the 'extension' field from 'text.nameext_from_url()'.
 2019-06-28 20:34:43 +02:00
Mike Fährmann
a9c89085fb
[instagram] implement login support (#195) 2019-06-26 23:58:47 +02:00
Mike Fährmann
b1985d6579
test default format strings during extractor result tests 
A missing value or an invalid "syntax" for a format replacement field
will raise an exception.
 2019-06-25 18:12:32 +02:00
Mike Fährmann
95b1e4c3c0
implement R<old>/<new>/ format option (#318) 2019-06-23 22:45:44 +02:00
Mike Fährmann
70713f0f28
fix extractor result tests 2019-06-20 18:12:36 +02:00
Mike Fährmann
ee4d7c3d89
update downloader.find() and related code 
Instead of replacing 'https' with 'http' for every URL in
'get_downloader()', this now only happens once during downloader
initialization. Also unit tests.
 2019-06-20 16:59:44 +02:00
Mike Fährmann
179d112083
[downloader] overhaul http and text modules 
Get rid of the modular structure and simplify/specialize those modules.
 2019-06-19 22:56:11 +02:00
Mike Fährmann
a77340c647
[keenspot] fix extraction for "TwoKinds" 2019-06-17 19:49:39 +02:00
Mike Fährmann
b171befa87
implement 'parse_unicode_escapes()' 2019-06-16 21:47:24 +02:00
Mike Fährmann
e05a96db5e
[deviantart] rename 'stash' to 'extra' (#302) 
'stash' is already used as a name for the StashExtractor and therefore
expected to be a dictionary.
 2019-06-10 21:05:25 +02:00
Mike Fährmann
7c6cb908f9
[xhamster] update test results 2019-06-07 16:28:49 +02:00
Mike Fährmann
62335b9015
[paheal] adjust test results 2019-06-05 11:42:01 +02:00
Mike Fährmann
6a34f4b0c1
skip tests on read timeouts; print list of skipped tests 2019-06-01 20:47:31 +02:00
Mike Fährmann
d33f5a7423
[wallhaven] rewrite 
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
  IDs that can't be translated to the new ID system
- support direct links to wallpapers
 2019-05-31 14:53:02 +02:00
Mike Fährmann
5499934ae2
[ngomik] fix extraction 2019-05-30 20:18:36 +02:00
Mike Fährmann
2b1999476e
implement 'text.rextract()' 2019-05-28 21:03:41 +02:00
Mike Fährmann
e30ada162d
fix cookie tests 
update _get_extractor():
- always return an Extractor instance with a _login_impl() method
- use Extractor.from_url()
 2019-05-26 20:22:04 +02:00
Mike Fährmann
2316e0ed3d
fix strptime workaround from b0e85a4 
Don't return a modified version of 'date_time' if strptime fails.
 2019-05-25 23:22:26 +02:00
Mike Fährmann
6764847349
fix cookie tests 
'cookies' is a CookieJar, not a dict,
and removing the call to '.keys()' doesn't have the same effect
 2019-05-14 22:32:40 +02:00
Mike Fährmann
a5b060765d
improve code in tests 
- use 'assertRaises' as context manager
- remove calls to .keys()
 2019-05-13 11:48:20 +02:00
Mike Fährmann
b0e85a42e3
apply workaround from 4736912 in parse_datetime() itself 2019-05-09 21:53:17 +02:00
Mike Fährmann
4736912d4e
[pixiv] work around strptime limitations in Python < 3.7 
"%z" doesn't allow a colon separator in older Python versions:
    - "+0900" is OK
    - "+09:00" raises an exception
 2019-05-08 18:08:03 +02:00
Mike Fährmann
d09864b581
implement text.parse_datetime() 2019-05-08 15:43:59 +02:00
Mike Fährmann
5582b06ae4
fix tests with 'urllist' messages 2019-04-30 16:31:48 +02:00
Mike Fährmann
5018781898
allow type tests by name 2019-04-29 17:27:59 +02:00
Mike Fährmann
6264a46212
use 'utcfromtimestamp()' 
'fromtimestamp()' converts its results to the local timezone and causes
problems when running tests on a different machine.
 2019-04-21 16:22:53 +02:00
Mike Fährmann
d670de0344
implement 'text.parse_timestamp()' 2019-04-21 15:28:27 +02:00
Mike Fährmann
21a7e395a7
implement convenience wrapper for text.extract functionality 2019-04-19 22:30:11 +02:00
Mike Fährmann
e25ebc4bff
don't disable certificate checks anymore 
Executables generated with PyInstaller auto-include the root certificate
file and certificate checks now work out-of-the-box.
 2019-04-17 13:27:19 +02:00
Mike Fährmann
d6ddb74cde
update test results 
- deviantart: 'index' is now an integer
- flickr: image file with lower quality
- paheal: image server name changed
- rule34: post got deleted
 2019-04-12 09:59:48 +02:00
Mike Fährmann
d9b94a585d
[mangoxo] add login support (#184) 
A very recent change: It is now only possible to see more
than the first 5 images of an album if you are logged in.
 2019-04-10 18:55:25 +02:00
Mike Fährmann
e730fc9045
[twitter] add login support (#214) 2019-04-09 09:27:49 +02:00
Mike Fährmann
790f15a56f
[photobucket] use HTTPS 2019-04-03 18:30:45 +02:00
Mike Fährmann
c70b21248d
[wikiart] add extractors (#179) 
for
- artists:          https://www.wikiart.org/en/thomas-cole
- artist-listings:  https://www.wikiart.org/en/artists-by-century/12
- artwork-listings: https://www.wikiart.org/en/paintings-by-media/grisaille
 2019-04-02 17:34:57 +02:00
Mike Fährmann
0c991a3155
add convenience targets to Makefile 2019-03-29 15:35:00 +01:00
Mike Fährmann
6277a739e4
[35photo] add user-, genre-, and image-extractors (#162) 2019-03-18 01:11:30 +01:00
Mike Fährmann
973a720a7a
[weibo] fix unit test URL patterns 2019-03-15 15:19:39 +01:00
Mike Fährmann
6f57d44ec2
[seaotterscans] remove extractor 
http://seaotterscans.com/ now redirects to their MangaDex profile
 2019-03-13 22:02:45 +01:00
Mike Fährmann
0887fb61f4
[komikcast] update test results 2019-03-07 14:55:52 +01:00
Mike Fährmann
a881537b91
more util.py tests 2019-03-06 21:09:37 +01:00
Mike Fährmann
976ccb267f
[myportfolio] combine gallery and user extractors 
An URL alone isn't good enough to distinguish between a gallery or a
gallery-listing, so the new extractor decides what to do based on the
page's content.
 2019-03-06 19:45:01 +01:00
Mike Fährmann
9c0e2f294b
[shopify] add generic collection and product extractors (#175) 
with fashionnova.com  as a default domain
 2019-03-05 22:33:37 +01:00
Mike Fährmann
176b7253a1
update function signature for config.load() 2019-03-01 14:13:34 +01:00
Mike Fährmann
e687a6095e
[luscious] raise exception if album is not available 2019-02-19 13:30:39 +01:00
Mike Fährmann
b09a8184ca
move TestJob into test module; test _extractor values 2019-02-17 18:18:31 +01:00
Mike Fährmann
5530871b5a
change results of text.nameext_from_url() 
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)

Example: "https://example.org/path/filename.ext"

before:
- filename : filename.ext
- name     : filename
- extension: ext

now:
- filename : filename
- extension: ext
 2019-02-14 16:07:17 +01:00
Mike Fährmann
148b8f15d0
update tests for util.py 2019-02-14 11:15:19 +01:00
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor 2019-02-11 13:31:10 +01:00
Mike Fährmann
1f3422c28b
[mangahere] fix extraction 2019-02-10 22:10:53 +01:00
Mike Fährmann
84ae72b8d8
[ngomik] fix extraction 2019-02-10 14:19:08 +01:00
Mike Fährmann
9a9cd32461
implement alternative constructor for extractors 2019-02-09 14:42:25 +01:00
Mike Fährmann
abbd45d0f4
update handling of extractor URL patterns 
When loading extractor classes during 'extractor.find(…)', their
'pattern' attribute will be replaced with a compiled version of itself.
 2019-02-08 20:08:16 +01:00
Mike Fährmann
6284731107
simplify extractor constants 
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
 2019-02-08 13:45:40 +01:00
Mike Fährmann
bc0951d974
allow for simplified test data structures 
Instead of a strict list of (URL, RESULTS)-tuples, extractor result
tests can now be a single (URL, RESULTS)-tuple, if it's just one test,
and "only matching" tests can now be a simple string.
 2019-02-06 17:24:44 +01:00
Mike Fährmann
347398f692
fix various tests 2019-02-04 14:40:21 +01:00
Mike Fährmann
e1d3e9a926
add 'ext_from_url' to text.py 2019-01-31 12:23:25 +01:00
Mike Fährmann
2d2953a5bf
add 'text.parse_float()' + cleanup in text.py 2019-01-29 16:46:21 +01:00
Mike Fährmann
0c32dc5858
[hentaifox] add extractor for search results (#160) 2019-01-28 22:38:32 +01:00
Mike Fährmann
217a0687ef
[behance] add 'collection' extractor (closes #157) 2019-01-19 18:11:20 +01:00
Mike Fährmann
b8fed34548
add generalized extractors for Mastodon instances (#144) 
Extractors for Mastodon instances can now be dynamically generated,
based on the instance names in the 'extractor.mastodon.*' config path.

Example:
{
    "extractor": {
        "mastodon": {
            "pawoo.net": { ... },
            "mastodon.xyz": { ... },
            "tabletop.social": { ... },
            ...
        }
    }
}

Each entry requires an 'access-token' value, which can be generated with
'gallery-dl oauth:mastodon:<instance URL>'.
An 'access-token' (as well as a 'client-id' and 'client-secret') for
pawoo.net is always available, but can be overwritten as necessary.
 2019-01-19 14:28:59 +01:00
Mike Fährmann
66460337f1
[mangapark] fix extraction 2019-01-17 21:24:53 +01:00
Mike Fährmann
79c01ec7ae
implement J<separator>/ format option 
J joins list elements by calling <separator>.join(list):

Example:
{f:J - /} -> "a - b - c" (if "f" is ["a", "b", "c"])
 2019-01-17 17:01:58 +01:00
Mike Fährmann
9bbbadd93a
[hbrowse] use HTTPS 2019-01-15 18:07:39 +01:00
Mike Fährmann
98c6520384
[pinterest] update root URL of API calls 2019-01-14 15:22:04 +01:00
Mike Fährmann
751e535948
[nhentai] fix extraction (closes #156) 
Use JSON embedded in webpage since API endpoints have been disabled
 2019-01-14 07:57:50 +01:00
Mike Fährmann
1734a6c879
[reactor] detect "circular" redirects (#148) 2019-01-09 14:59:15 +01:00
Mike Fährmann
e53cdfd6a8
update build_supportedsites.py 2019-01-09 14:58:35 +01:00
Mike Fährmann
0afa913de4
[tumblr] add tests for hidden and private blogs (#145) 
Hidden / dashboard-only blogs are pretty straightforward and "only"
require a valid 'access-token' and 'access-token-secret' for the given
'api-key' and 'api-secret', so that signed OAuth1.0 requests are possible.

Private / password protected blogs on the other hand are a bit
cumbersome. In addition to a valid 'access-token' and
'access-token-secret', they also require the account belonging to those
tokens to be a member of the blog itself. Knowing the password and
entering it in the website isn't enough to access a blog through the
API. Following a private blog is also impossible, so that option can't
work either.
 2019-01-03 16:12:24 +01:00
Mike Fährmann
fa7fa2f8ff
[deviantart1 update tests] 2019-01-01 15:39:34 +01:00
Mike Fährmann
259123732f
[readcomiconline] improve comic-page parsing 2018-12-30 13:19:23 +01:00
Mike Fährmann
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor (#113) 
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
 2018-12-26 18:56:57 +01:00
Mike Fährmann
c5d4f558c9
allow missing field access keys in format strings (#136) 2018-12-22 13:54:14 +01:00
Mike Fährmann
4d73cc785d
update test results 2018-12-14 16:07:32 +01:00
Mike Fährmann
010da8372a
[instagram] relax test pattern 2018-12-11 19:59:28 +01:00
Mike Fährmann
15890930ea
[mangafox] fix extraction 
use mobile version since desktop version is obfuscated
 2018-11-26 16:13:41 +01:00
Mike Fährmann
fb53b5dd55
fix control+c during -j and range tests 2018-11-25 18:54:05 +01:00
Mike Fährmann
59bb434ba5
[flickr] add ability to download all albums of a user 
for example with 'https://www.flickr.com/photos/shona_s/albums'
 2018-11-23 09:09:37 +01:00
Mike Fährmann
041bd501fc
[hentaifoundry] unescape YII_CSRF_TOKEN value 
This fixes the POST requests to /site/filters
 2018-11-19 21:46:17 +01:00
Mike Fährmann
d4b2b73bef
release version 1.6.0 2018-11-17 18:28:02 +01:00
Mike Fährmann
3c25fa2dad
update build_testresult_db.py script 2018-11-15 22:58:14 +01:00
Mike Fährmann
7f6a0be982
adjust some tests 2018-11-15 22:50:04 +01:00
Mike Fährmann
966a9ca3a0
update test results 2018-11-10 19:14:54 +01:00
Mike Fährmann
c9861ca812
adjust message for status_code based exceptions 
from: 5xx HTTP Error: Reason
to  : 5xx: Reason

The "HTTP Error" part was in there to emulate Request's error messages
from response.raise_for_status(), but it reads a lot better without.
 2018-10-18 15:09:49 +02:00
Mike Fährmann
c00dce2adc
[behance] enable 'categorytransfer' 2018-10-09 23:40:49 +02:00
Mike Fährmann
1532d1b690
fix 'range' tests and update a few test results 2018-10-08 23:53:58 +02:00
Mike Fährmann
0514d6a0ae
make --filter and --range config-file options 
The functionality of --(chapter-)filter and --(chapter-)range are now
also exposed as the following config-file options:

- extractor.*.image-filter
- extractor.*.image-range
- extractor.*.chapter-filter
- extractor.*.chapter-range

TODO: update configuration.rst
 2018-10-07 21:39:56 +02:00
Mike Fährmann
4a348990f4
adjust value resolution for retries/timeout/verify options 
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.

'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.

Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
 2018-10-07 21:13:39 +02:00
Mike Fährmann
ca6ac4db6a
fix 'content' tests 2018-10-05 21:10:33 +02:00
Mike Fährmann
d70db2d555
Revert "[komikcast] fix extraction" 
This reverts commit 5507f5ce2e.
 2018-10-02 20:38:42 +02:00
Mike Fährmann
5507f5ce2e
[komikcast] fix extraction 2018-09-29 16:37:30 +02:00
Mike Fährmann
17611bfec0
update build_supportedsites.py script 2018-09-28 12:43:19 +02:00
Mike Fährmann
e066f35118
update extractor tests 2018-09-21 11:25:56 +02:00
Mike Fährmann
22ab509a70
[bobx] rename "model" to "idol" extractor 2018-09-14 18:11:36 +02:00
Mike Fährmann
8a23b21d0e
[tests] let 'pattern' require at least 1 URL 2018-09-02 21:19:44 +02:00
Mike Fährmann
0bc8ef51c8
[smugmug] Handle albums with no explicit owner (#100) 2018-09-01 12:55:02 +02:00
Mike Fährmann
590c0b3ad5
re-implement and improve filename formatter 
A format string now gets parsed only once instead of re-parsing it each
time it is applied to a set of data.

The initial parsing causes directory path creation to be at about 2x
slower than before, since each format string there is used only once,
but building a filename, the more common operation, is at least 2x
faster. The "directory slowness" cancels at about 5 filenames and
everything above that is significantly faster.
 2018-08-25 10:45:14 +02:00
Mike Fährmann
34b556922d
update/restore tests 2018-08-23 15:47:40 +02:00
Mike Fährmann
e3055d356c
release version 1.5.1 2018-08-17 13:21:36 +02:00
Mike Fährmann
f9ded38d89
[test:results] add support for "range" options in tests 2018-08-15 21:49:44 +02:00
Mike Fährmann
c9e6ccbd7c
[test:extractor] small fixes and improvements 2018-08-15 21:49:33 +02:00
Mike Fährmann
7f4e41c989
increase timeout during extractor tests 
cloudflare's 522 response takes longer than 30 seconds
 2018-08-10 16:51:05 +02:00
Mike Fährmann
b55e39d1ee
[mangadex] improve extraction 
- cache manga API results
- add artist, author and date fields to chapter metadata
- remove Manga-/ChapterExtractor inheritance
- minor code simplifications and improvements
 2018-08-10 16:50:07 +02:00
Mike Fährmann
2a9f3341a2
[behance] fix title extraction 2018-08-08 10:48:58 +02:00
Mike Fährmann
a86f2bfc80
[pinterest] update not-found redirects 2018-08-07 12:13:19 +02:00
Mike Fährmann
7442d2940c
release version 1.5.0 2018-08-03 17:50:27 +02:00
Mike Fährmann
b040ca0718
[rule34] small unit test fixes 2018-08-03 17:28:47 +02:00
Mike Fährmann
f3793660ef
update tests 2018-08-02 14:57:28 +02:00
Mike Fährmann
42a346413b
fix "re:" prefix for keyword tests 2018-08-02 14:48:51 +02:00
Mike Fährmann
e0dd8dff5f
implement L<maxlen>/<replacement>/ format option 
The L option allows for the contents of a format field to be replaced
with <replacement> if its length is greater than <maxlen>.

Example:
{f:L5/too long/} -> "foo"      (if "f" is "foo")
                 -> "too long" (if "f" is "foobar")

(#92) (#94)
 2018-07-29 13:52:07 +02:00
Mike Fährmann
bb89a1e6d7
[mangahere] use http:// 
invalid SSL cert for quite some time now
 2018-07-26 18:11:31 +02:00
Mike Fährmann
ce34d82cb4
fix skipping tests on 5xx status codes 2018-07-19 18:47:23 +02:00
Mike Fährmann
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor 2018-07-14 09:53:16 +02:00
Mike Fährmann
0ba93650e0
[8chan] replace unit test URL 
the other thread is no longer accessible
 2018-07-14 09:53:16 +02:00
Mike Fährmann
8fe9056b16
implement string slicing for format strings 
It is now possible to slice string (or list) values of format string
replacement fields with the same syntax as in regular Python code.

"{digits}"       -> "0123456789"
"{digits[2:-2]}" -> "234567"
"{digits[:5]}"   -> "01234"

The optional third parameter (step) has been left out to simplify things.
 2018-07-14 09:53:15 +02:00
Mike Fährmann
269dc2bbd5
[sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
764331823b
release version 1.4.2 2018-07-06 16:02:40 +02:00
Mike Fährmann
2eefaa99a3
[mangapark] support .net and .com mirrors 2018-07-05 14:45:05 +02:00
Mike Fährmann
188e956c4e
[imagefap] use HTTPS + update test results 2018-06-30 19:40:46 +02:00
Mike Fährmann
a699787d01
[deviantart] update URL patterns to new format 
DeviantArt changed its URL format from
https://<name>.deviantart.com/...
to
https://www.deviantart.com/<name>/...

With this change both formats will be supported.
 2018-06-28 20:21:59 +02:00
Mike Fährmann
0055fdd714
change OAuth test server 
DNS record for oauthbin.com expired
 2018-06-28 14:32:02 +02:00
Mike Fährmann
b8c97d2295
use 'extractor.request()' for more HTTP requests 2018-06-25 23:40:59 +02:00
Mike Fährmann
7a98cc9798
[smugmug] update tests 
My test account expired and all uploaded images got deleted.
 2018-06-22 15:04:31 +02:00
Mike Fährmann
4eb94aca17
[postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
a9e276bc37
reset delete-flag 
Since 'PathFormat' objects are being reused, setting `delete`
to True once caused all files downloaded after to be deleted as well.
 2018-06-20 18:12:59 +02:00
Mike Fährmann
6ac403c5d3
add postprocessor config example 2018-06-08 18:31:59 +02:00
Mike Fährmann
2403c405e3
Merge branch 'postprocessor' 2018-06-08 17:43:11 +02:00
Mike Fährmann
b344f2290f
fix downloader tests 2018-06-07 22:27:36 +02:00
Mike Fährmann
a47c6136cd
[simplyhentai] avoid redirects for all-pages.json (#89) 2018-06-01 22:06:34 +02:00
Mike Fährmann
ae9a37a528
implement text.split_html() 2018-05-27 15:00:41 +02:00
Mike Fährmann
0a1863fce3
[pixiv] respect more query parameters for user URLs 
The API endpoint responsible for user illustrations does not
provide sufficient filter capabilities* to match the actual
website, so we are spinning our own filters.

Respected parameters are
    'type': illust, manga, ugoira
    'tag' : any image tag (this was already supported)
    'p'   : the page to start on

*
- API can filter for illustrations and manga, but not for ugoira.
- 'offset' is applied before filtering
- no 'tag' filter
 2018-05-18 15:36:30 +02:00
Mike Fährmann
7f899bd5d8
Merge branch 'master' into 1.4-dev 2018-05-14 14:50:02 +02:00
Mike Fährmann
4cea886177
[imgur] allow longer album hashes 2018-05-13 11:21:51 +02:00
Mike Fährmann
e1e23165a0
[pinterest] catch JSON decode errors 2018-05-11 17:37:27 +02:00
Mike Fährmann
6a31ada9e3
re-implement OAuth1.0 code 
OAuth support for SmugMug needs some additional features
(auth-rebuild on redirect, query parameters in URL, ...)
and fixing this in the old code wouldn't work all that well.
 2018-05-10 18:47:05 +02:00
Mike Fährmann
e2157f594e
[mangadex] fix manga extraction (closes #84) 
Chapter listings for manga now use
https://mangadex.org/manga/<id>/_/chapters/2/
as URL instead of
https://mangadex.org/manga/<id>/_//2/
 2018-05-06 17:43:50 +02:00
Mike Fährmann
69a5e6ddb3
Merge branch 'master' into 1.4-dev 2018-05-04 10:19:02 +02:00
Mike Fährmann
3fe653d940
fix test_results for empty sets 
{} is an empty dict and doesn't support set operations
 2018-04-29 22:43:37 +02:00
Mike Fährmann
d96b3474e5
[puremashiro] remove module 
site has been unreachable for a couple of weeks
and now the DNS record is gone as well
 2018-04-28 14:24:20 +02:00
Mike Fährmann
b44a296404
[gomanga] remove module 
site has been unreachable for a couple of weeks
and the cloudflare status page shows host errors
 2018-04-28 14:24:21 +02:00
Mike Fährmann
2395d870dd
[pinterest] unquote board and user names, better errors 2018-04-26 16:38:12 +02:00
Mike Fährmann
55d4d23860
[pinterest] use Pinterest's "Web" API (#83) 
no access tokens, no user credentials of any kind ...
 2018-04-24 22:28:10 +02:00
Mike Fährmann
f471161920
Merge branch 'master' into 1.4-dev 2018-04-21 12:15:40 +02:00
Mike Fährmann
cc36f88586
rename safe_int to parse_int; move parse_* to text module 2018-04-20 14:53:21 +02:00
Mike Fährmann
10cc59f3b5
fix extractor names 2018-04-18 18:12:57 +02:00
Mike Fährmann
b1325d4d2c
fix extractor docstrings 2018-04-18 18:03:43 +02:00
Mike Fährmann
df7e18399e
[luscious] fix image order 2018-04-17 17:32:21 +02:00
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions 
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
 2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest (#83) 
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
 2018-04-16 20:03:28 +02:00
Mike Fährmann
dbe250f7e5
[pinterest] update access_token (#83) 2018-04-16 09:46:45 +02:00