Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-22 10:42:34 +01:00
Code Issues Releases Wiki Activity
6,062 Commits 6 Branches 164 Tags 15 MiB
b2fa149598
Commit Graph

384 Commits

Author SHA1 Message Date
Mike Fährmann
559462789d
add some tests for job.py 2021-05-14 19:44:16 +02:00
Mike Fährmann
c6c4a73f87
update fanbox entry in supportedsites.md 2021-04-25 19:44:19 +02:00
thatfuckingbird
e47952ac14
add extractors for fantia and fanbox (#1459) 
* add extractors for fantia and fanbox

* appease linter

* make docstrings unique

* [fantia] refactor post extraction

* [fantia] capitalize

* [fantia] improve regex pattern

* code style

* capitalize

* [fanbox] use BASE_PATTERN for url regexes

* [fanbox] refactor metadata and post extraction

* [fanbox] improve url base pattern

* [fanbox] accept creator page links ending with /posts

* [fanbox] more tests

* [fantia] improved pagination

* [fanbox] misc. code logic improvements

* [fantia] finish restructuring pagination code

* [fanbox] avoid making a request for each individual post when processing a creator page

* [fanbox] support embedded videos

* [fanbox] fix errors

* [fanbox] document extractor.fanbox.videos

* [fanbox] handle "article" and "entry" post types, all embeds

* [fanbox] fix downloading of embedded fanbox posts
 2021-04-25 19:39:13 +02:00
Mike Fährmann
77a9cc6fd6
update supportedsites.md entry for Instagram 2021-04-23 23:21:01 +02:00
Hans Christian Gunawan
334d690687
[hentaicosplays] Add extractor (#1473) 2021-04-18 20:28:00 +02:00
Mike Fährmann
086925e685
[shopify] support omgmiamiswimwear.com (closes #1280) 2021-04-13 23:54:03 +02:00
Mike Fährmann
78d7ee3ef4
[yuki] remove module for yuki.la 2021-04-12 21:42:32 +02:00
Mike Fährmann
d5aad999dc
[tapas] implement login with username & password (#692) 2021-03-30 01:45:28 +02:00
Mike Fährmann
62cfee4d28
[vk] initial support for albums (#474) 2021-03-23 19:02:16 +01:00
Mike Fährmann
91c2e15da9
[deviantart] add support for posts from watched users (#794) 2021-03-22 19:25:04 +01:00
Mike Fährmann
69ca4e29f1
[deviantart] add 'watch' extractor (#794) 2021-03-17 22:50:02 +01:00
Mike Fährmann
c485d0a956
[philomena] add generalized extractors for philomena sites 
(closes #1379)
 2021-03-14 17:19:57 +01:00
Mike Fährmann
72fe9ac0f3
[gelbooru_v01] support some more boorus by default 
- https://drawfriends.booru.org/
- https://vidyart.booru.org/
- https://tlb.booru.org/
 2021-03-14 17:19:56 +01:00
Mike Fährmann
ec98b2c56f
categorize sites in supportedsites.md by basecategory 2021-03-12 16:53:36 +01:00
Mike Fährmann
a67e002f40
update docs/supportedsites 
- use Markdown with inline HTML instead of reStructuredText
- move file from docs/supportedsites.rst to docs/supportedsites.md
- update Makefile, README, etc
 2021-03-12 03:03:48 +01:00
Mike Fährmann
4be27ff0fe
[nozomi] support '/index-N.html' URLs (closes #1365) 
and '/index-Popular-N.html'
 2021-03-11 01:06:47 +01:00
Seonghyeon Cho
665499924d
Support naver webtoon (#1331) 
* Support naver webtoon (WIP)

* Apply patch

* Change filename format

* Fill test results

* Fill test result
 2021-03-03 15:21:13 +01:00
topozorra
a9119da4d4
support tumblrgallery.xyz (#1298) 
* support `tumblrgallery.xyz`

* fix format issues

* Refactor and add post and search page support

* Fix warnings

* Few improvments

* Better file names

* Fix linting errors

* move id closer to the begining of the file name

Co-authored-by: topozorra <none>
 2021-03-03 15:20:47 +01:00
Mike Fährmann
e165e6c265
[wallhaven] add 'collections' extractor (#1351) 2021-03-02 01:32:26 +01:00
Mike Fährmann
1a38fae785
add option to use different youtube-dl modules (fixes #1330) 
by setting the 'downloader.ytdl.module' value. For example

{
    "downloader": {
        "ytdl": {
            "module": "yt_dlp"
        }
    }
}

or '-o module=yt_dlp'
 2021-03-01 03:10:42 +01:00
Mike Fährmann
e39aea42cd
fix supportedsites.py for modules without docstring 
(fixes #1332)
 2021-02-21 22:57:37 +01:00
Mike Fährmann
03b219db9d
fix supportedsites.py 2021-02-20 22:31:21 +01:00
Mike Fährmann
8974f0361c
[pixiv] update (#1304) 
- remove login with username & password
- require a refresh token
- add 'oauth:pixiv' functionality

See also:
- https://github.com/upbit/pixivpy/issues/158
- https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362
 2021-02-12 18:07:16 +01:00
Mike Fährmann
ae530f6365
[erome] add extractors for albums, users, searches (closes #409) 2021-02-07 22:58:19 +01:00
Mike Fährmann
e1bdd2ab67
add a 'generated by …' comment to supportedsites.rst 2021-01-23 17:24:02 +01:00
Mike Fährmann
0eea424a52
update supportedsites.rst 2021-01-22 19:41:50 +01:00
Mike Fährmann
fa8ee6eac4
[derpibooru] add search and gallery extractors (#862) 2021-01-07 18:05:32 +01:00
Mike Fährmann
b0beed7a06
[sankaku] add support for book searches (closes #1204) 2020-12-29 17:36:37 +01:00
Mike Fährmann
6cdbab07b5
[pinterest] add support for getting all boards of a user 
(#1205)
 2020-12-29 16:57:03 +01:00
Mike Fährmann
df9e40ab35
small updates to release.sh 
- use git push --atomic
- update VM name
- rename (git|pypi)-upload
 2020-12-26 19:01:35 +01:00
Mike Fährmann
bf629a2818
[instagram] add 'include' option (closes #1180) 
Split the functionality of the old 'user' extractor into separate
'posts' and 'highlights' extractors, which respond to virtual URLs
('/<user>/posts' and '/<user>/highlights')
 2020-12-21 02:06:28 +01:00
Mike Fährmann
465015f75a
[sankaku] reimplement login support (#1176, #1182) 2020-12-17 16:12:59 +01:00
Mike Fährmann
b2c55f0a72
[sankaku] remove login support 
The old login method for 'https://chan.sankakucomplex.com/user/login'
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
 2020-12-08 21:05:47 +01:00
Mike Fährmann
de0c57886d
[twitter] add 'list-members' extractor (closes #1096) 2020-11-13 06:47:45 +01:00
Mike Fährmann
05d7009cc6
rename 'Authentication' entries in supportedsites.rst 
- change 'Optional' to 'Supported'
- use 'OAuth' and 'Cookies' in their own
- add link to weasyl API key option
 2020-10-19 20:16:17 +02:00
Zanny
3ec60e894a
[weasyl] api-key authentication (#1057) 
* [weasyl] support api keys

* [weasyl] document api-key authentication

* [weasyl] usernames can contain ~
 2020-10-15 15:12:09 +02:00
Mike Fährmann
b9bdd2c564
[hentaifoundry] add support for stories (closes #734) 2020-09-27 02:27:40 +02:00
Mike Fährmann
117582d798
use definition lists instead of tables in docs/configuration.rst 
and strip some clutter from generated man pages
 2020-09-26 13:33:46 +02:00
Mike Fährmann
e9cc719497
[weasyl] update and simplify 
- simplify 'pattern' regexps
- parse 'posted_at' as 'date'
- use unaltered 'title' ({title!l:R /_/} to lowercase and replace spaces)
 2020-09-26 02:10:45 +02:00
Mike Fährmann
c43b3894be
[myhentaigallery] update and fix extraction (#1001) 
- extract more metadata
- match "/show/" URLs
- complete test results
- fix missing images for lines starting with " <img"
- fix missing comma in supportedsites.py
 2020-09-17 18:14:23 +02:00
choeronline
05b9ac8d37
[myhentaigallery] add extractor (#1001) 
* adds support for myhentaigallery

* fixes linting issues in myhentaigallery extractor
 2020-09-17 17:32:54 +02:00
ArtaxIsSleeping
0e941553ec
[aryion] Add username/password support (#960) 
* Add username/password support to aryion extractor

* Update docs to match

* Fix code style
 2020-08-27 22:45:30 +02:00
Mike Fährmann
901a897e6f
[inkbunny] mention login support 2020-07-26 23:52:52 +02:00
Mike Fährmann
d5fcffcced
[subscribestar] add login capabilities (#852) 2020-07-17 22:18:01 +02:00
Mike Fährmann
df7217c36d
update supportedsites.py (#889, #893) 
- mention optional auth access for more sites
- link to OAuth and Cookies sections in README
 2020-07-16 15:00:31 +02:00
jakem72360
7dfdcc3fbf
[mangakakalot] Added extractors for MangaKakalot (#876) 2020-07-13 21:20:09 +02:00
Mike Fährmann
74494b43d3
let zsh completion immediately suggest cmdline options 
instead of expecting an URL and trying to complete it.
 2020-07-05 22:36:00 +02:00
Mike Fährmann
d0cd86e0d5
add zsh completion script (#150) 2020-07-04 18:05:31 +02:00
Mike Fährmann
e62ebb4643
update CHANGELOG before building sdist and wheel packages 2020-06-27 19:45:09 +02:00
Mike Fährmann
0f459f340b
[instagram] fix and re-enable login with username&password 
This reverts commit 3e0848a482.
(#756, #771, #797, #803)

https://github.com/althonos/InsaLooter/issues/287#issuecomment-630456522
 2020-05-31 00:29:09 +02:00
Mike Fährmann
3e0848a482
[instagram] disable login with username&password (#756) 2020-05-29 23:29:40 +02:00
Leonardo Taccari
39cd389679
[webtoons] Add a new extractor for webtoons.com (#761) 
The webtoons extractor can extract episode and entire comic (all
episodes) from webtoons.com.

All the logic of the extractors should be trivial except for a couple
of kludges needed:

 - `ageGatePass' cookie is always set to avoid possible redirect and stop of
    extraction, especially in the comic extractor
 - The image URLs returned by the episode extractor could not be fetched
   directly and the `Referer:' HTTP header needs to be passed to fetch them

Close #593.
 2020-05-18 19:04:20 +02:00
Bepis
7b5711ee04
[imagechest] Add new extractor for ImageChest (#750) 
* [imagechest] Add new extractor for ImageChest

* [imagechest] Fix flake8 compliance issues
 2020-05-18 19:02:56 +02:00
Mike Fährmann
f557cac074
[redgifs] add image extractor (#724) 2020-05-10 00:31:42 +02:00
Leonardo Taccari
b47cfc5ac9
[speakerdeck] Add a new extractor for speakerdeck.com (#726) 2020-05-01 22:32:22 +02:00
Mike Fährmann
6143050980
[aryion] add gallery and post extractors (#390, #673) 2020-04-08 21:52:51 +02:00
Mike Fährmann
2587296deb
[mastodon] add access tokens for mastodon.social and baraag.net 
(closes #665)
 2020-04-02 22:34:32 +02:00
Mike Fährmann
c8db2a87e9
fix create_test_data.py script 2020-03-25 22:06:38 +01:00
Mike Fährmann
88ebbadc58
remove dashes from subcategory names in supportedsites.rst 2020-03-24 02:49:00 +01:00
Mike Fährmann
90d15e3682
[instagram] use 'itertools.chain()' 2020-03-17 15:52:44 +01:00
Mike Fährmann
5b676ea59d
[e621] document username & password support (#640) 2020-03-14 01:15:22 +01:00
Mike Fährmann
1b3ba86110
improve lists in man pages 2020-03-14 01:08:31 +01:00
Mike Fährmann
fdd2dd5136
[kabeuchi] add 'user' extractor (closes #561) 2020-03-13 16:45:42 +01:00
Mike Fährmann
402025c3c3
fix some build issues 
- use 'os.name' to decide between Windows/Linux build
- don't check Windows executable version number,
  since Wine fails to run the executable and causes release.sh to stop
 2020-03-06 15:15:59 +01:00
Mike Fährmann
ec85bf90de
use context managers in cache.py & add tests 2020-02-25 23:08:47 +01:00
Mike Fährmann
765b2a0527
[hentaihand] add extractors (closes #605) 2020-02-19 21:55:47 +01:00
Mike Fährmann
138135c190
[furaffinity] add extractors (#284) 2020-02-11 19:51:24 +01:00
Mike Fährmann
719b63d0ca
[bcy] add user and post extractors (#592) 2020-02-09 02:37:14 +01:00
Mike Fährmann
cf7a67d67f
[yaplog] remove module 
Yaplog! ended its service on 2020-01-31
 2020-01-31 12:56:54 +01:00
Mike Fährmann
a6b98d3bec
fix auto-generation of supportedsites.rst 2020-01-14 09:48:32 +01:00
Mike Fährmann
5cac79c3d9
[erolord] remove extractor 2020-01-10 14:40:30 +01:00
Mike Fährmann
173a93454e
fix config.set calls in build_testresults_db.py 2019-12-03 16:29:09 +01:00
Mike Fährmann
d4d8c1c01f
update links to standalone executables in README.rst 2019-11-26 12:05:31 +01:00
Mike Fährmann
3ece3976ae
[newgrounds] implement login support (#394) 2019-11-16 23:45:32 +01:00
Mike Fährmann
ce98a86c0e
fix data file inclusion in source distributions 2019-11-09 00:47:13 +01:00
Mike Fährmann
4f39d2ae64
install bash completion into share/bash-completion/completions 2019-11-07 23:23:48 +01:00
Mike Fährmann
8f38a35b91
[imgur] use API with "public" client_id (#446) 
Using the API endpoints makes it possible to access NSFW content
without logging in.
 2019-10-23 21:43:55 +02:00
Mike Fährmann
b23c822b23
[luscious] use GraphQL 2019-10-22 21:17:08 +02:00
Mike Fährmann
5882b00f2f
[imgur] implement login support (#446) 2019-10-15 22:00:22 +02:00
Mike Fährmann
6779512fc7
[nozomi] add post and tag extractors (#388) 2019-10-13 22:16:03 +02:00
Mike Fährmann
5a1a0f5325
change text representation of user extractors to "User Profiles" 2019-09-22 22:21:48 +02:00
Mike Fährmann
6fe9a134bf
[lineblog] add blog and post extractors (closes #404) 2019-09-06 22:16:42 +02:00
Mike Fährmann
dd72ae7164
add postprocessor tests 2019-08-12 21:46:30 +02:00
Mike Fährmann
f687052daf
have e-hentai and exhentai on supportedsites.rst (#365) 2019-08-03 11:42:28 +02:00
Mike Fährmann
2c839f3760
[imgbb] add user extractor + login support (#361) 2019-08-01 21:39:20 +02:00
Mike Fährmann
2153206093
[imgbb] add album extractor (#361) 2019-07-30 23:11:19 +02:00
Mark Henrick
923e1bb714 [docs] Fix inconsistency about which sites have optional authentication (#359) 
* [docs] Fix inconsistency about which sites have optional authentication

* update authentication docs
 2019-07-29 18:22:31 +02:00
Mike Fährmann
dc73d02d87
[exhentai] always use e-hentai.org as domain + set nw cookie 2019-07-28 10:54:17 +02:00
Mike Fährmann
60cf40380a
[vsco] add user extractor (#331) 2019-07-23 16:23:11 +02:00
Mike Fährmann
3fe5ccdfa6
[adultempire] add gallery extractor (closes #340) 2019-07-21 22:29:57 +02:00
Mike Fährmann
fb875d1ab8
add warning about NSFW sites in supportedsites.rst (#335) 2019-07-15 21:44:34 +02:00
Mike Fährmann
2ff73873f0
[erolord] add gallery extractor (closes #326) 2019-07-04 20:28:04 +02:00
Mike Fährmann
40da44b17f
Merge branch 'v1.9.0' 2019-06-29 15:39:52 +02:00
Mike Fährmann
a9c89085fb
[instagram] implement login support (#195) 2019-06-26 23:58:47 +02:00
Mike Fährmann
18a1f8c6cd
[vanillarock] add post and tag extractors (closes #254) 2019-06-23 22:45:36 +02:00
Mike Fährmann
f0c5093812
[nsfwalbum] add album extractor (closes #287) 2019-06-23 22:45:07 +02:00
Mike Fährmann
2184e3a86b
[slickpic] add album extractor (#249) 2019-06-09 21:59:22 +02:00
Mike Fährmann
096009367b
[xhamster] add gallery & user extractor (#281) 2019-06-05 11:11:51 +02:00
Mike Fährmann
d33f5a7423
[wallhaven] rewrite 
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
  IDs that can't be translated to the new ID system
- support direct links to wallpapers
 2019-05-31 14:53:02 +02:00
Mike Fährmann
74c2415138
[sankakucomplex] move article extractor to its own module (#258) 2019-05-27 23:49:23 +02:00
Mike Fährmann
a247c94c34
[sexcom] add pin and board extractors (#147) 2019-04-24 22:09:19 +02:00
Mike Fährmann
166a721c19
include PyInstaller executable in release script 2019-04-17 12:44:45 +02:00
Mike Fährmann
18345381f3
add PyInstaller script and hook (#166) 2019-04-17 12:43:29 +02:00
Mike Fährmann
96c7ccd380
update/cleanup Python dev scripts 
- put common code in its own util.py file
- same Python3 shebang for all scripts
- add file docstrings
- fix format string replacement fields in man page template
 2019-04-16 21:21:34 +02:00
Mike Fährmann
d9b94a585d
[mangoxo] add login support (#184) 
A very recent change: It is now only possible to see more
than the first 5 images of an album if you are logged in.
 2019-04-10 18:55:25 +02:00
Mike Fährmann
e730fc9045
[twitter] add login support (#214) 2019-04-09 09:27:49 +02:00
Mike Fährmann
35919a9bb8
[livedoor] add blog- and post-extractors (#190) 2019-04-06 16:27:48 +02:00
Mike Fährmann
2ff043edfa
[yaplog] add user- and post-extractors (#190) 2019-04-04 17:56:56 +02:00
Mike Fährmann
c70b21248d
[wikiart] add extractors (#179) 
for
- artists:          https://www.wikiart.org/en/thomas-cole
- artist-listings:  https://www.wikiart.org/en/artists-by-century/12
- artwork-listings: https://www.wikiart.org/en/paintings-by-media/grisaille
 2019-04-02 17:34:57 +02:00
Mike Fährmann
089923e3dd
parse configuration.rst to build gallery-dl.conf.5 (#150) 
… a man-page containing all of gallery-dl's configuration file options.

This implementation relies on Python dicts preserving their insertion
order. Python 3.4 and 3.5 need to use OrderedDict or they produce
randomly ordered man-page sections.

The man-page formatting is a bit rough around the edges, but it works
for the most part. The only real "problem" are inline-links, but it's
better if they are left in there.
 2019-03-28 16:20:52 +01:00
Mike Fährmann
e5f44a5160
add Makefile; include manpage&completion in setup.py (#150) 2019-03-24 11:03:02 +01:00
Mike Fährmann
e47a24afc7
script to generate a simple man page (#150) 2019-03-24 11:03:01 +01:00
Mike Fährmann
b0f88c2ab5
script to generate a simple bash completion file (#150) 2019-03-23 09:50:39 +01:00
Mike Fährmann
6277a739e4
[35photo] add user-, genre-, and image-extractors (#162) 2019-03-18 01:11:30 +01:00
Mike Fährmann
9c0e2f294b
[shopify] add generic collection and product extractors (#175) 
with fashionnova.com  as a default domain
 2019-03-05 22:33:37 +01:00
Mike Fährmann
ea6fbfdd39
simplify build_supportedsites.py 2019-02-21 14:29:19 +01:00
Mike Fährmann
ee2ad76620
update and improve create_test_data.py script 2019-02-20 15:48:31 +01:00
Mike Fährmann
abbd45d0f4
update handling of extractor URL patterns 
When loading extractor classes during 'extractor.find(…)', their
'pattern' attribute will be replaced with a compiled version of itself.
 2019-02-08 20:08:16 +01:00
Mike Fährmann
53c2fd4664
add mastodon/foolslide/foolfuuka examples to example config 2019-02-05 16:17:25 +01:00
Mike Fährmann
373cb07b28
update .travis.yml and run_tests.sh 
- add python3.8 and pypy3 builds
- remove deprecated 'sudo: true' and 'sudo: false'
- enable builds for 'test-...' branches
 2019-01-31 15:58:52 +01:00
Mike Fährmann
bfbbac4495
[tsumino] add login capabilities (#161) 2019-01-30 17:58:48 +01:00
Mike Fährmann
0156189468
[hentaifox] add chapter extractor (#160) 2019-01-28 18:00:32 +01:00
Mike Fährmann
e4171d6baf
[luscious] add login capabilities (closes #159) 2019-01-28 17:14:15 +01:00
Mike Fährmann
e53cdfd6a8
update build_supportedsites.py 2019-01-09 14:58:35 +01:00
Mike Fährmann
6c71e9cf5d
[deviantart] add separate 'sta.sh' extractor (#113) 
- supports multiple stashed deviations per page
- explicitly mentions sta.sh support on supportedsites.rst
 2018-12-26 18:56:57 +01:00
Mike Fährmann
fb53b5dd55
fix control+c during -j and range tests 2018-11-25 18:54:05 +01:00
Mike Fährmann
3c25fa2dad
update build_testresult_db.py script 2018-11-15 22:58:14 +01:00
Mike Fährmann
2cf3f53839
[yuki] add thread extractor (closes #111) 2018-09-28 12:47:32 +02:00
Mike Fährmann
17611bfec0
update build_supportedsites.py script 2018-09-28 12:43:19 +02:00
Mike Fährmann
c402cc4047
[hentaifoundry] add 'popular' and 'recent' extractors 
for "Popular Pictures" and "Recent Pictures" listings
 2018-09-24 13:11:18 +02:00
Mike Fährmann
6ea9a78588
[wallhaven] add login capabilities 
Being logged in is required to access NSFW wallpapers.
 2018-09-19 21:04:01 +02:00
Mike Fährmann
69fd61ea86
[bobx] add gallery and model extractors 2018-09-13 20:13:12 +02:00
Mike Fährmann
3ecea4cf36
[hentaicafe] add chapter and manga extractors (#101) 2018-09-05 21:08:40 +02:00
Mike Fährmann
75862715ac
[behance] add user extractor 2018-08-31 17:42:09 +02:00
Mike Fährmann
e9dd2eff1d
[twitter] add extractor for media-tweet timelines (#96) 
For example "https://twitter.com/PicturesEarth/media".
They are different from normal timelines in that they do not contain
any (re)tweets from other users and feature all media the user ever
posted, including responses to other tweets.
 2018-08-19 20:46:12 +02:00
Mike Fährmann
63fa0b2006
[pinterest] add extractors for related pins 
Related pins can not be accessed by adding a "#related" fragment
to the end of a Pinterest URL, for example:
- https://www.pinterest.com/pin/858146903966145189/#related
- https://www.pinterest.com/g1952849/test-/#related

There are no explicit real URLs for related pins,
using an option to enable them results in "clunky" code,
and a custom "related:<URL>" scheme doesn't feel right either.
 2018-08-15 21:49:45 +02:00
Mike Fährmann
84854fcad7
[myportfolio] add user and gallery extractors (#95) 2018-07-19 18:56:45 +02:00
Mike Fährmann
a6fe2bb594
[whatisthisimnotgoodwithcomputers] remove extractor 2018-07-14 09:53:16 +02:00
Mike Fährmann
55b0913412
[simplyhentai] add gallery extractor (#89) 2018-05-27 15:25:04 +02:00
Mike Fährmann
f3483a2b7c
[smugmug] add OAuth support 2018-05-10 18:58:05 +02:00
Mike Fährmann
82c50fa609
release version 1.3.5 2018-05-04 10:03:20 +02:00
Mike Fährmann
16e014baaa
[smugmug] added image and album extractor 
just some initial code that still requires a lot of work ...

TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth

It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
 2018-04-29 21:27:25 +02:00
Mike Fährmann
d10579edb5
[pinterest] improve PinterestAPI code; remove OAuth mentions 
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
 2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107
[pinterest] implement oauth:pinterest (#83) 
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
 2018-04-16 20:03:28 +02:00
Mike Fährmann
7b562907c3
[nijie] add favorites extractor 
adds support for 'https://nijie.info/user_like_illust_view.php?id=...'
 2018-03-31 18:54:25 +02:00
Mike Fährmann
299ae24996
[test] add a few downloader tests 2018-03-25 15:10:25 +02:00
Mike Fährmann
dd314279fb
[test] add unit tests for extractor module functions 2018-03-25 11:49:42 +02:00
Mike Fährmann
d75dc71394
fix travis tests 2018-03-19 18:07:19 +01:00
Mike Fährmann
8ef790de12
update .travis.yml 
- restrict builds to master branch and release tags
- implement 'core' and 'results' test categories
 2018-03-19 17:57:32 +01:00
Mike Fährmann
7121eeae8b
check supportedsites.rst in release script 2018-03-17 15:35:38 +01:00
Mike Fährmann
1400868f53
[mangadex] general improvements 
- support >100 chapter entries per manga
- custom archive ID format
- detect non-existing chapters
 2018-03-06 14:15:15 +01:00
Mike Fährmann
40d35c87bc
[paheal] add tag- and post-extractors (closes #69) 2018-01-15 16:39:05 +01:00
Mike Fährmann
35e09869d1
[mangapark] fix image URLs and use HTTPS 2018-01-12 14:59:49 +01:00
Mike Fährmann
29d75fc3fa
[tumblr] add support for OAuth authentication (#65) 2018-01-11 14:11:37 +01:00
Mike Fährmann
b33efc99a4
[idolcomplex] add support for idol.sankakucomplex.com 2018-01-09 17:54:37 +01:00
Mike Fährmann
5b094328b5
[puremashiro] add chapter- and manga-extractor (closes #66) 
Also adds support for region subtags in language codes (e.g. en-us)
 2018-01-07 21:50:43 +01:00
Mike Fährmann
fc7d165c97
[deviantart] add support for OAuth2 authentication 
Some user galleries [*] require you to be either logged in or
authenticated via OAuth2 to access their deviations.

[*] e.g. https://polinaegorussia.deviantart.com/gallery/

--------------

known issue:
A deviantart 'refresh_token' can only be used once and gets updated
whenever it is used to request a new 'access_token', so storing its
initial value in a config file and reusing it again and again is not
possible.
 2017-12-18 01:16:46 +01:00
Mike Fährmann
0a9a07a6e1
[slideshare] improve metadata; flake8 
- added 'views' and 'published' keywords
- fixed longer titles and descriptions
 2017-12-13 21:16:49 +01:00
Mike Fährmann
2c1adda784
update release.sh script 
- update CHANGELOG on new releases
  - change issue references to actual links
  - replace "Unreleased" with new version and date
- fix filenames of old Windows executables

[no ci]
 2017-11-08 17:47:52 +01:00
Mike Fährmann
2d0cfb33e1
[xvideos] add user profile extractor (#45) 2017-11-02 17:28:35 +01:00
Mike Fährmann
035ef655f1
[imagefap] update unit tests 
old gallery/image has been deleted
 2017-10-27 12:22:16 +02:00
Mike Fährmann
6af921a952
[sankaku] rewrite/improve (fixes #44) 
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
 2017-10-14 23:01:33 +02:00
Mike Fährmann
0386503c80
fix (sub)category-transfer for DownloadJob instances (#41) 
... and extend "parent" parameters to TestJob- and DataJob-classes
as well.
 2017-10-06 15:38:35 +02:00
Mike Fährmann
cebf800a7f
[foolfuuka] add support for more sites (#18) 
- https://arch.b4k.co
- https://archive.whatisthisimnotgoodwithcomputers.com
- https://archive.yeet.net

Notes:
- The name "whatisthisimnotgoodwithcomputers" is way too long ...
- archive.yeet.net is out of date and also blocked by 4chan servers
  - newest threads are 2 weeks old
  - using "https://archive.yeet.net" as Referer header results in
    "403 Forbidden" when accessing 4chan
 2017-09-16 21:36:16 +02:00
Mike Fährmann
18e6ed1c7e
[booru] add extractors for "Popular" images 2017-08-24 21:29:22 +02:00
Mike Fährmann
d7cb3c668a
update supportedsites.rst 2017-07-24 10:50:40 +02:00
Mike Fährmann
74a2b5af06
update supportedsites.rst 2017-07-15 15:01:30 +02:00
Mike Fährmann
1dac76fd1c
update extractor docstrings 2017-06-28 17:39:07 +02:00
Mike Fährmann
92a11528d1
smaller changes 2017-06-28 09:42:49 +02:00
Mike Fährmann
2993206c4b
smaller fixes and "security" measures 
- move the OAuthSession class into util.py
- block special extractors for reddit and recursive
- ignore 'only matching' tests for testresults script
 2017-06-16 21:01:40 +02:00
Mike Fährmann
fb1904dd59
add 'Authentication' column to supportedsites.rst 2017-06-15 21:06:20 +02:00
Mike Fährmann
c921b4f32a
code cleanup and fixing tests 2017-06-02 09:10:58 +02:00
Mike Fährmann
659c65dbb0
[flickr] add image extractor 2017-05-30 17:43:02 +02:00
Mike Fährmann
f79320e35b
fix tests 2017-05-27 11:47:15 +02:00
Mike Fährmann
398506da45
update release script 2017-05-22 08:47:58 +02:00
Mike Fährmann
1e416e5227
update README 2017-05-09 15:43:55 +02:00
Mike Fährmann
4332694492
add simple imagehosts to list of supported sites 2017-04-23 17:08:45 +02:00
Mike Fährmann
9f32cf1f4e
improve supportedsites.rst and build script 2017-04-20 16:56:50 +02:00
Mike Fährmann
385285932a
add script to collect test results in JSON format 2017-04-14 14:45:35 +02:00
Mike Fährmann
e9b445a0f8
update release script 2017-03-31 17:04:44 +02:00
Mike Fährmann
2b9cf4fd83
add script to build and release new versions 2017-03-20 13:39:43 +01:00
Mike Fährmann
ed94d9b92d
fix/improve various things 2017-03-17 09:39:46 +01:00
Mike Fährmann
9538a873cb
move create_test_data script 2017-01-19 01:35:59 +01:00
Mike Fährmann
659577ae54
add a script to automatically build a list of supported sites 2017-01-15 21:31:21 +01:00