1
0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-24 11:42:33 +01:00
Commit Graph

33 Commits

Author SHA1 Message Date
Mike Fährmann
dbf841ebd1
prevent unhandled exception on Cloudflare challenges (#868)
The relatively new v2 challenges aren't supported (*), but retrying
often enough may yield a v1 challenge which can be solved.

(*) and probably never will. They are far too complicated to do without
a real browser.
2020-07-08 23:22:33 +02:00
Mike Fährmann
1489712325
resolve redirects after solving Cloudflare challanges 2020-06-10 21:14:57 +02:00
Mike Fährmann
b0b1feaa67
request 'transparent.gif' when solving Cloudflare challenges
This currently also works without, but they might be using these to
detect potential bots in the future.
2020-06-10 21:04:33 +02:00
Mike Fährmann
d17e9628b3
fix/improve Cloudflare bypass code (#728, #757)
- support changing values for 'k'
- use XML parser to get request parameters
  (some input fields are now embedded in an HTML comment)
2020-05-18 19:04:50 +02:00
Mike Fährmann
714566b6e3
fix/improve Cloudflare bypass code 2020-05-01 23:35:43 +02:00
Mike Fährmann
5d7ca76885
retry Cloudflare challenges 2020-04-24 22:47:27 +02:00
Mike Fährmann
8fe858eb0e
improve parameter extraction when solving Cloudflare challenge 2020-04-22 22:08:17 +02:00
Mike Fährmann
c76c8b765a
[cloudflare] unescape challenge URL 2020-02-09 18:51:29 +01:00
Mike Fährmann
f9e74320de
retain trailing zeroes in Cloudflare challenge answers 2019-12-16 15:32:49 +01:00
Mike Fährmann
e2710702d4
fix Cloudflare bypss 2019-12-01 01:07:24 +01:00
Mike Fährmann
76df628b13
rewrite invalid cloudflare redirect locations
After solving a challenge on komikcast.com, cloudflare would redirect to
https:/komikcast.com (with only one '/') when testing on TravisCI.
2019-04-27 16:22:42 +02:00
Mike Fährmann
49a6522c38
ensure consistent headers and params ordering
Necessary to avoid being labeled a bot and getting a CAPTCHA response
after solving a Cloudflare challenge.
2019-04-09 10:52:27 +02:00
Mike Fährmann
9ebd29fcc1
update cloudflare bypass (wip)
This commit adds support for the two new JS expressions embedded in the
overall challenge code.

It does compute the correct 'js_answer' value, but the HTTP request to
/cdn-cgi/l/chk_jschl to get the 'cf_clearance' cookie always results in
a 403 response with a CAPTCHA inside (hence 'wip')

All steps to make this HTTP request indistinguishable from a regular web
browser (which passes the test) show no effect. This includes:
- using the exact same HTTP headers as a web browser
- follow query argument order
- different wait times
2019-04-01 15:14:59 +02:00
Mike Fährmann
5476404a5c
update and fix Cloudflare bypass 2019-03-25 22:53:36 +01:00
Mike Fährmann
f612284d24
cache cfclearance cookies 2019-03-14 16:14:29 +01:00
Mike Fährmann
6dae6bee37
automatically detect and bypass cloudflare challenge pages
TODO: cache and re-apply cfclearance cookies
2019-03-10 15:31:33 +01:00
Mike Fährmann
0328a04a65
[cloudflare] don't output the whole challenge page
thanks to the embedded animated gifs this is just a bit too much
2018-12-30 13:09:23 +01:00
Mike Fährmann
95392554ee
use text.urljoin() 2018-04-26 17:00:26 +02:00
Mike Fährmann
ff643793bd
improve and document cloudflare bypass code 2018-04-19 21:32:10 +02:00
Mike Fährmann
d11fcf4804
smaller changes and fixes
- fix the cloudflare challenge result if the last decimal places
  are zero (JS`s toFixed() removes trailing zeroes)
- fix downloading of kissmanga chapter-pages hosted on blogspot
  (accessing blogspot with "kissmanga.com" as referrer yields a 401)
- disable certificate validation for 'mangahere' tests
- update flickr test result
2018-04-06 15:30:09 +02:00
Mike Fährmann
f6c95dccf9
[cloudflare] fix bypass procedure
Cloudflare challenges, at least for kissmanga and readcomiconline,
now use slightly different Javascript expressions.

Instead of a single value per expression, they now have a numerator
and a denominator of a fractional value, which in the end gets
truncated to 10 decimal places.
2018-04-05 20:28:04 +02:00
Mike Fährmann
67d4462d26
[batoto] rudimentary Cloudflare bypass 2018-01-11 18:49:19 +01:00
Mike Fährmann
cf79a47b59
update unit tests 2017-04-08 11:02:32 +02:00
Mike Fährmann
cdecd0b37b
add notification when solving cloudflare challenge 2017-03-25 12:23:30 +01:00
Mike Fährmann
4f123b8513
code adjustments according to pep8 2017-01-30 19:40:15 +01:00
Mike Fährmann
40dbea7ed2
rewrite parts of the cloudflare bypass system 2016-12-16 13:28:36 +01:00
Mike Fährmann
9e3788175e
implement decorator for cloudflare bypass
this method for enabling and caching a cloudflare bypass for a
requests.session object allows for different cache-timeouts for
different domains
2016-11-20 18:05:49 +01:00
Mike Fährmann
b634ace39e
[readcomiconline] add comic-issue and comic extractor 2016-11-14 18:29:45 +01:00
Mike Fährmann
3c94d85409
fix cloudflare bypass 2016-07-12 12:03:25 +02:00
Mike Fährmann
8accf4b790
use cache-decorator for cloudflare bypass 2016-03-07 16:34:42 +01:00
Mike Fährmann
e78937a580
cache cookies from cloudflare-bypass 2015-11-27 21:32:07 +01:00
Mike Fährmann
079330785a
document and fix smaller bugs 2015-11-07 13:06:23 +01:00
Mike Fährmann
822527a2ce
module to bypass cloudflare protection 2015-11-07 02:30:08 +01:00