Mirrors/gallery-dl
1
0
Fork 0
mirror of https://github.com/mikf/gallery-dl.git synced 2024-11-24 03:32:33 +01:00
Code Issues Releases Wiki Activity
d3b3b30107
gallery-dl/CHANGELOG.md
Mike Fährmann 08674a91f3
[patreon] fix hash extraction from download URLs (closes #693) 
The old method was assuming every URL path ends with '/1'. For URLs
where this is not the case, the segment containing the post ID was
used as file hash.
2020-04-15 23:28:57 +02:00

62 KiB
Raw Blame History

Changelog

Unreleased

1.13.4 - 2020-04-12

Additions

  • [aryion] add gallery and post extractors (#390, #673)
  • [deviantart] detect and handle folders in sta.sh listings (#659)
  • [hentainexus] add circle, event, and title_conventional metadata fields (#661)
  • [hiperdex] add artist extractor (#606)
  • [mastodon] add access tokens for mastodon.social and baraag.net (#665)

Changes

  • [deviantart] retrieve all download URLs through the OAuth API
  • automatically read config files in PyInstaller executable directories (#682)

Fixes

  • [deviantart] handle "Request blocked" errors (#655)
  • [deviantart] improve JPEG quality replacement pattern
  • [hiperdex] fix extraction
  • [mastodon] handle API rate limits (#665)
  • [mastodon] update OAuth credentials for pawoo.net (#665)
  • [myportfolio] fix extraction of galleries without title
  • [piczel] fix extraction of single images
  • [vsco] fix collection extraction
  • [weibo] accept status URLs with non-numeric IDs (#664)

1.13.3 - 2020-03-28

Additions

  • [instagram] Add support for user's saved medias (#644)
  • [nozomi] support multiple images per post (#646)
  • [35photo] add tag extractor

Changes

  • [mangadex] transform timestamps from date fields to datetime objects

Fixes

  • [deviantart] handle decode errors for extended_fetch results (#655)
  • [e621] fix bug in API rate limiting and improve pagination (#651)
  • [instagram] update pattern for user profile URLs
  • [mangapark] fix metadata extraction
  • [nozomi] sort search results (#646)
  • [piczel] fix extraction
  • [twitter] fix typo in x-twitter-auth-type header (#625)
  • remove trailing dots from Windows directory names (#647)
  • fix crash with missing stdout/stderr/stdin handles (#653)

1.13.2 - 2020-03-14

Additions

  • [furaffinity] extract more metadata
  • [instagram] add post_shortcode metadata field (#525)
  • [kabeuchi] add extractor (#561)
  • [newgrounds] add extractor for favorited posts (#394)
  • [pixiv] implement avatar option (#595, #623)
  • [twitter] add extractor for bookmarked Tweets (#625)

Fixes

  • [bcy] reduce number of HTTP requests during data extraction
  • [e621] update to new interface (#635)
  • [exhentai] handle incomplete MIME types (#632)
  • [hitomi] improve metadata extraction
  • [mangoxo] fix login
  • [newgrounds] improve error handling when extracting post data

1.13.1 - 2020-03-01

Additions

  • [hentaihand] add extractors (#605)
  • [hiperdex] add chapter and manga extractors (#606)
  • [oauth] implement option to write DeviantArt refresh-tokens to cache (#616)
  • [downloader:http] add more MIME types for .bmp and .rar files (#621, #628)
  • warn about expired cookies

Fixes

  • [bcy] fix partial image URLs (#613)
  • [danbooru] fix Ugoira downloads and metadata
  • [deviantart] check availability of /intermediary/ URLs (#609)
  • [hitomi] follow multiple redirects & fix image URLs
  • [piczel] improve and update
  • [tumblr] replace - with in tag searches (#611)
  • [vsco] update gallery URL pattern
  • fix --verbose and --quiet command-line options

1.13.0 - 2020-02-16

Additions

  • Support for
  • [blogger] implement video extraction (#587)
  • [oauth] add option to specify port number used by local server (#604)
  • [pixiv] add rating metadata field (#595)
  • [pixiv] recognize tags at the end of new bookmark URLs
  • [reddit] add videos option
  • [weibo] use youtube-dl to download from m3u8 manifests
  • implement parent-directory option (#551)
  • extend filename formatting capabilities:
    • implement field name alternatives (#525)
    • allow multiple "special" format specifiers per replacement field (#595)
    • allow for numeric list and string indices

Changes

  • [reddit] handle reddit-hosted images and videos natively (#551)
  • [twitter] change default value for videos to true

Fixes

  • [cloudflare] unescape challenge URLs
  • [deviantart] fix video extraction from extended_fetch results
  • [hitomi] implement workaround for "broken" redirects
  • [khinsider] fix and improve metadata extraction
  • [patreon] filter duplicate files per post (#590)
  • [piczel] fix extraction
  • [pixiv] fix user IDs for bookmarks API calls (#596)
  • [sexcom] fix image URLs
  • [twitter] force old login page layout (#584, #598)
  • [vsco] skip "invalid" entities
  • improve functions to load/save cookies.txt files (#586)

Removals

  • [yaplog] remove module

1.12.3 - 2020-01-19

Additions

  • [hentaifoundry] extract more metadata (#565)
  • [twitter] add option to extract TwitPic embeds (#579)
  • implement a post-processor module to compare file versions (#530)

Fixes

  • [hitomi] update image URL generation
  • [mangadex] revert domain to mangadex.org
  • [pinterest] improve detection of invalid pin.it links
  • [pixiv] update URL patterns for user profiles and bookmarks (#568)
  • [twitter] Fix stop before real end (#573)
  • remove temp files before downloading from fallback URLs

Removals

  • [erolord] remove extractor

1.12.2 - 2020-01-05

Additions

  • [deviantart] match new search/popular URLs (#538)
  • [deviantart] match /favourites/all URLs (#555)
  • [deviantart] add extractor for followed users (#515)
  • [pixiv] support listing followed users (#515)
  • [imagefap] handle beta.imagefap.com URLs (#552)
  • [postprocessor:metadata] add directory option (#520)

Fixes

  • [artstation] fix search result pagination (#537)
  • [directlink] send Referer headers (#536)
  • [exhentai] restrict default directory name length (#545)
  • [mangadex] change domain to mangadex.cc (#559)
  • [mangahere] send isAdult cookies (#556)
  • [newgrounds] fix tags metadata extraction
  • [pixiv] retry after rate limit errors (#535)
  • [twitter] handle quoted tweets (#526)
  • [twitter] handle API rate limits (#526)
  • [twitter] fix URLs forwarded to youtube-dl (#540)
  • prevent infinite recursion when spawning new extractors (#489)
  • improve output of --list-keywords for "parent" extractors (#548)
  • provide fallback for SQLite versions with missing WITHOUT ROWID support (#553)

1.12.1 - 2019-12-22

Additions

  • [4chan] add extractor for entire boards (#510)
  • [realbooru] add extractors for pools, posts, and tag searches (#514)
  • [instagram] implement a videos option (#521)
  • [vsco] implement a videos option
  • [postprocessor:metadata] implement a bypost option for downloading the metadata of an entire post (#511)

Changes

  • [reddit] change the default value for comments to 0
  • [vsco] improve image resolutions
  • make filesystem-related errors during file downloads non-fatal (#512)

Fixes

  • [foolslide] add fallback for chapter data extraction
  • [instagram] ignore errors during post-page extraction
  • [patreon] avoid errors when fetching user info (#508)
  • [patreon] improve URL pattern for single posts
  • [reddit] fix errors with t1 submissions
  • [vsco] fix user profile extraction … again
  • [weibo] handle unavailable/deleted statuses
  • [downloader:http] improve rate limit handling
  • retain trailing zeroes in Cloudflare challenge answers

1.12.0 - 2019-12-08

Additions

  • [flickr] support 3k, 4k, 5k, and 6k photo sizes (#472)
  • [imgur] add extractor for subreddit links (#500)
  • [newgrounds] add extractors for audio listings and general media files (#394)
  • [newgrounds] implement login support (#394)
  • [postprocessor:metadata] implement a extension-format option (#477)
  • --exec-after

Changes

  • [deviantart] ensure consistent username capitalization (#455)
  • [directlink] split {path} into {path}/{filename}.{extension}
  • [twitter] update metadata fields with user/author information
  • [postprocessor:metadata] filter private entries & rename format to content-format
  • Enable cookies-update by default

Fixes

  • [2chan] fix metadata extraction
  • [behance] get images from 'media_collection' modules
  • [bobx] fix image downloads by randomly generating session cookies (#482)
  • [deviantart] revert to getting download URLs from OAuth API calls (#488)
  • [deviantart] fix URL generation from '/extended_fetch' results (#505)
  • [flickr] adjust OAuth redirect URI (#503)
  • [hentaifox] fix extraction
  • [imagefap] adapt to new image URL format
  • [imgbb] fix error in galleries without user info (#471)
  • [instagram] prevent errors with missing 'video_url' fields (#479)
  • [nijie] fix date parsing
  • [pixiv] match new search URLs (#507)
  • [plurk] fix comment pagination
  • [sexcom] send specific Referer headers when downloading videos
  • [twitter] fix infinite loops (#499)
  • [vsco] fix user profile and collection extraction (#480)
  • Fix Cloudflare DDoS protection bypass

Removals

  • --abort-on-skip

1.11.1 - 2019-11-09

Fixes

  • Fix inclusion of bash completion and man pages in source distributions

1.11.0 - 2019-11-08

Additions

Changes

  • Rewrite imgur using its public API (#446)
  • Rewrite luscious using GraphQL queries (#457)
  • Adjust default nijie filenames to match pixiv
  • Change enumeration index for gallery extractors from page to num
  • Return non-zero exit status when errors occurred
  • Forward proxy settings to youtube-dl downloader
  • Install bash completion script into share/bash-completion/completions

Fixes

  • Adapt to new instagram page layout when logged in (#391)
  • Support protected twitter videos (#452)
  • Extend hitomi URL pattern and fix gallery extraction
  • Restore OAuth2 authentication error messages
  • Miscellaneous fixes for patreon (#444), deviantart (#455), sexcom (#464), imgur (#467), simplyhentai

1.10.6 - 2019-10-11

Additions

  • --exec command-line option to specify a command to run after each file download (#421)

Changes

  • Include titles in gfycat default filenames (#434)

Fixes

  • Fetch working download URLs for deviantart (#436)
  • Various fixes and improvements for yaplog blogs (#443)
  • Fix image URL generation for hitomi galleries
  • Miscellaneous fixes for behance and xvideos

1.10.5 - 2019-09-28

Additions

  • instagram.highlights option to include highlighted stories when downloading user profiles (#329)
  • Support for /user/ URLs on reddit (#350)
  • Support for imgur user profiles and favorites (#420)
  • Additional metadata fields on nijie(#423)

Fixes

  • Improve handling of private deviantart artworks (#414) and 429 status codes (#424)
  • Prevent fatal errors when trying to open download-archive files (#417)
  • Detect and ignore unavailable videos on weibo (#427)
  • Update the scope of new reddit refresh-tokens (#428)
  • Fix inconsistencies with the reddit.comments option (#429)
  • Extend URL patterns for hentaicafe manga and pixiv artworks
  • Improve detection of unavailable albums on luscious and imgbb
  • Miscellaneous fixes for tsumino

1.10.4 - 2019-09-08

Additions

Fixes

  • Download the correct files from twitter replies (#403)
  • Prevent crash when trying to use unavailable downloader modules (#405)
  • Fix pixiv authentication (#411)
  • Improve exhentai image limit checks
  • Miscellaneous fixes for hentaicafe, simplyhentai, tumblr

1.10.3 - 2019-08-30

Additions

  • Provide filename metadata for all deviantart files (#392, #400)
  • Implement a ytdl.outtmpl option to let youtube-dl handle filenames by itself (#395)
  • Support seiga mobile URLs (#401)

Fixes

  • Extract more than the first 32 posts from piczel galleries (#396)
  • Fix filenames of archives created with --zip (#397)
  • Skip unavailable images and videos on flickr (#398)
  • Fix filesystem paths on Windows with Python 3.6 and lower (#402)

1.10.2 - 2019-08-23

Additions

  • Support for instagram stories and IGTV (#371, #373)
  • Support for individual imgbb images (#363)
  • deviantart.quality option to set the JPEG compression quality for newer images (#369)
  • enumerate option for extractor.skip (#306)
  • adjust-extensions option to control filename extension adjustments
  • path-remove option to remove control characters etc. from filesystem paths

Changes

  • Rename restrict-filenames to path-restrict
  • Adjust pixiv metadata and default filename format (#366)
    • Set filename to "{category}_{user[id]}_{id}{suffix}.{extension}" to restore the old default
  • Improve and optimize directory and filename generation

Fixes

  • Allow the classify post-processor to handle files with unknown filename extension (#138)
  • Fix rate limit handling for OAuth APIs (#368)
  • Fix artwork and scraps extraction on deviantart (#376, #392)
  • Distinguish between imgur album and gallery URLs (#380)
  • Prevent crash when using --ugoira-conv (#382)
  • Handle multi-image posts on patreon (#383)
  • Miscellaneous fixes for *reactor, simplyhentai

1.10.1 - 2019-08-02

Fixes

  • Use the correct domain for exhentai.org input URLs

1.10.0 - 2019-08-01

Warning

  • Prior to version 1.10.0 all cache files were created world readable (mode 644) leading to possible sensitive information disclosure on multi-user systems
  • It is recommended to restrict access permissions of already existing files (/tmp/.gallery-dl.cache) with chmod 600
  • Windows users should not be affected

Additions

Changes

  • The default cache file location on non-Windows systems is now
    • $XDG_CACHE_HOME/gallery-dl/cache.sqlite3 or
    • ~/.cache/gallery-dl/cache.sqlite3
  • New cache files are created with mode 600
  • exhentai extractors will always use e-hentai.org as domain

Fixes

  • Better handling of exhentai image limits and errors (#356, #360)
  • Try to prevent ZIP file corruption (#355)
  • Miscellaneous fixes for behance, ngomik

1.9.0 - 2019-07-19

Additions

  • Support for
  • Add login support for instagram (#195)
  • Add --no-download and extractor.*.download disable file downloads (#220)
  • Add -A/--abort to specify the number of consecutive download skips before aborting
  • Interpret -1 as infinite retries (#300)
  • Implement custom log message formats per log-level (#304)
  • Implement an mtime post-processor that sets file modification times according to metadata fields (#332)
  • Implement a twitter.content option to enable tweet text extraction (#333, #338)
  • Enable date-min/-max/-format options for tumblr (#337)

Changes

  • Set file modification times according to their Last-Modified header when downloading (#236, #277)
    • Use --no-mtime or downloader.*.mtime to disable this behavior
  • Duplicate download URLs are no longer silently ignored (controllable with extractor.*.image-unique)
  • Deprecate --abort-on-skip

Fixes

  • Retry downloads on OpenSSL exceptions (#324)
  • Ignore unavailable pins on sexcom instead of raising an exception (#325)
  • Use Firefox's SSL/TLS ciphers to prevent Cloudflare CAPTCHAs (#342)
  • Improve folder name matching on deviantart (#343)
  • Forward cookies to youtube-dl to allow downloading private videos
  • Miscellaneous fixes for 35photo, 500px, newgrounds, simplyhentai

1.8.7 - 2019-06-28

Additions

Changes

  • Delete empty archives created by the zip post-processor (#316)

Fixes

  • Handle hitomi Game CG galleries correctly (#321)
  • Miscellaneous fixes for deviantart, hitomi, pururin, kissmanga, keenspot, mangoxo, imagefap

1.8.6 - 2019-06-14

Additions

Changes

  • Detect directlink URLs with upper case filename extensions (#296)

Fixes

  • Improved error handling for tumblr API calls (#297)
  • Fixed extraction of livedoor blogs (#301)
  • Fixed extraction of special deviantart Sta.sh items (#307)
  • Fixed pagination for specific keenspot comics

1.8.5 - 2019-06-01

Additions

Changes

  • Adjust wallhaven extractors to new page layout:
    • use API and add api-key option
    • removed traditional login support
  • Provide original filenames for patreon downloads (#268)
  • Use e-hentai.org or exhentai.org depending on input URL (#278)

Fixes

  • Fix pagination over sankaku popular listings (#265)
  • Fix folder and collection extraction on deviantart (#271)
  • Detect "AreYouHuman" redirects on readcomiconline (#279)
  • Miscellaneous fixes for hentainexus, livedoor, ngomik

1.8.4 - 2019-05-17

Additions

Changes

  • Improved flickr metadata and video extraction (#246)

Fixes

  • Download original GIF animations from deviantart (#242)
  • Ignore missing edge_media_to_comment fields on instagram (#250)
  • Fix serialization of datetime objects for --write-metadata (#251, #252)
  • Allow multiple post-processor command-line options at once (#253)
  • Prevent crash on booru sites when no tags are available (#259)
  • Fix extraction on instagram after rhx_gis field removal (#266)
  • Avoid Cloudflare CAPTCHAs for Python interpreters built against OpenSSL < 1.1.1
  • Miscellaneous fixes for luscious

1.8.3 - 2019-05-04

Additions

Changes

  • Standalone executables are now built using PyInstaller:
    • uses the latest CPython interpreter (Python 3.7.3)
    • available on several platforms (Windows, Linux, macOS)
    • includes the certifi CA bundle, youtube-dl, and pyOpenSSL on Windows

Fixes

  • Patch urllib3's default list of SSL/TLS ciphers to prevent Cloudflare CAPTCHAs (#227) (Windows users need to install pyOpenSSL for this to take effect)
  • Provide fallback URLs for twitter images (#237)
  • Send Referer headers when downloading from hitomi (#239)
  • Updated login procedure on mangoxo

1.8.2 - 2019-04-12

Additions

Changes

  • Increased required Requests version to 2.11.0

Fixes

  • Improved image quality on reactor sites (#210)
  • Support imagebam galleries with more than 100 images (#219)
  • Updated Cloudflare bypass code

1.8.1 - 2019-03-29

Additions

Changes

  • Better FFmpeg arguments for --ugoira-conv
  • Adjusted metadata for luscious albums

Fixes

  • Proper handling of instagram multi-image posts (#178, #201)
  • Fixed tumblr avatar URLs when not using OAuth1.0 (#193)
  • Miscellaneous fixes for exhentai, komikcast

1.8.0 - 2019-03-15

Additions

Changes

  • More metadata for nhentai galleries
  • Combined myportfolio extractors into one
  • Renamed name metadata field to filename and removed the original filename field
  • Simplified and improved internal data structures
  • Optimized creation of child extractors

Fixes

  • Filter empty tumblr URLs (#165)
  • Filter ads and improve connection speed on hentaifoundry
  • Show proper error messages if luscious galleries are unavailable
  • Miscellaneous fixes for mangahere, ngomik, simplyhentai, imgspice

Removals

  • seaotterscans

1.7.0 - 2019-02-05

1.6.3 - 2019-01-18

  • Added metadata post-processor to write image metadata to an external file (#135)
  • Added option to reverse chapter order of manga extractors (#149)
  • Added authentication support for danbooru (#151)
  • Added tag metadata for exhentai and hbrowse galleries
  • Improved *reactor extractors (#148)
  • Fixed extraction issues for nhentai (#156), pinterest, mangapark

1.6.2 - 2019-01-01

  • Added support for:
  • Added support for multiple items on sta.sh pages (#113)
  • Added option to download tumblr avatars (#137)
  • Changed defaults for visited post types and inline media on tumblr
  • Improved inline extraction of tumblr posts (#133, #137)
  • Improved error handling and retry behavior of all API calls
  • Improved handling of missing fields in format strings (#136)
  • Fixed hash extraction for unusual tumblr URLs (#129)
  • Fixed image subdomains for hitomi galleries (#142)
  • Fixed and improved miscellaneous issues for kissmanga (#20), luscious, mangapark, readcomiconline

1.6.1 - 2018-11-28

1.6.0 - 2018-11-17

  • Added support for:
  • Added youtube-dl integration and video downloads for twitter (#99), behance, artstation
  • Added per-extractor options for network connections (retries, timeout, verify)
  • Added a --no-check-certificate command-line option
  • Added ability to specify the number of skipped downloads before aborting/exiting (#115)
  • Added extractors for scraps, favorites, popular and recent images on hentaifoundry (#110)
  • Improved login procedure for pixiv to avoid unwanted emails on each new login
  • Improved album metadata and error handling for flickr (#109)
  • Updated default User-Agent string to Firefox 62 (#122)
  • Fixed twitter API response handling when logged in (#123)
  • Fixed issue when converting Ugoira using H.264
  • Fixed miscellaneous issues for 2chan, deviantart, fallenangels, flickr, imagefap, pinterest, turboimagehost, warosu, yuki (#112)

1.5.3 - 2018-09-14

  • Added support for:
  • Added black-/whitelist options for post-processor modules
  • Added support for tumblr inline videos (#102)
  • Fixed extraction of smugmug albums without owner (#100)
  • Fixed issues when using default config values with reddit extractors (#104)
  • Fixed pagination for user favorites on sankaku (#106)
  • Fixed a crash when processing deviantart journals (#108)

1.5.2 - 2018-08-31

  • Added support for twitter timelines (#96)
  • Added option to suppress FFmpeg output during ugoira conversions
  • Improved filename formatter performance
  • Improved inline image quality on tumblr (#98)
  • Fixed image URLs for newly released mangadex chapters
  • Fixed a smaller issue with deviantart journals
  • Replaced subapics with ngomik

1.5.1 - 2018-08-17

  • Added support for:
  • Added support for related pins on pinterest
  • Fixed accessing "offensive" galleries on exhentai (#97)
  • Fixed extraction issues for mangadex, komikcast and behance
  • Removed original-image functionality from tumblr, since "raw" images are no longer accessible

1.5.0 - 2018-08-03

  • Added support for:
  • Added custom format string options to handle long strings (#92, #94)
    • Slicing: "{field[10:40]}"
    • Replacement: "{field:L40/too long/}"
  • Improved frame rate handling for ugoira conversions
  • Improved private access token usage on deviantart
  • Fixed metadata extraction for some images on nijie
  • Fixed chapter extraction on mangahere
  • Removed whatisthisimnotgoodwithcomputers
  • Removed support for Python 3.3

1.4.2 - 2018-07-06

  • Added image-pool extractors for safebooru and rule34
  • Added option for extended tag information on booru sites (#92)
  • Added support for DeviantArt's new URL format
  • Added support for mangapark mirrors
  • Changed imagefap extractors to use HTTPS
  • Fixed crash when skipping downloads for files without known extension

1.4.1 - 2018-06-22

  • Added an ugoira post-processor to convert pixiv animations to WebM
  • Added --zip and --ugoira-conv command-line options
  • Changed how ugoira frame information is handled
    • instead of being written to a separate file, it is now made available as metadata field of the ZIP archive
  • Fixed manga and chapter titles for mangadex
  • Fixed file deletion by post-processors

1.4.0 - 2018-06-08

  • Added support for:
  • Added extractors for
    • pixiv search results and followed users
    • deviantart search results and popular listings
  • Added post-processors to perform actions on downloaded files
  • Added options to configure logging behavior
  • Added OAuth support for smugmug
  • Changed pixiv extractors to use the AppAPI
    • this breaks favorite archive IDs and changes some metadata fields
  • Changed the default filename format for tumblr and renamed offset to num
  • Fixed a possible UnicodeDecodeError during installation (#86)
  • Fixed extraction of mangadex manga with more than 100 chapters (#84)
  • Fixed miscellaneous issues for imgur, reddit, komikcast, mangafox and imagebam

1.3.5 - 2018-05-04

  • Added support for:
  • Added title information for mangadex chapters
  • Improved the pinterest API implementation (#83)
  • Improved error handling for deviantart and tumblr
  • Removed gomanga and puremashiro

1.3.4 - 2018-04-20

  • Added support for custom OAuth2 credentials for pinterest
  • Improved rate limit handling for tumblr extractors
  • Improved hentaifoundry extractors
  • Improved imgur URL patterns
  • Fixed miscellaneous extraction issues for luscious and komikcast
  • Removed loveisover and spectrumnexus

1.3.3 - 2018-04-06

  • Added extractors for
    • nhentai search results
    • exhentai search results and favorites
    • nijie doujins and favorites
  • Improved metadata extraction for exhentai and nijie
  • Improved tumblr extractors by avoiding unnecessary API calls
  • Fixed Cloudflare DDoS protection bypass
  • Fixed errors when trying to print unencodable characters

1.3.2 - 2018-03-23

  • Added extractors for artstation albums, challenges and search results
  • Improved URL and metadata extraction for hitomiand nhentai
  • Fixed page transitions for danbooru API results (#82)

1.3.1 - 2018-03-16

  • Added support for:
  • Added Cloudflare DDoS protection bypass to komikcast extractors
  • Changed archive ID formats for deviantart folders and collections
  • Improved error handling for deviantart API calls
  • Removed imgchili and various smaller image hosts

1.3.0 - 2018-03-02

  • Added --proxy to explicitly specify a proxy server (#76)
  • Added options to customize archive ID formats and undefined replacement fields
  • Changed various archive ID formats to improve their behavior for favorites / bookmarks / etc.
    • Affected modules are deviantart, flickr, tumblr, pixiv and all …boorus
  • Improved sankaku and idolcomplex support by
    • respecting page and next URL parameters (#79)
    • bypassing the page-limit for unauthenticated users
  • Improved directlink metadata by properly unquoting it
  • Fixed pixiv ugoira extraction (#78)
  • Fixed miscellaneous extraction issues for mangastream and tumblr
  • Removed yeet, chronos, coreimg, hosturimage, imageontime, img4ever, imgmaid, imgupload

1.2.0 - 2018-02-16

  • Added support for:
  • Added --download-archive to record downloaded files in an archive file
  • Added --write-log to write logging output to a file
  • Added a filetype check on download completion to fix incorrectly assigned filename extensions (#63)
  • Added the tumblr:... pseudo URI scheme to support custom domains for Tumblr blogs (#71)
  • Added fallback URLs for tumblr images (#64)
  • Added support for reddit-hosted images (#68)
  • Improved the input file format by allowing comments and per-URL options
  • Fixed OAuth 1.0 signature generation for Python 3.3 and 3.4 (#75)
  • Fixed smaller issues for luscious, hentai2read, hentaihere and imgur
  • Removed the batoto module

1.1.2 - 2018-01-12

  • Added support for:
  • Added an option to filter reblogs on tumblr (#61)
  • Added OAuth user authentication for tumblr (#65)
  • Added support for slideshare mobile URLs (#67)
  • Improved pagination for various …booru sites to work around page limits
  • Fixed chapter information parsing for certain manga on kissmanga (#58) and batoto (#60)

1.1.1 - 2017-12-22

  • Added support for:
  • Added pool- and post-extractors for sankaku
  • Added OAuth user authentication for deviantart
  • Updated luscious to support members.luscious.net URLs (#55)
  • Updated mangahere to use their new domain name (mangahere.cc) and support mobile URLs
  • Updated gelbooru to not be restricted to the first 20,000 images (#56)
  • Fixed extraction issues for nhentai and khinsider

1.1.0 - 2017-12-08

  • Added the -r/--limit-rate command-line option to set a maximum download rate
  • Added the --sleep command-line option to specify the number of seconds to sleep before each download
  • Updated gelbooru to no longer use their now disabled API
  • Fixed SWF extraction for sankaku (#52)
  • Fixed extraction issues for hentai2read and khinsider
  • Removed the deprecated --images and --chapters options
  • Removed the mangazuki module

1.0.2 - 2017-11-24

  • Added an option to set a custom user-agent string
  • Improved retry behavior for failed HTTP requests
  • Improved seiga by providing better metadata and getting more than the latest 200 images
  • Improved tumblr by adding support for all post types, scanning for inline images and following external links (#48)
  • Fixed extraction issues for hbrowse, khinsider and senmanga

1.0.1 - 2017-11-10

  • Added support for:
  • Fixed exception handling during file downloads which could lead to a premature exit
  • Fixed an issue with tumblr where not all images would be downloaded when using tags (#48)
  • Fixed extraction issues for imgbox (#47), mangastream (#49) and mangahere

1.0.0 - 2017-10-27

  • Added support for:
  • Added support for pixiv ranking lists
  • Added support for booru popular lists (danbooru, e621, konachan, yandere, 3dbooru)
  • Added the --cookies command-line and cookies config option to load additional cookies
  • Added the --filter and --chapter-filter command-line options to select individual images or manga-chapters by their metadata using simple Python expressions (#43)
  • Added the verify config option to control certificate verification during file downloads
  • Added config options to overwrite internally used API credentials (API Tokens & IDs)
  • Added -K as a shortcut for --list-keywords
  • Changed the --images and --chapters command-line options to --range and --chapter-range
  • Changed keyword names for various modules to make them accessible by --filter. In general minus signs have been replaced with underscores (e.g. gallery-id -> gallery_id).
  • Changed default filename formats for manga extractors to optionally use volume and title information
  • Improved the downloader modules to use .part files and support resuming incomplete downloads (#29)
  • Improved deviantart by distinguishing between users and groups (#26), always using HTTPS, and always downloading full-sized original images
  • Improved sankaku by adding authentication support and fixing various other issues (#44)
  • Improved URL pattern for direct image links (#30)
  • Fixed an issue with luscious not getting original image URLs (#33)
  • Fixed various smaller issues for batoto, hentai2read (#38), jaiminisbox, khinsider, kissmanga (#28, #46), mangahere, pawoo, twitter
  • Removed kisscomic and yonkouprod modules

0.9.1 - 2017-07-24

0.9.0 - 2017-06-28

  • Added support for:
  • Added support for direct image links
  • Added user authentication via OAuth for reddit and flickr
  • Added support for user authentication data from .netrc files (#22)
  • Added a simple progress indicator for multiple URLs (#19)
  • Added the --write-unsupported command-line option to write unsupported URLs to a file
  • Added documentation for all available config options (configuration.rst)
  • Improved pixiv to support tags for user downloads (#17)
  • Improved pixiv to support shortened and http://pixiv.me/... URLs (#23)
  • Improved imgur to properly handle .gifv images and provide better metadata
  • Fixed an issue with kissmanga where metadata parsing for some series failed (#20)
  • Fixed an issue with getting filename extensions from Content-Type response headers

0.8.4 - 2017-05-21

  • Added the --abort-on-skip option to stop extraction if a download would be skipped
  • Improved the output format of the --list-keywords option
  • Updated deviantart to support all media types and journals
  • Updated fallenangels to support their Vietnamese version
  • Fixed an issue with multiple tags on ...booru sites
  • Removed the yomanga module

0.8.3 - 2017-05-01

  • Added support for https://pawoo.net/
  • Added manga extractors for all FoOlSlide-based modules
  • Added the -q/--quiet and -v/--verbose options to control output verbosity
  • Added the -j/--dump-json option to dump extractor results in JSON format
  • Added the --ignore-config option
  • Updated the exhentai extractor to fall back to using the e-hentai version if no username is given
  • Updated deviantart to support sta.sh URLs
  • Fixed an issue with kissmanga which prevented image URLs from being decrypted properly (again)
  • Fixed an issue with pixhost where for an image inside an album it would always download the first image of that album (#13)
  • Removed the mangashare and readcomics modules

0.8.2 - 2017-04-10

  • Fixed an issue in kissmanga which prevented image URLs from being decrypted properly

0.8.1 - 2017-04-09

  • Added new extractors:
  • Added a favourites extractor for deviantart
  • Re-enabled the kissmanga module
  • Updated nijie to support multi-page image listings
  • Updated mangastream to support readms.net URLs
  • Updated exhentai to support e-hentai.org URLs
  • Updated fallenangels to support their new domain and site layout

0.8.0 - 2017-03-28

  • Added logging support
  • Added the -R/--retries option to specify how often a download should be retried before giving up
  • Added the --http-timeout option to set a timeout for HTTP connections
  • Improved error handling/tolerance during HTTP file downloads (#10)
  • Improved option parsing and the help message from -h/--help
  • Changed the way configuration values are used by prioritizing top-level values
    • This allows for cmdline options like -u/--username to overwrite values set in configuration files
  • Fixed an issue with imagefap.com where incorrectly reported gallery sizes would cause the extractor to fail (#9)
  • Fixed an issue with seiga.nicovideo.jp where invalid characters in an API response caused the XML parser to fail
  • Fixed an issue with seiga.nicovideo.jp where the filename extension for the first image would be used for all others
  • Removed support for old configuration paths on Windows
  • Removed several modules:
    • mangamint: site is down
    • whentai: now requires account with VIP status for original images
    • kissmanga: encrypted image URLs (will be re-added later)

0.7.0 - 2017-03-06

  • Added --images and --chapters options
    • Specifies which images (or chapters) to download through a comma-separated list of indices or index-ranges
    • Example: --images -2,4,6-8,10- will select images with index 1, 2, 4, 6, 7, 8 and 10 up to the last one
  • Changed the -g/--get-urls option
    • The amount of how often the -g option is given now determines up until which level URLs are resolved.
    • See 3bca866185
  • Changed several option keys:
    • directory_fmt -> directory
    • filename_fmt -> filename
    • download-original -> original
  • Improved FoOlSlide-based extractors
  • Fixed URL extraction for hentai2read
  • Fixed an issue with deviantart, where the API access token wouldn't get refreshed

0.6.4 - 2017-02-13

  • Added new extractors:
    • fallenangels (famatg.com)
  • Fixed url- and data-extraction for:
    • nhentai
    • mangamint
    • twitter
    • imagetwist
  • Disabled InsecureConnectionWarning when no certificates are available

0.6.3 - 2017-01-25

  • Added new extractors:
    • gomanga
    • yomanga
    • mangafox
  • Fixed deviantart extractor failing - switched to using their API
  • Fixed an issue with SQLite on Python 3.6
  • Automated test builds via Travis CI
  • Standalone executables for Windows

0.6.2 - 2017-01-05

  • Added new extractors:
    • kisscomic
    • readcomics
    • yonkouprod
    • jaiminisbox
  • Added manga extractor to batoto-module
  • Added user extractor to seiga-module
  • Added -i/--input-file argument to allow local files and stdin as input (like wget)
  • Added basic support for file:// URLs
    • this allows for the recursive extractor to be applied to local files:
    • $ gallery-dl r:file://[path to file]
  • Added a utility extractor to run unit test URLs
  • Updated luscious to deal with API changes
  • Fixed twitter to provide the original image URL
  • Minor fixes to hentaifoundry
  • Removed imgclick extractor

0.6.1 - 2016-11-30

  • Added new extractors:
    • whentai
    • readcomiconline
    • sensescans, worldthree
    • imgmaid, imagevenue, img4ever, imgspot, imgtrial, pixhost
  • Added base class for extractors of FoOlSlide-based sites
  • Changed default paths for configuration files on Windows
    • old paths are still supported, but that will change in future versions
  • Fixed aborting downloads if a single one failed (#5)
  • Fixed cloudflare-bypass cache containing outdated cookies
  • Fixed image URLs for hitomi and 8chan
  • Updated deviantart to always provide the highest quality image
  • Updated README.rst
  • Removed doujinmode extractor

0.6.0 - 2016-10-08

  • Added new extractors:
    • hentaihere
    • dokireader
    • twitter
    • rapidimg, picmaniac
  • Added support to find filename extensions by Content-Type response header
  • Fixed filename/path issues on Windows (#4):
    • Enable path names with more than 260 characters
    • Remove trailing spaces in path segments
  • Updated Job class to automatically set category/subcategory keywords

0.5.2 - 2016-09-23

  • Added new extractors:
    • pinterest
    • rule34
    • dynastyscans
    • imagebam, coreimg, imgcandy, imgtrex
  • Added login capabilities for batoto
  • Added --version cmdline argument to print the current program version and exit
  • Added --list-extractors cmdline argument to print names of all extractor classes together with descriptions and example URLs
  • Added proper error messages if an image/user does not exist
  • Added unittests for every extractor

0.5.1 - 2016-08-22

  • Added new extractors:
    • luscious
    • doujinmode
    • hentaibox
    • seiga
    • imagefap
  • Changed error output to use stderr instead of stdout
  • Fixed broken pipes causing an exception-dump by catching BrokenPipeErrors

0.5.0 - 2016-07-25

0.4.1 - 2015-12-03

  • New modules (imagetwist, turboimagehost)
  • Manga-extractors: Download entire manga and not just single chapters
  • Generic extractor (provisional)
  • Better and configurable console output
  • Windows support

0.4.0 - 2015-11-26

0.3.3 - 2015-11-10

0.3.2 - 2015-11-04

0.3.1 - 2015-10-30

0.3.0 - 2015-10-05

0.2.0 - 2015-06-28

0.1.0 - 2015-05-27