This adresses bug #69.
All fields in the filename pattern that should be replaced by a date
encoding string (excluding {post.date_local}) are now evaluated using
UTC while ensuring that the replacing string indicates the use of UTC.
{date_utc} encodes the post creation date in UTC rather than the
current local timezone, as {date} does.
This was proposed in #69. Encoding the post creation date in
local time zone induces problems regarding --fast-update when the
time zone is changed.
- Added owner_id and mediaid to Post class properties.
- In case of not downloading stories, the attributes of the Post class
can now be used in filename-pattern, e.g. {post.owner_id} or {post.mediaid}.
Closes #53.
- If when retrying a failed connection ^C is hit, the retrying is
aborted,
- Option --max-connection-attempts to override default of how many
connection attempts should be made.
Closes #51.
With --metadata-json, a JSON file for each post is created saving the
Post properties defined in instaloader.Post class, i.e. caption, number
of likes, people tagged in caption or the picture itself, etc.
This closes #33 and closes #47.
caption_mentions is a list of all lowercased profiles that are mentioned
in the Post's caption, without preceeding '@'.
tagged_users is a list of all lowercased users that are tagged in the
Post. This was requested in #47.
Just like all properties of instaloader.Post class, caption_mentions and
tagged_users are available for --only-if filters.
caption_hashtags is a list of all hashtags that are mentioned in the
Post's caption. It allows to easily filter Posts that have multiple
hashtags, and as such fixes #24.
Further, the documentation of --only-if has been completed by linking to
a description of the syntax in the Python documentation, and by linking
to a list of all defined properties with their meanings. So, this commit
also closes #42.
I've come across several implausible values for `device_timestamp` (such as 182428140, which would be in october 1975, assuming millisecond). I guess it is due to improperly configured phones, or maybe some third party software that's mangeling the EXIF data on images before posting. Anyway. Since by the nature of stories, the `taken_at` timestamp (presumably when the instagram servers received the post) ought to be approximately when an image was actually taken. So there's no real value trying to use the timestamp provided by the photo-taking-device.
Additional sleeps are necessary because Instagram is rate limiting
GraphQL queries. The error does not occur if not more than 100 queries
are made in a sliding window of eleven minutes.
Ports a894c2d to version 3.
If --only-if='likes>1000 or viewer_has_liked' is given, it is not
neccessary to evaluate viewer_has_liked if the post has more than 1000
likes. The new implementation smartly handles this case.
download_stories() may trow a BadResponseException, which should not
cause abortion of download_profile(). Now, all calls of
download_stories() are within an _error_catcher context.
These are now adapted to how many requests have already been done. With
the current settings, Instaloader does not more than
12 request in the first ten seconds,
28 requests in the first minute,
40 requests in the first two minutes,
63 requests in the first five minutes,
90 requests in the first ten minutes,
and after that 50 requests per ten minutes.
This should make it less likely that Instaloader is rate-limited by
Instagram, while still being fast if downloading only a few posts.
Further, option --no-sleep is hidden in --help output and README.rst.
Fixes a KeyError which occurred when fetching some information note
available in Post._node. Makes all properties not throw exceptions,
rather they return None if the information cannot be obtained.
sidecar_edges is now a method get_sidecar_edges(). get_comments() does
not do any additional requests if the post does not have any comment.
Fixes #39.
where FILTER is a boolean expression in python syntax where all names
are evaluated to instaloader.Post properties.
Examples:
instaloader --login=USER --only-if='viewer_has_liked' :feed
instaloader --only-if='likes>1000 and comments>5' profile
The default Instaloader headers aren't passed with a simple `requests.head()`, so leakage of user agents such as `"python-requests/2.18.1"` will occur.
This simplifies accessing properties of a Post. Method download_post()
remains to class Instaloader rather than Post, as it fits there better.
Also, since it is now easily possible, all download_*() functions now
have a filter_func parameter. Its meaning has been reverted to be
consistent of how a filter is commonly understood: A post is downloaded
iff filter_func is None or evaluates to True.
Post.get_comments() foreports commit 86fb80d ("Avoid GraphQL queries if
all comments in metadata").
The original method of substituting 2048x2048 for whatever resolution was given seemed somewhat convoluted. This accomplishes the same thing, except raising an exception if the given url is not on the right domain.
Instead of retrying a download attempt answered with a 404, the download
is aborted after the first attempt. Thanks to the _error_catcher(), a
message is printed and Instaloader goes on with the next files to
download.
Further, this commit removes the unneeded NodeUnavailableException and
adjusts docstrings accordingly.
Now, setup.py does not assume to be called from the path where the
source tree resides. This fixes getting the long_description and the
version if setup.py is called from outside.
These options instruct instaloader to not save captions or geotags
respectively, even if the regarding information can be obtained without
any additional queries to Instagram.
This feature was proposed in #25, and thus this commit should close #25.
Get rid of NonfatalException (an exception is nonfatal iff it is
catched somewhere)
Foreport fixes for #26 and #30.
The current __sersion__ string is now kept in instaloader.py rather than
setup.py. This lets instaloader.__version__ always deliver the version
of the actually-loaded Instaloader module.
Minor changes to README.rst, error handling and which class methods are
public.
With these and the changes of the previous commit, we saved 31 lines of
code, indicating that it might be easier to understand and to maintain.
Remove many code duplications, merely by using more pythonic idioms.
Use GraphQL more often.
Better cope with errors: All requests can be retried; failed requests do
not cause program termination; all error strings are repeated to the
user at the end of execution.
download_post() (formerly download_node()) does not repeat node metadata
request (before this commit, this request was executed up to three
times).
In case a node can not be downloaded or its metadata is needed and can
not be retrieved, a NodeUnavailableException is raised and the according
node will be skipped.
Concerns #26
download_stories() did and does not check if a story is "unseen". The
response to the query of the '/feed/reels_tray/' URL provides all
available stories of the user's followees. Nevertheless, some of them do
not contain an 'items' field which has no causal relationship with their
status in terms of "seen" or "unseen". Therefore, to overcome this lack
of 'items' the 'feed/user/TARGET_USERID/reel_media/' URL needs to be
queried for each relevant followee whose 'items' were not provided in
the first place.
Unless --no-sleep is given, Instaloader waits between requests to the
Instagram servers.
This commit fixes and enhances this behavior. Now, --no-sleep is always
obeyed. Between requests to the instagram.com servers, there is now a
delay of 250 ms ~ 2000 ms. Requests to the file servers do not cause a
delay.
Instaloader downloads all posts in
<DIRNAME>/<FILENAME>+(suffix and extension)
which are now generated by the templates given with --dirname-pattern
and --filename-pattern. These templates may contain specifiers such as
'{target}', '{profile}', '{date}' and '{shortcode}'.
Default for --dirname-pattern is '{target}', default for
--filename-pattern is '{date:%Y-%m-%d_%H-%M-%S}'
The former options --no-profile-subdir and --hashtag-username were
removed, because their behavior can now be achieved like this:
--no-profile-subdir and --hashtag-username:
--dirname-pattern='.' --filename-pattern='{profile}__{date:%Y-%m-%d_%H-%M-%S}'
--no-profile-subdir, but not --hashtag-username:
--dirname-pattern='.' --filename-pattern='{target}__{date:%Y-%m-%d_%H-%M-%S}'
--hashtag-username but not --no-profile-subdir:
--dirname-pattern='{profile}'
This adds the option proposed in #23, to encode both the hashtag and the
profile name in the file's path when downloading by hashtag, e.g.:
--dirname-pattern='{target}' --filename-pattern='{profile}_{date:%Y-%m-%d_%H-%M-%S}'
(Closes #23)
On module level:
Cleaner exception handling for load_session_from file
interactive_login logs in interactively now, always asking the user for
password. Before, it had an optional password parameter determining
whether it was interactive or not.
On application level:
Warn if profile specifiers are used which require login, but not --login
flag was given (@profile, :feed-all, :feed-liked).
Clearly warn that --password is insecure.
With --hashtag-username given, if downloading per #hashtag, instead of
per username, for each picture an additional request to the Instagram
server is issued to lookup the picture's username. Instead of storing
files in #hashtag/timestamp.jpg, files are stored in
username/timestamp.jpg as it is the default when not downloading per
hashtag.
This closes #22.
Instaloader is now capable of downloading all pictures associated with
one #hashtag with:
instaloader #hashtag
This implements the feature requested with #18.
The check_id() step, including the get_username_by_id(), which is used
to determine whether a profile's account name has changed since the last
download using it's unique id when Instaloader is operating logged-in,
is actually optional and should not cause termination in any case.
