This commit makes the DNS resolution code better suited for both
filtering on cname and ip address. The change allows early availability
of ip address so that `ipaddress=` option can be matched at
onBeforeRequest time.
As a result, it is now possible to block root document using
`ipaddress=` option -- so long as an ip address can be extracted
before first onBeforeRequest() call.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2792
Caveat
------
the ip address used is the first one among the list of ip
addresses returned by dns.resolve() method. There is no way for uBO
to know which exact ip address will be used by the browser when
sending the request, so this is at most a best guess. The exact IP
address used by the browser is available at onHeadersReceived time,
and uBO will also filter according to this value, but by then the
network request has already been sent to the remote server.
Possibly a future improvement would make available the whole list
of ip addresses to the filtering engine, but even then it's impossible
to know with certainty which ip address will ultimately be used by the
browser -- it is entirely possible that the ip address used by the
browser might not be in the list received through dns.resolve().
If an IP address can be extracted from the hostname portion of
a URL, the IP address matching will be performed at onBeforeRequest()
time.
Regardless, IP address matching will subsequently always be performed
at onHeadersReceived() time as the request details at that point
contain a reliable IP address value on supported platforms (Firefox-
only as of now).
The `cap_ipaddress` now evaluates to `true` in Chromium-based
browsers. Even though these browsers are unable to provide reliable
IP address value at onHeadersReceived() time, they can still
perform IP address matching for IP address extracted from hostname
portion of a URL.
A new icon has been added to the popup panel, to open a popup
window with a terse list of DNR events for the current tab, in
reverse chronological order (most recent DNR event appears at
the top).
The new ability is available only when the extension is sideloaded,
as per `declarativeNetRequestFeedback` documentation. Ref:
https://developer.chrome.com/docs/extensions/reference/api/declarativeNetRequest#event-onRuleMatchedDebug
Purposefully minimal, so as to have something rather than nothing
when having to diagnose filtering issue with the DNR API. Example:
https://github.com/uBlockOrigin/uBOL-home/issues/156
The content of the popup window does not dynamically update, force
a refresh (F5) to get the most recent DNR events. This might be
improved in the future.
The DNR event buffer is not persisted, so the buffer is empty when
service worker is restarted. This might be improved in the future
by using session storage API.
There is no output filtering ability in this first draft. This
might be improved in the future.
DNR rules are reported. The filter from which a DNR rule
originates is not reported. Given that the rulesets are optimized
after conversion from original filter lists to reduce the DNR rule
count, this is unlikely to ever be possible.
For internal use by filter list maintainers, do not open issues
about this. Left undocumented on purpose.
This new procedural operator allows to target elements in the
shadow root of an element.
subject:shadow(arg)
- Description: Look-up matching elements inside the shadow root (if
present) of _subject_.
- Chainable: Yes
- _subject_: Can be a plain or procedural selector.
- _arg_: A plain or a procedural selector for the elements to target
inside the shadowroot.
Example:
..##body > div:not([class]):shadow(div[style]):has(:shadow([data-i18n^="#ad"]))
In uBO, the "cache storage" is used to save resources which can
be safely discarded, though at the cost of having to fetch or
recompute them again.
Extension storage (browser.storage.local) is now always used as
cache storage backend. This has always been the default for
Chromium-based browsers.
For Firefox-based browsers, IndexedDB was used as backend for
cache storage, with fallback to extension storage when using
Firefox in private mode by default.
Extension storage is reliable since it works in all contexts,
though it may not be the most performant one.
To speed-up loading of resources from extension storage, uBO will
now make use of Cache API storage, which will mirror content of
key assets saved to extension storage. Typically loading resources
from Cache API is faster than loading the same resources from
the extension storage.
Only resources which must be loaded in memory as fast as possible
will make use of the Cache API storage layered on top of the
extension storage.
Compiled filter lists and memory snapshot of filtering engines
(aka "selfies") will be mirrored to the Cache API storage, since
these must be loaded into memory as fast as possible, and reloading
filter lists from their compiled counterpart is a common
operation.
This new design makes it now seamless to work in permanent private
mode for Firefox-based browsers, since extension storage now
always contains cache-related assets.
Support for IndexedDB is removed for the time being, except to
support migration of cached assets the first time uBO runs with
the new cache storage design.
In order to easily support all choices of storage, a new serializer
has been introduced, which is capable of serializing/deserializing
structure-cloneable data to/from a JS string.
Because of this new serializer, JS data structures can be stored
directly from their native representation, and deserialized
directly to their native representation from uBO's point of view,
since the serialization occurs (if needed) only at the storage
interface level.
This new serializer simplifies many code paths where data
structures such as Set, Map, TypedArray, RegExp, etc. had to be
converted in a disparate manner to be able to persist them to
extension storage.
The new serializer supports workers and LZ4 compression. These
can be configured through advanced settings.
With this new layered design, it's possible to introduce more
storage layers if measured as beneficial (i.e. maybe
browser.storage.session)
References:
- https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/local
- https://developer.mozilla.org/en-US/docs/Web/API/Cache
- https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm
This is to reduce the diff size of rulesets in new
releases. Beside smaller diff size, this also makes it
easier to investigate rule changes across releases.
This commit brings the following changes to the logger:
All logging output generated by injected scriptlets are now sent to
the logger, the developer console will no longer be used to log
scriptlet logging information.
When the logger is not opened, the scriplets will not output any
logging information.
The goal with this new approach is to allow filter authors to
more easily assess the working of scriptlets without having to
go through scriptlet parameters to enable logging.
Consequently all the previous ways to tell scriptlets to log
information are now obsolete: if the logger is opened, the
scriptlets will log information to the logger.
Another benefit of this approach is that the dev tools do not
need to be open to obtain scriptlets logging information.
Accordingly, new filter expressions have been added to the logger:
"info" and "error". Selecting the "scriptlet" expression will also
keep the logging information from scriptlets.
A new button has been added to the logger (not yet i18n-ed): a
"volume" icon, which allows to enable verbose mode. When verbose
mode is enabled, the scriptlets may choose to output more
information regarding their inner working.
The entries in the logger will automatically expand on mouse hover.
This allows to scroll through entries which text does not fit into
a single row.
Clicking anywhere on an entry in the logger will open the detailed
view when applicable.
Generic information/errors will now be rendered regardless of which
tab is currently selected in the logger (similar to how tabless
entries are already being rendered).
Procedural filters with `:xpath` operator were silently rejected
at conversion time because the parser was failing to evaluate the
xpath expression due to the absence of a `document` object in
nodejs.
If `document` object is not present, the parser will assume the
xpath expression is valid.
The idea is to remove as many dependencies as possible for
low-level ScriptletFilteringEngine in order to make it easier
to reuse the module outside uBO itself.
The high-level derived class takes care of caching and
injection of scriptlets into documents, which requires
more knowledge about the environment in which scriptlets
are to be used.
Also improve scriptlet cache usage to minimize overhead of
retrieving scriptlets.
