As reported internally to ubo-security by https://github.com/distinctmondaylila
One issue is a regression from the rewriting of the static filtering
parser in version 1.47.0, specifically the following commit:
https://github.com/gorhill/uBlock/commit/8ea3b0f64c
The existing regex was no longer suitable to properly detect
some usage of `report-xxx` in the rwritten parser.
Another issue which predates 1.47.0 is that the regex used for
validation was case-sensititive, while the `report-uri` directive
can be written using uppercase letters, i.e. `Report-uri`.
Avoid race conditions between isolated world-side broadcast channel
and main-side broadcast channel, so as to not lose logging
information if the isolated world-side is not yet ready to
receive through its broadcast channel.
Additionally, added new scriptlet: `trusted-replace-argument`.
[...]##+js(trusted-replace-argument, fn, argpos, argval [,condition, pattern])
Where:
- `fn` is the function we want to proxy through an `apply` handler.
This can also be a class, in which case the scriptlet will proxy
through `construct` handler. At the moment, `fn` must exist at the
time the scriptlet executes.
- `argpos` is the 0-based position of the argument we want to change
- `argval` is the value we want to have for the argument -- the value
is interpreted the same way the value for `set-constant` is
interpreted.
- `condition, pattern` is a vararg which tells the scriptlet to act
only if `pattern` is found in the argument to overwrite.
Example of usage:
alliptvlinks.com##+js(trusted-replace-argument, MutationObserver, 0, noopFunc)
This commit brings the following changes to the logger:
All logging output generated by injected scriptlets are now sent to
the logger, the developer console will no longer be used to log
scriptlet logging information.
When the logger is not opened, the scriplets will not output any
logging information.
The goal with this new approach is to allow filter authors to
more easily assess the working of scriptlets without having to
go through scriptlet parameters to enable logging.
Consequently all the previous ways to tell scriptlets to log
information are now obsolete: if the logger is opened, the
scriptlets will log information to the logger.
Another benefit of this approach is that the dev tools do not
need to be open to obtain scriptlets logging information.
Accordingly, new filter expressions have been added to the logger:
"info" and "error". Selecting the "scriptlet" expression will also
keep the logging information from scriptlets.
A new button has been added to the logger (not yet i18n-ed): a
"volume" icon, which allows to enable verbose mode. When verbose
mode is enabled, the scriptlets may choose to output more
information regarding their inner working.
The entries in the logger will automatically expand on mouse hover.
This allows to scroll through entries which text does not fit into
a single row.
Clicking anywhere on an entry in the logger will open the detailed
view when applicable.
Generic information/errors will now be rendered regardless of which
tab is currently selected in the logger (similar to how tabless
entries are already being rendered).
Procedural filters with `:xpath` operator were silently rejected
at conversion time because the parser was failing to evaluate the
xpath expression due to the absence of a `document` object in
nodejs.
If `document` object is not present, the parser will assume the
xpath expression is valid.
Scriptlets parameters which are quoted must be re-quoted when
output to the logger to be sure they can be properly looked up
in the list, and that they can be used through copy-paste
operations.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/3083
This will not completely eliminate the issue but it should
lower the likelihood it will occur -- so at least uBO can
still benefit from reliable scriptlet execution in Firefox.
The idea is to remove as many dependencies as possible for
low-level ScriptletFilteringEngine in order to make it easier
to reuse the module outside uBO itself.
The high-level derived class takes care of caching and
injection of scriptlets into documents, which requires
more knowledge about the environment in which scriptlets
are to be used.
Also improve scriptlet cache usage to minimize overhead of
retrieving scriptlets.
If the `manual` parameter is assigned a date in the form of
`YYMMDD`, this will tell uBO to update lists from origin sources
when the current time is within the range of the specified date,
otherwise lists will be updated from CDNs. Updating from CDNs
is always strongly recommended since this enables differential
updates.
For the time being, `manual=1` will always cause to update lists
from origin, but this form will be deprecated once next stable
release is widespread. The idea is to not leave behind stale
and obsolete links which would be detrimental to differential
updates should someone click on one of these old links left
behind.
Purging all the lists from cache storage is detrimental to
differential update, and cause filter lists to be updated less
often and consequently to be less up to date then when letting
differential updater do its work.
Possibly related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2136
Also reported internally, steps to reproduce the issue fixed here:
- Open uBO's dashboard through 3-dot > Add-ons > uBO > Settings
- Bring forth "Filter lists" pane
We want the tab to be already opened at next launch
- Quit Firefox for Android
- Launch Firefox for Android
Result:
Very long launch time, lists marked as out of date.
In Support pane's troubleshooting information, a delta sign will
be added to list update information when the list was last updated
through differential update.
Related commit:
https://github.com/uBlockOrigin/uAssets/commit/20312c2178
To work around incompatibity with `urltransform` in 1.53.0.
With this commit support for `urltransform` is dropped in
favor of `uritransform`. Since `uritransform` won't be parsed
by older versions of uBO, the change in syntax for `urltransform`
between 1.53.0 and 1.54.0 will no longer cause issue.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2969
Changes:
Use browser.alarms to trigger selfie creation. Presence of a selfie
improve markedly time to readiness when uBO is unsuspended.
Mirror content of storage.local to (in-memory) storage.session for
faster load to readiness when uBO is ususpended.
Broadcast channels are more suited to uBO than DOM events to dispatch
notifications to different parts of uBO.
DOM events can only be dispatched to local context, broadcast channels
dispatch to all contexts (i.e. background process, workers, auxiliary
pages) -- this last behavior is better suited to uBO to communicate
internal changes to all potential listeners, not just those in the local
context.
Additionally, broadcasting to content scripts is now done through
tabs.sendMessage() instead of through potentially opened message
ports, this simplifies broadcasting to content scripts, and this
doesn't require to have long-lived message ports in content
scripts.
Fortunately, getAvailableLists() filters out non-filter
list resources thus preventing those mistakenly selected
resources from being loaded as filter lists.
There is a lot asynchronicity in the auto-update code, and
the fix here is to detect then fix instances of out of sync
state between a cached filter list and its metadata stored
separately.
When done compiling, force CSSTree to parse an empty string, so
as to ensure it doesn't keep a reference to that string.
Typically, the string passed to CSSTree is a small slice of a
larger string which is a whole filter list. This means that
holding a reference to the sliced string causes the JS engine
to hold in memory to the whole filter list last parsed.
Reference documentation:
https://adguard.com/kb/general/ad-filtering/create-own-filters/#replace-modifier
This is a network filter option which can only be loaded from a
trusted source.
Since this filter is about modifying the response body, it currently
only works in Firefox.
As discussed with filter list maintainers.
Manual update of one or more lists will cause the most recent version
of these lists to be fetched from the "origin" server, and since the
lists from "origin" servers cannot be updated through differential
update, the lists will be subsequently updated according to their
`Expires` directive.
When the lists are auto-updated, the "CDN" servers will be used,
and as a result the lists will start to be updated trhough
differential updates every 6-hour (currently).
Thus it is recommended and optimal to let the lists auto-update,
since you will benefit from a much shorter delay to get up-to-date
lists (i.e. every 6-hour instead of every 6-day).
You can force the auto-updater to fetch all the lists by clicking
"Purge all caches", then restart uBO without clicking "Update".
This will cause uBO to perform an emergency auto-update at restart
time, after which you will have all the lists which are candidates
for differential update.
The "Update now" button in the "Support" pane will also cause lists
to be fetched from their "origin" server.
Related discussion:
https://github.com/ameshkov/diffupdates
The benefits of diff-patching filter lists is much shorter update
schedule and significantly less bandwidth consumed.
At the moment, only default filter lists are subject to be
diff-patched.
External filter lists can make their lists diff-patchable by
following the specification link above.
Only filter lists fetched by the auto-updater are candidate for
diff-patching.
Forcing a manual update of the filter lists will prevent the
diff-patcher from kicking in until one or more lists are
auto-updated.
Some back-of-the-envelop calculations regarding the load on free
CDN solutions used by uBO to distribute its own filter lists:
Currently, for each CDN (with lists updating after days):
~560 M req/month, ~78 TB/month
With diff-patching lists on a 6-hour schedule:
~390 M req/month, 1 TB/month
Those estimates were done according to statistics shown by
jsDelivr, which is one of 4 CDNs picked randomly when a list
updates:
https://www.jsdelivr.com/package/gh/uBlockOrigin/uAssetsCDN?tab=stats
I worked through some of the websites listed in the google-ima shim
script issue[1], to see what was going wrong. It turned out the
addEventListener method supports an optional context Object, which is
bound to the listener if provided. Some websites make use of that,
and then break when `this` is not bound correctly when events are
dispatched.
See also https://github.com/duckduckgo/tracker-surrogates/pull/24
1 - https://github.com/uBlockOrigin/uBlock-issues/issues/2265
Related discussion:
https://github.com/uBlockOrigin/uBlock-issues/discussions/2895
Changes:
The _content of the My filters_ pane is now considered untrusted by
default, and only uBO's own lists are now trusted by default.
It has been observed that too many people will readily copy-paste
filters from random sources. Copy-pasting filters which require trust
represents a security risk to users with no understanding of how the
filters work and their potential abuse.
Using a filter which requires trust in a filter list from an untrusted
source will cause the filter to be invalid, i.e. shown as an error.
A new advanced setting has been added to control which lists are
considered trustworthy: `trustedListPrefixes`, which is a space-
separated list of tokens. Examples of possible values:
- `ublock-`: trust only uBO lists, exclude everything else including
content of _My filters_ (default value)
- `ublock- user-`: trust uBO lists and content of _My filters_
- `-`: trust no list, essentially disabling all filters requiring
trust (admins or people who don't trust us may want to use this)
One can also decide to trust lists maintained elsewhere. For example,
for stock AdGuard lists add ` adguard-`. To trust stock EasyList lists,
add ` easylist-`.
To trust a specific regional stock list, look-up its token in
assets.json and add to `trustedListPrefixes`.
The matching is made with String.startsWith(), hence why `ublock-`
matches all uBO's own filter lists.
This also allows to trust imported lists, for example add
` https://filters.adtidy.org/extension/ublock/filters/` to trust all
non-stock AdGuard lists.
Add the complete URL of a given imported list to trust only that one
list.
URLs not starting with `https://` or `file:///` will be rejected,
i.e. `http://example.org` will be ignored.
Invalid URLs are rejected.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2896
TODO: Eventually, distinguish between filtering profile increasing
or decreasing so as to avoid flushing caches when increasing
filtering, which should not affect the scriptlets cache.
