***
New procedural cosmetic operator: `:remove()`
Related issue:
- https://github.com/gorhill/uBlock/issues/2252
The purpose is to outright remove elements from the
DOM tree. Since `:remove()` is an "action" operator,
it must only be used as a trailing operator (just
like the `:style()` operator).
AdGuard's cosmetic filter syntax `{ remove: true; }`
will be converted to uBO's `:remove()` operator
internally.
***
New procedural cosmetic operator: `:upward(...)`
The purpose is to lookup an ancestor element.
When used with an integer argument, it is synonym of
`:nth-ancestor()`, which will be deprecated and which
will no longer be supported once no longer used in
mainstream filter lists.
Filter lists maintainers must only use `:upward(int)`
instead of `:nth-ancestor(int)` once the new operator
become available in all stable releases of uBO.
`:upward()` can also accept a CSS selector as argument,
in which case the nearest ancestor which matches the
CSS selector will be selected.
Override the Backspace key, not just the Delete key, as Mac keyboards have Backspace as the only delete key and label it Delete.
Source of key value: https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/key/Key_Values#Editing_keys
More background on Mac keyboard shortcuts: Mac keyboards can emulate Delete using fn+Delete, meaning Forward Delete, but Mac software does not use it except in text editing. When deletion is dangerous, Mac software requires holding a modifier key in conjuction with Delete, but I think it’s better to make deletion easy in this case.
This new binding has a potential downside: if the user Backspace key normally goes Back in history (which can differ across OSs and browsers), this will change the behavior to delete the selected element instead. If the user really wants to go back in history, they will have to press Escape to leave the mode and then press Backspace, or they will have to press an alternative keyboard shortcuts such as Alt+Left. I think the user will rarely want to go back in history in the middle of picking an element, though.
That downside could be mitigated by conditioning the key check on `runtime.PlatformOs` (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/PlatformOs). But that would complicate the code a lot compared to the importance of this feature, and such detection would still fail to capture the user’s intent accurately in all cases. I think it’s better to unconditionally accept both Backspace (Delete) and Delete (Forward Delete).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/899
browser.storage.local is now used to store non-critical
local settings.
These settings are all collated under the key
`localStorage`, and vAPI.localStorage is an API to
handle access to these values stored under this key.
vAPI.localStorage.getItem() is still synchronous but
its purpose is to return internally cached values --
this minimizes code changes throughout uBO.
New advanced setting: `benchmarkDatasetURL`
Default value: `unset`
To specify a URL from where the benchmark dataset will be
fetched. This allows to launch benchmark operations from
within published versions of uBO, rather than from just
a locally built version.
A specific cosmetic filter of the following form...
*##.selector
... will be unconditionally injected into all web pages,
whereas a cosmetic filter of the form...
##.selector
... would be injected only when uBO's DOM surveyor finds
at least one matching element in a web page.
The new specific-generic form will also be disabled when a
web page is subject to a `generichide` exception filter,
since the filter is essentially a generic one -- the only
difference from the usual generic form is that the filter
is injected unconditionally instead of through the DOM
surveyor.
Specific-generic cosmetic filters will NOT be discarded
when checking the "Ignore generic cosmetic filters"
option in the "Filter lists" pane -- since the purpose
of this option is primarily to disable the DOM surveyor.
Specific-generic cosmetic filters should be used
parcimoniously and only when using a normal specific
filter is really impractical.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/803
Hostnames which have unpunycoded IDN or canonical
name information will have this information reported
as an extra line under the hostname name in the popup
panel, rather than as a tooltip.
For the fenix UI flavor, the save/revert tools are
now shown regardless of whether the overview area is
visible or not.
More specifically, this commit makes it so that
the new default UI for Firefox Preview will be
automatically selected.
A default platform-specific UI can be cancelled
by setting the advanced setting `uiFlavor` to `-`.
The toolip of a canonical name entry (blue)
in the popup panel will provide the alias
hostname related to the canonical hostname
in order to help with rule-making decisions.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/882
Related commits:
- https://github.com/gorhill/uBlock/commit/a95ef16e064a
- https://github.com/gorhill/uBlock/commit/7971b223855d
Leading wildcards before valid token characters need to
be kept in order to respect the semantic of the filter.
A leading wildcard in such case changes the semantic of
a filter, i.e. two following filters are semantically
different:
example/abc
*example/abc
As a result, µBlock.BidiTrieContainer.indexOf() is now
able to deal with a needle of length zero -- which is
what happens in FilterPatternLeft(Ex) with filter
patterns starting with `*` (or `^*`) and followed by
valid token characters (0-9, a-z and %).
First draft of changes as discussed with Firefox
Preview people.
In order to allow testing/evaluating these changes,
the new advanced setting `uiFlavor` has been added.
Default to `unset`; and can currently only be set
to `fenix`.
The new setting takes effect at launch only. This
new setting is not to be mentioned in official
documentation for now.
This is ongoing work, not open to external feedback.
This commit make it so that if an *unexpected* error is
thrown when trying to load the selfie at launch, the filter
lists will still be loaded as a result.
Tokens which are as long or longer than the max token
length possible do not need to have the prefix part
evaluated against special regex characters. This will
help increase the likelihood of extracting a valid
token from regex-based filters.
Actual case found in EasyPrivacy:
/^https?:\/\/eulerian..*\/[a-z0-9]{2,12}\.js/$script
Before this commit, uBO was not able to extract a
valid token, while now uBO is able to extract `eulerian`
as a valid token (consequently the regex-based filter
will now be evaluated only when the token `euleria` is
found in a URL).
This is related to the list of domains/subdomains in
the dynamic filtering pane of the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/284
Clicking on the empty space of a row will toggle
the visibility of the subdomains.
Additionally, the root context will always be visible
regardless of the expand/collspase state, along with
a visual indicator that a specific domain or subdomain
is the actual root context. (the root context is the
hostname in which local rules are created).
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/issues/805#issuecomment-561500819
uBO was testing for regex-like plain patterns after
prepending `@@` in the case of exception filters, thus
preventing proper detection of regex-like plain
patterns. The filtering engine was not affected, only
the proper rendering of the filter in the logger was
affected.
Advanced setting `cnameAliasList` has been removed.
New advanced settings:
cnameUncloak:
Boolean
Default value:
true
Description:
Whether to CNAME-uncloak hostnames.
cnameIgnoreExceptions:
Boolean
Default value:
true
Description:
Whether to bypass the uncloaking of network requests
which were excepted by filters/rules. This is
necessary so as to avoid undue breakage by having
exception filters being rendered useless as a result
of CNAME-uncloaking.
For example, `google-analytics.com` uncloaks to
`www-google-analytics.l.google.com` and both hostnames
appear in Peter Lowe's list, which means exception
filters for `google-analytics.com` (to fix site
breakage) would be rendered useless as the uncloaking
would cause the network request to be ultimately
blocked.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
Related commit:
- https://github.com/gorhill/uBlock/commit/3a564c199260
This adds two new advanced settings:
- cnameIgnoreRootDocument
- Default to `true`
- Tells uBO to skip CNAME-lookup for root document.
- cnameReplayFullURL
- Default to `false`
- Tells uBO whether to replay the whole URL or just
the origin part of it.
Replaying only the origin part is meant to lower
undue breakage and improve performance by avoiding
repeating the pattern-matching of the whole URL --
which pattern-matching was most likely already
accomplished with the original request.
This commit is meant to explore enabling CNAME-lookup
by default for the next stable release while:
- Eliminating a development burden by removing the
need to create a new filtering syntax to deal with
undesirable CNAME-cloaked hostnames
- Eliminating a filter list maintainer burden by
removing the need to find/deal with all base
domains which engage in undesirable CNAME-cloaked
hostnames
The hope is that the approach implemented in this
commit should require at most a few unbreak rules
with no further need for special filtering syntax
or filter list maintance efforts.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dzw57l/
Each token requires two slots in the token indices
array. This commit fixes uBO breaking when dealing
with very long URLs with lot of distinct tokens in
them.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
- Handle DNS lookup failure
- Skip DNS lookup for non network-based URLs
- Benchmark code to be able to provide an estimate
based on objective measurements regarding added
overhead when DNS lookup is enabled
(quick answer: a complete non-issue)
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
New webext permission added: `dns`, which purpose is
to allow an extension to fetch the DNS record of
specific hostnames, reference documentation:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/dns
The webext API `dns` is available in Firefox 60+ only.
The new API will enable uBO to "uncloak" the actual
hostname used in network requests. The ability is
currently disabled by default for now -- this is only
a first commit related to the above issue to allow
advanced users to immediately use the new ability.
Four advanced settings have been created to control the
uncloaking of actual hostnames:
cnameAliasList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to "uncloak" the hostnames in the list will.
cnameIgnoreList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to NOT re-run the network request through uBO's
filtering engine with the CNAME hostname. This is
useful to exclude commonly used actual hostnames
from being re-run through uBO's filtering engine, so
as to avoid pointless overhead.
cnameIgnore1stParty: boolean.
Default value: true.
Whether uBO should ignore to re-run a network request
through the filtering engine when the CNAME hostname
is 1st-party to the alias hostname.
cnameMaxTTL: number of minutes.
Default value: 120.
This tells uBO to clear its CNAME cache after the
specified time. For efficiency purpose, uBO will
cache alias=>CNAME associations for reuse so as
to reduce calls to `browser.dns.resolve`. All the
associations will be cleared after the specified time
to ensure the map does not grow too large and too
ensure uBO uses up to date CNAME information.
Related issue:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1588916
Due to browser behavior changes during development,
uBO can end up using obsolete cached compiled filter
lists or selfie.
Saving magic format values in the cache storage
itself ensure uBO will properly detect actually
obsolete compiled lists/selfie.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
As per various feedbacks:
Added an advanced setting to keep the original behavior,
which can be potentially costly CPU-wise on some sites:
popupCosmeticFilterBadgeSlow
Default to `false`. Set to `true` to restore original
method of surveying the number of elements hidden as
a result of applying cosmetic filtering.
As suggested by <https://github.com/gwarser>, skip
descendant of nodes which have been found to be a
match in order to potentially increase the number
of nodes which can be surveyed in the alloted time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
When trying the number of hidden elements as a result of
cosmetic filtering for popup panel badge purpose, the
code will bail out if this takes too long, and in such
case the badge will be set to `?`, meaning the number
of hidden elements is undetermined.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The code used to count expensive-to-survey particular
inline javascript occurrences will be executed if and
only if no other script tags have been found in the
DOM. This is acceptable since the badge count is to be
deemed approximate (as per documentation).
Related feedback:
- c090d2fde4 (commitcomment-35767596)
Mind that there might not be selectors to match as a
result of `generichide` or `no-cosmetic-filtering`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The badge value for the no-cosmetic-filtering
switch will be evaluated on-demand only, when
the user hover over the switch with the mouse
cursor.
For touch screen displays, a tap on the switch
will cause the badge to be rendered if not
already done, otherwise this will toggle the
switch as usual.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dpcvfx/
Regression from:
- 7971b22385
Leading/trailing wildcards are useless when a valid
token can be found, and in such case they need to
be removed so as to ensure the proper filter class
is used.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
This is the code used to find out the count values
displayed as badge on the cosmetic filtering and
scripting per-site switches in the popup panel.
The issue is that document.querySelector*() -- used to
find out the number of hidden elements -- is unduly
expensive on large DOM.
The changes in this commit have focused on avoiding the
use of document.querySelector*() as much as possible.
Also, the results are cached for reuse unless DOM
mutations are detected.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/759
This commit adds code to rely less on the state of the
cache storage to decide whether filter lists should be
re-compiled or whether the selfie is currently valid
at launch time when a change in compiled/selfie format
is detected.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/759
If for some reasons the launch code throws, uBO could
find itself in permanent suspend mode, thus preventing
web pages from loading in Firefox.
Though uBO should not have exceptions thrown during
it's initialization code, this commit will ensure
uBO complete its initialization process should it
ever happen for whatever reason.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/764
WebAssembly.Memory.grow() preserves the buffer content when
we grow it, no need to manually copy it. Doing so was
causing an access to a no longer valid ArrayBuffer.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
Changes related to above issue made it possible to
create WASM versions of methods used in the bidi-trie.
In this commit, WASM versions for startsWith(), indexOf()
and lastIndexOf() have been implemented.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dn9zso/
When a network request is tabless and contextless, i.e.
`tabId === -1` and `frameId === -1`, use the URL of
the network request as the context.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
- https://github.com/uBlockOrigin/uBlock-issues/issues/528
The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.
Example of filters that could be moved into a bidi-trie
data structure:
&ad_box_
/w/d/capu.php?z=$script,third-party
||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif
Examples of filters that could NOT be moved to a bidi-trie:
-adap.$domain=~l-adap.org
/tsc.php?*&ses=
||ibsrv.net/*forumsponsor$domain=[...]
@@||imgspice.com/jquery.cookie.js|$script
||view.atdmt.com^*/iview/$third-party
||postimg.cc/image/$csp=[...]
Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).
Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.
This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.
- Added ability to associate a pattern with an integer
in the bidi-trie [1].
- The bidi-trie match code passes this externally
provided integer when calling an externally
provided method used for testing extra conditions
that may be present for a plain pattern found to
be matching in the bidi-trie.
- Decomposed existing filters into smaller logical units:
- FilterPlainLeftAnchored =>
FilterPatternPlain +
FilterAnchorLeft
- FilterPlainRightAnchored =>
FilterPatternPlain +
FilterAnchorRight
- FilterExactMatch =>
FilterPatternPlain +
FilterAnchorLeft +
FilterAnchorRight
- FilterPlainHnAnchored =>
FilterPatternPlain +
FilterAnchorHn
- FilterWildcard1 =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
]
- FilterWildcard1HnAnchored =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
] +
FilterAnchorHn
- FilterGenericHnAnchored =>
FilterPatternGeneric +
FilterAnchorHn
- FilterGenericHnAndRightAnchored =>
FilterPatternGeneric +
FilterAnchorRight +
FilterAnchorHn
- FilterOriginMixedSet =>
FilterOriginMissSet +
FilterOriginHitSet
- Instances of FilterOrigin[...], FilterDataHolder
can also be added to a composite filter to
represent `domain=` and `csp=` options.
- Added a new filter class, FilterComposite, for
filters which are a combination of two or more
logical units. A FilterComposite instance is a
match when *all* filters composing it are a
match.
Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.
Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.
Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.
The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.
Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.
A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.
This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.
As a result of the above changes, the map of `token`
(integer) => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.
Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.
`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.
New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.
All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.
Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.
FilterParser is now instanciated on demand and
released when no longer used.
***
[1] 135a45a878/src/js/strie.js (L120)
[2] e94024d350
[3] 135a45a878/src/js/static-net-filtering.js (L1630)
[4] 135a45a878/src/js/static-net-filtering.js (L1566)
The readTime property is unused for compiled or selfie
assets, and thus writing back to storage to persist
this property is useless for those assets, and an undue
overhead when reading such assets, especially at launch
time.
Assets are always loaded from their compiled or selfie
counterparts at launch.
This commit will prevent those unnecessary storage
write operations.
Related issue:
- https://github.com/gorhill/uBlock/issues/127
Procedural cosmetic exception filters were the
last class of cosmetic exception filters not
being reported in the logger; this commit fixes
this.
Additionally, ensure that a single DOM listener
can't prevent other listeners from being
processed by throwing an exception. Such approach
would have prevented regression leading to
emergency release 1.22.4:
- https://github.com/gorhill/uBlock/releases/tag/1.22.4
In rare circumstances, it's possible the content
script lose access to the background page, best
to check against this to avoid spurious console
errors.
Using pairs of integers allows the use of a
single integer-only array to store lists of
string indices associated to a specific
hostname.
Memory usage of instances of HostnameBasedDB
as per Chromium's heap snaphshot (bytes):
Before:
2,459,256 => specific cosmetic filters
944,152 => scriptlet filtering
736
---------
3,404,144
After:
1,947,448 => "
757,936 => "
632
---------
2,706,016
Ultimately, using 2 integers for each entry
instead of a single one is still worth it
because this allows the use of one single
integer-only array instead of having to use
an array of arrays for hostnames which have
multiple entries.
This commits make it so that `csp=` filters
are now stored in the same data structures as
all other static network filters rather than
being stored in a separate one.
This internal change is motivated by the wish
to bring session filters to the static network
filtering engine, as has already been done for
the static extended filtering engine in the
following commit:
59c9a34d34
Related commit:
- 87d0e456f1
Ensure that the code which depends on extending
`vapi-client.js` is ready to deal with
`vapi-client-extra.js` failing to load.
This is a feature under development, hidden behind
a new advanced setting, `filterAuthorMode` which
default to `false`.
Ability to point-and-click to create temporary
exception filters for static extended filters (i.e.
cosmetic, scriptlet & html filters) from within
the summary pane in the logger. The button to
toggle on/off temporary exception filter is
labeled `#@#`.
The created exceptions are temporary and will be
lost when restarting uBO, or manually toggling off
the exception filters.
Creating temporary exception filters does not
cause the filter lists to reloaded, and thus there
is no overhead in creating/removing these temporary
exception filters.
