Hostnames which have unpunycoded IDN or canonical
name information will have this information reported
as an extra line under the hostname name in the popup
panel, rather than as a tooltip.
For the fenix UI flavor, the save/revert tools are
now shown regardless of whether the overview area is
visible or not.
More specifically, this commit makes it so that
the new default UI for Firefox Preview will be
automatically selected.
A default platform-specific UI can be cancelled
by setting the advanced setting `uiFlavor` to `-`.
The toolip of a canonical name entry (blue)
in the popup panel will provide the alias
hostname related to the canonical hostname
in order to help with rule-making decisions.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/882
Related commits:
- https://github.com/gorhill/uBlock/commit/a95ef16e064a
- https://github.com/gorhill/uBlock/commit/7971b223855d
Leading wildcards before valid token characters need to
be kept in order to respect the semantic of the filter.
A leading wildcard in such case changes the semantic of
a filter, i.e. two following filters are semantically
different:
example/abc
*example/abc
As a result, µBlock.BidiTrieContainer.indexOf() is now
able to deal with a needle of length zero -- which is
what happens in FilterPatternLeft(Ex) with filter
patterns starting with `*` (or `^*`) and followed by
valid token characters (0-9, a-z and %).
First draft of changes as discussed with Firefox
Preview people.
In order to allow testing/evaluating these changes,
the new advanced setting `uiFlavor` has been added.
Default to `unset`; and can currently only be set
to `fenix`.
The new setting takes effect at launch only. This
new setting is not to be mentioned in official
documentation for now.
This is ongoing work, not open to external feedback.
This commit make it so that if an *unexpected* error is
thrown when trying to load the selfie at launch, the filter
lists will still be loaded as a result.
Tokens which are as long or longer than the max token
length possible do not need to have the prefix part
evaluated against special regex characters. This will
help increase the likelihood of extracting a valid
token from regex-based filters.
Actual case found in EasyPrivacy:
/^https?:\/\/eulerian..*\/[a-z0-9]{2,12}\.js/$script
Before this commit, uBO was not able to extract a
valid token, while now uBO is able to extract `eulerian`
as a valid token (consequently the regex-based filter
will now be evaluated only when the token `euleria` is
found in a URL).
This is related to the list of domains/subdomains in
the dynamic filtering pane of the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/284
Clicking on the empty space of a row will toggle
the visibility of the subdomains.
Additionally, the root context will always be visible
regardless of the expand/collspase state, along with
a visual indicator that a specific domain or subdomain
is the actual root context. (the root context is the
hostname in which local rules are created).
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/issues/805#issuecomment-561500819
uBO was testing for regex-like plain patterns after
prepending `@@` in the case of exception filters, thus
preventing proper detection of regex-like plain
patterns. The filtering engine was not affected, only
the proper rendering of the filter in the logger was
affected.
Advanced setting `cnameAliasList` has been removed.
New advanced settings:
cnameUncloak:
Boolean
Default value:
true
Description:
Whether to CNAME-uncloak hostnames.
cnameIgnoreExceptions:
Boolean
Default value:
true
Description:
Whether to bypass the uncloaking of network requests
which were excepted by filters/rules. This is
necessary so as to avoid undue breakage by having
exception filters being rendered useless as a result
of CNAME-uncloaking.
For example, `google-analytics.com` uncloaks to
`www-google-analytics.l.google.com` and both hostnames
appear in Peter Lowe's list, which means exception
filters for `google-analytics.com` (to fix site
breakage) would be rendered useless as the uncloaking
would cause the network request to be ultimately
blocked.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
Related commit:
- https://github.com/gorhill/uBlock/commit/3a564c199260
This adds two new advanced settings:
- cnameIgnoreRootDocument
- Default to `true`
- Tells uBO to skip CNAME-lookup for root document.
- cnameReplayFullURL
- Default to `false`
- Tells uBO whether to replay the whole URL or just
the origin part of it.
Replaying only the origin part is meant to lower
undue breakage and improve performance by avoiding
repeating the pattern-matching of the whole URL --
which pattern-matching was most likely already
accomplished with the original request.
This commit is meant to explore enabling CNAME-lookup
by default for the next stable release while:
- Eliminating a development burden by removing the
need to create a new filtering syntax to deal with
undesirable CNAME-cloaked hostnames
- Eliminating a filter list maintainer burden by
removing the need to find/deal with all base
domains which engage in undesirable CNAME-cloaked
hostnames
The hope is that the approach implemented in this
commit should require at most a few unbreak rules
with no further need for special filtering syntax
or filter list maintance efforts.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dzw57l/
Each token requires two slots in the token indices
array. This commit fixes uBO breaking when dealing
with very long URLs with lot of distinct tokens in
them.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
- Handle DNS lookup failure
- Skip DNS lookup for non network-based URLs
- Benchmark code to be able to provide an estimate
based on objective measurements regarding added
overhead when DNS lookup is enabled
(quick answer: a complete non-issue)
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
New webext permission added: `dns`, which purpose is
to allow an extension to fetch the DNS record of
specific hostnames, reference documentation:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/dns
The webext API `dns` is available in Firefox 60+ only.
The new API will enable uBO to "uncloak" the actual
hostname used in network requests. The ability is
currently disabled by default for now -- this is only
a first commit related to the above issue to allow
advanced users to immediately use the new ability.
Four advanced settings have been created to control the
uncloaking of actual hostnames:
cnameAliasList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to "uncloak" the hostnames in the list will.
cnameIgnoreList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to NOT re-run the network request through uBO's
filtering engine with the CNAME hostname. This is
useful to exclude commonly used actual hostnames
from being re-run through uBO's filtering engine, so
as to avoid pointless overhead.
cnameIgnore1stParty: boolean.
Default value: true.
Whether uBO should ignore to re-run a network request
through the filtering engine when the CNAME hostname
is 1st-party to the alias hostname.
cnameMaxTTL: number of minutes.
Default value: 120.
This tells uBO to clear its CNAME cache after the
specified time. For efficiency purpose, uBO will
cache alias=>CNAME associations for reuse so as
to reduce calls to `browser.dns.resolve`. All the
associations will be cleared after the specified time
to ensure the map does not grow too large and too
ensure uBO uses up to date CNAME information.
Related issue:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1588916
Due to browser behavior changes during development,
uBO can end up using obsolete cached compiled filter
lists or selfie.
Saving magic format values in the cache storage
itself ensure uBO will properly detect actually
obsolete compiled lists/selfie.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
As per various feedbacks:
Added an advanced setting to keep the original behavior,
which can be potentially costly CPU-wise on some sites:
popupCosmeticFilterBadgeSlow
Default to `false`. Set to `true` to restore original
method of surveying the number of elements hidden as
a result of applying cosmetic filtering.
As suggested by <https://github.com/gwarser>, skip
descendant of nodes which have been found to be a
match in order to potentially increase the number
of nodes which can be surveyed in the alloted time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
When trying the number of hidden elements as a result of
cosmetic filtering for popup panel badge purpose, the
code will bail out if this takes too long, and in such
case the badge will be set to `?`, meaning the number
of hidden elements is undetermined.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The code used to count expensive-to-survey particular
inline javascript occurrences will be executed if and
only if no other script tags have been found in the
DOM. This is acceptable since the badge count is to be
deemed approximate (as per documentation).
Related feedback:
- c090d2fde4 (commitcomment-35767596)
Mind that there might not be selectors to match as a
result of `generichide` or `no-cosmetic-filtering`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The badge value for the no-cosmetic-filtering
switch will be evaluated on-demand only, when
the user hover over the switch with the mouse
cursor.
For touch screen displays, a tap on the switch
will cause the badge to be rendered if not
already done, otherwise this will toggle the
switch as usual.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dpcvfx/
Regression from:
- 7971b22385
Leading/trailing wildcards are useless when a valid
token can be found, and in such case they need to
be removed so as to ensure the proper filter class
is used.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
This is the code used to find out the count values
displayed as badge on the cosmetic filtering and
scripting per-site switches in the popup panel.
The issue is that document.querySelector*() -- used to
find out the number of hidden elements -- is unduly
expensive on large DOM.
The changes in this commit have focused on avoiding the
use of document.querySelector*() as much as possible.
Also, the results are cached for reuse unless DOM
mutations are detected.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/759
This commit adds code to rely less on the state of the
cache storage to decide whether filter lists should be
re-compiled or whether the selfie is currently valid
at launch time when a change in compiled/selfie format
is detected.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/759
If for some reasons the launch code throws, uBO could
find itself in permanent suspend mode, thus preventing
web pages from loading in Firefox.
Though uBO should not have exceptions thrown during
it's initialization code, this commit will ensure
uBO complete its initialization process should it
ever happen for whatever reason.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/764
WebAssembly.Memory.grow() preserves the buffer content when
we grow it, no need to manually copy it. Doing so was
causing an access to a no longer valid ArrayBuffer.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
Changes related to above issue made it possible to
create WASM versions of methods used in the bidi-trie.
In this commit, WASM versions for startsWith(), indexOf()
and lastIndexOf() have been implemented.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dn9zso/
When a network request is tabless and contextless, i.e.
`tabId === -1` and `frameId === -1`, use the URL of
the network request as the context.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
- https://github.com/uBlockOrigin/uBlock-issues/issues/528
The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.
Example of filters that could be moved into a bidi-trie
data structure:
&ad_box_
/w/d/capu.php?z=$script,third-party
||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif
Examples of filters that could NOT be moved to a bidi-trie:
-adap.$domain=~l-adap.org
/tsc.php?*&ses=
||ibsrv.net/*forumsponsor$domain=[...]
@@||imgspice.com/jquery.cookie.js|$script
||view.atdmt.com^*/iview/$third-party
||postimg.cc/image/$csp=[...]
Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).
Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.
This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.
- Added ability to associate a pattern with an integer
in the bidi-trie [1].
- The bidi-trie match code passes this externally
provided integer when calling an externally
provided method used for testing extra conditions
that may be present for a plain pattern found to
be matching in the bidi-trie.
- Decomposed existing filters into smaller logical units:
- FilterPlainLeftAnchored =>
FilterPatternPlain +
FilterAnchorLeft
- FilterPlainRightAnchored =>
FilterPatternPlain +
FilterAnchorRight
- FilterExactMatch =>
FilterPatternPlain +
FilterAnchorLeft +
FilterAnchorRight
- FilterPlainHnAnchored =>
FilterPatternPlain +
FilterAnchorHn
- FilterWildcard1 =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
]
- FilterWildcard1HnAnchored =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
] +
FilterAnchorHn
- FilterGenericHnAnchored =>
FilterPatternGeneric +
FilterAnchorHn
- FilterGenericHnAndRightAnchored =>
FilterPatternGeneric +
FilterAnchorRight +
FilterAnchorHn
- FilterOriginMixedSet =>
FilterOriginMissSet +
FilterOriginHitSet
- Instances of FilterOrigin[...], FilterDataHolder
can also be added to a composite filter to
represent `domain=` and `csp=` options.
- Added a new filter class, FilterComposite, for
filters which are a combination of two or more
logical units. A FilterComposite instance is a
match when *all* filters composing it are a
match.
Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.
Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.
Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.
The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.
Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.
A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.
This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.
As a result of the above changes, the map of `token`
(integer) => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.
Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.
`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.
New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.
All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.
Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.
FilterParser is now instanciated on demand and
released when no longer used.
***
[1] 135a45a878/src/js/strie.js (L120)
[2] e94024d350
[3] 135a45a878/src/js/static-net-filtering.js (L1630)
[4] 135a45a878/src/js/static-net-filtering.js (L1566)
The readTime property is unused for compiled or selfie
assets, and thus writing back to storage to persist
this property is useless for those assets, and an undue
overhead when reading such assets, especially at launch
time.
Assets are always loaded from their compiled or selfie
counterparts at launch.
This commit will prevent those unnecessary storage
write operations.
Related issue:
- https://github.com/gorhill/uBlock/issues/127
Procedural cosmetic exception filters were the
last class of cosmetic exception filters not
being reported in the logger; this commit fixes
this.
Additionally, ensure that a single DOM listener
can't prevent other listeners from being
processed by throwing an exception. Such approach
would have prevented regression leading to
emergency release 1.22.4:
- https://github.com/gorhill/uBlock/releases/tag/1.22.4
In rare circumstances, it's possible the content
script lose access to the background page, best
to check against this to avoid spurious console
errors.
Using pairs of integers allows the use of a
single integer-only array to store lists of
string indices associated to a specific
hostname.
Memory usage of instances of HostnameBasedDB
as per Chromium's heap snaphshot (bytes):
Before:
2,459,256 => specific cosmetic filters
944,152 => scriptlet filtering
736
---------
3,404,144
After:
1,947,448 => "
757,936 => "
632
---------
2,706,016
Ultimately, using 2 integers for each entry
instead of a single one is still worth it
because this allows the use of one single
integer-only array instead of having to use
an array of arrays for hostnames which have
multiple entries.
This commits make it so that `csp=` filters
are now stored in the same data structures as
all other static network filters rather than
being stored in a separate one.
This internal change is motivated by the wish
to bring session filters to the static network
filtering engine, as has already been done for
the static extended filtering engine in the
following commit:
59c9a34d34
Related commit:
- 87d0e456f1
Ensure that the code which depends on extending
`vapi-client.js` is ready to deal with
`vapi-client-extra.js` failing to load.
This is a feature under development, hidden behind
a new advanced setting, `filterAuthorMode` which
default to `false`.
Ability to point-and-click to create temporary
exception filters for static extended filters (i.e.
cosmetic, scriptlet & html filters) from within
the summary pane in the logger. The button to
toggle on/off temporary exception filter is
labeled `#@#`.
The created exceptions are temporary and will be
lost when restarting uBO, or manually toggling off
the exception filters.
Creating temporary exception filters does not
cause the filter lists to reloaded, and thus there
is no overhead in creating/removing these temporary
exception filters.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/d6zbqv/
For static filter `popup` filter purpose, the URL of the
embedded frame from which the popup was launched will
be used in the matching algorithm.
As reported by https://github.com/uBlock-user,
adding the two following filters:
example.com##div:style(width: 1016px !important)
example.com##div:style(display: block !important)
The logger would report only one of the filter when
navigating to https://example.com/
While testing for the new `elemhide` option using ABP's
test page[1], I found out that the placeholder of the
blocked image on that page was not properly collapsed
by uBO.
The reason was because the page is very simple and
flushing the blocked-elements cache at
webNavigation.onCommitted time was causing the loss
of information collected between
webRequest.onBeforeRequest and
webNavigation.onCommitted, preventing uBO from
properly collapsing the placeholders.
The blocked-elements cache is now flushed
ONLY at webRequest.onBeforeRequest time.
[1] https://testpages.adblockplus.org/en/exceptions/elemhide
... from platform-independent code.
This should complete the de-chromiumification of
uBO. Next step will be to swap the content of the
`platform/chromium` and `platform/firefox`
folders so as to reflect that Firefox is the
natural platform for uBO.
Related documentation:
- https://help.eyeo.com/en/adblockplus/how-to-write-filters#element-hiding
Related feedback/discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/d6vxzj/
The `elemhide` filter option as per ABP semantic is
now supported. Previously uBO would consider `elemhide`
to be an alias of `generichide`.
The support of `elemhide` is through the convenient
conversion of `elemhide` option into existing
`generichide` option and new `specifichide` option.
The purpose of the new `specifichide` filter option
is to disable all specific cosmetic filters, i.e.
those who target a specific site.
Additionally, for convenience purpose, the filter
options `generichide`, `specifichide` and `elemhide`
can be aliased using the shorter forms `ghide`,
`shide` and `ehide` respectively.
Little-used code from vapi-client.js has been moved
to vapi-client-extra.js. Given that vapi-client.js
is injected in all web pages, this means less dead
code being injected in all pages.
Swathes of code in vapi-client.js was used only in
a few very specific cases, such as when the logger's
DOM inspector is opened or when the "Filter lists"
pane in the dashboard is opened -- and thus to avoid
that little used code to be loaded in every web page
unconditionally, it has been moved to its own
separate file, vapi-client.extra.js.
vapi-client-extra.js is loaded declaratively or
programmatically only where needed.
No need to store mouse coordinates in background
page, thus no need to post mouse coordinates
information for every click.
Rename/group element picker arguments and popup
arguments separately.
Reported by:
- https://github.com/uBlock-user:
Imported custom list were incorrectly seen as out of
date immediately after import operation.
Regression from:
- e27328f931
A few lines of code were improperly removed during
refactoring.
Also, coallesce calls to selfieManager.destroy() so as
to avoid undue repeated calls to storage deletion of
selfie assets.
Related commit:
- e27328f931
Related commit:
- e27328f931
The regression was preventing the compiled filter
lists from being properly loaded by uBO, thus
always causing a full parsing/compiling at
launch time.
Issue reported by @uBlock-user in team channel.
Creating cosmetic procedural exception filters was
causing `cosmetic-logger.js` scriptlet to throw at
and thus further breaking the logging of cosmetic
filters overall.
Reuse permanent instances instead. The trailing `$` is
used to denote these variables are register-like
instances, i.e. their content is valid only for the
duration of the call. (From now on I will use this
convention throughout the code base.)
Entity-based filters where not properly looked-up if
they used subdomains. Example:
- `example.*##^script` => ok
- `www.example.*##^script` => failed on `https://www.example.com/`
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/710
Messages from unprivileged ports (i.e. from content scripts)
are no longer relayed to message handlers which are to be
strictly used to execute privileged code.
The last remaining case of unprivileged messages which
should be converted into a privileged ones will be taken
care of when the following issue is fixed:
- https://github.com/gorhill/uBlock/issues/3497
Related commit:
- fb4e94f92c
A bidi-trie can't store strings longer than 255 characters
because the string segment lengths are encoded into a single
byte. This commit ensures only strings smaller than
256 characters are stored in the bidi-tries.
Plain filters can be any length, while the bidi-trie was
assuming max length of 256. The origin of this error
is from when the bidi-trie code was originally imported
from the hntrie code as start to develop bidi-trie.
The erroneous code could cause issue when the following
conditions were met:
- Plain filter longer than 256 characters
- Free space in bidi-trie's character buffer was less
than 256 bytes
Likely a rare occurrence, but some filter lists do contains
long plain filters, for example:
https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter.txt
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/cs1y26/
The purpose is to wholly disable scriptlet injection
for a given site without having to create exceptions
for all matching scriptlet injection filters.
The following exception filter will cause scriptlet
injection to be wholly disable for `example.com`:
`example.com#@#+js()`
Or to disable scriptlet injection everywhere:
`#@#+js()`
The following form is meaningless and will be
ignored:
`example.com##+js()`
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/701
The filter option `empty` is converted to `redirect=empty`
by uBO internally; however unlike when the `redirect=`
option is used expressly, the `empty` option does not
require a resource type.
When `empty` is used, only network requests which are meant
to return a text response will be redirected to an empty
response body by uBO -- so `empty` will not work for
resources such as images, media, or other binary resources.
Related commit & feedback:
- 7ff750eaf6
The color value for the icon badge is now
"attached" to the blocking profile value.
Additionally, as per feedback, `3p` rules
will be relaxing before master JavaScript
switch rules.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/cmh910/
Additionally, the `3p` rule has been made distinct from
`3p-script`/`3p-frame` for the purpose of
"Relax blocking mode" command.
The badge color will hint at the current blocking mode.
There are four colors for the four following blocking
modes:
- JavaScript wholly disabled
- All 3rd parties blocked
- 3rd-party scripts and frames blocked
- None of the above
The default badge color will be used when JavaScript is not
wholly disabled and when there are no rules for `3p`,
`3p-script` or `3p-frame`.
A new advanced setting has been added to let the user choose
the badge colors for the various blocking modes,
`blockingProfileColors`. The value *must* be a sequence of
4 valid CSS color values that match 6 hexadecimal digits
prefixed with`#` -- anything else will be ignored.
The ability to redirect xmlhttprequest to binary
resources was lost when redirectable/injectable
resources became immutable in commit
152cea2dfe.
This commit restores the ability to redirect a
xmlhttprequest to a binary resource by making
it possible to derive a data: URI from the
content of binary resources such as images.
Addtionally a redirect to a data: URI can be
forced by prefixing the resource token with `%`.
This is a non-official feature at this point,
i.e. it could be removed at any time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/310
The purpose of this new option is to add the ability
to create standalone redirect rule without being forced
to create a block filter (a corresponding block filter
is always created when using the `redirect=`).
Additionally:
The syntax `*$redirect=token,...` is now supported, there
is no need to "trick" the filter parser with
`*/*$redirect=token,...` in order to create redirect rules
which are meant to match all paths.
Filters of the form `|http*://` will be normalized into
two corresponding filters `|https://` and `|http://` so as
to reduce the number of filters in the buckets of
untokenizable filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/682#issuecomment-515197130
The following advanced setting has been added:
updateAssetBypassBrowserCache
Default to `false`. If set to `true`, uBO will ensure the
browser cache is bypassed when fetching a remote resource.
This is for the convenience of filter list maintainers who
may want to test the latest version of their lists when
fetched from their remote location.
Related feedback:
- a54cb2e38f (commitcomment-34387041)
Regression from converting uBO resources into
immutable resources. This affects only Chromium-based
browsers.
This is a quick fix for the dev build, to at least unbreak
the reported case.
I need to research whether the root issue (conflict with HTTPS-E)
is still occurring, and if so the fix is not trivial: I will need
to add code to uBO to fetch and convert binary data into `data:`
URIs.
Theoretically, if a selfie is loaded from corrupted
storage resulting in a thrown exception from the
constructor, this would cause an improperly initialized
HostnameBasedDB instance and overall potentially a
broken uBO.
Related feedback:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1543880#c7
Since https://github.com/uBlockOrigin/uBlock-issues/issues/156
won't be fixed in next release, no need to ship
with code which will be unused, and anyways only once
the fix is worked on will it be clear exactly what needs
to be used by scriptlets to deal harmoniously with
property listener collisions.
Make sure the parser is safely compatible with old
resources format -- for those users still using
custom resources (via `userResourcesLocation`).
Prepare code for future fix to
<https://github.com/uBlockOrigin/uBlock-issues/issues/156>:
This commit introduces a new private Map() object,
`uBOSafe`, accessible by all injected scriptlets. This
private safe can be used to store data which can be shared
with different scriptlets. The idea is for scriptlets to
use that safe to graciously deal with the need to install
multiple listeners for the same property.
Since resources are now immutable, by default they are
only compiled once each time uBO updates to a new
version. However I need a way to force a re-compiling
of the resource in the dev build. This commit adds code
to invalidate the resources selfie when forcing the
update of any filter list.
With hindsight, I revised decisions made earlier during
this development cycle:
Un-redirectable scriptlets have been removed from
/web_accessible_resources and instead put in the new
/assets/resources/scriptlets.js, which contains all
scriptlets used for web page injection purpose.
uBO will no longer fetch a remote version of built-in
resources.
Advanced setting `userResourcesLocation` will still be
honoured by uBO, and if set, will be fetched every
time at least one asset is updated.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/651
The `behnind-the-scene` context was wrongly used to
evaluate the whitelist status of the context of tabless
network requests. The document origin must be used
instead when it's available.
Additionally, much code has been revisited for better
ES6 syntax compliance.
The goal is to be able to specify a scriptlet token
without the `.js` part at the end, because that part
is essentially redundant with the `+js` part of
the syntax.
When the next stable release is in widespread use
(to determine), scriptlet tokens will have to be
specified without the `.js` part, and with this
commit the logger will already report the normalized
version of scriptlets.
Eventually, when the migration to sans-`.js` is
complete (also to determine), the internal
normalization of the token will be removed and this
will become official syntax.
Filter list maintainers will have to mind that
uAssets is becoming in use beyond uBO (i.e. Brave)
when skipping the `.js` part -- hopefully Brave will
go along with the change here, which is to remove a
bit of tediousness for filter list maintainers.
This is a first step, the ultimate goal is to remove
the need for resources.txt, or at least to reduce to
only hotfixes or for trivial resources targeting very
specific websites.
Most resources will become immutable, i.e. they will
be part of uBO's code base. Advantages include easier
code maintenance (jshint, syntax highlight), and to
make scriptlets more easy to code review by external
parties (for example extension store reviewers).
TODO:
- More scriptlets need to be imported before next
release.
- Need to make legacy versions of uBO use a legacy
version of resources.txt, as all the now obsolete
scriptlets will have to be removed once uBO's
next release become widespread.
- Possibly need to add code to load binary
resources so that they can be injected as
data: URI. So far it's unclear whether this is
really needed. For example, this would be needed
if a xmlhttprequest is redirected to an image
resource.
This works only for platforms supporting the return of
Promise by network listeners, i.e. only Firefox at this
point.
When filter lists are reloaded[1], there is a small
time window in which some network requests which should
have normally been blocked are not being blocked
because the static network filtering engine may not
have yet loaded all the filters in memory
This is now addressed by suspending the network request
handler when filter lists are reloaded -- again, this
works only on supported platforms.
[1] Examples: when a filter list update session
completes; when user filters change, when
adding/removing filter lists.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/c5do7w/
Make the element picker better reflect network filters as
parsed by the static network filtering engine. Additionally,
discard single alphanumeric character-based filters.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/c62irc/
Inject newly created cosmetic filters into the DOM
filterer, in order for these filters to be enforced by
the DOM filterer in subsequent dynamic DOM changes.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/371
By default, no specific keyboard shortcut is predefined,
this will have to be assigned by the user. The command
name in English is "Toggle blocking profile".
The default behavior is to toggle down according to one
of the following scenarios.
a) If script execution is disabled through the no-scripting
switch, the no-scripting switch will be locally toggled
so as to allow script execution. The page will be
automatically reloaded.
b) If script execution is not blocked but the 3rd-party
script and/or frame cells are blocked, local no-op rules
will be set so as to no longer block 3rd-party scripts
and/or frames. The page will be automatically reloaded.
Given this, it may take more than one toggle down command
to reach the lowest blocking profile, which is one where
JavaScript execution is not blocked and 3rd-party scripts
and frames resources block rules, if any, are bypassed
with local no-op rules.
TODO: At this point, I haven't yet decided whether
toggling from the lowest profile should restore the
original highest blocking profile.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/c5do7w/
Fixed:
- Expect the differ can return the first input as is when
there is no difference between the two items.
- Better deal with extraneous whitespaces in `srcset`
The purpose is to avoid having to iterate through
all input nodes at each operator implementation
level. The `transpose` method deals with only one
input node, and the iteration is performed by the
main procedural filtering entry points.
Additionally:
- Add `:spath` to HTML filtering
- Rename `:watch-attrs` to `:watch-attr`
- `:watch=attrs` is deprecated and will be kept around
until it is safe to remove it completely
Where `x` is the minimal text length of the subject
DOM element. DOM elements whose text length is
greater than or equal to `x` will be selected.
The original rationale for such procedural cosmetic
operator[1] is to be able to remove inline script
elements according to a minimum text length using
HTML filtering.
[1] As a result of internal discussion with filter
list maintainers @ uAssets.
- Remove HNTrieContainer class from global context by
storing it as a property of µBlock.
- Use block scope to isolate HNTrie-related constants
from global context.
- Prevent filters which are pure IP address from
being stored in an HNTrie instance -- as this
could cause false positives.
The bidirectional trie allows storing the right
and left parts of a string into a trie given a
pivot position.
Releated issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/528
Additionally, the mandatory token-at-index-0 rule
for FilterPlainHnAnchored has been lifted, thus
allowing the engine to pick a potentially better token
at any position in the filter string.
***
TODO: Eventually rename `strie.js` to `biditrie.js`.
TODO: Fix dump() method, it currently only show the
right-hand side of a filter string.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/610
The service worker-related issue affects both
Chromium/Firefox: the type of resources fetched
from a service worker are uniformly set to
`xmlhttprequest`, hence losing a key piece of
information for the purpose of accurate content
filtering.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/591
Additionally, I added a link to the logger in the
"About" pane in the dashboard in order to be able
to access the logger without having to go through
the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/983
- https://github.com/gorhill/uBlock/issues/1353
The current implementation reports statistics for all
static filters, and the presentation/featureset is
intentionally minimal: *Do not open issues about this.*
It's still a work in progress and it will be worked on
slowly and thoughtfully over time and as time allows.
Pausing the logger will not pause the collation of
filter hit statistics, thus it is possible to lower
the logger overhead by pausing logger output without
losing filter hit collation.
The motivations for the re-arrangement:
- Reducing the number of entry points:
matchStringExactString() has been removed and
matchString() is simply reused with a modifier parameter
to enable matching variants.
- Presumption that most matches, if any, occur early with
the left-most tokens in a URL. This gives a very small
marginal performance gain as per built-in benchmark.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/bqnsoa/
The `all` option is equivalent to specifying all
network-based types + `popup`, `document`,
`inline-font`, `inline-script`.
Example from discussion:
||bet365.com^$all
Above will block all network requests, block all popups,
prevent inline fonts/scripts from `bet365.com`. EasyList-
compatible syntax does not allow to accomplish that
semantic when using only `||bet365.com^`.
If using specific negated type options along with `all`,
the order in which the options appear is important. In
such case `all` should always be first, followed by
the negated type option(s).
Related issue:
- https://github.com/gorhill/uBlock/issues/2394
Additionally, I added a new advanced setting to control
how long after launch an auto-update session should be
started -- value is in seconds:
autoUpdateDelayAfterLaunch 180
Related issue:
- https://github.com/gorhill/uBlock/issues/3271
When navigating away by clicking another pane tab button,
there will be an embedded warning, which can be ignore
in order to proceed to the new pane, or dismissed by
either clicking on the "Stay" button or anywhere else
in the dashboard.
When navigating away by trying to close the tab, there will
be a built-in browser warning asking for confirmation.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/131
The new advanced setting and its default value is:
allowGenericProceduralFilters false
Whenever this setting is toggled, the user is responsible
of forcing a reload of all filter lists so as to allow uBO
to process differently any existing generic procedural
cosmetic filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/214
Built-in whitelist directives are now rendered differently
than user-defined whitelist directives. Also, removing a
built-in whitelist directive will only cause that directive
to be commented out, so that users do not have to remember
built-in directives should they want to bring them back.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/494
The built-in per-site switch rule
`no-scripting: behind-the-scene false` has been removed,
it should not ever be needed since there will always be a
valid root context for main- and sub-frames.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/551
The issue fixes previewing the hiding/unhiding of targeted
elements in the element picker.
However it does not address the case of previewing
`:style(...)` operators -- this would require a much
more complex fix, which I am not sure is worth the
amount of work and increased code complexity.
Related issue:
- https://github.com/gorhill/uBlock/issues/127
Additionally, the extended exception filters in the
logger will be rendered with a line-through to more
easily distinguish them from non-exception ones.
Also, opportunistically converted revisited code to
ES6 syntax.
This was a TODO item:
- 07cbae66a4/src/js/cosmetic-filtering.js (L375)
µBlock.staticExtFilteringEngine.HostnameBasedDB has been
re-factored to accomodate the storing of specific cosmetic
filters.
As a result of this refactoring:
- Memory usage has been further decreased
- Performance of selector retrieval marginally
improved
- New internal representation opens the door
to use a specialized version of HNTrie, which
should further improve performance/memory
usage
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/572
Wildcards are now allowed in the hostname part of redirect
filters. There will be an attempt to find the longest
right-hand portion of the hostname with no wildcard. If
no non-empty hostname can be extracted, `*` will be used.
To be used at the console, as an investigation tool for
development purpose.
Using it to verify the content of the largest
FilterHostnameDict instance, I spotted an all-uppercase
hostname in the HNTrieRef instance:
µBlock.staticNetFilteringEngine.categories.get(0).get(0x10000000).dict.dump();
Thus the changes to static-net-filtering.js are to fix
the erroneous insertion of filters with uppercase
characters. The single instance found was a hostname entry
in Malware Domain List (TRIANGLESERVICESLTD dot COM).
The `null` placeholder are not necessary, we can just use
default arguments instead, and add the HNTrieContainer
references if and only if they are instanciated.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/550
Related Chromium issue (I can't access it):
- https://bugs.chromium.org/p/chromium/issues/detail?id=957866
Findings so far: affects browsers based on Chromium 74.
I could not reproduce the issue with either Chromium 73 or
Google Chrome 75.
This commit is a mitigation: to prevent sites from using
uBO's internal WAR secret for tracking purpose. A secret
can be used for at most one second, after which a new secret
is generated.
The original issue related to the implementation of
secret-gated web accessible resources is:
- https://github.com/gorhill/uBlock/issues/2823
The purpose of this new `:nth-ancestor(n)` operator is to
lookup the nth ancestor relative to the currently selected
node.
It is essentially equivalent to `:xpath(..)`, where
ancestor distance is expressed as a number rather than a
sequence of slash-separated `..`.
The rationale to introduce this new procedural selector
is to have a low overhead way to accomplish ancestor
selection.
This commit implements the alphabetical ordering of HNTrie
nodes, so as to make it possible to bail out early at
HNTrie.matches() time.
Contrary to what I expected, there is no performance gain
observed to HNTrie.matches() as per benchmarks -- I find
the results perplexing.
Because of this I will revert this commit immediately.
The purpose of this commit is to record the changes so
that I can bring them back to life in the future whenever
I want to investigate further.
Consider the two following filters:
example.com
www.example.com
This commit make it so that if the first filter is
already present in a given HNTrie, the second filter
will not be stored, since HNTrie will _always_
return the first filter as a match whenever the
hostname to match is example.com or any subdomain
of example.com.
The detection of such pointless filters is
virtually free when adding a hostname to an HNTrie
instance (given how data is stored in the trie), so
in practice no overhead is incurred to detect such
pointless filters.
The ability to ignore impossible to match filters
in HNTrie instances will _especially_ benefit those
using large hosts files.
Examples of how this helps using real configurations:
- Default lists:
444 filters out of 100,382 were ignored as a result
of this commit.
- Default lists + "Energized Ultimate Protection":
283,669 filters out of 903,235 were ignored as a
result of this commit.
Side note: There was no measurable difference between
the two configurations above in the performance of
the matching algorithm as reported by the built-in
benchmark tool.
The staticNetFilteringEngine uses token hashes to store/lookup
filters into Map objects.
Before this commit, the tokens were encoded into token hashes
as JS numbers (not exceeding MAX_SAFE_INTEGER) using at most
the 8 first characters of the token.
With this commit, token hashes are now restricted to fit
into 32-bit integers, and are derived from at most the 7 first
characters. This improves filter look-up performance as per
built-in benchmark().
Related commit:
- 69a43e07c4
Using 32 bits of token hash rather than just the 16 lower
bits does help discard more unknown tokens.
Using the default filter lists, the known-token lookup
table is populated by 12,276 entries, out of 65,536, thus
making the case that theoretically there is a lot of
possible tokens which can be discarded.
In practice, running the built-in
staticNetFilteringEngine.benchmark() with default filter
lists, I find that 1,518,929 tokens were skipped out of
4,441,891 extracted tokens, or 34%.
Related commit:
- 3f3a1543ea
The regression was preventing uBO to find from which list a filter
originated. This affected only filters for which the `domain=`
option had multiple hostnames.
Given that all tokens extracted from one single URL are potentially
iterated multiple times in a single URL-matching cycle, it pays to
ignore extracted tokens which are known to not be used anywhere in
the static filtering engine.
The gain in processing a single network request in the static
filtering engine can become especially high when dealing with
long and random-looking URLs, which URLs have a high likelihood
of containing a majority of tokens which are known to not be in
use.
