The new parser no longer uses the browser DOM to validate
that a cosmetic filter is valid or not, this is now done
through a JS library, CSSTree.
This means filter list authors will have to be more careful
to ensure that a cosmetic filter is really valid, as there is
no more guarantee that a cosmetic filter which works for a
given browser/version will still work properly on another
browser, or different version of the same browser.
This change has become necessary because of many reasons,
one of them being the flakiness of the previous parser as
exposed by many issues lately:
- https://github.com/uBlockOrigin/uBlock-issues/issues/2262
- https://github.com/uBlockOrigin/uBlock-issues/issues/2228
The new parser introduces breaking changes, there was no way
to do otherwise. Some current procedural cosmetic filters will
be shown as invalid with this change. This occurs because the
CSSTree library gets confused with some syntax which was
previously allowed by the previous parser because it was more
permissive.
Mainly the issue is with the arguments passed to some procedural
cosmetic filters, and these issues can be solved as follow:
Use quotes around the argument. You can use either single or
double-quotes, whichever is most convenient. If your argument
contains a single quote, use double-quotes, and vice versa.
Additionally, try to escape a quote inside an argument using
backslash. THis may work, but if not, use quotes around the
argument.
When the parser encounter quotes around an argument, it will
discard them before trying to process the argument, same with
escaped quotes inside the argument. Examples:
Breakage:
...##^script:has-text(toscr')
Fix:
...##^script:has-text(toscr\')
Breakage:
...##:xpath(//*[contains(text(),"VPN")]):upward(2)
Fix:
...##:xpath('//*[contains(text(),"VPN")]'):upward(2)
There are not many filters which break in the default set of
filter lists, so this should be workable for default lists.
Unfortunately those fixes will break the filter for previous
versions of uBO since these to not deal with quoted argument.
In such case, it may be necessary to keep the previous filter,
which will be discarded as broken on newer version of uBO.
THis was a necessary change as the old parser was becoming
more and more flaky after being constantly patched for new
cases arising, The new parser should be far more robust and
stay robist through expanding procedural cosmetic filter
syntax.
Additionally, in the MV3 version, filters are pre-compiled
using a Nodejs script, i.e. outside the browser, so validating
cosmetic filters using a live DOM no longer made sense.
This new parser will have to be tested throughly before stable
release.
If the removed stock list is labelled a "bad list", do not
convert it into an imported list.
This will allow to seamlessly merge resource-abuse stock list
with privacy stock list when 1.42.0 is widespread.
Related discussion:
- a0a9497b4a (commitcomment-62560291)
The new setting, when disabled (enabled by default), allows a user
to prevent uBO from waiting for all filter lists to be loaded
before allowing network activity at launch. The setting is enabled
by default, meaning uBO waits for all filter lists to be loaded in
memory before unsuspending network activity. Some users may find
this behavior undesirable, hence the new setting.
This gives the option to potentially speed up page load at launch,
at the cost of potentially not properly filtering network requests
as per filter lists/rules.
For platforms not supporting the suspension of network activity,
the setting will merely prevent whatever mechanism exists on the
platform to mitigate improper filtering of network requests at
launch. For example, in Chromium-based browsers, unchecking the
new setting will prevent the browser from re-loading tabs for
which there was network activity while in "suspended" state at
launch.
So as to allow nodejs usage to better deal with
out of date serialization/compilation.
Additionally, use FilterImportant() only when a
"block-important" filter is stored in the "block" realm.
In the static network filtering engine (snfe), the
compiling-related code was spread across two classes.
This commit makes it so that all the compiling-related
code is in FilterCompiler class, which clear purpose is
to compile raw filters into a form which can be persisted
and later fed to the snfe with no parsing overhead.
To compile raw static network filter, the new approach is:
snfe.createCompiler(parser);
Then for each single raw filter to compile:
compiler.compile(parser, writer);
The caller is responsible to keep a reference to the
compiler instance for as long as it is needed. This removes
the need for the clunky code used to keep an instance of
compiler alive in the snfe.
Additionally, snfe.tokenHistograms() has been moved to
benchmarks.js, as it has no dependency on the snfe, it's
just a utility function.
The code exported to nodejs package was revised to use modern
JavaScript syntax. A few issues were fixed at the same time.
The exported classes are:
- DynamicHostRuleFiltering
- DynamicURLRuleFiltering
- DynamicSwitchRuleFiltering
These related to the content the of "My rules" pane in the
uBlock Origin extension.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
This change allows to add the redirect engine into the
nodejs package. The purpose of the redirect engine is to
resolve a redirect token into a path to a local resource,
to be used by the caller as wished.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1513
Prior to this commit, the ability to enable/disable the
uncloaking of canonical names was only available to advanced
users. This commit make it so that the setting can be
toggled from the _Settings_ pane.
The setting is enabled by default. The documentation should
be clear that the setting should not be disabled unless it
actually solves serious network issues, for example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1694404
Also, as a result, the advanced setting `cnameUncloak` is no
longer available from within the advanced settings editor.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1480
Forward compatiblity was broken due to `externalLists`
being converted into an Array from a string, i.e.
downgrading to uBO 1.32.4 was completely breaking uBO.
This commit restores `externalLists` as a string which
is what older versions of uBO expect.
A new property `importedLists` has been created to
hold the imported lists as an array, while
`externalLists` will be kept around for a while until
it is completely removed in some future.
The managed `userSettings` entry is an array of entries,
where each entry is a name/value pair encoded into an array
of strings.
The first item in the entry array is the name of a setting,
and the second item is the stringified value for the
setting.
This is a more convenient way for administrators to set
specific user settings. The settings set through
`userSettings` policy will always be set at uBO launch
time.
The new entry is an array of strings, each representing a
distinct line, and all entries are used to populate the
"My filters" pane.
This offers an more straightforward way for administrators
to specify a list of custom filters to use for all
installations.
The entry `toOverwrite.filterLists` is an array of
string, where each string is a token identifying a
stock filter list, or a URL for an external filter
list.
This new entry is to make it easier for an
administrator to centrally configure uBO with a
custom set of filter lists.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1433
The new "extraTrustedSiteDirectives" policy is an array
of strings, each of which is parsed as a trusted-site
directive to append to a user's own set of trusted-site
directives at launch time.
The added trusted-site directives will be considered as
part of the default set of directives by uBO.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1365
This commit adds the compiled magic version number to the
compiled data itself, and consequently this allows uBO
to no longer require that any given compiled list with a
mismatched format to be detected and discarded at launch
time.
Given this change, uBO no longer needs to rely on the
deletion of cached data at launch time to ensure it
won't use no longer valid compiled lists.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1365
When compiled data format changes, do not rely on order
of operations at launch to assume deletion of storage
occurs before attempts to access it. It's unclear this
commit will fix the reported issue, as I could not
reproduce it except when outright commenting out the code
to prevent the storage deletion from occurring.
Notably, make `queryprune` option available only
to filter list authors, until there are guards
against bad filters in some future and until the
option syntax and behavior is fully settled.
Instances of `queryprune` in filter lists will be
compiled, however instances of `queryprune` in
_"My filters"_ will be ignored unless users
indicated they are a filter list author.
filterUnits is now treated as a buffer which is
pre-allocated and which will grow in chunks so as
to minimize memory allocations. Entries are never
released, just null-ed.
Additionally, move urlTokenizer into the static
network filtering engine, since it's not used
anywhere else.
In addition to what is deemed really bad lists by consensus,
some lists will also be labelled "really bad list"
temporarily so as to force-remove them from the set of
filter lists.
This will be the case for filter lists which are not
necessarily "bad lists" but which were once part of
uBO's stock filter lists and have been removed since
then for various reasons.
This will ensure that the majority of users who do not
modifies uBO's default listset will still have a
configuration which matches the official default listset.
This commit adds concept of "really bad list" to the
badlists infrastructure. Really bad lists won't be
fetched from a remote server, while plain bad list
will be fetched but won't be compiled.
A really bad list is denoted by the `nofetch` token
following the URL.
Really bad lists can cause more serious issues such
as causing undue launch delays because the remote
server where a really bad list is hosted fails to
respond properly and times out.
Such an example of really bad list is hpHosts which
original server no longer exist.
Many filter lists are known to cause serious filtering
issues in uBO and are not meant to be used in uBO.
Unfortunately, unwitting users keep importing these
filter lists and as a result this ends up causing
filtering issues for which the resolution is always
to remove the incompatible filter list.
Example of inconpatible filter lists:
- Reek's Anti-Adblock Killer
- AdBlock Warning Removal List
- ABP anti-circumvention filter list
uBO will use the following resource to know
which filter lists are incompatible:
- https://github.com/uBlockOrigin/uAssets/blob/master/filters/badlists.txt
Incompatible filter lists can still be imported into
uBO, useful for asset-viewing purpose, but their content
will be discarded at compile time.
