Fixed serious regression in previous dev build in applying
`csp=` filters. Reported internally by uBO team.
Promote usage of `removeparam` in code instead of `queryprune`,
which is to be deprecated.
Removed test against previously tested hostname in
FilterHostnameDict since as per various benchmark, the
test does not really help.
Remove serialization API in Node.js code as the API is now
present in SNFE itself.
Not sure this can really happen, but if ever Math.random() would
return `0.9999999999999999`, the attribute name would start with
`{`, i.e. an invalid attribute name.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1732
The regression affect filter with the `important` option when
the following conditions were fulfilled:
- The filter pattern is pure hostname
- The filter has not one of the following options:
- domain
- denyallow
- header
- strict1p, strict3p
- csp
- removeparam
- There is a matching exception filter
Related commit:
- a2a8ef7e85
A related mocha test has been added in order to detect this
specific regression in the future through `make test`.
This is a replacement for the dubious approach when the
extension itself was used to run benchmarks to detect
performance and filtering behavior regressions.
The main nodejs flavor is "npm", which is to be used to
lint/test and the publication of an official npm
package -- and by design it has dependencies on mocha,
eslint, etc.
A new flavor "dig" has been created with minimal
dependencies and which purpose is to easily allow to
write specialized code to investigate local code changes
in uBO -- and it's not meant for publication.
Consequently, "make nodejs" has been replaced with
"make npm", and a new "dig" target has been added to the
makefile, to be used for instrumenting local code changes
for investigation purpose.
Whereas before the string segment was encoded as:
LL OOOOOOOOOOOO
where L are the upper 8 bits and used to encode the length
of the segment, and O are the lower 24 bits and used to
encode the offset of the string data in the character
buffer, the new code encode as follow:
OOOOOOOOOOOO LL
And furthermore the most significant bit of the length
LL is now used to mark whether the current string segment
is a label boundary.
This means a cell can't reference a segment longer then
127 characters. To work around this limitation for when a
segment is longer than 127 characters (a rare occurrence),
the algorithm will simply split the segment into multiple
adjacent cells.
As a result, there is no longer a need to encode
"boundariness" into special cells, which simplifies
both the storing and matching algorithms.
Additionally, added minimal documentation for the NPM
package on how to import and use HNTrieContainer as a
standalone API.
For clients who may wish to persist the intermediate compiled form
in order to be able to skip costly parsing operation when the
list is fed to the static network filtering engine.
In the static network filtering engine (snfe), the
compiling-related code was spread across two classes.
This commit makes it so that all the compiling-related
code is in FilterCompiler class, which clear purpose is
to compile raw filters into a form which can be persisted
and later fed to the snfe with no parsing overhead.
To compile raw static network filter, the new approach is:
snfe.createCompiler(parser);
Then for each single raw filter to compile:
compiler.compile(parser, writer);
The caller is responsible to keep a reference to the
compiler instance for as long as it is needed. This removes
the need for the clunky code used to keep an instance of
compiler alive in the snfe.
Additionally, snfe.tokenHistograms() has been moved to
benchmarks.js, as it has no dependency on the snfe, it's
just a utility function.
Ports can be disconnected under certain conditions,
and in such case runtime.lastError must be accessed
to avoid the extensions framework from emitting
error messages in the console.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
Disconnected ports could still happen Even when the port
was still seen as valid internally. Using a try-catch
block makes invalid port detection more reliable. This
is an occurrence I often encountered when stepping into
content script code, causing suprious error messages to
be thrown into uBO's background dev console.
Though Firefox shares a lot of WebExtensions code with Chromium,
these platforms have their own specific code paths, for various
reasons.
The reorganization here makes it clear that Chromium platform is
just one flavor of WebExtensions, and as such all Chromium-specific
code paths should no longer be automatically pulled by other
platforms where these code paths are not needed.
Given that the filepath of many files changed, here is the
parent commit to quickly browse back to the previous directory
layout:
ec7db30b2f
There were spurious error messages in the dev console of
uBO in Firefox, because Firefox does not silently ignore
duplicate contextual menu entries, which could occur
transiently when the contextual menu entries were updated.
The fix simplifies contextual menu code, and actually
fulfill the original goal of avoiding to call extensions
framework API as much as possible.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1513
Prior to this commit, the ability to enable/disable the
uncloaking of canonical names was only available to advanced
users. This commit make it so that the setting can be
toggled from the _Settings_ pane.
The setting is enabled by default. The documentation should
be clear that the setting should not be disabled unless it
actually solves serious network issues, for example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1694404
Also, as a result, the advanced setting `cnameUncloak` is no
longer available from within the advanced settings editor.
The managed `userSettings` entry is an array of entries,
where each entry is a name/value pair encoded into an array
of strings.
The first item in the entry array is the name of a setting,
and the second item is the stringified value for the
setting.
This is a more convenient way for administrators to set
specific user settings. The settings set through
`userSettings` policy will always be set at uBO launch
time.
The new entry is an array of strings, each representing a
distinct line, and all entries are used to populate the
"My filters" pane.
This offers an more straightforward way for administrators
to specify a list of custom filters to use for all
installations.
The entry `toOverwrite.filterLists` is an array of
string, where each string is a token identifying a
stock filter list, or a URL for an external filter
list.
This new entry is to make it easier for an
administrator to centrally configure uBO with a
custom set of filter lists.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1433
The new "extraTrustedSiteDirectives" policy is an array
of strings, each of which is parsed as a trusted-site
directive to append to a user's own set of trusted-site
directives at launch time.
The added trusted-site directives will be considered as
part of the default set of directives by uBO.
Related issue:
- https://github.com/gorhill/uBlock/issues/1744
A new context menu entry, "Block element in frame...", will
be present when right-clicking on a frame element. When
this entry is clicked, uBO's element picker will be
launched from within the embedded frame and function the
same way as when launched from within the page.
The condition has been spotted occurring when bringing
up the DOM inspector for a page on which cosmetic filters
are being applied.
Not clear why this happens, but uBO must be ready to
graciously handle such condition.
Reported internally by @gwarser.
In rare occasion, a timing issue could cause uBO to redirect
to a web accessible resource meant to be used for another
network request. This is a regression introduced with the
following commit:
- 2e5d32e967
Additionally, I identified another issue which would cause
cached redirection to fail when a cache entry with redirection
to a web accessible resource was being reused, an issue which
could especially affect pages which are generated dynamically
(i.e. without full page reload).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/760
The purpose of this new network filter option is to remove
query parameters form the URL of network requests.
The name `queryprune` has been picked over `querystrip`
since the purpose of the option is to remove some
parameters from the URL rather than all parameters.
`queryprune` is a modifier option (like `csp`) in that it
does not cause a network request to be blocked but rather
modified before being emitted.
`queryprune` must be assigned a value, which value will
determine which parameters from a query string will be
removed. The syntax for the value is that of regular
expression *except* for the following rules:
- do not wrap the regex directive between `/`
- do not use regex special values `^` and `$`
- do not use literal comma character in the value,
though you can use hex-encoded version, `\x2c`
- to match the start of a query parameter, prepend `|`
- to match the end of a query parameter, append `|`
`queryprune` regex-like values will be tested against each
key-value parameter pair as `[key]=[value]` string. This
way you can prune according to either the key, the value,
or both.
This commit introduces the concept of modifier filter
options, which as of now are:
- `csp=`
- `queryprune=`
They both work in similar way when used with `important`
option or when used in exception filters. Modifier
options can apply to any network requests, hence the
logger reports the type of the network requests, and no
longer use the modifier as the type, i.e. `csp` filters
are no longer reported as requests of type `csp`.
Though modifier options can apply to any network requests,
for the time being the `csp=` modifier option still apply
only to top or embedded (frame) documents, just as before.
In some future we may want to apply `csp=` directives to
network requests of type script, to control the behavior
of service workers for example.
A new built-in filter expression has been added to the
logger: "modified", which allow to see all the network
requests which were modified before being emitted. The
translation work for this new option will be available
in a future commit.
Months ago, usage of synchronous localStorage was replaced
with asynchronous extension storage. There was code for the
conversion to be seamless by importing the content of now
obsolete localStorage.
This code is no longer needed as majority of users are
assumed to use versions of uBO above 1.25.0.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/899
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/763
Changes:
From now on, uBO will allow click-to-subscribe on only
a few select domains, currently:
- https://filterlists.com/
- https://github.com/
- https://github.io/
More domains can be added if and only the demonstration
is made that more than a marginal number of filter lists
can be subscribed from those domains.
The browser alert box is no longer used to confirm
subscription to a filter list. Instead, the asset
viewer has been expanded to serve that purpose. This
way, users can peruse at the content of a filter list
before subscribing to it.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1226
Related commit:
- 9eb455ab5e
In the previous commit, the element picker dialog was
isolated from the page content. This commit is to also
isolate the svg layers from the page content.
With this commit, there is no longer a need for an anonymous
iframe and the isolated world iframe is now directly
embedded in the page.
As a result, pages are now unable to interfere with any
of the element picker user interface. Pages can now only
see an iframe, but are unable to see the content of that
iframe. The styles applied to the iframe are from a user
stylesheet, so as to ensure pages can't override the
iframe's style properties set by uBO.
This commit fixes deleting all entries when cloud
storage usage is beyond allowed limit.
The issue would prevent pushing new data well within
quota limit because the old data beyond limit was
never removed in the first place.
Cloud storage is a limited resource, and thus it
makes sense to support data compression before
sending the data to cloud storage.
A new hidden setting allows to toggle on
cloud storage compression:
name: cloudStorageCompression
default: false
By default, this hidden setting is `false`, and a
user must set it to `true` to enable compression
of cloud storage items.
This hidden setting will eventually be toggled
to `true` by default, when there is good confidence
a majority of users are using a version of uBO
which can properly handle compressed cloud storage
items.
A cursory assessment shows that compressed items
are roughly 40-50% smaller in size.
The pseudo user styles code served only browsers based
on Chromium 65 and earlier -- Chromium 66 supports
native user styles and was first released more than two
years ago.
In Chromium-based browsers, the pseudo user styles code
is being unconditionally injected in every page/frame
just in case the browser is version 65 or earlier.
Removing pseudo user styles reduce uBO's main content
script in Chromium-based browsers by more than 20K.
Related thread:
- https://github.com/NanoAdblocker/NanoCore/issues/348#issuecomment-653646507
