Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1692
The ids/classes from html/body elements will leave out
looking up lowly generic cosmetic filters made of a single
identifier.
This does not absolutely guarantee that html/body elements
will never be targeted, but it should greatly mitigate the
probability that this erroneously happens.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1690
New procedural operator: `:matches-path(...)`
Description: this is a all-or-nothing passthrough operator, which
on/off behavior is dictated by whether the argument match the
path of the current location. The argument can be either plain
text to be found at any position in the path, or a literal regex
against which the path is tested.
Whereas cosmetic filters can be made specific to whole domain,
the new `:matches-path()` operator allows to further narrow
the specificity according to the path of the current document
lcoation.
Typically this procedural operator is used as first operator in
a procedural cosmetic filter, so as to ensure that no further
matching work is performed should there be no match against the
current path of the current document location.
Example of usage:
example.com##:matches-path(/shop) p
Will hide all `p` elements when visiting `https://example.com/shop/stuff`,
but not when visiting `https://example.com/` or any other page
on `example.com` which has no instance of `/shop` in the path part
of the URL.
So as to allow nodejs usage to better deal with
out of date serialization/compilation.
Additionally, use FilterImportant() only when a
"block-important" filter is stored in the "block" realm.
When matching a network request in the static network filtering
engine ("snfe"), these are the possible outcomes, from most
to least likely:
- No block
- Block
- Unblock ("exception" filter overriding the block)
- Block-important ("important" filter override the unblock)
Hence why the matching in the snfe always check for a match in
the "block" realm, and the "unblock" realm would be checked
if and only if there was a match in the "block" realm.
However the "block-important" realm was always matched against
first, and when a match in that realm was found, there would
be no need to check in other realms since nothing can override
the "important" option. The problem with this approach though
is that matches in the "block-important" realm are most
unlikely, which means pointless work being done for vast
majority of network requests.
This commit makes it so that the "block-important" realm is
matched against ONLY when there is a matched "unblock" filter.
The result is a measurable improvement in the snfe-related
benchmarks (though given the numbers involved, end users won't
perceive a difference).
Somewhat related discussion which was the motivation to look
more into this:
https://github.com/cliqz-oss/adblocker/discussions/2170#discussioncomment-1168125
Whereas before the string segment was encoded as:
LL OOOOOOOOOOOO
where L are the upper 8 bits and used to encode the length
of the segment, and O are the lower 24 bits and used to
encode the offset of the string data in the character
buffer, the new code encode as follow:
OOOOOOOOOOOO LL
And furthermore the most significant bit of the length
LL is now used to mark whether the current string segment
is a label boundary.
This means a cell can't reference a segment longer then
127 characters. To work around this limitation for when a
segment is longer than 127 characters (a rare occurrence),
the algorithm will simply split the segment into multiple
adjacent cells.
As a result, there is no longer a need to encode
"boundariness" into special cells, which simplifies
both the storing and matching algorithms.
Additionally, added minimal documentation for the NPM
package on how to import and use HNTrieContainer as a
standalone API.
The erroneous test does not seem to interfere
with the proper functioning of the trie, due
to the fact that nodes are never split without
a OR node or boundary node being present.
The issue was found when undertaking a rewrite
of the algorithm to avoid having to create
boundary nodes.
In the static network filtering engine (snfe), the
compiling-related code was spread across two classes.
This commit makes it so that all the compiling-related
code is in FilterCompiler class, which clear purpose is
to compile raw filters into a form which can be persisted
and later fed to the snfe with no parsing overhead.
To compile raw static network filter, the new approach is:
snfe.createCompiler(parser);
Then for each single raw filter to compile:
compiler.compile(parser, writer);
The caller is responsible to keep a reference to the
compiler instance for as long as it is needed. This removes
the need for the clunky code used to keep an instance of
compiler alive in the snfe.
Additionally, snfe.tokenHistograms() has been moved to
benchmarks.js, as it has no dependency on the snfe, it's
just a utility function.
The code exported to nodejs package was revised to use modern
JavaScript syntax. A few issues were fixed at the same time.
The exported classes are:
- DynamicHostRuleFiltering
- DynamicURLRuleFiltering
- DynamicSwitchRuleFiltering
These related to the content the of "My rules" pane in the
uBlock Origin extension.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
This change allows to add the redirect engine into the
nodejs package. The purpose of the redirect engine is to
resolve a redirect token into a path to a local resource,
to be used by the caller as wished.
Invalid URLs like "http://" and "http://foo@" trigger TypeErrors
when they are passed to the URL constructor. These TypeErrors
caused the scriptlet to stop processing subsequent noscript nodes
due to uncaught exceptions.
These exceptions are now caught to allow all noscript nodes to
be processed.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The various filtering engine benchmarking functions are best
isolated in their own file since they have specific
dependencies that should not be suffered by the filtering
engines.
Additionally, moved decomposeHostname() into uri-utils.js
as it's a hostname-related function required by many
filtering engine cores -- this allows to further reduce
or outright remove dependency on `µb`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
The title of tabs in uBO is solely to have a better
presentation in the logger -- no other purpose.
This commit simplify keeping track of the titles, from
an active approach by directly querying it from tabs
whenever a change occurs, to a passive approach by
storing it when the title string become available in
some tab event handlers.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/oicch9/
The new replacement script contains the smallest API
possible to resolve the reported case.
Please report instances where it's not sufficient to
unbreak a site, in which case I will extend the neutered
API to address these cases on an on-demand basis.
Related issue:
- https://github.com/gorhill/uBlock/issues/3212
The element picker will now properly work on sites where
cosmetic filtering is disabled, but will not allow the
creation of cosmetic filters when specific cosmetic filters
are not meant to be enforced in the current page.
When specific cosmetic filters are not meant to be enforced,
the element picker will still allow the creation of network
filters, that is unless the current page is trusted, in which
case using the element picker is pointless.
Related issue:
- https://github.com/gorhill/uBlock/issues/3037
This takes care of the specific case reported. There are
other edge cases which are likely not addressed though, i.e.
those involving wildcards -- those should be rather rare and
at this point I rather leave them unaddressed to not
risk regressions (as they are less trivial to address).
Related feedback:
- https://ilakovac.com/teespring-ublock-issue/
The surrogate script googletagmanager_gtm.js was essentially a
subset of surrogate script google-analytics_analytics.js. This
commit makes it a plain alias so that the whole GA API -- often
expected by clients of GTM -- is properly stubbed.
Reported internally:
> STR --
>
> Import https://cdn.statically.io/gh/uBlockOrigin/uAssets/master/filters/filters.txt
> as a Custom filter list.
>
> Observe the filter count at 24K instead of true count
> being 29K.
>
> Force updating is sometimes compiling 24K filters and
> on subsequent updates 29K, so I narrowed it down to
> this host.
Findings:
One of the sublists was erroring when being fetched on
this particular CDN server (reason unknown).
uBO was not properly handling network errors when
fetching a sublist. This commit make it so that if
a sublist can't be fetched, then the error is propagated
as if it affected the whole list, in which case uBO will
use an alternative URL if any.
This advanced setting is not really needed, as the
same can be accomplished with a broad exception
filter such as `#@#+js()`.
Related feedback:
- f5b453fae3 (commitcomment-49499082)
Ever since the `redirect` code was refactored:
157cef6034
This advanced setting is no longer needed, as the same
can be accomplished with a plain network filter:
@@*$redirect-rule
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1553
This commit ensures FLoC is opt-in. The generic filter
`*##+js(no-floc)` in "uBlock filters -- Privacy" ensures
the feature is disabled when using default settings/lists.
Users can opt-in to FLoC by adding a generic exception
filter to their custom filters, `#@#+js(no-floc)`; or they
can opt-in only for a specific set of websites through a
more specific exception filter:
example.com,shopping.example#@#+js(no-floc)
The syntax to remove response header is a special case
of HTML filtering, whereas the response headers are
targeted, rather than the response body:
example.com##^responseheader(header-name)
Where `header-name` is the name of the header to
remove, and must always be lowercase.
The removal of response headers can only be applied to
document resources, i.e. main- or sub-frames.
Only a limited set of headers can be targeted for
removal:
location
refresh
report-to
set-cookie
This limitation is to ensure that uBO never lowers the
security profile of web pages, i.e. we wouldn't want to
remove `content-security-policy`.
Given that the header removal occurs at onHeaderReceived
time, this new ability works for all browsers.
The motivation for this new filtering ability is instance
of website using a `refresh` header to redirect a visitor
to an undesirable destination after a few seconds.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1513
Prior to this commit, the ability to enable/disable the
uncloaking of canonical names was only available to advanced
users. This commit make it so that the setting can be
toggled from the _Settings_ pane.
The setting is enabled by default. The documentation should
be clear that the setting should not be disabled unless it
actually solves serious network issues, for example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1694404
Also, as a result, the advanced setting `cnameUncloak` is no
longer available from within the advanced settings editor.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1501
Exception filters for `document` option are complying with
uBO's own semantic for `document` option, i.e. an exception
filter for `document` option will only allow to bypass a
block filter for `document` (either explicit or implicit)
and nothing else.
Exception filters using `document` option are *not*
compatible with ABP's interpretation of these filters.
Whereas in ABP the purpose of a `document` exception filter
is to wholly disable content blocking, in uBO the same
filter will just cause strict-blocking to be disabled while
leaving content blocking intact.
Additionally, the logger was fixed to properly report pages
which are being strict-blocked.
The procedural cosmetic filtering code has been split from
the content script code injected unconditionally and will
from now on be injected only when it is needed, i.e. when
there are procedural cosmetic filters to enforce.
The motivation for this is:
https://www.debugbear.com/blog/2020-chrome-extension-performance-report#what-can-extension-developers-do-to-keep-their-extensions-fast
Though uBO's content script injected unconditionally in all
pages/frames is relatively small, I still wanted to further
reduce the amount of content script code injected
unconditionally: The procedural cosmetic filtering code
represents roughly 14KB of code the browser won't have to
parse/execute unconditionally unless there exists procedural
cosmetic filters to enforce for a page or frame.
At the time the above article was published, the total
size of unconditional content scripts injected by uBO was
~101 KB, while after this commit, the total size will be
~57 KB (keeping in mind uBO does not minify and does not
remove comments from its JavaScript code).
Additionally, some refactoring on how user stylesheets are
injected so as to ensure that `:style`-based procedural
filters which are essentially declarative are injected
earlier along with plain, non-procedural cosmetic filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/210
Additionally, a small (experimental) widget has been added
to emphasize/de-emphasize rows which have 3rd-party
scripts/frames, so as to more easily identify which rows
are "affected" by 3rd-party scripts and/or frames.
Tooltip localization for the new widget is not available
yet as I want wait for the feature to be fully settled.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1480
Forward compatiblity was broken due to `externalLists`
being converted into an Array from a string, i.e.
downgrading to uBO 1.32.4 was completely breaking uBO.
This commit restores `externalLists` as a string which
is what older versions of uBO expect.
A new property `importedLists` has been created to
hold the imported lists as an array, while
`externalLists` will be kept around for a while until
it is completely removed in some future.
The managed `userSettings` entry is an array of entries,
where each entry is a name/value pair encoded into an array
of strings.
The first item in the entry array is the name of a setting,
and the second item is the stringified value for the
setting.
This is a more convenient way for administrators to set
specific user settings. The settings set through
`userSettings` policy will always be set at uBO launch
time.
The new entry is an array of strings, each representing a
distinct line, and all entries are used to populate the
"My filters" pane.
This offers an more straightforward way for administrators
to specify a list of custom filters to use for all
installations.
Content scripts can't properly look up effective context
for sandboxed frames. This commit add ability to extract
effective context from already existing store of frames
used for each tab.
The entry `toOverwrite.filterLists` is an array of
string, where each string is a token identifying a
stock filter list, or a URL for an external filter
list.
This new entry is to make it easier for an
administrator to centrally configure uBO with a
custom set of filter lists.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1433
The new "extraTrustedSiteDirectives" policy is an array
of strings, each of which is parsed as a trusted-site
directive to append to a user's own set of trusted-site
directives at launch time.
The added trusted-site directives will be considered as
part of the default set of directives by uBO.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1241
uBO will not discard secondary requests fired before a root
frame is committed, by ensuring that if newly uncommitted
root frames are of the same origin as previous one(s), the
uncommited journal slot pointer is not updated.
Related commit:
- 6ac09a2856
Patternless `queryprune` ar enow preserved as being
pattern-less while still attempting to extract a token
from the `queryprune` value. This allows to report the
filter in the logger same as its original form.
The following regex are not rejected as invalid when
using built-in regex objects:
/abc]/
/a7,18}/
/a{7,18/
However, as per documentation, they are not supposed to
be valid, as `{` and `}` are special characters and as
such should be escaped:
/abc\]/
/a7,18\}/
/a\{7,18/
With this commit, the regexes will additionally be
validated using the regex analyzer library in the editor
to ensure strict regex syntax compliance so as to avoid
what are likely mistakes in regex crafting by authors.
This commit fixes mouse double-click-and-drag operations,
which was broken due to the implementation of a custom
word selection in the filter list editor/viewer.
Regex-based static network filters are those most likely to
cause performance degradation, and as such the best guard
against undue performance degradation caused by regex-based
filters is the ability to extract valid and good tokens
from regex patterns.
This commit introduces a complete regex parser so that the
static network filtering engine can now safely extract
tokens regardless of the complexity of the regex pattern.
The regex parser is a library imported from:
https://github.com/foo123/RegexAnalyzer
The syntax highlighter adds an underline to regex-based
filters as a visual aid to filter authors so as to avoid
mistakenly creating regex-based filters. This commit
further colors the underline as a warning when a regex-based
filter is found to be untokenizable.
Filter list authors are invited to spot these untokenizable
regex-based filters in their lists to verify that no
mistake were made for those filters, causing them to be
untokenizabke. For example, what appears to be a mistake:
/^https?:\/\/.*\/sw.js?.[a-zA-Z0-9%]{50,}/
Though the mistake is minor, the regex-based filter above
is untokenizable as a result, and become tokenizable when
the `.` is properly escaped:
/^https?:\/\/.*\/sw\.js?.[a-zA-Z0-9%]{50,}/
Filter list authors can use this search expression in the
asset viewer to find instances of regex-based filters:
/^(@@)?\/[^\n]+\/(\$|$)/
This should improve usability of uBO's hard-mode
and "relax blocking mode" operations. This is the
new default behavior.
The previous behavior of forcing a reload of the
page can be re-enabled by simply setting the `3p`
bit of the advanced setting `blockingProfiles`
to 1.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1204
Not much can be done beside reporting to tabless network
requests to all tabs for which the context is a match.
A short term local cache is used to avoid having to iterate
through all existing tabs for each tabless network request
just to find and report to the matching ones -- users
reporting having a lot of opened tabs at once is not so
uncommon.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/857
The recognized resources are:
- abp-resource:blank-mp3
- abp-resource:blank-js
ABP's tokens are excluded from auto-complete so as to not
get in the way of uBO's filter list maintainers.
Reported internally by @uBlock-user.
Also, fixed broken caching of `cname` exception, which forced
uBO to repeatedly evaluate whether a `cname` exception exists
when a block `cname`-cloaked request is encountered.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1365
This commit adds the compiled magic version number to the
compiled data itself, and consequently this allows uBO
to no longer require that any given compiled list with a
mismatched format to be detected and discarded at launch
time.
Given this change, uBO no longer needs to rely on the
deletion of cached data at launch time to ensure it
won't use no longer valid compiled lists.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1388
Fixed the special `none` redirect resource no longer being
enforced.
Fixed the enforcement of `important` redirect rules over
exceptions and non-important ones.
Related issue:
- https://github.com/gorhill/uBlock/issues/1744
A new context menu entry, "Block element in frame...", will
be present when right-clicking on a frame element. When
this entry is clicked, uBO's element picker will be
launched from within the embedded frame and function the
same way as when launched from within the page.
This is particularly helpful for static network filters
used with filter options causing the same pattern to be
reused across multiple filter instances, i.e. `all` or
`~css`, etc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1365
When compiled data format changes, do not rely on order
of operations at launch to assume deletion of storage
occurs before attempts to access it. It's unclear this
commit will fix the reported issue, as I could not
reproduce it except when outright commenting out the code
to prevent the storage deletion from occurring.
All matching `redirect-rule` directives will now be reported
in the logger, instead of just the effective one.
The highest-ranked redirect directive will be the one
effectively used for redirection. This way filter list
authors can see whether a lower priority redirect is
being overriden by a higher priority one.
The default priority has been changed to 10, so as to allow
more leeway to create lower ranked redirect directives.
Additonally, rendering of redirect directives with explicit
priority has been fixed in the logger, they will no longer
be rendered as unknown redirect tokens.
The header value is no longer implicitly a regex-based literal, but
a plain string against which the header name is compared. The value can
be set to a regex literal by bracing the header value with the usual
forward slashes, `/.../`.
Examples:
*$1p,strict3p,script,header=via:1.1 google
*$1p,strict3p,script,header=via:/1\.1\s+google/
The first form will cause a strict comparison with the value of the header
named `via` against the string `1.1 google`.
The second form will cause a regex-based test with the value of the header
named `via` against the regex `/1\.1\s+google/`.
The header value can be prepended with `~` to reverse the comparison:
*$1p,strict3p,script,header=via:~1.1 google
The header value is optional and may be ommitted to test only for the
presence of a specific header:
*$1p,strict3p,script,header=via
Related discussions:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1356#issuecomment-732411286
- https://github.com/AdguardTeam/CoreLibs/issues/1384
Changes:
Negation character is `~` (instead of `!`).
Drop special anchor character `|` -- leading `|`
will be supported until no such filter is present
in uBO's own filter lists. For example, instance
of `queryprune=|ad` will have to be replaced with
`queryprune=/^ad/` (or `queryprune=ad` if the name
of the parameter to remove is exactly `ad`).
Align semantic with that of AdGuard's `removeparam=`,
except that specifying multiple `|`-separated names
is not supported.
`match-case`
------------
Related issue:
- https://github.com/uBlockOrigin/uAssets/issues/8280#issuecomment-735245452
The new filter option `match-case` can be used only for
regex-based filters. Using `match-case` with any other
sort of filters will cause uBO to discard the filter.
`redirect=`
-----------
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1366
`redirect=` filters with unresolvable resource token at
runtime will be discarded.
Additionally, the implicit priority is now set to 1
(was 0). The idea is to allow custom `redirect=` filters
to be used strictly as fallback `redirect=` filters in case
another `redirect=` filter is not picked up.
For example, one might create a `redirect=click2load.html:0`
filter, to be taken if and only if the blocked resource is
not already being redirected by another "official" filter
in one of the enabled filter lists.
Related issue:
- https://github.com/gorhill/uBlock/issues/3590
Since the `redirect=` option was refactored into a modifier
filter, presence of a type (`script`, `xhr`, etc.) is no
longer a requirement.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1356
Related commit:
- bde3164eb4
It is not possible to achieve perfect compatiblity at this
point, but reasonable compatibility should be achieved for
a majority of instances of `removeparam=`.
Notable differences:
--------------------
uBO always matches in a case insensitive manner, there is
no need to ask for case-insensitivity, and no need to use
uppercase characters in `queryprune=` values.
uBO does not escape special regex characters since the
`queryprune=` values are always assumed to be literal
regex expression (leaving out the documented special
characters). This means `removeparam=` with characters
which are special regex characters won't be properly
translated and are unlikely to work properly in uBO.
For example, the `queryprune` value of a filter such as
`$removeparam=__xts__[0]` internally become the literal
regex `/__xts__[0]/`, and consequently would not match
a query parameter such as `...?__xts__[0]=...`.
Notes:
------
Additionally, for performance reason, when uBO encounter
a pattern-less `queryprune=` (or `removeparam=`) filter,
it will try to extract a valid pattern from the
`queryprune=` value. For instance, the following filter:
$queryprune=utm_campaign
Will be translated internally into:
utm_campaign$queryprune=utm_campaign
The logger will reflect this internal translation.
New filter options
==================
Strict partyness: `1P`, `3P`
----------------------------
The current options 1p/3p are meant to "weakly" match partyness, i.e. a
network request is considered 1st-party to its context as long as both the
context and the request share the same base domain.
The new partyness options are meant to check for strict partyness, i.e. a
network request will be considered 1st-party if and only if both the context
and the request share the same hostname.
For examples:
- context: `www.example.org`
- request: `www.example.org`
- `1p`: yes, `1P`: yes
- `3p`: no, `3P`: no
- context: `www.example.org`
- request: `subdomain.example.org`
- `1p`: yes, `1P`: no
- `3p`: no, `3P`: yes
- context: `www.example.org`
- request: `www.example.com`
- `1p`: no, `1P`: no
- `3p`: yes, `3P`: yes
The strict partyness options will be visually emphasized in the editor so as
to prevent mistakenly using `1P` or `3P` where weak partyness is meant to be
used.
Filter on response headers: `header=`
-------------------------------------
Currently experimental and under evaluation. Disabled by default, enable by
toggling `filterOnHeaders` to `true` in advanced settings.
Ability to filter network requests according to whether a specific response
header is present and whether it matches or does not match a specific value.
For example:
*$1p,3P,script,header=via:1\.1\s+google
The above filter is meant to block network requests which fullfill all the
following conditions:
- is weakly 1st-party to the context
- is not strictly 1st-party to the context
- is of type `script`
- has a response HTTP header named `via`, which value matches the regular
expression `1\.1\s+google`.
The matches are always performed in a case-insensitive manner.
The header value is assumed to be a literal regular expression, except for
the following special characters:
- to anchor to start of string, use leading `|`, not `^`
- to anchor to end of string, use trailing `|`, not `$`
- to invert the test, use a leading `!`
To block a network request if it merely contains a specific HTTP header is
just a matter of specifying the header name without a header value:
*$1p,3P,script,header=via
Generic exception filters can be used to disable specific block `header=`
filters, i.e. `@@*$1p,3P,script,header` will override the block `header=`
filters given as example above.
Dynamic filtering's `allow` rules override block `headers=` filters.
Important: It is key that filter authors use as many narrowing filter options
as possible when using the `header=` option, and the `header=` option should
be used ONLY when other filter options are not sufficient.
More documentation justifying the purpose of `header=` option will be
provided eventually if ever it is decided to move it from experimental to
stable status.
To be decided: to restrict usage of this filter option to only uBO's own
filter lists or "My filters".
Changes
=======
Fine tuning `queryprune=`
-------------------------
The following changes have been implemented:
The special value `*` (i.e. `queryprune=*`) means "remove all query
parameters".
If the `queryprune=` value is made only of alphanumeric characters
(including `_`), the value will be internally converted to regex equivalent
`^value=`. This ensures a better future compatibility with AdGuard's
`removeparam=`.
If the `queryprune=` value starts with `!`, the test will be inverted. This
can be used to remove all query parameters EXCEPT those who match the
specified value.
Other
-----
The legacy code to test for spurious CSP reports has been removed. This
is no longer an issue ever since uBO redirects to local resources through
web accessible resources.
Notes
=====
The following new and recently added filter options are not compatible with
Chromium's manifest v3 changes:
- `queryprune=`
- `1P`
- `3P`
- `header=`
The auto-complete feature in the _"My filters"_ pane will
use hostname/domain from the set of opened tabs to assist
in entering values for `domain=` option. This also works
for the implict `domain=` option ṗrepending static extended
filters.
`about:srcdoc` frames are their own origin, trying to
use the origin of the parent context causes an
exception to be thrown when accessing location.href.
Notably, add clickable link to open the widget
in its own tab. Also, allows the URL to be text-
selected so that it becomes possible to use the
selection in a browser contextual menu's "Open
in a new tab" option.
Notably, make `queryprune` option available only
to filter list authors, until there are guards
against bad filters in some future and until the
option syntax and behavior is fully settled.
Instances of `queryprune` in filter lists will be
compiled, however instances of `queryprune` in
_"My filters"_ will be ignored unless users
indicated they are a filter list author.
The important bit is now considered an action bit
so that there is no more a need for the `important`
property in the parser. The modify bit is now
considered a realm bit.
When the modify bit is set, the action bits become
available to be used to further narrow the realm.
This could be useful in the future if we want to
spread the population of modifier filters across
different buckets.
Reusing the same iterator instance for all cases
of `domain=` option parsing should reduce memory
churning.
Additonally, fine tune regex used to extract
valid token from regex-based filters to increase
likelihood of being able to extract a valid
token.
Reported internally by @gwarser.
In rare occasion, a timing issue could cause uBO to redirect
to a web accessible resource meant to be used for another
network request. This is a regression introduced with the
following commit:
- 2e5d32e967
Additionally, I identified another issue which would cause
cached redirection to fail when a cache entry with redirection
to a web accessible resource was being reused, an issue which
could especially affect pages which are generated dynamically
(i.e. without full page reload).
filterUnits is now treated as a buffer which is
pre-allocated and which will grow in chunks so as
to minimize memory allocations. Entries are never
released, just null-ed.
Additionally, move urlTokenizer into the static
network filtering engine, since it's not used
anywhere else.
Notably, defer the post-load optimization operations
to a few seconds after the filters have been all
loaded in memory -- this is not a critical step for
the filtering engine to work properly, hence this
can be delayed in order to ensure readiness as soon
as possible.
Most notably, the `denyallow=` option now requires
the presence of a valid `domain=` option to not be
rejected.
Using `denyallow=` without narrowing down using the
`domain=` option leads to catastrophic blocking
behvior, hence the requirement for a valid `domain=`
option.
Related commit:
- b265f2644d
The optimization in the commit above was meant to
improve the performance of lookup operations of
modifier filters, but I forgot to enable the
optimisation for that class of filters.
This means this commit brings another significant
performance gain on top of the previous commit, as
shown by the built-in benchmark.
Additionally a few minor code rearrangements.
Performance-related work.
There is a fair number of filters which can't be tokenized
in uBO's own filter lists. Majority of those filters also
declare a `domain=` option, examples:
*$script,redirect-rule=noopjs,domain=...
*$script,3p,domain=...,denyallow=...
*$frame,3p,domain=...
Such filters can be found in uBO's asset viewer using the
following search expression:
/^\*?\$[^\n]*?domain=/
Some filter buckets will contain many of those filters, for
instance one of the bucket holding untokenizable `redirect=`
filters has over 170 entries, which must be all visited when
collating all matching `redirect=` filters.
When a bucket contains many such filters, I found that it's
worth to extract all the non-negated hostname values from
`domain=` options into a single hntrie and perform a pre-test
at match() time to find out whether the current origin of a
network request matches any one of the collected hostnames,
so as to avoid iterating through all the filters.
Since there is rarely a match() for vast majority of network
requests with `domain=` option, this pre-test saves a good
amount of work, and this is measurable with the built-in
benchmark.
This commit moves the parsing, compiling and enforcement
of the `redirect=` and `redirect-rule=` network filter
options into the static network filtering engine as
modifier options -- just like `csp=` and `queryprune=`.
This solves the two following issues:
- https://github.com/gorhill/uBlock/issues/3590
- https://github.com/uBlockOrigin/uBlock-issues/issues/1008#issuecomment-716164214
Additionally, `redirect=` option is not longer afflicted
by static network filtering syntax quirks, `redirect=`
filters can be used with any other static filtering
modifier options, can be excepted using `@@` and can be
badfilter-ed.
Since more than one `redirect=` directives could be found
to apply to a single network request, the concept of
redirect priority is introduced.
By default, `redirect=` directives have an implicit
priority of 0. Filter authors can declare an explicit
priority by appending `:[integer]` to the token of the
`redirect=` option, for example:
||example.com/*.js$1p,script,redirect=noopjs:100
The priority dictates which redirect token out of many
will be ultimately used. Cases of multiple `redirect=`
directives applying to a single blocked network request
are expected to be rather unlikely.
Explicit redirect priority should be used if and only if
there is a case of redirect ambiguity to solve.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/760
The purpose of this new network filter option is to remove
query parameters form the URL of network requests.
The name `queryprune` has been picked over `querystrip`
since the purpose of the option is to remove some
parameters from the URL rather than all parameters.
`queryprune` is a modifier option (like `csp`) in that it
does not cause a network request to be blocked but rather
modified before being emitted.
`queryprune` must be assigned a value, which value will
determine which parameters from a query string will be
removed. The syntax for the value is that of regular
expression *except* for the following rules:
- do not wrap the regex directive between `/`
- do not use regex special values `^` and `$`
- do not use literal comma character in the value,
though you can use hex-encoded version, `\x2c`
- to match the start of a query parameter, prepend `|`
- to match the end of a query parameter, append `|`
`queryprune` regex-like values will be tested against each
key-value parameter pair as `[key]=[value]` string. This
way you can prune according to either the key, the value,
or both.
This commit introduces the concept of modifier filter
options, which as of now are:
- `csp=`
- `queryprune=`
They both work in similar way when used with `important`
option or when used in exception filters. Modifier
options can apply to any network requests, hence the
logger reports the type of the network requests, and no
longer use the modifier as the type, i.e. `csp` filters
are no longer reported as requests of type `csp`.
Though modifier options can apply to any network requests,
for the time being the `csp=` modifier option still apply
only to top or embedded (frame) documents, just as before.
In some future we may want to apply `csp=` directives to
network requests of type script, to control the behavior
of service workers for example.
A new built-in filter expression has been added to the
logger: "modified", which allow to see all the network
requests which were modified before being emitted. The
translation work for this new option will be available
in a future commit.
Additionally, add a button in the About pane
to launch benchmark sessions. The button will
be available only when advanced setting
`benchmarkDatasetURL` is set and pointing to
a valid dataset.
Related issues:
- https://github.com/gorhill/uBlock/issues/1390
- https://github.com/gorhill/uBlock/issues/2334
The deadline to interactively load a specific media
element has been extended from 2sec to 5sec.
Clicking over a blocked large media element will cause
uBO to lookup and handle all potentially blocked large
elements at the cursor position. This should take care
of being able to unblock media elements hidden under
other DOM object.
The CSS style applied to blocked large media elements
has been fine tuned to improve interactivity.
uBO will now remember the specific media elements which
were unblocked and keep them exempted from being
further blocked. This would be an issue when unblocking
a video and then a bit later seeking to another point
in the video, in which case uBO would again block
network requests for that video.
The specificity slider will now be more intuitive
by ordering candidates by match count from highest
match count to the left to the lowest match count
to the right.
Candidates with same match counts will be discarded
and replaced with the shortest candidate.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Double-clicking on...
... a filter option will cause the option to be
wholly selected, including `=[value]` if present;
... a value assigned to a filter option will cause
the value to be wholly selected, except when the
value is a hostname/entity, in which case all the
labels from the cursor position to the right-most
label will be selected.
This allows to bring in all the benefits of
syntax highlighting and enhanced editing
features in the element picker, like auto-
completion, etc.
This is also a necessary step to possibly solve
the following issue:
- https://github.com/gorhill/uBlock/issues/2035
Additionally, incrementally improved the behavior
of uBO's custom CodeMirror static filtering syntax
mode when double-clicking somewhere in a static
extended filter:
- on a class/id string will cause the whole
class/id string to be selected, including the
prepending `.`/`#`.
- somewhere in a hostname/entity will cause all
the labels from the cursor position to the
right-most label to be selected (subject to
change/fine-tune as per feedback of filter
list maintainers).
Related feedback:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134#issuecomment-679421316
The redirectable resource has been renamed
`click2load.html`, so as to avoid uses of dash
characters and to also allow for future different
click-to-load resources.
Additionally, as a requirement to support click-to-load
feature, redirected resources will from now on no
longer be collapsed.
Related issues:
- https://github.com/gorhill/uBlock/issues/2688
- https://github.com/gorhill/uBlock/issues/3619
- https://github.com/gorhill/uBlock/issues/1899
This new feature should considered in its draft
stage and it needs to be fine-tuned as per
feedback.
Important: Only embedded frames can be converted
into click-to-load widgets, as only these can be
properly shieded from access by page content.
Examples of usage:
||youtube.com/embed/$3p,frame,redirect=clicktoload
||scribd.com/embeds/$3p,frame,redirect=clicktoload
||player.vimeo.com/video/$3p,frame,redirect=clicktoload
Related feedback:
- https://github.com/uBlockOrigin/uBlock-issues/issues/401#issuecomment-703075797
Name: `uiTheme`
Default: `unset`
Values:
- `unset`: uBO will pick the theme according to
browser's `prefers-color-scheme`
- `light`: force light scheme
- `dark`: force dark theme
This advanced setting is not to be documented yet as
it has not been decided this is a long term solution.
Double-cliking on a URL will cause the whole URL to be
selected, thus making it easier to navigate to this
URL (through your browser "Open in new tab" entry in
contextual menu).
Unrecognized scriptlet names will be highlighted so as
to warn that the filter is not going to be effective.
Injecting declarative CSS `:style()` selector could cause
the instatiation of the procedural filterer, even when
there was no actual procedural cosmetic filter to
enforce.
This commit ensure that the procedural cosmetic filterer
is instantiated only when there are actual procedural
filters to enforce.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/763
Changes:
From now on, uBO will allow click-to-subscribe on only
a few select domains, currently:
- https://filterlists.com/
- https://github.com/
- https://github.io/
More domains can be added if and only the demonstration
is made that more than a marginal number of filter lists
can be subscribed from those domains.
The browser alert box is no longer used to confirm
subscription to a filter list. Instead, the asset
viewer has been expanded to serve that purpose. This
way, users can peruse at the content of a filter list
before subscribing to it.
Now that the element picker user interface is isolated
frompage content world, there is no need to use the
`important` modifier for the element picker's own
CSS styles.
Further iterating on the work done in following commit:
- 1268f0ae43
This commit adds a new widget to the element picker to
control the depth of a cosmetic filter selector. The
new widget is essentially just another way of selecting
the depth, which is still controllable through picking
one of the cosmetic filters in the list of candidates.
In addition to what is deemed really bad lists by consensus,
some lists will also be labelled "really bad list"
temporarily so as to force-remove them from the set of
filter lists.
This will be the case for filter lists which are not
necessarily "bad lists" but which were once part of
uBO's stock filter lists and have been removed since
then for various reasons.
This will ensure that the majority of users who do not
modifies uBO's default listset will still have a
configuration which matches the official default listset.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/851
The ctrl key is no longer used to adjust specificity of
a candidate filter.
A new widget has been added to adjust the specificity of
a candidate filter to various level. The widget will be
visible as long as the candidate filter matches one entry
in the list of suggested candidate cosmetic filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1226
Related commit:
- 9eb455ab5e
In the previous commit, the element picker dialog was
isolated from the page content. This commit is to also
isolate the svg layers from the page content.
With this commit, there is no longer a need for an anonymous
iframe and the isolated world iframe is now directly
embedded in the page.
As a result, pages are now unable to interfere with any
of the element picker user interface. Pages can now only
see an iframe, but are unable to see the content of that
iframe. The styles applied to the iframe are from a user
stylesheet, so as to ensure pages can't override the
iframe's style properties set by uBO.
This commit adds concept of "really bad list" to the
badlists infrastructure. Really bad lists won't be
fetched from a remote server, while plain bad list
will be fetched but won't be compiled.
A really bad list is denoted by the `nofetch` token
following the URL.
Really bad lists can cause more serious issues such
as causing undue launch delays because the remote
server where a really bad list is hosted fails to
respond properly and times out.
Such an example of really bad list is hpHosts which
original server no longer exist.
Many filter lists are known to cause serious filtering
issues in uBO and are not meant to be used in uBO.
Unfortunately, unwitting users keep importing these
filter lists and as a result this ends up causing
filtering issues for which the resolution is always
to remove the incompatible filter list.
Example of inconpatible filter lists:
- Reek's Anti-Adblock Killer
- AdBlock Warning Removal List
- ABP anti-circumvention filter list
uBO will use the following resource to know
which filter lists are incompatible:
- https://github.com/uBlockOrigin/uAssets/blob/master/filters/badlists.txt
Incompatible filter lists can still be imported into
uBO, useful for asset-viewing purpose, but their content
will be discarded at compile time.
Cloud storage is a limited resource, and thus it
makes sense to support data compression before
sending the data to cloud storage.
A new hidden setting allows to toggle on
cloud storage compression:
name: cloudStorageCompression
default: false
By default, this hidden setting is `false`, and a
user must set it to `true` to enable compression
of cloud storage items.
This hidden setting will eventually be toggled
to `true` by default, when there is good confidence
a majority of users are using a version of uBO
which can properly handle compressed cloud storage
items.
A cursory assessment shows that compressed items
are roughly 40-50% smaller in size.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1196
For yet unknown reasons, in Chromium 84 the rendering of
`svg/use`-based icons in the dashboard causes page load
stall and undue high memory usage when the dashboard is
opened, especially the Filter list pane.
The fix in this commit is to avoid using `svg/use` and to
directly create and insert the `path` element defining an
icon.
Added a dotted box around found text occurrences,
as just pale yellow to highlight the text is not
enough to visually distinguish from surrounding text.
Iterating through found text occurrences will now
ensure they are vertically positioned in the middle
of the editor.
Code review following latest changes.
Also, move the input field to the left so that it
renders properly on smaller displays and does not
jump around when the result position/count numbers
change.
This also makes it easier to add more functionality
to the editor's toolbar in the future.
Related commit:
- 23332400f5
Since the search worker can go away after its time-to-live
elapsed, we may need to pass again the haystack on which
search operations are performed.
Before this commit, CodeMirror's add-on for search occurrences
was limited to find at most 1000 first occurrences, because of
performance considerations.
This commit removes this low limit by having the search
occurrences done in a dedicated worker. The limit is now
time-based, and highly unlikely to ever be hit under normal
condition.
With this change, all search occurrences are gathered,
and as a result:
- All occurrences are reported in the scrollbar instead of
just the 1,000 first
- The total count of all occurrences is now reported, instead
of capping at "1000+".
- The current occurrence rank at the cursor or selection
position is now reported -- this was not possible to report
this before.
The number of occurrences is line-based, it's not useful to
report finer-grained occurences in uBO.
Since it's possible to execute specific code paths according
to whether the context is that of a worker or not, it's possible
to keep the main/thread code in a single file. Keeping the
main/worker code paths into a single file is more convenient
for both code maintenance and code review.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1118
Usage of space in network filter patterns will now be
strictly interpreted as the filter being a hosts file
entry.
Usage of space in any other scenario will cause the
pattern of the network filter to be rejected as
erroneous.
The pseudo user styles code served only browsers based
on Chromium 65 and earlier -- Chromium 66 supports
native user styles and was first released more than two
years ago.
In Chromium-based browsers, the pseudo user styles code
is being unconditionally injected in every page/frame
just in case the browser is version 65 or earlier.
Removing pseudo user styles reduce uBO's main content
script in Chromium-based browsers by more than 20K.
Related thread:
- https://github.com/NanoAdblocker/NanoCore/issues/348#issuecomment-653646507
Related commit:
- aed850978e
No need to mind rejected promise after all,
vAPI.tabs.executeScript() is designed to
not fail -- I had forgotten about this.
Related discussion:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1652925
It's not clear the code here will fix the reported
issue, but I did identify that the subframe
dictionary of a very long-lived web page can
theoretically grow unbound.
Regression from:
- 16727d68c8
The issue was causing the element picker to being
unable to select elements with no valid `srcset`
property.
Test case -- trying to select one of the embedded
frames in the following page would fail:
- http://raymondhill.net/ublock/tiles1.html
Lines in AdGuard filter lists have trailing `\r`
characters, and these caused the redirect engine
compile code to reject as invalid the redirect
token.
This is trivially fixed by trimming the raw option
strings before parsing it in the redirect engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
CodeMirror's code folding reference:
- https://codemirror.net/doc/manual.html#addon_foldcode
This commit adds support for code-folding to the filter
list editor/viewer.
The following blocks of code are foldable by clicking the
corresponding marker in the gutter:
- !#if/#endif blocks
- !#include blocks
Addtionally, the following changes:
- The `!#include` line is now preserved when importing a
sublist
- The `!#if` directives will be syntax-colored according
to whether they evaluate to true or false on the current
platform
- Double-clicking on a foldable line in the gutter will
select the content of the foldable block
- Minor visual improvement to matching brackets
Auto-completion will work only for uBO's own
tokens, compatibility-related tokens[1] will not be
taken into account for auto-completion.
The reason is to not have the compatibility-related
tokens get in the way of auto-completion in order
to not inconvenience uBO's filter list maintainers.
[1] `adguard_ext_chromium`, `adguard_ext_firefox`,
etc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Invalid values for `!#if ...` will be highlighted as errors.
Auto completion is now supported for both the directives
themselves and the valid values for `!#if ...`.
For examples, when pressing ctrl-space:
- `!#e` will auto-complete to `!#endif`
- `!#i` will offer to choose between `!#if ` or `!#include `
- `!#if fir` will auto-complete to `!#if env_firefox`
Additionally, support for some of AdGuard preparsing
directives, i.e. `!#if adguard` is now a valid and will be
honoured -- it always evaluate to `false` in uBO.
In case of invalid `Expires` value -- i.e. `NaN` -- do
not use `1` as default value, just let uBO pick the
value according to the global default (which is `5` as
of commit time).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Specifically;
- `beacon`, `ping`, and `websocket` cannot be redirected;
- it's ok to not specify a type when redirecting to `empty`
resource;
- `csp=` option can't be mixed with other types, redirec
directives, and more `csp=` options.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1071
Additionally, match unconditionally against `srcset` attribute
when trying to find matching elements in the page. For example,
sometimes an img element may set both `src` and `srcset`
properties, they should not be deemed mutually exclusive.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/772
Unicode characters inside hostname part of a filter will
be converted to punycode.
Unicode characters anywhere else in the pattern will be
percent-encoded.
Unicode characters which cannot be encoded will cause a
filter to be invalid.
This fix the case of the following filter:
trk*.vidible.tv
Not matching:
https://trk.vidible.tv/trk/.vidible.tv
The wildcard is supposed to match any number of
characters, including zero characters. The issue
is that the code was not matching zero characters.
This is due to an incorrect comparison in
BidiTrieContainer.indexOf(), causing the code to
bail out before testing for the zero character
condition.
The regression broke filters of the form:
||trk*.vidible.tv^
The new parser will eventually interpret differently
wildcard characters when they are used in a manner
meant to represent only hostname-valid characters,
but this will come in a future version -- for now
the default meaning must be preserved until the
static network filtering engine is modified to
enforce the new interpretation.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/hbpo86/
For unknown reasons at this point, it appears some users
end up with a truncated version of EasyList, leading to
improper blocking in uBO.
This commit adds a heuristic to discard a downloaded list
when its new size is more than 25% smaller than the
currently cached version.
This commit adds CodeMirror's auto-completion capability
to the _My filters_ pane.
Currently, auto-completion is available for scriptlet
tokens: pressing ctrl-space while the text cursor is
positioned where a scriptlet token should appear will
cause auto-completion to kick-in. In case of ambiguity,
CodeMirror's widget to pick a specific scriptlet will
appear.
Rename `l` property to `len`, to avoid ambiguity as
`l` could mean _left_ or _length_. Typically `l` is
to be used for _left_ (whereas `r` is to be used for
_right_).
Additionally, add CodeMirror's bracket-matching and
bracket auto-closing to _My filters_ pane and and
bracket-matching to asset viewer page.
Specifically, do not invalidate valid hostnames when
there are extraneous separators: that sort of error
will be visually highlighted but will not otherwise
prevent a filter from being properly enforced.
This ensures proper garbage collection once the parser
is no longer referenced -- this is important now that
the parser is instantiated on-demand only.
Reported internally.
Potential regex flags are passed as is to RegExp contructor,
and in case of failure the query is deemed a plain text one.
Related commit:
- 8de67d22bd (diff-3f4aa453cefa49f6431f1bba3bb53a8e)
This commit moves some of the parsing logic of static
extended filtering into the static filtering parser; this
allows better syntax highlighting and creation-time
error-catching for cosmetic, HTML, and scriptlet filters.
Specifically:
There can't be more than one `redirect` or `csp`
option.
There can be no type specified when there is a
`csp` option.
There must be one single network-related type when
there is a `redirect` option. Since `empty` and
`mp4` imply a `redirect` and a network-related
type, these can't be used along another `redirect`
or network-related type.
Related commit:
- 01b1ed9a98
Reported internally. The issue could cause the logger
to be unable to successfully reverse-lookup a filter
list for a filter which had tokens longer than 6
characters followed by wildcard.
Regression from:
- 01b1ed9a98
Reported internally.
Regression from:
- 01b1ed9a98
The regression was made apparent be the fact that
AdGuard filter lists have lines ending with the
CR character.
Computing storage-used figures can take longer than
usual sometimes when a lot of filter lists are enabled,
and this can cause the Settings pane to take longer to
be filled the first time it is opened. Fetching
settings and storage-used figures separately removes
that potential delay (they were fetched together in
a single Promise.all() call).
Regression from:
- 01b1ed9a98
The new parser needs to be able to deal with trailing
newline characters, which if present will be interpreted
as trailing spaces.
A new standalone static filtering parser is introduced,
vAPI.StaticFilteringParser. It's purpose is to parse
line of text into representation suitable for
compiling filters. It can additionally serves for
syntax highlighting purpose.
As a side effect, this solves:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1038
This is a first draft, there are more work left to do
to further perfect the implementation and extend its
capabilities, especially those useful to assist filter
authors.
For the time being, this commits break line-continuation
syntax highlighting -- which was already flaky prior to
this commit anyway.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1008
This commit adds support entity-matching in the filter
option `domain=`. Example:
pattern$domain=google.*
The `*` above is meant to match any suffix from the Public
Suffix List. The semantic is exactly the same as the
already existing entity-matching support in static
extended filtering:
- https://github.com/gorhill/uBlock/wiki/Static-filter-syntax#entity
Additionally, in this commit:
Fix cases where "just-origin" filters of the form `|http*://`
were erroneously normalized to `|http://`. The proper
normalization of `|http*://` is `*`.
Add support to store hostname strings into the character
buffer of a hntrie container. As of commit time, there are
5,544 instances of FilterOriginHit, and 732 instances of
FilterOriginMiss, which filters require storing/matching a
single hostname string. Those strings are now stored in the
character buffer of the already existing origin-related
hntrie container. (The same approach is used for plain
patterns which are not part of a bidi-trie.)
Pressing spacebar will toggle god mode in popup
panel.
This capability shouldn't encourage people to
misuse _allow_ rules as having to press the
space bar act as an extra necessary step which
purpose is to make the creation of _allow_ rules
more mindful.
There have been too many examples out there of users
opting-in to "I am an advanced user" and yet still misusing
dynamic filtering by creating _allow_ rules where _noop_
rules should be used.
Creating _allow_ rules has serious consequences as these
override blocking static filters and can potentially
disable other advanced filtering ability such as
HTML filtering and scriptlet injection -- often used
to deal with anti-blocker mechanisms.
The ability to point-and-click to create _allow_ rules
from the popup panel is no longer allowed by default.
An new advanced setting has been added to enable
the ability to create _allow_ rules from the popup
panel, `popupPanelGodMode`, which default to `false`.
Set to `true` to restore ability to set _allow_ rules
from popup panel.
Since the creation of _allow_ rules is especially useful
to filter list authors, to diagnose and narrow down site
breakage as a result of problematic blocking filter,
the creation of _allow_ rules will still be available
when the advanced setting `filterAuthorMode` is `true`.
This change is probably going to be problematic to all
those users who were misusing dynamic filtering by
creating _allow_ rules instead of _noop_ rules -- but
the breakage is going to bring their misusing to their
attention, a positive outcome.
Default to `unset`.
To allow users to bypass uBO's default CSS styles in
case they are causing issues to specific users. It is
the responsibility of the user to ensure the value of
`uiStyles` contains valid CSS property declarations.
uBO will assign the value to `document.body.style.cssText`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1044
For example, in the case of the issue above, one could
set `uiStyles` to `font-family: sans-serif` to force uBO
to the system font for its user interface.
FilterPlainHostname, an atomic filter unit, has been
removed and is being replaced with a composite filter
made of a pattern filter and a filter which test
hostname boundaries.
Doing so enables filters formerly being represented
by FilterPlainHostname to be now represented as a
plain pattern, and thus to be potentially stored in
a bidi-trie.
Comparing the new filter histogram with the previous
one:
FilterPatternPlain 24612 26432 1820
FilterComposite 17656 17125 -531
FilterPlainTrie Content 12977 13519 542
FilterPlainHostname 2904 0 -2904
FilterBucket 2121 1961 -160
FilterPlainTrie 1418 1578 160
Which means:
- An extra 542 patterns could be stored in bidi-tries
- There are 531 less composite filters needed
- An extra 160 buckets could be aggregated into 160
bidi-trie
Memory-wise, it's a marginal gain (as per Chromium's
Javascript VM instance figure) -- i.e. not worth
talking about). CPU-wise, no measurable difference.
The benefit is that I consider this conceptually
simplifies slightly the static network filtering
code base.
Specifically, the sticky controls box is re-parented
pre-emptively instead of waiting for the decision as
to whether the panel must be toggled into a vertical
layout mode.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1004
The fix is trivial enough that I feel confident to make
it part of current RC cycle. There are many reports of
block page at launch on Chromium.
The fix is to let the platform pick it's natural suspend
state rather than force it. Once advanced settings are
read, the suspend state will for toggled only if
`suspendTabsUntilReady` is either `yes` or `no`. This
there should no longer be blocked page on Chromium at
launch when using default settings.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/gac2h8/
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1005
There is probably more work to do regarding the
shown URL -- it might be better to truncate it
on small screen display since it can be quite
long sometimes.
In rare instances -- though it definitely happens
eventually -- the popup panel viewport width is seen as
insufficiently wide enough and as a result the popup panel
is toggled into vertical-layout mode.
The added code uses animation frames to delay the
code testing the viewport width. Hopefully this will
work.
The old "classic" popup panel will still be used
when at least one of the following is true:
- advanced setting `uiFlavor` is set to `classic`; or
- the browser is Chromium 65 or older; or
- the browser is Firefox 67 or older
The default configuration of the new popup panel
at installation time is to show the power button,
statistics and the basic tool icons, i.e. access
to dashboard, logger, pickers.
For existing installations, the new popup panel
will be configured by respecting the existing
configuration of the classic one.
The new popup panel is currently already in use
on Firefox for Android, and the visual redesign
was made according to suggestions and feedback
from <https://github.com/brampitoyo> to be
optimal for Firefox for Android.
The new popup panel will allow closing the following
pending issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/255
- https://github.com/uBlockOrigin/uBlock-issues/issues/178
The blocking-stuff button icons have been brought
closer to the power button.
The go-somewhere button icons have been moved at
the bottom.
The hostname of the current site is now shown below
the power button. The base domain is bolded.
This re-ordering is for a few reasons:
- The blocking-stuff button icons are now closer
to the revert/persist/reload transient buttons
which show up when modifying the blocking
behavior of uBO.
- The makes it easier to potentially bring in a
very basic view where only the power button and
the statistics are shown and all the button
icons are hidden from view.
This commit specifically address bringing the
desktop version of the new popup panel's look
and feel more inline with the classic one:
- Hide tool captions on desktop
- Bring back no-popups switch on desktop
- Bring back tooltips on desktop (though they
are now rendered natively by the browser)
- Use the Photon icons suggested by @brampitoyo
for the no-popups and no-remote-fonts
switches
Noteworthy:
- Make new popup panel better resize in desktop
environment
- Fix button visuals in "My rules" pane
- Keep "Update now" button visible until update
is completed
- Renaming CSS variables for consistency
Better constrast for warning color. Related feedback:
- 5bee33253f (commitcomment-38700085)
Revisit how large numbers are rendered in a
compact form in the new popup panel. Feedback
from https://crowdin.com/project/ublock pointed
out that the string to translate was flawed for
Chinese locale (and possibly in some other
locales as well).
The rendering of the total number of blocked
requests will now be abbreviated using `M` and
`G` when the block count is respectively above
1 million and 1 billion.
The storage used figure in the Settings pane
will be rendered using KB, MB or GB.
As per email feedback from Mozilla's
https://github.com/brampitoyo
This is yet another incremental step toward
redesigning the UI, much more is left to do.
The idea is to align uBO's UI to that of
Firefox Preview.
Additionally, code has been added to reset
the new popup panel to vertical layout should
the viewport be not wide enough to
accomodate the horizontal layout.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/g4ufvi/
Related commit:
- 2ac288397c
Instead of having the `localStorage` data being accessed
from different locations, all accesses are now funnelled
to the main process.
Doing so simplifies the code in auxiliary processes and
also remove the need for browser.storage.local.onChanged()
listeners.
No longer using an onChanged() listener also happens to
remove spurious warnings from the Firefox console.
Related commits:
- 0662767dd6
- 99b2a0a761
- a8e3118fea
- 453f5450b6
- a45a21f337
This commit focuses on implementing the use of
CSS `var(...)` throughout so as to make it easy
to create themes -- a requirement for uBO to
support a dark theme.
There is still work to do regarding converting
uBO's CSS to completely support `var(...)` but
being able to start using theming will help
complete and fine tune CSS `var(...)` support.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/401
Related commits:
- a8e3118fea
- 453f5450b6
- a45a21f337
Roughly, the changes in this commit:
- Make it easier to implement a dark theme eventually:
- Flatten the look of buttons
- Remove the use of `opacity` style property as a
color modifier
- Some work toward removing dependency on FontAwesome
font
Related commits:
- 453f5450b6
- a45a21f337
This commit fixes parts of the re-design not
rendering as intended as per <https://github.com/brampitoyo>'s
feedback screenshots.
The dashboard tab buttons have been converted to
`span` tags (from `a` tags), thus eliminating
the unwanted side effect of the text being
rendered as unvisited links (blueish).
The font size intended for touch screens was not
taking effect due to bad cut & paste.
Other minor adjustments to improve consistency
in spacing.
This commit add the ability to fetch from CDN servers
when an asset is fetched as a result of auto-update.
If an asset has a `cdnURLs` entry in `assets.json`,
the asset will be auto-updated using one of those
CDN URLs. When many CDN URLs are specified, those
URLs will be shuffled in order to spread the bandwidth
across all specified CDN servers. If all specified CDN
servers fail to respond, uBO will fall back to usual
`contentURLs` entry.
The `cdnURLs` are used only when an asset is
auto-updated, this ensures a user will get the more
recent available version of an asset when manually
updating.
The motivation of this new feature is to relieve
GitHub from acting as a CDN (which it is not) for
uBO -- an increasing concern with the growing adoption
of uBO along with the growing size of key uBO assets.
Related commit:
- 453f5450b6
The changes in this commit are mostly related
to the "Filter lists" pane.
An eye icon has been added, which purpose is
to view the content of a list. Clicking on the
name of the list will toggle the list's
checkbox.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/974
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/fuscia/
The race condition was that a content script could
query the main process to retrieve cosmetic filters
while the cosmetic filters had not been yet fully
loaded into memory. The fix ensure that an already
injected content script will re-query once the
cosmetic filters are fully loaded in memory at
browser launch time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/911
Since cname-uncloaking is available only on Firefox
at the moment, the fix is relevant only to Firefox.
By default uBO will no longer cname-uncloak when it
detects that network requests are being being proxied.
This default behavior can be overriden by setting the
new advanced setting `cnameUncloakProxied` to `true`.
The new setting default to `false`, i.e. cname-uncloaking
is disabled when uBO detects that a proxy is in use.
This new advanced setting may disappear once the
following Firefox issue is fixed:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1618271
Provide a way to optionally deduplicate filter
instances, useful for filter instances with:
- high likelihood of duplication; and
- non-trivial memory footprint per instance
- For examples, filter instances to implement
`domain=`, `denyallow=`, `csp=`.
Cursory tests show this helps further reduce
uBO's memory footprint.
This concerns the static network filtering engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943
* * *
New static network filter type: `cname`
By default, network requests which are result of
resolving a canonical name are subject to filtering.
This filtering can be bypassed by creating exception
filters using the `cname` option. For example:
@@*$cname
The filter above tells the network filtering engine
to except network requests which fulfill all the
following conditions:
- network request is blocked
- network request is that of an unaliased hostname
Filter list authors are discouraged from using
exception filters of `cname` type, unless there no
other practical solution such that maintenance
burden become the greater issue. Of course, such
exception filters should be as narrow as possible,
i.e. apply to specific domain, etc.
* * *
New static network filter option: `denyallow`
The purpose of `denyallow` is bring
default-deny/allow-exceptionally ability into static
network filtering arsenal. Example of usage:
*$3p,script, \
denyallow=x.com|y.com \
domain=a.com|b.com
The above filter tells the network filtering engine that
when the context is `a.com` or `b.com`, block all
3rd-party scripts except those from `x.com` and `y.com`.
Essentially, the new `denyallow` option makes it easier
to implement default-deny/allow-exceptionally in static
filter lists, whereas before this had to be done with
unwieldy regular expressions[1], or through the mix of
broadly blocking filters along with exception filters[2].
[1] https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0
[2] Typically filters which pattern are of the
form `|http*://`
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/923
Use current page's hostname instead of that of
image URLs to decide whether to reset pattern
union with previous picker sessions.
The fixed issue arose from the fact that the
page uses URLs from different origins in a
single srcset attribute.
Related commit:
- https://github.com/gorhill/uBlock/commit/703c525b01aa
This adds an indentation requirement for line
continuation to take place. The conditions are now
as follow:
- Current line ends with ` \`: ASCII space + backslash
- Next line starts with ` `: four ASCII spaces
If a line in a filter list ends with a space
(ASCII code 32) followed by a backslash
(ASCII code 92), those two characters will be
removed, the line will be trimmed and the next
line will be trimmed and concatenated to form
a new, longer line.
The purpose is to give filter list authors
a way to visually break apart unduly long
filters and thus make maintenance easier.
When line continuation is used, it is suggested
that the extra lines are prepended with four
space so as to make it more visually obvious that
the extra line(s) are the continuation of a
previous line.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943
The filter referenced in the above issue was the
motivation to implement this feature:
- https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0
I verified and could not find any instance in major
filter lists of lines ending with ` \`, thus the
change should be safe.
Reported internally. The following invalid filter was not
discarded by uBO:
123tvnow.com##.123tv-ads
The correct form should be:
123tvnow.com##.\31 23tv-ads
Not discarding invalid CSS selector-based cosmetic
filter may break CSS selector-based cosmetic
filtering.
***
New procedural cosmetic operator: `:remove()`
Related issue:
- https://github.com/gorhill/uBlock/issues/2252
The purpose is to outright remove elements from the
DOM tree. Since `:remove()` is an "action" operator,
it must only be used as a trailing operator (just
like the `:style()` operator).
AdGuard's cosmetic filter syntax `{ remove: true; }`
will be converted to uBO's `:remove()` operator
internally.
***
New procedural cosmetic operator: `:upward(...)`
The purpose is to lookup an ancestor element.
When used with an integer argument, it is synonym of
`:nth-ancestor()`, which will be deprecated and which
will no longer be supported once no longer used in
mainstream filter lists.
Filter lists maintainers must only use `:upward(int)`
instead of `:nth-ancestor(int)` once the new operator
become available in all stable releases of uBO.
`:upward()` can also accept a CSS selector as argument,
in which case the nearest ancestor which matches the
CSS selector will be selected.
Override the Backspace key, not just the Delete key, as Mac keyboards have Backspace as the only delete key and label it Delete.
Source of key value: https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/key/Key_Values#Editing_keys
More background on Mac keyboard shortcuts: Mac keyboards can emulate Delete using fn+Delete, meaning Forward Delete, but Mac software does not use it except in text editing. When deletion is dangerous, Mac software requires holding a modifier key in conjuction with Delete, but I think it’s better to make deletion easy in this case.
This new binding has a potential downside: if the user Backspace key normally goes Back in history (which can differ across OSs and browsers), this will change the behavior to delete the selected element instead. If the user really wants to go back in history, they will have to press Escape to leave the mode and then press Backspace, or they will have to press an alternative keyboard shortcuts such as Alt+Left. I think the user will rarely want to go back in history in the middle of picking an element, though.
That downside could be mitigated by conditioning the key check on `runtime.PlatformOs` (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/PlatformOs). But that would complicate the code a lot compared to the importance of this feature, and such detection would still fail to capture the user’s intent accurately in all cases. I think it’s better to unconditionally accept both Backspace (Delete) and Delete (Forward Delete).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/899
browser.storage.local is now used to store non-critical
local settings.
These settings are all collated under the key
`localStorage`, and vAPI.localStorage is an API to
handle access to these values stored under this key.
vAPI.localStorage.getItem() is still synchronous but
its purpose is to return internally cached values --
this minimizes code changes throughout uBO.
New advanced setting: `benchmarkDatasetURL`
Default value: `unset`
To specify a URL from where the benchmark dataset will be
fetched. This allows to launch benchmark operations from
within published versions of uBO, rather than from just
a locally built version.
A specific cosmetic filter of the following form...
*##.selector
... will be unconditionally injected into all web pages,
whereas a cosmetic filter of the form...
##.selector
... would be injected only when uBO's DOM surveyor finds
at least one matching element in a web page.
The new specific-generic form will also be disabled when a
web page is subject to a `generichide` exception filter,
since the filter is essentially a generic one -- the only
difference from the usual generic form is that the filter
is injected unconditionally instead of through the DOM
surveyor.
Specific-generic cosmetic filters will NOT be discarded
when checking the "Ignore generic cosmetic filters"
option in the "Filter lists" pane -- since the purpose
of this option is primarily to disable the DOM surveyor.
Specific-generic cosmetic filters should be used
parcimoniously and only when using a normal specific
filter is really impractical.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/803
Hostnames which have unpunycoded IDN or canonical
name information will have this information reported
as an extra line under the hostname name in the popup
panel, rather than as a tooltip.
For the fenix UI flavor, the save/revert tools are
now shown regardless of whether the overview area is
visible or not.
More specifically, this commit makes it so that
the new default UI for Firefox Preview will be
automatically selected.
A default platform-specific UI can be cancelled
by setting the advanced setting `uiFlavor` to `-`.
The toolip of a canonical name entry (blue)
in the popup panel will provide the alias
hostname related to the canonical hostname
in order to help with rule-making decisions.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/882
Related commits:
- https://github.com/gorhill/uBlock/commit/a95ef16e064a
- https://github.com/gorhill/uBlock/commit/7971b223855d
Leading wildcards before valid token characters need to
be kept in order to respect the semantic of the filter.
A leading wildcard in such case changes the semantic of
a filter, i.e. two following filters are semantically
different:
example/abc
*example/abc
As a result, µBlock.BidiTrieContainer.indexOf() is now
able to deal with a needle of length zero -- which is
what happens in FilterPatternLeft(Ex) with filter
patterns starting with `*` (or `^*`) and followed by
valid token characters (0-9, a-z and %).
First draft of changes as discussed with Firefox
Preview people.
In order to allow testing/evaluating these changes,
the new advanced setting `uiFlavor` has been added.
Default to `unset`; and can currently only be set
to `fenix`.
The new setting takes effect at launch only. This
new setting is not to be mentioned in official
documentation for now.
This is ongoing work, not open to external feedback.
This commit make it so that if an *unexpected* error is
thrown when trying to load the selfie at launch, the filter
lists will still be loaded as a result.
Tokens which are as long or longer than the max token
length possible do not need to have the prefix part
evaluated against special regex characters. This will
help increase the likelihood of extracting a valid
token from regex-based filters.
Actual case found in EasyPrivacy:
/^https?:\/\/eulerian..*\/[a-z0-9]{2,12}\.js/$script
Before this commit, uBO was not able to extract a
valid token, while now uBO is able to extract `eulerian`
as a valid token (consequently the regex-based filter
will now be evaluated only when the token `euleria` is
found in a URL).
This is related to the list of domains/subdomains in
the dynamic filtering pane of the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/284
Clicking on the empty space of a row will toggle
the visibility of the subdomains.
Additionally, the root context will always be visible
regardless of the expand/collspase state, along with
a visual indicator that a specific domain or subdomain
is the actual root context. (the root context is the
hostname in which local rules are created).
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/issues/805#issuecomment-561500819
uBO was testing for regex-like plain patterns after
prepending `@@` in the case of exception filters, thus
preventing proper detection of regex-like plain
patterns. The filtering engine was not affected, only
the proper rendering of the filter in the logger was
affected.
Advanced setting `cnameAliasList` has been removed.
New advanced settings:
cnameUncloak:
Boolean
Default value:
true
Description:
Whether to CNAME-uncloak hostnames.
cnameIgnoreExceptions:
Boolean
Default value:
true
Description:
Whether to bypass the uncloaking of network requests
which were excepted by filters/rules. This is
necessary so as to avoid undue breakage by having
exception filters being rendered useless as a result
of CNAME-uncloaking.
For example, `google-analytics.com` uncloaks to
`www-google-analytics.l.google.com` and both hostnames
appear in Peter Lowe's list, which means exception
filters for `google-analytics.com` (to fix site
breakage) would be rendered useless as the uncloaking
would cause the network request to be ultimately
blocked.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
Related commit:
- https://github.com/gorhill/uBlock/commit/3a564c199260
This adds two new advanced settings:
- cnameIgnoreRootDocument
- Default to `true`
- Tells uBO to skip CNAME-lookup for root document.
- cnameReplayFullURL
- Default to `false`
- Tells uBO whether to replay the whole URL or just
the origin part of it.
Replaying only the origin part is meant to lower
undue breakage and improve performance by avoiding
repeating the pattern-matching of the whole URL --
which pattern-matching was most likely already
accomplished with the original request.
This commit is meant to explore enabling CNAME-lookup
by default for the next stable release while:
- Eliminating a development burden by removing the
need to create a new filtering syntax to deal with
undesirable CNAME-cloaked hostnames
- Eliminating a filter list maintainer burden by
removing the need to find/deal with all base
domains which engage in undesirable CNAME-cloaked
hostnames
The hope is that the approach implemented in this
commit should require at most a few unbreak rules
with no further need for special filtering syntax
or filter list maintance efforts.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dzw57l/
Each token requires two slots in the token indices
array. This commit fixes uBO breaking when dealing
with very long URLs with lot of distinct tokens in
them.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
- Handle DNS lookup failure
- Skip DNS lookup for non network-based URLs
- Benchmark code to be able to provide an estimate
based on objective measurements regarding added
overhead when DNS lookup is enabled
(quick answer: a complete non-issue)
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
New webext permission added: `dns`, which purpose is
to allow an extension to fetch the DNS record of
specific hostnames, reference documentation:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/dns
The webext API `dns` is available in Firefox 60+ only.
The new API will enable uBO to "uncloak" the actual
hostname used in network requests. The ability is
currently disabled by default for now -- this is only
a first commit related to the above issue to allow
advanced users to immediately use the new ability.
Four advanced settings have been created to control the
uncloaking of actual hostnames:
cnameAliasList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to "uncloak" the hostnames in the list will.
cnameIgnoreList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to NOT re-run the network request through uBO's
filtering engine with the CNAME hostname. This is
useful to exclude commonly used actual hostnames
from being re-run through uBO's filtering engine, so
as to avoid pointless overhead.
cnameIgnore1stParty: boolean.
Default value: true.
Whether uBO should ignore to re-run a network request
through the filtering engine when the CNAME hostname
is 1st-party to the alias hostname.
cnameMaxTTL: number of minutes.
Default value: 120.
This tells uBO to clear its CNAME cache after the
specified time. For efficiency purpose, uBO will
cache alias=>CNAME associations for reuse so as
to reduce calls to `browser.dns.resolve`. All the
associations will be cleared after the specified time
to ensure the map does not grow too large and too
ensure uBO uses up to date CNAME information.
Related issue:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1588916
Due to browser behavior changes during development,
uBO can end up using obsolete cached compiled filter
lists or selfie.
Saving magic format values in the cache storage
itself ensure uBO will properly detect actually
obsolete compiled lists/selfie.
