This fixes https://github.com/uBlockOrigin/uBlock-issues/issues/2240 and
should get the desired behavior regardless of browser.
Delay showing the iframe until load to prevent flashing a white
background on the initial about:blank.
This commit deprecates matches-css-before() and matches-css-after(): these
should no longer be used once 1.45.0 is published and widespread. The
deprecated syntax will eventually be removed in some future.
The syntax of procedural operator matches-css() has been extended to also
be able to target pesudo elements. Examples:
Same as before:
example.com##p:matches-css(opacity: 0.5)
This is the new way to target an `::after` pseudo-element:
example.com##p:matches-css(after, content: Ads)
This is the new way to target a `::before` pseudo-element:
example.com##p:matches-css(before, content: Ads)
The new syntax also means any valid pseudo-element can now be used as
a target:
example.com##p:matches-css(first-letter, opacity: 0.5)
If the first argument does not match the pattern "property name: value",
then it will be deemed a pseudo-element to target, and the second argument
will be the "property name: value".
Related issue:
- https://github.com/AdguardTeam/ExtendedCss/issues/150
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/2185
The argument must be a valid media query as documented on MDN, i.e.
what appears between the `@media` at-rule and the first opening
curly bracket (including the parentheses when required):
- https://developer.mozilla.org/en-US/docs/Web/CSS/Media_Queries/Using_media_queries
Best practice:
Use `:matches-media()` after plain CSS selectors, if any.
Good:
example.com###target-1 > .target-2:matches-media((min-width: 800px))
Bad (though this will still work):
example.com##:matches-media((min-width: 800px)) #target-1 > .target-2
The reason for this is to keep the door open for a future optimisation
where uBO could convert `:matches-media()`-based filters into CSS media
rules injected declaratively in a user stylesheet.
These functions were renamed in 2018, before the WebAssembly 1.0 spec
was finalized. wabt 1.0.25 dropped support for pre-1.0 names and the
sources fail to compile with errors like:
```
$ wat2wasm lz4-block-codec.wat
lz4-block-codec.wat:71:5: error: unexpected token get_local, expected ).
get_local $ilen
^^^^^^^^^
lz4-block-codec.wat:78:5: error: unexpected token get_local.
get_local $ilen
^^^^^^^^^
```
If the removed stock list is labelled a "bad list", do not
convert it into an imported list.
This will allow to seamlessly merge resource-abuse stock list
with privacy stock list when 1.42.0 is widespread.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/2041
The value 0xFFFFFFFF will be used instead of 0 to mark the end of
a sequence of tokens, as the value 0xFFFFFFFF can't happen as a
result of computing a token hash, since the four most significant
bits are always 0 in a computed token hash.
The purpose of this new procedural operator is to target
all elements _outside_ than the currently selected set of
elements.
For any element feeding into `others()`, the resultset
of the `others()` operator will include everything else
except:
- the descendants of a subject element
- the ancestors of a subject element
The resultset will contains the siblings of a subject
element _except_ when those siblings are either a
descendant or ancestor of another subject element.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/slyjzp/
Though this operator is unlikely to be used in default lists,
it opens the door to create specialized filter lists which
purpose is some sort of "reader mode", where everything
_else_ than a selected set of elements are hidden from view.
Examples of usage:
twitter.com##:matches-path(/^/home/) [data-testid="primaryColumn"]:others()
nature.com##:matches-path(/^/articles//) :is(.c-breadcrumbs,.c-article-main-column):others()
The status is currently considered experimental and support
might be removed in the future if it turns out there is no
sufficient usage or if unforeseen difficult issues arise
implementation-wise.
The Create button was being unduly enabled for a fraction of
second while editing the content of the filter text.
This commit is candidate for a revision to current stable
release.
uBO will use the information from prefers-color-scheme to reduce
likelihood of background color flash. However this works only for
when prefers-color-scheme is properly set by the browser, and only
when uBO's theme selection is "auto", or when it happens to
match that of prefers-color-scheme.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1284
Too many changes to list here, essentially there is now a
user interface setting to enable/disable dark theme, and
I've rearranged a bit the Settings pane as a result and
also altered other visuals in various places.
There are places which I know have not been thoroughly
tested (i.e. logger inspector).
Will fine-tune as per feedback.
Issues with the classic popup panel will not be addressed,
and if feedback is that it has become unusuable, it will be
outright removed.
Reported internally.
The issue involves `removeparam` filters with a regex value. When
such filter was visited before a selfie was created, this would cause
the created selfie to persist a RegExp object, which can't be
serialized. This would cause exceptions to be thrown when uBO would
be subsequently loaded with the tainted selfie, since uBO would try
to execute a plain Object as a RegExp.
This will allow to find out what is specifically blocked on the
page reported as having issues, potentially saving time
when volunteers try to diagnose issues.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1912
Related commit:
- 7713597e3e
In above related commit, uBO was modified to cache the URL of the
last clicked link, and to use this cached URL to test for the
legitimacy of the newly opened tab.
The current commit add back a test against the URL of the last
clicked link to avoid false positives when navigating from within
the newly opened tab.
Related discussion:
- a0a9497b4a (commitcomment-62560291)
The new setting, when disabled (enabled by default), allows a user
to prevent uBO from waiting for all filter lists to be loaded
before allowing network activity at launch. The setting is enabled
by default, meaning uBO waits for all filter lists to be loaded in
memory before unsuspending network activity. Some users may find
this behavior undesirable, hence the new setting.
This gives the option to potentially speed up page load at launch,
at the cost of potentially not properly filtering network requests
as per filter lists/rules.
For platforms not supporting the suspension of network activity,
the setting will merely prevent whatever mechanism exists on the
platform to mitigate improper filtering of network requests at
launch. For example, in Chromium-based browsers, unchecking the
new setting will prevent the browser from re-loading tabs for
which there was network activity while in "suspended" state at
launch.
As the trie is not immediately created, in order to speed up
launch time, the `domain=` option was stored in the filterRefs
array until it was moved to the trie.
This commit instead stores the `domain=` option into the trie
container's character buffer.
This commit will force-reload active tabs at launch for
environments not supporting suspend network request listeners,
or configured to not suspend network request listeners.
Refactored heuristics to collate set of origin-related
filter units are collated into a hostname trie, and
for better reuse of existing classes.
Generalized pre-test idea for bucket of filters, such
that in addition to origin-related filter units, there is
now a class to collate regex-based pattern-related units
into a new pre-test bucket class, FilterBucketIfRegexHits,
in order to test with a single regex test whether there is
a chance of a hit in the underlying bucket of filters.
Instances of these are rare, but at time of commit I found
this occurs with AdGuard France filter list.
Fine-tuned the "SNFE: Dump" output -- this new ability to
see the internal details of the SNFE has been really key
into finding/fixing issues during refactoring.
As was done with generic pattern-based filters, the source
string of regex-based filters is now stored into the
bidi-trie (pattern) buffer.
Additionally, added a new "dev tools" page to more
conveniently peer into uBO's internals at run time, without
having to do so from the browser's dev console -- something
which has become more difficult with the use of JS modules.
The new page can be launched from the Support pane through
the "More" button in the troubleshooting section.
The benchmark button in the About pane has been moved to this
new "dev tools" page.
The new "dev tools" page is for development purpose only,
do not open issues about it.
There are currently over 160 patterns with such pointless
trailing `*^` in uBO's filter lists, which ended up being
compiled as generic pattern filters (i.e. regex-based
internally), while the trailing `*^` accomplishes nothing
since it will always match the end of a URL ( `^` can
also match the end of URL).
This commit discards pointless trailing `*^` in patterns,
thus allowing most of those filters to be compiled as
plain pattern filters.
The syntax highlighter will reflect that a trailing
`*^` is pointless.
Rearrange logic to instantiate and add `important` filters
to the block realm when compiled lists are loaded instead
of when lists are compiled.
Additionally, removed now unused properties following
commit 68e14793cc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1863
As per internal discussion with team, best to have a simpler
scriplet, and which is hard-coded to work only on a specific
set of domains -- only those seen used by BAB.
Turns out the various benchmarks show no benefits when compiling
filters whose pattern contains a single wildcard character into
specialized classes which threat the pattern as two sub-patterns,
and actually there is a slight improvement in performance as per
benchamrks when treating these patterns as generic ones.
This also fixes the following related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1207
Fixed serious regression in previous dev build in applying
`csp=` filters. Reported internally by uBO team.
Promote usage of `removeparam` in code instead of `queryprune`,
which is to be deprecated.
Removed test against previously tested hostname in
FilterHostnameDict since as per various benchmark, the
test does not really help.
Remove serialization API in Node.js code as the API is now
present in SNFE itself.
All the auxiliary data structures must be fully loaded before
the data structure used as entry point is populated. The race
condition could lead to a case of the entry point data structure
being populated while the auxiliary data structures are still
unpopulated, potentially causing exceptions to be thrown at
launch when the static network filtering engine is queried.
I haven't been able to reproduce such exceptions -- but it
could happen on browsers which do not support being suspended
at launch time (i.e. chromium-based browsers).
Additionally, added convenience methods to easily
serialize/unserialize when SNFE is used as a npm package.
Related feedback:
- https://github.com/orgs/uBlockOrigin/teams/ublock-issues-volunteers/discussions/293
Related commit:
- 725e6931f5
Through all the changes, forgot to pay attention to scenarios
where the `filterData` needs to grow -- the buffer's defautl
size is set to accomodate default filter lists, and subscribing
to more lists would cause the static network filtering engine
to fail because the buffer was not resized when needed.
The original motivation is to further speed up launch time
for either non-selfie-based and selfie-based initialization
of the static network filtering engine (SNFE).
As a result of the refactoring:
Filters are no longer instance-based, they are sequence-of-
integer-based. This eliminates the need to create instances
of filters at launch, and consequently eliminates all the
calls to class constructors, the resulting churning of memory,
and so forth.
All the properties defining filter instances are now as much
as possible 32-bit integer-based, and these are allocated in a
single module-scoped typed array -- this eliminates the need
to allocate memory for every filter being instantiated.
Not all filter properties can be represented as a 32-bit
integer, and in this case a filter class can allocate slots
into another module-scoped array of references.
As a result, this eliminates a lot of memory allocations when
the SNFE is populated with filters, and this makes the saving
and loading of selfie more straightforward, as the operation
is reduced to saving/loading two arrays, one of 32-bit
integers, and the other, much smaller, an array JSON-able
values.
All filter classes now only contain static methods, and all
of these methods are called with an index to the specific
filter data in the module-scoped array of 32-bit integers.
The filter sequences (used to avoid the use of JS arrays) are
also allocated in the single module-scoped array of 32-bit
integers -- they used to be stored in their own dedicated
array.
Additionally, some filters are now loaded more in a deferred
way, so as reduce uBO's time-to-readiness -- the outcome of
this still needs to be evaluated, time-to-readiness is
especially a concern in Firefox for Android or less powerful
computers.
Add ability to bring back logger button in popup panel through
the advanced setting `uiPopupConfig`. Adding `+logger` token
to `uiPopupConfig` will bring back the logger icon in the mobile
version of the popup panel.
Additionally, the link to the logger in the Support pane will
take into account whether the <Shift> key is pressed, so as
to behave like the logger icon in the popup panel.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1847
The troubleshooting information has been further fine-tuned to
report popup panel data related to the reported page, for better
diagnosis by disclosing any customization to uBO which was
affecting the reported page.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1836
The URL to report can now be picked from a list of related
URLs in order to allow the reporter to publish edited version
of the reported URL.
Additionally, the hash, user name, and password which could be
present in a reported URL are always removed.
Unredacted settings is unlikely to be useful after all,
and removing the ability to unredact ensure users won't
mistakenly publish unredacted information.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1541
A "chat" icon has been added to the popup panel to make
it easy to report filter issue on specific sites.
Reporting filter issues require a GitHub account, since
uBO does not have a home server through which reports could
be sent.
The report icon is available only for when uBO is enabled
on a given site.
On mobile devices, the logger icon is replaced by the "chat"
icon since it is more likely to be useful on small display
devices. The logger can always be opened from the Support
pane in the dashboard.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1791
The following case of invalid syntax was not reported as
invalid by the syntax highlighter:
... example.com image ...
With dynamic filtering, there can't be a specific
hostname when a specific type is used, or a
specific type when a specific hostname is used, one
or the other must be `*`.
uBO support's `#?#`, which in AdGuard and ABP means that a
cosmetic filter is procedural.
However, uBO interprets this syntax as "probably procedural"
and will use the filter in a declarative way if the filter
is found to be stylesheet-compatible.
In reality though, the likelihood that a "probably procedural"
filter is sheet-selectable is very low, so treating the filter
as procedural a priori help saves pointless tests against
sheet-selectability when using lists primarily designed for
AdGuard or ABP.
Related commits:
- 4f923384de
- 97a33c9572
- ef07171f5a
For instance, with "Experimental Web Platform features" enabled, the
following filter becomes natively query-selectable:
.fail:has(+ a > b)
Meaning uBO won't need to emulate the `:has()` operator, it will
be executed natively using `querySelectorAll()`.
This commit fixes the erroneous assumption that a query-selectable
is also sheet-selectable.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1751
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/qgd6fe/
It turns out Chromium has started to implement the `:has()`
operator, which becomes recognized when the browser flag
"Experimental Web Platform features" is enabled. However the
hic is that `:has()` is not supported as a declarative CSS
style rule and is only supported through `querySelector()`
et al.
The fix is to no longer detect plain CSS selectors through
`querySelector` et al. but rather use an actual stylesheet
to validate that a cosmetic filter can be injected into a
stylesheet in a declarative way.
Additionally, I added support to enforce ABP's semantic
regarding cosmetic filter with the `#?#` anchor: when using
such anchor, uBO will _first_ try to compile the filter as
a procedural one rather than a declarative one.
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1011#issuecomment-884824166
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/757
Sometimes a tab event may fire for a tab which is not
yet known to uBO. In such case, bind the tab internally
so that it can be processed properly in the future.
Related commit:
- a3a43c7cb4
Specifically:
- Support i18n
- Autofill issues opened through Support pane with configuration
information
- Remove from About pane items found in Support pane
For now the language locales are not available as the text on
the page needs to stabilize before asking translation
volunteers to contribute their time working on the new text.
By default uBO assumed the Shortcut pane was needed,
unless it found the current version of FF was higher
than 73. This commit reverses the test, it assumes
the Shortcut pane is not needed, unless the current
version is lower than 74.
isBlockImportant() was relying strictly on the hash bits
to detect whether a matching filter was `important`, but
this approach regressed with changes with how `important`
filters are compiled. This commit fixed this by no longer
relying on the hash bits but rather on an internal
register variable being set by `important` filters when
they match.
I couldn't find any actual cases in default filter lists
(including a couple of default regional lists) that the
regression is having any effect, due to the limited cases
for which isBlockImportant() is called.
A test was added in a previous commit to detect such
regression in the future:
- a76935b232
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1732
The regression affect filter with the `important` option when
the following conditions were fulfilled:
- The filter pattern is pure hostname
- The filter has not one of the following options:
- domain
- denyallow
- header
- strict1p, strict3p
- csp
- removeparam
- There is a matching exception filter
Related commit:
- a2a8ef7e85
A related mocha test has been added in order to detect this
specific regression in the future through `make test`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1730
A new filter unit, FilterNotType, is introduced to enforce
negated filter type options.
Before this commit, there was no actual negated types in the
static network filtering engine, as a negated type was internally
converted to non-negated types at compile time. As a result,
the logger would never output a matching filter with its original
negated type options.
This commit no longer causes an internal conversion to take place
at compile time, but explicitly enforce negated types at match time,
and as a result the logger will from now on output matching filter
with their original negated type options.
Name: modifyWebextFlavor
Value: A list of space-separated tokens to be added/removed from the
computed default webext flavor.
The primary purpose is to give filter list authors the ability to
test mobile flavor on desktop computers. Though mobile versions of
web pages can be emulated using browser dev tools, it's not
possible to do so for uBO itself.
By using `+mobile` as a value for this setting will force uBO
to act as if it's being executed on a mobile device.
Important: this setting is best used in a dedicated browser
profile, as this affects how filter lists are compiled. So best
to set it in a new browser profile, then force all filter lists
to be recompiled, and use the profile in the future when there
is a need to test the specific webext flavor.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1692
The ids/classes from html/body elements will leave out
looking up lowly generic cosmetic filters made of a single
identifier.
This does not absolutely guarantee that html/body elements
will never be targeted, but it should greatly mitigate the
probability that this erroneously happens.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1690
New procedural operator: `:matches-path(...)`
Description: this is a all-or-nothing passthrough operator, which
on/off behavior is dictated by whether the argument match the
path of the current location. The argument can be either plain
text to be found at any position in the path, or a literal regex
against which the path is tested.
Whereas cosmetic filters can be made specific to whole domain,
the new `:matches-path()` operator allows to further narrow
the specificity according to the path of the current document
lcoation.
Typically this procedural operator is used as first operator in
a procedural cosmetic filter, so as to ensure that no further
matching work is performed should there be no match against the
current path of the current document location.
Example of usage:
example.com##:matches-path(/shop) p
Will hide all `p` elements when visiting `https://example.com/shop/stuff`,
but not when visiting `https://example.com/` or any other page
on `example.com` which has no instance of `/shop` in the path part
of the URL.
So as to allow nodejs usage to better deal with
out of date serialization/compilation.
Additionally, use FilterImportant() only when a
"block-important" filter is stored in the "block" realm.
When matching a network request in the static network filtering
engine ("snfe"), these are the possible outcomes, from most
to least likely:
- No block
- Block
- Unblock ("exception" filter overriding the block)
- Block-important ("important" filter override the unblock)
Hence why the matching in the snfe always check for a match in
the "block" realm, and the "unblock" realm would be checked
if and only if there was a match in the "block" realm.
However the "block-important" realm was always matched against
first, and when a match in that realm was found, there would
be no need to check in other realms since nothing can override
the "important" option. The problem with this approach though
is that matches in the "block-important" realm are most
unlikely, which means pointless work being done for vast
majority of network requests.
This commit makes it so that the "block-important" realm is
matched against ONLY when there is a matched "unblock" filter.
The result is a measurable improvement in the snfe-related
benchmarks (though given the numbers involved, end users won't
perceive a difference).
Somewhat related discussion which was the motivation to look
more into this:
https://github.com/cliqz-oss/adblocker/discussions/2170#discussioncomment-1168125
Whereas before the string segment was encoded as:
LL OOOOOOOOOOOO
where L are the upper 8 bits and used to encode the length
of the segment, and O are the lower 24 bits and used to
encode the offset of the string data in the character
buffer, the new code encode as follow:
OOOOOOOOOOOO LL
And furthermore the most significant bit of the length
LL is now used to mark whether the current string segment
is a label boundary.
This means a cell can't reference a segment longer then
127 characters. To work around this limitation for when a
segment is longer than 127 characters (a rare occurrence),
the algorithm will simply split the segment into multiple
adjacent cells.
As a result, there is no longer a need to encode
"boundariness" into special cells, which simplifies
both the storing and matching algorithms.
Additionally, added minimal documentation for the NPM
package on how to import and use HNTrieContainer as a
standalone API.
The erroneous test does not seem to interfere
with the proper functioning of the trie, due
to the fact that nodes are never split without
a OR node or boundary node being present.
The issue was found when undertaking a rewrite
of the algorithm to avoid having to create
boundary nodes.
In the static network filtering engine (snfe), the
compiling-related code was spread across two classes.
This commit makes it so that all the compiling-related
code is in FilterCompiler class, which clear purpose is
to compile raw filters into a form which can be persisted
and later fed to the snfe with no parsing overhead.
To compile raw static network filter, the new approach is:
snfe.createCompiler(parser);
Then for each single raw filter to compile:
compiler.compile(parser, writer);
The caller is responsible to keep a reference to the
compiler instance for as long as it is needed. This removes
the need for the clunky code used to keep an instance of
compiler alive in the snfe.
Additionally, snfe.tokenHistograms() has been moved to
benchmarks.js, as it has no dependency on the snfe, it's
just a utility function.
The code exported to nodejs package was revised to use modern
JavaScript syntax. A few issues were fixed at the same time.
The exported classes are:
- DynamicHostRuleFiltering
- DynamicURLRuleFiltering
- DynamicSwitchRuleFiltering
These related to the content the of "My rules" pane in the
uBlock Origin extension.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
This change allows to add the redirect engine into the
nodejs package. The purpose of the redirect engine is to
resolve a redirect token into a path to a local resource,
to be used by the caller as wished.
Invalid URLs like "http://" and "http://foo@" trigger TypeErrors
when they are passed to the URL constructor. These TypeErrors
caused the scriptlet to stop processing subsequent noscript nodes
due to uncaught exceptions.
These exceptions are now caught to allow all noscript nodes to
be processed.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The various filtering engine benchmarking functions are best
isolated in their own file since they have specific
dependencies that should not be suffered by the filtering
engines.
Additionally, moved decomposeHostname() into uri-utils.js
as it's a hostname-related function required by many
filtering engine cores -- this allows to further reduce
or outright remove dependency on `µb`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
The title of tabs in uBO is solely to have a better
presentation in the logger -- no other purpose.
This commit simplify keeping track of the titles, from
an active approach by directly querying it from tabs
whenever a change occurs, to a passive approach by
storing it when the title string become available in
some tab event handlers.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/oicch9/
The new replacement script contains the smallest API
possible to resolve the reported case.
Please report instances where it's not sufficient to
unbreak a site, in which case I will extend the neutered
API to address these cases on an on-demand basis.
Related issue:
- https://github.com/gorhill/uBlock/issues/3212
The element picker will now properly work on sites where
cosmetic filtering is disabled, but will not allow the
creation of cosmetic filters when specific cosmetic filters
are not meant to be enforced in the current page.
When specific cosmetic filters are not meant to be enforced,
the element picker will still allow the creation of network
filters, that is unless the current page is trusted, in which
case using the element picker is pointless.
Related issue:
- https://github.com/gorhill/uBlock/issues/3037
This takes care of the specific case reported. There are
other edge cases which are likely not addressed though, i.e.
those involving wildcards -- those should be rather rare and
at this point I rather leave them unaddressed to not
risk regressions (as they are less trivial to address).
Related feedback:
- https://ilakovac.com/teespring-ublock-issue/
The surrogate script googletagmanager_gtm.js was essentially a
subset of surrogate script google-analytics_analytics.js. This
commit makes it a plain alias so that the whole GA API -- often
expected by clients of GTM -- is properly stubbed.
Reported internally:
> STR --
>
> Import https://cdn.statically.io/gh/uBlockOrigin/uAssets/master/filters/filters.txt
> as a Custom filter list.
>
> Observe the filter count at 24K instead of true count
> being 29K.
>
> Force updating is sometimes compiling 24K filters and
> on subsequent updates 29K, so I narrowed it down to
> this host.
Findings:
One of the sublists was erroring when being fetched on
this particular CDN server (reason unknown).
uBO was not properly handling network errors when
fetching a sublist. This commit make it so that if
a sublist can't be fetched, then the error is propagated
as if it affected the whole list, in which case uBO will
use an alternative URL if any.
This advanced setting is not really needed, as the
same can be accomplished with a broad exception
filter such as `#@#+js()`.
Related feedback:
- f5b453fae3 (commitcomment-49499082)
Ever since the `redirect` code was refactored:
157cef6034
This advanced setting is no longer needed, as the same
can be accomplished with a plain network filter:
@@*$redirect-rule
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1553
This commit ensures FLoC is opt-in. The generic filter
`*##+js(no-floc)` in "uBlock filters -- Privacy" ensures
the feature is disabled when using default settings/lists.
Users can opt-in to FLoC by adding a generic exception
filter to their custom filters, `#@#+js(no-floc)`; or they
can opt-in only for a specific set of websites through a
more specific exception filter:
example.com,shopping.example#@#+js(no-floc)
The syntax to remove response header is a special case
of HTML filtering, whereas the response headers are
targeted, rather than the response body:
example.com##^responseheader(header-name)
Where `header-name` is the name of the header to
remove, and must always be lowercase.
The removal of response headers can only be applied to
document resources, i.e. main- or sub-frames.
Only a limited set of headers can be targeted for
removal:
location
refresh
report-to
set-cookie
This limitation is to ensure that uBO never lowers the
security profile of web pages, i.e. we wouldn't want to
remove `content-security-policy`.
Given that the header removal occurs at onHeaderReceived
time, this new ability works for all browsers.
The motivation for this new filtering ability is instance
of website using a `refresh` header to redirect a visitor
to an undesirable destination after a few seconds.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1513
Prior to this commit, the ability to enable/disable the
uncloaking of canonical names was only available to advanced
users. This commit make it so that the setting can be
toggled from the _Settings_ pane.
The setting is enabled by default. The documentation should
be clear that the setting should not be disabled unless it
actually solves serious network issues, for example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1694404
Also, as a result, the advanced setting `cnameUncloak` is no
longer available from within the advanced settings editor.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1501
Exception filters for `document` option are complying with
uBO's own semantic for `document` option, i.e. an exception
filter for `document` option will only allow to bypass a
block filter for `document` (either explicit or implicit)
and nothing else.
Exception filters using `document` option are *not*
compatible with ABP's interpretation of these filters.
Whereas in ABP the purpose of a `document` exception filter
is to wholly disable content blocking, in uBO the same
filter will just cause strict-blocking to be disabled while
leaving content blocking intact.
Additionally, the logger was fixed to properly report pages
which are being strict-blocked.
The procedural cosmetic filtering code has been split from
the content script code injected unconditionally and will
from now on be injected only when it is needed, i.e. when
there are procedural cosmetic filters to enforce.
The motivation for this is:
https://www.debugbear.com/blog/2020-chrome-extension-performance-report#what-can-extension-developers-do-to-keep-their-extensions-fast
Though uBO's content script injected unconditionally in all
pages/frames is relatively small, I still wanted to further
reduce the amount of content script code injected
unconditionally: The procedural cosmetic filtering code
represents roughly 14KB of code the browser won't have to
parse/execute unconditionally unless there exists procedural
cosmetic filters to enforce for a page or frame.
At the time the above article was published, the total
size of unconditional content scripts injected by uBO was
~101 KB, while after this commit, the total size will be
~57 KB (keeping in mind uBO does not minify and does not
remove comments from its JavaScript code).
Additionally, some refactoring on how user stylesheets are
injected so as to ensure that `:style`-based procedural
filters which are essentially declarative are injected
earlier along with plain, non-procedural cosmetic filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/210
Additionally, a small (experimental) widget has been added
to emphasize/de-emphasize rows which have 3rd-party
scripts/frames, so as to more easily identify which rows
are "affected" by 3rd-party scripts and/or frames.
Tooltip localization for the new widget is not available
yet as I want wait for the feature to be fully settled.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1480
Forward compatiblity was broken due to `externalLists`
being converted into an Array from a string, i.e.
downgrading to uBO 1.32.4 was completely breaking uBO.
This commit restores `externalLists` as a string which
is what older versions of uBO expect.
A new property `importedLists` has been created to
hold the imported lists as an array, while
`externalLists` will be kept around for a while until
it is completely removed in some future.
The managed `userSettings` entry is an array of entries,
where each entry is a name/value pair encoded into an array
of strings.
The first item in the entry array is the name of a setting,
and the second item is the stringified value for the
setting.
This is a more convenient way for administrators to set
specific user settings. The settings set through
`userSettings` policy will always be set at uBO launch
time.
The new entry is an array of strings, each representing a
distinct line, and all entries are used to populate the
"My filters" pane.
This offers an more straightforward way for administrators
to specify a list of custom filters to use for all
installations.
Content scripts can't properly look up effective context
for sandboxed frames. This commit add ability to extract
effective context from already existing store of frames
used for each tab.
The entry `toOverwrite.filterLists` is an array of
string, where each string is a token identifying a
stock filter list, or a URL for an external filter
list.
This new entry is to make it easier for an
administrator to centrally configure uBO with a
custom set of filter lists.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1433
The new "extraTrustedSiteDirectives" policy is an array
of strings, each of which is parsed as a trusted-site
directive to append to a user's own set of trusted-site
directives at launch time.
The added trusted-site directives will be considered as
part of the default set of directives by uBO.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1241
uBO will not discard secondary requests fired before a root
frame is committed, by ensuring that if newly uncommitted
root frames are of the same origin as previous one(s), the
uncommited journal slot pointer is not updated.
Related commit:
- 6ac09a2856
Patternless `queryprune` ar enow preserved as being
pattern-less while still attempting to extract a token
from the `queryprune` value. This allows to report the
filter in the logger same as its original form.
The following regex are not rejected as invalid when
using built-in regex objects:
/abc]/
/a7,18}/
/a{7,18/
However, as per documentation, they are not supposed to
be valid, as `{` and `}` are special characters and as
such should be escaped:
/abc\]/
/a7,18\}/
/a\{7,18/
With this commit, the regexes will additionally be
validated using the regex analyzer library in the editor
to ensure strict regex syntax compliance so as to avoid
what are likely mistakes in regex crafting by authors.
This commit fixes mouse double-click-and-drag operations,
which was broken due to the implementation of a custom
word selection in the filter list editor/viewer.
Regex-based static network filters are those most likely to
cause performance degradation, and as such the best guard
against undue performance degradation caused by regex-based
filters is the ability to extract valid and good tokens
from regex patterns.
This commit introduces a complete regex parser so that the
static network filtering engine can now safely extract
tokens regardless of the complexity of the regex pattern.
The regex parser is a library imported from:
https://github.com/foo123/RegexAnalyzer
The syntax highlighter adds an underline to regex-based
filters as a visual aid to filter authors so as to avoid
mistakenly creating regex-based filters. This commit
further colors the underline as a warning when a regex-based
filter is found to be untokenizable.
Filter list authors are invited to spot these untokenizable
regex-based filters in their lists to verify that no
mistake were made for those filters, causing them to be
untokenizabke. For example, what appears to be a mistake:
/^https?:\/\/.*\/sw.js?.[a-zA-Z0-9%]{50,}/
Though the mistake is minor, the regex-based filter above
is untokenizable as a result, and become tokenizable when
the `.` is properly escaped:
/^https?:\/\/.*\/sw\.js?.[a-zA-Z0-9%]{50,}/
Filter list authors can use this search expression in the
asset viewer to find instances of regex-based filters:
/^(@@)?\/[^\n]+\/(\$|$)/
This should improve usability of uBO's hard-mode
and "relax blocking mode" operations. This is the
new default behavior.
The previous behavior of forcing a reload of the
page can be re-enabled by simply setting the `3p`
bit of the advanced setting `blockingProfiles`
to 1.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1204
Not much can be done beside reporting to tabless network
requests to all tabs for which the context is a match.
A short term local cache is used to avoid having to iterate
through all existing tabs for each tabless network request
just to find and report to the matching ones -- users
reporting having a lot of opened tabs at once is not so
uncommon.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/857
The recognized resources are:
- abp-resource:blank-mp3
- abp-resource:blank-js
ABP's tokens are excluded from auto-complete so as to not
get in the way of uBO's filter list maintainers.
Reported internally by @uBlock-user.
Also, fixed broken caching of `cname` exception, which forced
uBO to repeatedly evaluate whether a `cname` exception exists
when a block `cname`-cloaked request is encountered.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1365
This commit adds the compiled magic version number to the
compiled data itself, and consequently this allows uBO
to no longer require that any given compiled list with a
mismatched format to be detected and discarded at launch
time.
Given this change, uBO no longer needs to rely on the
deletion of cached data at launch time to ensure it
won't use no longer valid compiled lists.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1388
Fixed the special `none` redirect resource no longer being
enforced.
Fixed the enforcement of `important` redirect rules over
exceptions and non-important ones.
Related issue:
- https://github.com/gorhill/uBlock/issues/1744
A new context menu entry, "Block element in frame...", will
be present when right-clicking on a frame element. When
this entry is clicked, uBO's element picker will be
launched from within the embedded frame and function the
same way as when launched from within the page.
This is particularly helpful for static network filters
used with filter options causing the same pattern to be
reused across multiple filter instances, i.e. `all` or
`~css`, etc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1365
When compiled data format changes, do not rely on order
of operations at launch to assume deletion of storage
occurs before attempts to access it. It's unclear this
commit will fix the reported issue, as I could not
reproduce it except when outright commenting out the code
to prevent the storage deletion from occurring.
All matching `redirect-rule` directives will now be reported
in the logger, instead of just the effective one.
The highest-ranked redirect directive will be the one
effectively used for redirection. This way filter list
authors can see whether a lower priority redirect is
being overriden by a higher priority one.
The default priority has been changed to 10, so as to allow
more leeway to create lower ranked redirect directives.
Additonally, rendering of redirect directives with explicit
priority has been fixed in the logger, they will no longer
be rendered as unknown redirect tokens.
The header value is no longer implicitly a regex-based literal, but
a plain string against which the header name is compared. The value can
be set to a regex literal by bracing the header value with the usual
forward slashes, `/.../`.
Examples:
*$1p,strict3p,script,header=via:1.1 google
*$1p,strict3p,script,header=via:/1\.1\s+google/
The first form will cause a strict comparison with the value of the header
named `via` against the string `1.1 google`.
The second form will cause a regex-based test with the value of the header
named `via` against the regex `/1\.1\s+google/`.
The header value can be prepended with `~` to reverse the comparison:
*$1p,strict3p,script,header=via:~1.1 google
The header value is optional and may be ommitted to test only for the
presence of a specific header:
*$1p,strict3p,script,header=via
Related discussions:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1356#issuecomment-732411286
- https://github.com/AdguardTeam/CoreLibs/issues/1384
Changes:
Negation character is `~` (instead of `!`).
Drop special anchor character `|` -- leading `|`
will be supported until no such filter is present
in uBO's own filter lists. For example, instance
of `queryprune=|ad` will have to be replaced with
`queryprune=/^ad/` (or `queryprune=ad` if the name
of the parameter to remove is exactly `ad`).
Align semantic with that of AdGuard's `removeparam=`,
except that specifying multiple `|`-separated names
is not supported.
`match-case`
------------
Related issue:
- https://github.com/uBlockOrigin/uAssets/issues/8280#issuecomment-735245452
The new filter option `match-case` can be used only for
regex-based filters. Using `match-case` with any other
sort of filters will cause uBO to discard the filter.
`redirect=`
-----------
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1366
`redirect=` filters with unresolvable resource token at
runtime will be discarded.
Additionally, the implicit priority is now set to 1
(was 0). The idea is to allow custom `redirect=` filters
to be used strictly as fallback `redirect=` filters in case
another `redirect=` filter is not picked up.
For example, one might create a `redirect=click2load.html:0`
filter, to be taken if and only if the blocked resource is
not already being redirected by another "official" filter
in one of the enabled filter lists.
Related issue:
- https://github.com/gorhill/uBlock/issues/3590
Since the `redirect=` option was refactored into a modifier
filter, presence of a type (`script`, `xhr`, etc.) is no
longer a requirement.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1356
Related commit:
- bde3164eb4
It is not possible to achieve perfect compatiblity at this
point, but reasonable compatibility should be achieved for
a majority of instances of `removeparam=`.
Notable differences:
--------------------
uBO always matches in a case insensitive manner, there is
no need to ask for case-insensitivity, and no need to use
uppercase characters in `queryprune=` values.
uBO does not escape special regex characters since the
`queryprune=` values are always assumed to be literal
regex expression (leaving out the documented special
characters). This means `removeparam=` with characters
which are special regex characters won't be properly
translated and are unlikely to work properly in uBO.
For example, the `queryprune` value of a filter such as
`$removeparam=__xts__[0]` internally become the literal
regex `/__xts__[0]/`, and consequently would not match
a query parameter such as `...?__xts__[0]=...`.
Notes:
------
Additionally, for performance reason, when uBO encounter
a pattern-less `queryprune=` (or `removeparam=`) filter,
it will try to extract a valid pattern from the
`queryprune=` value. For instance, the following filter:
$queryprune=utm_campaign
Will be translated internally into:
utm_campaign$queryprune=utm_campaign
The logger will reflect this internal translation.
New filter options
==================
Strict partyness: `1P`, `3P`
----------------------------
The current options 1p/3p are meant to "weakly" match partyness, i.e. a
network request is considered 1st-party to its context as long as both the
context and the request share the same base domain.
The new partyness options are meant to check for strict partyness, i.e. a
network request will be considered 1st-party if and only if both the context
and the request share the same hostname.
For examples:
- context: `www.example.org`
- request: `www.example.org`
- `1p`: yes, `1P`: yes
- `3p`: no, `3P`: no
- context: `www.example.org`
- request: `subdomain.example.org`
- `1p`: yes, `1P`: no
- `3p`: no, `3P`: yes
- context: `www.example.org`
- request: `www.example.com`
- `1p`: no, `1P`: no
- `3p`: yes, `3P`: yes
The strict partyness options will be visually emphasized in the editor so as
to prevent mistakenly using `1P` or `3P` where weak partyness is meant to be
used.
Filter on response headers: `header=`
-------------------------------------
Currently experimental and under evaluation. Disabled by default, enable by
toggling `filterOnHeaders` to `true` in advanced settings.
Ability to filter network requests according to whether a specific response
header is present and whether it matches or does not match a specific value.
For example:
*$1p,3P,script,header=via:1\.1\s+google
The above filter is meant to block network requests which fullfill all the
following conditions:
- is weakly 1st-party to the context
- is not strictly 1st-party to the context
- is of type `script`
- has a response HTTP header named `via`, which value matches the regular
expression `1\.1\s+google`.
The matches are always performed in a case-insensitive manner.
The header value is assumed to be a literal regular expression, except for
the following special characters:
- to anchor to start of string, use leading `|`, not `^`
- to anchor to end of string, use trailing `|`, not `$`
- to invert the test, use a leading `!`
To block a network request if it merely contains a specific HTTP header is
just a matter of specifying the header name without a header value:
*$1p,3P,script,header=via
Generic exception filters can be used to disable specific block `header=`
filters, i.e. `@@*$1p,3P,script,header` will override the block `header=`
filters given as example above.
Dynamic filtering's `allow` rules override block `headers=` filters.
Important: It is key that filter authors use as many narrowing filter options
as possible when using the `header=` option, and the `header=` option should
be used ONLY when other filter options are not sufficient.
More documentation justifying the purpose of `header=` option will be
provided eventually if ever it is decided to move it from experimental to
stable status.
To be decided: to restrict usage of this filter option to only uBO's own
filter lists or "My filters".
Changes
=======
Fine tuning `queryprune=`
-------------------------
The following changes have been implemented:
The special value `*` (i.e. `queryprune=*`) means "remove all query
parameters".
If the `queryprune=` value is made only of alphanumeric characters
(including `_`), the value will be internally converted to regex equivalent
`^value=`. This ensures a better future compatibility with AdGuard's
`removeparam=`.
If the `queryprune=` value starts with `!`, the test will be inverted. This
can be used to remove all query parameters EXCEPT those who match the
specified value.
Other
-----
The legacy code to test for spurious CSP reports has been removed. This
is no longer an issue ever since uBO redirects to local resources through
web accessible resources.
Notes
=====
The following new and recently added filter options are not compatible with
Chromium's manifest v3 changes:
- `queryprune=`
- `1P`
- `3P`
- `header=`
The auto-complete feature in the _"My filters"_ pane will
use hostname/domain from the set of opened tabs to assist
in entering values for `domain=` option. This also works
for the implict `domain=` option ṗrepending static extended
filters.
`about:srcdoc` frames are their own origin, trying to
use the origin of the parent context causes an
exception to be thrown when accessing location.href.
Notably, add clickable link to open the widget
in its own tab. Also, allows the URL to be text-
selected so that it becomes possible to use the
selection in a browser contextual menu's "Open
in a new tab" option.
