Notably, make `queryprune` option available only
to filter list authors, until there are guards
against bad filters in some future and until the
option syntax and behavior is fully settled.
Instances of `queryprune` in filter lists will be
compiled, however instances of `queryprune` in
_"My filters"_ will be ignored unless users
indicated they are a filter list author.
Reported internally by @gwarser.
In rare occasion, a timing issue could cause uBO to redirect
to a web accessible resource meant to be used for another
network request. This is a regression introduced with the
following commit:
- 2e5d32e967
Additionally, I identified another issue which would cause
cached redirection to fail when a cache entry with redirection
to a web accessible resource was being reused, an issue which
could especially affect pages which are generated dynamically
(i.e. without full page reload).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/760
The purpose of this new network filter option is to remove
query parameters form the URL of network requests.
The name `queryprune` has been picked over `querystrip`
since the purpose of the option is to remove some
parameters from the URL rather than all parameters.
`queryprune` is a modifier option (like `csp`) in that it
does not cause a network request to be blocked but rather
modified before being emitted.
`queryprune` must be assigned a value, which value will
determine which parameters from a query string will be
removed. The syntax for the value is that of regular
expression *except* for the following rules:
- do not wrap the regex directive between `/`
- do not use regex special values `^` and `$`
- do not use literal comma character in the value,
though you can use hex-encoded version, `\x2c`
- to match the start of a query parameter, prepend `|`
- to match the end of a query parameter, append `|`
`queryprune` regex-like values will be tested against each
key-value parameter pair as `[key]=[value]` string. This
way you can prune according to either the key, the value,
or both.
This commit introduces the concept of modifier filter
options, which as of now are:
- `csp=`
- `queryprune=`
They both work in similar way when used with `important`
option or when used in exception filters. Modifier
options can apply to any network requests, hence the
logger reports the type of the network requests, and no
longer use the modifier as the type, i.e. `csp` filters
are no longer reported as requests of type `csp`.
Though modifier options can apply to any network requests,
for the time being the `csp=` modifier option still apply
only to top or embedded (frame) documents, just as before.
In some future we may want to apply `csp=` directives to
network requests of type script, to control the behavior
of service workers for example.
A new built-in filter expression has been added to the
logger: "modified", which allow to see all the network
requests which were modified before being emitted. The
translation work for this new option will be available
in a future commit.
Additionally, add a button in the About pane
to launch benchmark sessions. The button will
be available only when advanced setting
`benchmarkDatasetURL` is set and pointing to
a valid dataset.
Related issues:
- https://github.com/gorhill/uBlock/issues/1390
- https://github.com/gorhill/uBlock/issues/2334
The deadline to interactively load a specific media
element has been extended from 2sec to 5sec.
Clicking over a blocked large media element will cause
uBO to lookup and handle all potentially blocked large
elements at the cursor position. This should take care
of being able to unblock media elements hidden under
other DOM object.
The CSS style applied to blocked large media elements
has been fine tuned to improve interactivity.
uBO will now remember the specific media elements which
were unblocked and keep them exempted from being
further blocked. This would be an issue when unblocking
a video and then a bit later seeking to another point
in the video, in which case uBO would again block
network requests for that video.
Additionally, as a requirement to support click-to-load
feature, redirected resources will from now on no
longer be collapsed.
Related issues:
- https://github.com/gorhill/uBlock/issues/2688
- https://github.com/gorhill/uBlock/issues/3619
- https://github.com/gorhill/uBlock/issues/1899
This new feature should considered in its draft
stage and it needs to be fine-tuned as per
feedback.
Important: Only embedded frames can be converted
into click-to-load widgets, as only these can be
properly shieded from access by page content.
Examples of usage:
||youtube.com/embed/$3p,frame,redirect=clicktoload
||scribd.com/embeds/$3p,frame,redirect=clicktoload
||player.vimeo.com/video/$3p,frame,redirect=clicktoload
Related feedback:
- https://github.com/uBlockOrigin/uBlock-issues/issues/401#issuecomment-703075797
Name: `uiTheme`
Default: `unset`
Values:
- `unset`: uBO will pick the theme according to
browser's `prefers-color-scheme`
- `light`: force light scheme
- `dark`: force dark theme
This advanced setting is not to be documented yet as
it has not been decided this is a long term solution.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/763
Changes:
From now on, uBO will allow click-to-subscribe on only
a few select domains, currently:
- https://filterlists.com/
- https://github.com/
- https://github.io/
More domains can be added if and only the demonstration
is made that more than a marginal number of filter lists
can be subscribed from those domains.
The browser alert box is no longer used to confirm
subscription to a filter list. Instead, the asset
viewer has been expanded to serve that purpose. This
way, users can peruse at the content of a filter list
before subscribing to it.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1226
Related commit:
- 9eb455ab5e
In the previous commit, the element picker dialog was
isolated from the page content. This commit is to also
isolate the svg layers from the page content.
With this commit, there is no longer a need for an anonymous
iframe and the isolated world iframe is now directly
embedded in the page.
As a result, pages are now unable to interfere with any
of the element picker user interface. Pages can now only
see an iframe, but are unable to see the content of that
iframe. The styles applied to the iframe are from a user
stylesheet, so as to ensure pages can't override the
iframe's style properties set by uBO.
Cloud storage is a limited resource, and thus it
makes sense to support data compression before
sending the data to cloud storage.
A new hidden setting allows to toggle on
cloud storage compression:
name: cloudStorageCompression
default: false
By default, this hidden setting is `false`, and a
user must set it to `true` to enable compression
of cloud storage items.
This hidden setting will eventually be toggled
to `true` by default, when there is good confidence
a majority of users are using a version of uBO
which can properly handle compressed cloud storage
items.
A cursory assessment shows that compressed items
are roughly 40-50% smaller in size.
Auto-completion will work only for uBO's own
tokens, compatibility-related tokens[1] will not be
taken into account for auto-completion.
The reason is to not have the compatibility-related
tokens get in the way of auto-completion in order
to not inconvenience uBO's filter list maintainers.
[1] `adguard_ext_chromium`, `adguard_ext_firefox`,
etc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Invalid values for `!#if ...` will be highlighted as errors.
Auto completion is now supported for both the directives
themselves and the valid values for `!#if ...`.
For examples, when pressing ctrl-space:
- `!#e` will auto-complete to `!#endif`
- `!#i` will offer to choose between `!#if ` or `!#include `
- `!#if fir` will auto-complete to `!#if env_firefox`
Additionally, support for some of AdGuard preparsing
directives, i.e. `!#if adguard` is now a valid and will be
honoured -- it always evaluate to `false` in uBO.
This commit adds CodeMirror's auto-completion capability
to the _My filters_ pane.
Currently, auto-completion is available for scriptlet
tokens: pressing ctrl-space while the text cursor is
positioned where a scriptlet token should appear will
cause auto-completion to kick-in. In case of ambiguity,
CodeMirror's widget to pick a specific scriptlet will
appear.
This commit moves some of the parsing logic of static
extended filtering into the static filtering parser; this
allows better syntax highlighting and creation-time
error-catching for cosmetic, HTML, and scriptlet filters.
There have been too many examples out there of users
opting-in to "I am an advanced user" and yet still misusing
dynamic filtering by creating _allow_ rules where _noop_
rules should be used.
Creating _allow_ rules has serious consequences as these
override blocking static filters and can potentially
disable other advanced filtering ability such as
HTML filtering and scriptlet injection -- often used
to deal with anti-blocker mechanisms.
The ability to point-and-click to create _allow_ rules
from the popup panel is no longer allowed by default.
An new advanced setting has been added to enable
the ability to create _allow_ rules from the popup
panel, `popupPanelGodMode`, which default to `false`.
Set to `true` to restore ability to set _allow_ rules
from popup panel.
Since the creation of _allow_ rules is especially useful
to filter list authors, to diagnose and narrow down site
breakage as a result of problematic blocking filter,
the creation of _allow_ rules will still be available
when the advanced setting `filterAuthorMode` is `true`.
This change is probably going to be problematic to all
those users who were misusing dynamic filtering by
creating _allow_ rules instead of _noop_ rules -- but
the breakage is going to bring their misusing to their
attention, a positive outcome.
Default to `unset`.
To allow users to bypass uBO's default CSS styles in
case they are causing issues to specific users. It is
the responsibility of the user to ensure the value of
`uiStyles` contains valid CSS property declarations.
uBO will assign the value to `document.body.style.cssText`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1044
For example, in the case of the issue above, one could
set `uiStyles` to `font-family: sans-serif` to force uBO
to the system font for its user interface.
The old "classic" popup panel will still be used
when at least one of the following is true:
- advanced setting `uiFlavor` is set to `classic`; or
- the browser is Chromium 65 or older; or
- the browser is Firefox 67 or older
The default configuration of the new popup panel
at installation time is to show the power button,
statistics and the basic tool icons, i.e. access
to dashboard, logger, pickers.
For existing installations, the new popup panel
will be configured by respecting the existing
configuration of the classic one.
The new popup panel is currently already in use
on Firefox for Android, and the visual redesign
was made according to suggestions and feedback
from <https://github.com/brampitoyo> to be
optimal for Firefox for Android.
The new popup panel will allow closing the following
pending issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/255
- https://github.com/uBlockOrigin/uBlock-issues/issues/178
This commit specifically address bringing the
desktop version of the new popup panel's look
and feel more inline with the classic one:
- Hide tool captions on desktop
- Bring back no-popups switch on desktop
- Bring back tooltips on desktop (though they
are now rendered natively by the browser)
- Use the Photon icons suggested by @brampitoyo
for the no-popups and no-remote-fonts
switches
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/974
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/fuscia/
The race condition was that a content script could
query the main process to retrieve cosmetic filters
while the cosmetic filters had not been yet fully
loaded into memory. The fix ensure that an already
injected content script will re-query once the
cosmetic filters are fully loaded in memory at
browser launch time.
This concerns the static network filtering engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943
* * *
New static network filter type: `cname`
By default, network requests which are result of
resolving a canonical name are subject to filtering.
This filtering can be bypassed by creating exception
filters using the `cname` option. For example:
@@*$cname
The filter above tells the network filtering engine
to except network requests which fulfill all the
following conditions:
- network request is blocked
- network request is that of an unaliased hostname
Filter list authors are discouraged from using
exception filters of `cname` type, unless there no
other practical solution such that maintenance
burden become the greater issue. Of course, such
exception filters should be as narrow as possible,
i.e. apply to specific domain, etc.
* * *
New static network filter option: `denyallow`
The purpose of `denyallow` is bring
default-deny/allow-exceptionally ability into static
network filtering arsenal. Example of usage:
*$3p,script, \
denyallow=x.com|y.com \
domain=a.com|b.com
The above filter tells the network filtering engine that
when the context is `a.com` or `b.com`, block all
3rd-party scripts except those from `x.com` and `y.com`.
Essentially, the new `denyallow` option makes it easier
to implement default-deny/allow-exceptionally in static
filter lists, whereas before this had to be done with
unwieldy regular expressions[1], or through the mix of
broadly blocking filters along with exception filters[2].
[1] https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0
[2] Typically filters which pattern are of the
form `|http*://`
The toolip of a canonical name entry (blue)
in the popup panel will provide the alias
hostname related to the canonical hostname
in order to help with rule-making decisions.
This is related to the list of domains/subdomains in
the dynamic filtering pane of the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/284
Clicking on the empty space of a row will toggle
the visibility of the subdomains.
Additionally, the root context will always be visible
regardless of the expand/collspase state, along with
a visual indicator that a specific domain or subdomain
is the actual root context. (the root context is the
hostname in which local rules are created).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
As per various feedbacks:
Added an advanced setting to keep the original behavior,
which can be potentially costly CPU-wise on some sites:
popupCosmeticFilterBadgeSlow
Default to `false`. Set to `true` to restore original
method of surveying the number of elements hidden as
a result of applying cosmetic filtering.
As suggested by <https://github.com/gwarser>, skip
descendant of nodes which have been found to be a
match in order to potentially increase the number
of nodes which can be surveyed in the alloted time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
When trying the number of hidden elements as a result of
cosmetic filtering for popup panel badge purpose, the
code will bail out if this takes too long, and in such
case the badge will be set to `?`, meaning the number
of hidden elements is undetermined.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The badge value for the no-cosmetic-filtering
switch will be evaluated on-demand only, when
the user hover over the switch with the mouse
cursor.
For touch screen displays, a tap on the switch
will cause the badge to be rendered if not
already done, otherwise this will toggle the
switch as usual.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
This is the code used to find out the count values
displayed as badge on the cosmetic filtering and
scripting per-site switches in the popup panel.
The issue is that document.querySelector*() -- used to
find out the number of hidden elements -- is unduly
expensive on large DOM.
The changes in this commit have focused on avoiding the
use of document.querySelector*() as much as possible.
Also, the results are cached for reuse unless DOM
mutations are detected.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
- https://github.com/uBlockOrigin/uBlock-issues/issues/528
The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.
Example of filters that could be moved into a bidi-trie
data structure:
&ad_box_
/w/d/capu.php?z=$script,third-party
||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif
Examples of filters that could NOT be moved to a bidi-trie:
-adap.$domain=~l-adap.org
/tsc.php?*&ses=
||ibsrv.net/*forumsponsor$domain=[...]
@@||imgspice.com/jquery.cookie.js|$script
||view.atdmt.com^*/iview/$third-party
||postimg.cc/image/$csp=[...]
Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).
Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.
This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.
- Added ability to associate a pattern with an integer
in the bidi-trie [1].
- The bidi-trie match code passes this externally
provided integer when calling an externally
provided method used for testing extra conditions
that may be present for a plain pattern found to
be matching in the bidi-trie.
- Decomposed existing filters into smaller logical units:
- FilterPlainLeftAnchored =>
FilterPatternPlain +
FilterAnchorLeft
- FilterPlainRightAnchored =>
FilterPatternPlain +
FilterAnchorRight
- FilterExactMatch =>
FilterPatternPlain +
FilterAnchorLeft +
FilterAnchorRight
- FilterPlainHnAnchored =>
FilterPatternPlain +
FilterAnchorHn
- FilterWildcard1 =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
]
- FilterWildcard1HnAnchored =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
] +
FilterAnchorHn
- FilterGenericHnAnchored =>
FilterPatternGeneric +
FilterAnchorHn
- FilterGenericHnAndRightAnchored =>
FilterPatternGeneric +
FilterAnchorRight +
FilterAnchorHn
- FilterOriginMixedSet =>
FilterOriginMissSet +
FilterOriginHitSet
- Instances of FilterOrigin[...], FilterDataHolder
can also be added to a composite filter to
represent `domain=` and `csp=` options.
- Added a new filter class, FilterComposite, for
filters which are a combination of two or more
logical units. A FilterComposite instance is a
match when *all* filters composing it are a
match.
Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.
Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.
Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.
The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.
Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.
A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.
This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.
As a result of the above changes, the map of `token`
(integer) => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.
Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.
`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.
New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.
All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.
Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.
FilterParser is now instanciated on demand and
released when no longer used.
***
[1] 135a45a878/src/js/strie.js (L120)
[2] e94024d350
[3] 135a45a878/src/js/static-net-filtering.js (L1630)
[4] 135a45a878/src/js/static-net-filtering.js (L1566)
This commits make it so that `csp=` filters
are now stored in the same data structures as
all other static network filters rather than
being stored in a separate one.
This internal change is motivated by the wish
to bring session filters to the static network
filtering engine, as has already been done for
the static extended filtering engine in the
following commit:
59c9a34d34
This is a feature under development, hidden behind
a new advanced setting, `filterAuthorMode` which
default to `false`.
Ability to point-and-click to create temporary
exception filters for static extended filters (i.e.
cosmetic, scriptlet & html filters) from within
the summary pane in the logger. The button to
toggle on/off temporary exception filter is
labeled `#@#`.
The created exceptions are temporary and will be
lost when restarting uBO, or manually toggling off
the exception filters.
Creating temporary exception filters does not
cause the filter lists to reloaded, and thus there
is no overhead in creating/removing these temporary
exception filters.
... from platform-independent code.
This should complete the de-chromiumification of
uBO. Next step will be to swap the content of the
`platform/chromium` and `platform/firefox`
folders so as to reflect that Firefox is the
natural platform for uBO.
Related documentation:
- https://help.eyeo.com/en/adblockplus/how-to-write-filters#element-hiding
Related feedback/discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/d6vxzj/
The `elemhide` filter option as per ABP semantic is
now supported. Previously uBO would consider `elemhide`
to be an alias of `generichide`.
The support of `elemhide` is through the convenient
conversion of `elemhide` option into existing
`generichide` option and new `specifichide` option.
The purpose of the new `specifichide` filter option
is to disable all specific cosmetic filters, i.e.
those who target a specific site.
Additionally, for convenience purpose, the filter
options `generichide`, `specifichide` and `elemhide`
can be aliased using the shorter forms `ghide`,
`shide` and `ehide` respectively.
No need to store mouse coordinates in background
page, thus no need to post mouse coordinates
information for every click.
Rename/group element picker arguments and popup
arguments separately.
