Cloud storage is a limited resource, and thus it
makes sense to support data compression before
sending the data to cloud storage.
A new hidden setting allows to toggle on
cloud storage compression:
name: cloudStorageCompression
default: false
By default, this hidden setting is `false`, and a
user must set it to `true` to enable compression
of cloud storage items.
This hidden setting will eventually be toggled
to `true` by default, when there is good confidence
a majority of users are using a version of uBO
which can properly handle compressed cloud storage
items.
A cursory assessment shows that compressed items
are roughly 40-50% smaller in size.
Auto-completion will work only for uBO's own
tokens, compatibility-related tokens[1] will not be
taken into account for auto-completion.
The reason is to not have the compatibility-related
tokens get in the way of auto-completion in order
to not inconvenience uBO's filter list maintainers.
[1] `adguard_ext_chromium`, `adguard_ext_firefox`,
etc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Invalid values for `!#if ...` will be highlighted as errors.
Auto completion is now supported for both the directives
themselves and the valid values for `!#if ...`.
For examples, when pressing ctrl-space:
- `!#e` will auto-complete to `!#endif`
- `!#i` will offer to choose between `!#if ` or `!#include `
- `!#if fir` will auto-complete to `!#if env_firefox`
Additionally, support for some of AdGuard preparsing
directives, i.e. `!#if adguard` is now a valid and will be
honoured -- it always evaluate to `false` in uBO.
This commit adds CodeMirror's auto-completion capability
to the _My filters_ pane.
Currently, auto-completion is available for scriptlet
tokens: pressing ctrl-space while the text cursor is
positioned where a scriptlet token should appear will
cause auto-completion to kick-in. In case of ambiguity,
CodeMirror's widget to pick a specific scriptlet will
appear.
This commit moves some of the parsing logic of static
extended filtering into the static filtering parser; this
allows better syntax highlighting and creation-time
error-catching for cosmetic, HTML, and scriptlet filters.
There have been too many examples out there of users
opting-in to "I am an advanced user" and yet still misusing
dynamic filtering by creating _allow_ rules where _noop_
rules should be used.
Creating _allow_ rules has serious consequences as these
override blocking static filters and can potentially
disable other advanced filtering ability such as
HTML filtering and scriptlet injection -- often used
to deal with anti-blocker mechanisms.
The ability to point-and-click to create _allow_ rules
from the popup panel is no longer allowed by default.
An new advanced setting has been added to enable
the ability to create _allow_ rules from the popup
panel, `popupPanelGodMode`, which default to `false`.
Set to `true` to restore ability to set _allow_ rules
from popup panel.
Since the creation of _allow_ rules is especially useful
to filter list authors, to diagnose and narrow down site
breakage as a result of problematic blocking filter,
the creation of _allow_ rules will still be available
when the advanced setting `filterAuthorMode` is `true`.
This change is probably going to be problematic to all
those users who were misusing dynamic filtering by
creating _allow_ rules instead of _noop_ rules -- but
the breakage is going to bring their misusing to their
attention, a positive outcome.
Default to `unset`.
To allow users to bypass uBO's default CSS styles in
case they are causing issues to specific users. It is
the responsibility of the user to ensure the value of
`uiStyles` contains valid CSS property declarations.
uBO will assign the value to `document.body.style.cssText`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1044
For example, in the case of the issue above, one could
set `uiStyles` to `font-family: sans-serif` to force uBO
to the system font for its user interface.
The old "classic" popup panel will still be used
when at least one of the following is true:
- advanced setting `uiFlavor` is set to `classic`; or
- the browser is Chromium 65 or older; or
- the browser is Firefox 67 or older
The default configuration of the new popup panel
at installation time is to show the power button,
statistics and the basic tool icons, i.e. access
to dashboard, logger, pickers.
For existing installations, the new popup panel
will be configured by respecting the existing
configuration of the classic one.
The new popup panel is currently already in use
on Firefox for Android, and the visual redesign
was made according to suggestions and feedback
from <https://github.com/brampitoyo> to be
optimal for Firefox for Android.
The new popup panel will allow closing the following
pending issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/255
- https://github.com/uBlockOrigin/uBlock-issues/issues/178
This commit specifically address bringing the
desktop version of the new popup panel's look
and feel more inline with the classic one:
- Hide tool captions on desktop
- Bring back no-popups switch on desktop
- Bring back tooltips on desktop (though they
are now rendered natively by the browser)
- Use the Photon icons suggested by @brampitoyo
for the no-popups and no-remote-fonts
switches
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/974
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/fuscia/
The race condition was that a content script could
query the main process to retrieve cosmetic filters
while the cosmetic filters had not been yet fully
loaded into memory. The fix ensure that an already
injected content script will re-query once the
cosmetic filters are fully loaded in memory at
browser launch time.
This concerns the static network filtering engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943
* * *
New static network filter type: `cname`
By default, network requests which are result of
resolving a canonical name are subject to filtering.
This filtering can be bypassed by creating exception
filters using the `cname` option. For example:
@@*$cname
The filter above tells the network filtering engine
to except network requests which fulfill all the
following conditions:
- network request is blocked
- network request is that of an unaliased hostname
Filter list authors are discouraged from using
exception filters of `cname` type, unless there no
other practical solution such that maintenance
burden become the greater issue. Of course, such
exception filters should be as narrow as possible,
i.e. apply to specific domain, etc.
* * *
New static network filter option: `denyallow`
The purpose of `denyallow` is bring
default-deny/allow-exceptionally ability into static
network filtering arsenal. Example of usage:
*$3p,script, \
denyallow=x.com|y.com \
domain=a.com|b.com
The above filter tells the network filtering engine that
when the context is `a.com` or `b.com`, block all
3rd-party scripts except those from `x.com` and `y.com`.
Essentially, the new `denyallow` option makes it easier
to implement default-deny/allow-exceptionally in static
filter lists, whereas before this had to be done with
unwieldy regular expressions[1], or through the mix of
broadly blocking filters along with exception filters[2].
[1] https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0
[2] Typically filters which pattern are of the
form `|http*://`
The toolip of a canonical name entry (blue)
in the popup panel will provide the alias
hostname related to the canonical hostname
in order to help with rule-making decisions.
This is related to the list of domains/subdomains in
the dynamic filtering pane of the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/284
Clicking on the empty space of a row will toggle
the visibility of the subdomains.
Additionally, the root context will always be visible
regardless of the expand/collspase state, along with
a visual indicator that a specific domain or subdomain
is the actual root context. (the root context is the
hostname in which local rules are created).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
As per various feedbacks:
Added an advanced setting to keep the original behavior,
which can be potentially costly CPU-wise on some sites:
popupCosmeticFilterBadgeSlow
Default to `false`. Set to `true` to restore original
method of surveying the number of elements hidden as
a result of applying cosmetic filtering.
As suggested by <https://github.com/gwarser>, skip
descendant of nodes which have been found to be a
match in order to potentially increase the number
of nodes which can be surveyed in the alloted time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
When trying the number of hidden elements as a result of
cosmetic filtering for popup panel badge purpose, the
code will bail out if this takes too long, and in such
case the badge will be set to `?`, meaning the number
of hidden elements is undetermined.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The badge value for the no-cosmetic-filtering
switch will be evaluated on-demand only, when
the user hover over the switch with the mouse
cursor.
For touch screen displays, a tap on the switch
will cause the badge to be rendered if not
already done, otherwise this will toggle the
switch as usual.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
This is the code used to find out the count values
displayed as badge on the cosmetic filtering and
scripting per-site switches in the popup panel.
The issue is that document.querySelector*() -- used to
find out the number of hidden elements -- is unduly
expensive on large DOM.
The changes in this commit have focused on avoiding the
use of document.querySelector*() as much as possible.
Also, the results are cached for reuse unless DOM
mutations are detected.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
- https://github.com/uBlockOrigin/uBlock-issues/issues/528
The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.
Example of filters that could be moved into a bidi-trie
data structure:
&ad_box_
/w/d/capu.php?z=$script,third-party
||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif
Examples of filters that could NOT be moved to a bidi-trie:
-adap.$domain=~l-adap.org
/tsc.php?*&ses=
||ibsrv.net/*forumsponsor$domain=[...]
@@||imgspice.com/jquery.cookie.js|$script
||view.atdmt.com^*/iview/$third-party
||postimg.cc/image/$csp=[...]
Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).
Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.
This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.
- Added ability to associate a pattern with an integer
in the bidi-trie [1].
- The bidi-trie match code passes this externally
provided integer when calling an externally
provided method used for testing extra conditions
that may be present for a plain pattern found to
be matching in the bidi-trie.
- Decomposed existing filters into smaller logical units:
- FilterPlainLeftAnchored =>
FilterPatternPlain +
FilterAnchorLeft
- FilterPlainRightAnchored =>
FilterPatternPlain +
FilterAnchorRight
- FilterExactMatch =>
FilterPatternPlain +
FilterAnchorLeft +
FilterAnchorRight
- FilterPlainHnAnchored =>
FilterPatternPlain +
FilterAnchorHn
- FilterWildcard1 =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
]
- FilterWildcard1HnAnchored =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
] +
FilterAnchorHn
- FilterGenericHnAnchored =>
FilterPatternGeneric +
FilterAnchorHn
- FilterGenericHnAndRightAnchored =>
FilterPatternGeneric +
FilterAnchorRight +
FilterAnchorHn
- FilterOriginMixedSet =>
FilterOriginMissSet +
FilterOriginHitSet
- Instances of FilterOrigin[...], FilterDataHolder
can also be added to a composite filter to
represent `domain=` and `csp=` options.
- Added a new filter class, FilterComposite, for
filters which are a combination of two or more
logical units. A FilterComposite instance is a
match when *all* filters composing it are a
match.
Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.
Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.
Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.
The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.
Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.
A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.
This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.
As a result of the above changes, the map of `token`
(integer) => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.
Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.
`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.
New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.
All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.
Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.
FilterParser is now instanciated on demand and
released when no longer used.
***
[1] 135a45a878/src/js/strie.js (L120)
[2] e94024d350
[3] 135a45a878/src/js/static-net-filtering.js (L1630)
[4] 135a45a878/src/js/static-net-filtering.js (L1566)
This commits make it so that `csp=` filters
are now stored in the same data structures as
all other static network filters rather than
being stored in a separate one.
This internal change is motivated by the wish
to bring session filters to the static network
filtering engine, as has already been done for
the static extended filtering engine in the
following commit:
59c9a34d34
This is a feature under development, hidden behind
a new advanced setting, `filterAuthorMode` which
default to `false`.
Ability to point-and-click to create temporary
exception filters for static extended filters (i.e.
cosmetic, scriptlet & html filters) from within
the summary pane in the logger. The button to
toggle on/off temporary exception filter is
labeled `#@#`.
The created exceptions are temporary and will be
lost when restarting uBO, or manually toggling off
the exception filters.
Creating temporary exception filters does not
cause the filter lists to reloaded, and thus there
is no overhead in creating/removing these temporary
exception filters.
... from platform-independent code.
This should complete the de-chromiumification of
uBO. Next step will be to swap the content of the
`platform/chromium` and `platform/firefox`
folders so as to reflect that Firefox is the
natural platform for uBO.
Related documentation:
- https://help.eyeo.com/en/adblockplus/how-to-write-filters#element-hiding
Related feedback/discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/d6vxzj/
The `elemhide` filter option as per ABP semantic is
now supported. Previously uBO would consider `elemhide`
to be an alias of `generichide`.
The support of `elemhide` is through the convenient
conversion of `elemhide` option into existing
`generichide` option and new `specifichide` option.
The purpose of the new `specifichide` filter option
is to disable all specific cosmetic filters, i.e.
those who target a specific site.
Additionally, for convenience purpose, the filter
options `generichide`, `specifichide` and `elemhide`
can be aliased using the shorter forms `ghide`,
`shide` and `ehide` respectively.
No need to store mouse coordinates in background
page, thus no need to post mouse coordinates
information for every click.
Rename/group element picker arguments and popup
arguments separately.
Also, coallesce calls to selfieManager.destroy() so as
to avoid undue repeated calls to storage deletion of
selfie assets.
Related commit:
- e27328f931
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/710
Messages from unprivileged ports (i.e. from content scripts)
are no longer relayed to message handlers which are to be
strictly used to execute privileged code.
The last remaining case of unprivileged messages which
should be converted into a privileged ones will be taken
care of when the following issue is fixed:
- https://github.com/gorhill/uBlock/issues/3497
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/cmh910/
Additionally, the `3p` rule has been made distinct from
`3p-script`/`3p-frame` for the purpose of
"Relax blocking mode" command.
The badge color will hint at the current blocking mode.
There are four colors for the four following blocking
modes:
- JavaScript wholly disabled
- All 3rd parties blocked
- 3rd-party scripts and frames blocked
- None of the above
The default badge color will be used when JavaScript is not
wholly disabled and when there are no rules for `3p`,
`3p-script` or `3p-frame`.
A new advanced setting has been added to let the user choose
the badge colors for the various blocking modes,
`blockingProfileColors`. The value *must* be a sequence of
4 valid CSS color values that match 6 hexadecimal digits
prefixed with`#` -- anything else will be ignored.
With hindsight, I revised decisions made earlier during
this development cycle:
Un-redirectable scriptlets have been removed from
/web_accessible_resources and instead put in the new
/assets/resources/scriptlets.js, which contains all
scriptlets used for web page injection purpose.
uBO will no longer fetch a remote version of built-in
resources.
Advanced setting `userResourcesLocation` will still be
honoured by uBO, and if set, will be fetched every
time at least one asset is updated.
Related issue:
- https://github.com/gorhill/uBlock/issues/2394
Additionally, I added a new advanced setting to control
how long after launch an auto-update session should be
started -- value is in seconds:
autoUpdateDelayAfterLaunch 180
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/214
Built-in whitelist directives are now rendered differently
than user-defined whitelist directives. Also, removing a
built-in whitelist directive will only cause that directive
to be commented out, so that users do not have to remember
built-in directives should they want to bring them back.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/494
The built-in per-site switch rule
`no-scripting: behind-the-scene false` has been removed,
it should not ever be needed since there will always be a
valid root context for main- and sub-frames.
Related issue:
- https://github.com/gorhill/uBlock/issues/127
Additionally, the extended exception filters in the
logger will be rendered with a line-through to more
easily distinguish them from non-exception ones.
Also, opportunistically converted revisited code to
ES6 syntax.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/409
By default `indexedDB` is used in Firefox for purpose of cache storage
backend.
This commit allows to force the use of `browser.storage.local` instead
as cache storage backend. For this to happen, set `cacheStorageAPI` to
`browser.storage.local` in advanced settings.
Additionally, should `indexedDB` not be available for whatever reason,
uBO will automatically fallback to `browser.storage.local`.
Performance-related work: the logger data has been decoupled
from the DOM -- inspired from CodeMirror's way of efficiently
handling large amout of text data.
This decoupling now makes the logger highly efficient CPU- and
memory-wise, and open the way to more possibilities.
Ability to configure some aspect of the logger behavior and
visuals:
- The hard-coded limit of 5000 entries has been
removed and is now replaced with a variety of
user-configurable settings to enforce the discarding of
logger entries.
- Some columns in the logger output can now be hidden.
The filter list look-up feature has been merged into the
existing overlay dialog used to create URL rules or static
filters, as an entry in a new "Details" pane.
Other issues addressed during refactoring:
- https://github.com/uBlockOrigin/uBlock-issues/issues/280
- https://github.com/gorhill/uBlock/issues/1999
The minimum version supported on Firefox has been bumped
up to 55.0.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/372
- https://github.com/gorhill/uBlock/issues/93
A new advanced settings has been added: `autoCommentFilterTemplate`.
Default value is `{{date}} {{origin}}`.
Placeholders are identified by `{{...}}`. There are currently
only three placeholders supported:
- `{{date}}`: will be replaced with current date
- `{{time}}`: will be replaced with current time
- `{{origin}}`: will be replaced with site information on which
the filter(s) was created
If no placeholder is found in `autoCommentFilterTemplate`, this
will disable auto-commenting. So one can use `-` to disable
auto-commenting.
Additionally, if auto-commenting is enabled, uBO will not emit a
comment if an emitted comment would be a duplicate of the last
one found in the user filter list.
<https://github.com/gorhill/uBlock/issues/3436>: a new per-site switch
has been added, no-scripting, which purpose is to wholly disable/enable
javascript for a given site. This new switch has precedence over all
other ways javascript can be disabled, including precedence over dynamic
filtering rules.
The popup panel will report the number of script resources which have
been seen by uBO for the current page. There is a minor inaccuracy to
be fixed regarding the count, and which fix requires to extend request
journaling.
<https://github.com/gorhill/uBlock/issues/308>: the `noscript` tags will
now be respected when the new no-scripting switch is in effect on a given
site.
A default setting has been added to the _Settings_ pane to
disable/enable globally the new no-script switch, such that one can
work in default-deny mode regarding javascript execution.
<https://github.com/uBlockOrigin/uBlock-issues/issues/155>: a new
hidden setting, `requestJournalProcessPeriod`, has been added to
allow controlling the delay before uBO internally process it's
network request journal queue. Default to 1000 (milliseconds).
- collate together specific filters with same base domain
- replace string-based hash to integer-based hash
- revisit code to benefit from ES6-specific syntax
- Default to being detached
- Default to "Current tab"
- Append current tab title to "Current tab" entry
- Avoid iterating through all tabs when no change
