Issue reported by @uBlock-user in team channel.
Creating cosmetic procedural exception filters was
causing `cosmetic-logger.js` scriptlet to throw at
and thus further breaking the logging of cosmetic
filters overall.
Reuse permanent instances instead. The trailing `$` is
used to denote these variables are register-like
instances, i.e. their content is valid only for the
duration of the call. (From now on I will use this
convention throughout the code base.)
Entity-based filters where not properly looked-up if
they used subdomains. Example:
- `example.*##^script` => ok
- `www.example.*##^script` => failed on `https://www.example.com/`
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/710
Messages from unprivileged ports (i.e. from content scripts)
are no longer relayed to message handlers which are to be
strictly used to execute privileged code.
The last remaining case of unprivileged messages which
should be converted into a privileged ones will be taken
care of when the following issue is fixed:
- https://github.com/gorhill/uBlock/issues/3497
Related commit:
- fb4e94f92c
A bidi-trie can't store strings longer than 255 characters
because the string segment lengths are encoded into a single
byte. This commit ensures only strings smaller than
256 characters are stored in the bidi-tries.
Plain filters can be any length, while the bidi-trie was
assuming max length of 256. The origin of this error
is from when the bidi-trie code was originally imported
from the hntrie code as start to develop bidi-trie.
The erroneous code could cause issue when the following
conditions were met:
- Plain filter longer than 256 characters
- Free space in bidi-trie's character buffer was less
than 256 bytes
Likely a rare occurrence, but some filter lists do contains
long plain filters, for example:
https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter.txt
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/cs1y26/
The purpose is to wholly disable scriptlet injection
for a given site without having to create exceptions
for all matching scriptlet injection filters.
The following exception filter will cause scriptlet
injection to be wholly disable for `example.com`:
`example.com#@#+js()`
Or to disable scriptlet injection everywhere:
`#@#+js()`
The following form is meaningless and will be
ignored:
`example.com##+js()`
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/701
The filter option `empty` is converted to `redirect=empty`
by uBO internally; however unlike when the `redirect=`
option is used expressly, the `empty` option does not
require a resource type.
When `empty` is used, only network requests which are meant
to return a text response will be redirected to an empty
response body by uBO -- so `empty` will not work for
resources such as images, media, or other binary resources.
Related commit & feedback:
- 7ff750eaf6
The color value for the icon badge is now
"attached" to the blocking profile value.
Additionally, as per feedback, `3p` rules
will be relaxing before master JavaScript
switch rules.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/cmh910/
Additionally, the `3p` rule has been made distinct from
`3p-script`/`3p-frame` for the purpose of
"Relax blocking mode" command.
The badge color will hint at the current blocking mode.
There are four colors for the four following blocking
modes:
- JavaScript wholly disabled
- All 3rd parties blocked
- 3rd-party scripts and frames blocked
- None of the above
The default badge color will be used when JavaScript is not
wholly disabled and when there are no rules for `3p`,
`3p-script` or `3p-frame`.
A new advanced setting has been added to let the user choose
the badge colors for the various blocking modes,
`blockingProfileColors`. The value *must* be a sequence of
4 valid CSS color values that match 6 hexadecimal digits
prefixed with`#` -- anything else will be ignored.
The ability to redirect xmlhttprequest to binary
resources was lost when redirectable/injectable
resources became immutable in commit
152cea2dfe.
This commit restores the ability to redirect a
xmlhttprequest to a binary resource by making
it possible to derive a data: URI from the
content of binary resources such as images.
Addtionally a redirect to a data: URI can be
forced by prefixing the resource token with `%`.
This is a non-official feature at this point,
i.e. it could be removed at any time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/310
The purpose of this new option is to add the ability
to create standalone redirect rule without being forced
to create a block filter (a corresponding block filter
is always created when using the `redirect=`).
Additionally:
The syntax `*$redirect=token,...` is now supported, there
is no need to "trick" the filter parser with
`*/*$redirect=token,...` in order to create redirect rules
which are meant to match all paths.
Filters of the form `|http*://` will be normalized into
two corresponding filters `|https://` and `|http://` so as
to reduce the number of filters in the buckets of
untokenizable filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/682#issuecomment-515197130
The following advanced setting has been added:
updateAssetBypassBrowserCache
Default to `false`. If set to `true`, uBO will ensure the
browser cache is bypassed when fetching a remote resource.
This is for the convenience of filter list maintainers who
may want to test the latest version of their lists when
fetched from their remote location.
Related feedback:
- a54cb2e38f (commitcomment-34387041)
Regression from converting uBO resources into
immutable resources. This affects only Chromium-based
browsers.
This is a quick fix for the dev build, to at least unbreak
the reported case.
I need to research whether the root issue (conflict with HTTPS-E)
is still occurring, and if so the fix is not trivial: I will need
to add code to uBO to fetch and convert binary data into `data:`
URIs.
Theoretically, if a selfie is loaded from corrupted
storage resulting in a thrown exception from the
constructor, this would cause an improperly initialized
HostnameBasedDB instance and overall potentially a
broken uBO.
Related feedback:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1543880#c7
Since https://github.com/uBlockOrigin/uBlock-issues/issues/156
won't be fixed in next release, no need to ship
with code which will be unused, and anyways only once
the fix is worked on will it be clear exactly what needs
to be used by scriptlets to deal harmoniously with
property listener collisions.
Make sure the parser is safely compatible with old
resources format -- for those users still using
custom resources (via `userResourcesLocation`).
Prepare code for future fix to
<https://github.com/uBlockOrigin/uBlock-issues/issues/156>:
This commit introduces a new private Map() object,
`uBOSafe`, accessible by all injected scriptlets. This
private safe can be used to store data which can be shared
with different scriptlets. The idea is for scriptlets to
use that safe to graciously deal with the need to install
multiple listeners for the same property.
Since resources are now immutable, by default they are
only compiled once each time uBO updates to a new
version. However I need a way to force a re-compiling
of the resource in the dev build. This commit adds code
to invalidate the resources selfie when forcing the
update of any filter list.
With hindsight, I revised decisions made earlier during
this development cycle:
Un-redirectable scriptlets have been removed from
/web_accessible_resources and instead put in the new
/assets/resources/scriptlets.js, which contains all
scriptlets used for web page injection purpose.
uBO will no longer fetch a remote version of built-in
resources.
Advanced setting `userResourcesLocation` will still be
honoured by uBO, and if set, will be fetched every
time at least one asset is updated.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/651
The `behnind-the-scene` context was wrongly used to
evaluate the whitelist status of the context of tabless
network requests. The document origin must be used
instead when it's available.
Additionally, much code has been revisited for better
ES6 syntax compliance.
The goal is to be able to specify a scriptlet token
without the `.js` part at the end, because that part
is essentially redundant with the `+js` part of
the syntax.
When the next stable release is in widespread use
(to determine), scriptlet tokens will have to be
specified without the `.js` part, and with this
commit the logger will already report the normalized
version of scriptlets.
Eventually, when the migration to sans-`.js` is
complete (also to determine), the internal
normalization of the token will be removed and this
will become official syntax.
Filter list maintainers will have to mind that
uAssets is becoming in use beyond uBO (i.e. Brave)
when skipping the `.js` part -- hopefully Brave will
go along with the change here, which is to remove a
bit of tediousness for filter list maintainers.
This is a first step, the ultimate goal is to remove
the need for resources.txt, or at least to reduce to
only hotfixes or for trivial resources targeting very
specific websites.
Most resources will become immutable, i.e. they will
be part of uBO's code base. Advantages include easier
code maintenance (jshint, syntax highlight), and to
make scriptlets more easy to code review by external
parties (for example extension store reviewers).
TODO:
- More scriptlets need to be imported before next
release.
- Need to make legacy versions of uBO use a legacy
version of resources.txt, as all the now obsolete
scriptlets will have to be removed once uBO's
next release become widespread.
- Possibly need to add code to load binary
resources so that they can be injected as
data: URI. So far it's unclear whether this is
really needed. For example, this would be needed
if a xmlhttprequest is redirected to an image
resource.
This works only for platforms supporting the return of
Promise by network listeners, i.e. only Firefox at this
point.
When filter lists are reloaded[1], there is a small
time window in which some network requests which should
have normally been blocked are not being blocked
because the static network filtering engine may not
have yet loaded all the filters in memory
This is now addressed by suspending the network request
handler when filter lists are reloaded -- again, this
works only on supported platforms.
[1] Examples: when a filter list update session
completes; when user filters change, when
adding/removing filter lists.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/c5do7w/
Make the element picker better reflect network filters as
parsed by the static network filtering engine. Additionally,
discard single alphanumeric character-based filters.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/c62irc/
Inject newly created cosmetic filters into the DOM
filterer, in order for these filters to be enforced by
the DOM filterer in subsequent dynamic DOM changes.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/371
By default, no specific keyboard shortcut is predefined,
this will have to be assigned by the user. The command
name in English is "Toggle blocking profile".
The default behavior is to toggle down according to one
of the following scenarios.
a) If script execution is disabled through the no-scripting
switch, the no-scripting switch will be locally toggled
so as to allow script execution. The page will be
automatically reloaded.
b) If script execution is not blocked but the 3rd-party
script and/or frame cells are blocked, local no-op rules
will be set so as to no longer block 3rd-party scripts
and/or frames. The page will be automatically reloaded.
Given this, it may take more than one toggle down command
to reach the lowest blocking profile, which is one where
JavaScript execution is not blocked and 3rd-party scripts
and frames resources block rules, if any, are bypassed
with local no-op rules.
TODO: At this point, I haven't yet decided whether
toggling from the lowest profile should restore the
original highest blocking profile.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/c5do7w/
Fixed:
- Expect the differ can return the first input as is when
there is no difference between the two items.
- Better deal with extraneous whitespaces in `srcset`
The purpose is to avoid having to iterate through
all input nodes at each operator implementation
level. The `transpose` method deals with only one
input node, and the iteration is performed by the
main procedural filtering entry points.
Additionally:
- Add `:spath` to HTML filtering
- Rename `:watch-attrs` to `:watch-attr`
- `:watch=attrs` is deprecated and will be kept around
until it is safe to remove it completely
Where `x` is the minimal text length of the subject
DOM element. DOM elements whose text length is
greater than or equal to `x` will be selected.
The original rationale for such procedural cosmetic
operator[1] is to be able to remove inline script
elements according to a minimum text length using
HTML filtering.
[1] As a result of internal discussion with filter
list maintainers @ uAssets.
- Remove HNTrieContainer class from global context by
storing it as a property of µBlock.
- Use block scope to isolate HNTrie-related constants
from global context.
- Prevent filters which are pure IP address from
being stored in an HNTrie instance -- as this
could cause false positives.
The bidirectional trie allows storing the right
and left parts of a string into a trie given a
pivot position.
Releated issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/528
Additionally, the mandatory token-at-index-0 rule
for FilterPlainHnAnchored has been lifted, thus
allowing the engine to pick a potentially better token
at any position in the filter string.
***
TODO: Eventually rename `strie.js` to `biditrie.js`.
TODO: Fix dump() method, it currently only show the
right-hand side of a filter string.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/610
The service worker-related issue affects both
Chromium/Firefox: the type of resources fetched
from a service worker are uniformly set to
`xmlhttprequest`, hence losing a key piece of
information for the purpose of accurate content
filtering.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/591
Additionally, I added a link to the logger in the
"About" pane in the dashboard in order to be able
to access the logger without having to go through
the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/983
- https://github.com/gorhill/uBlock/issues/1353
The current implementation reports statistics for all
static filters, and the presentation/featureset is
intentionally minimal: *Do not open issues about this.*
It's still a work in progress and it will be worked on
slowly and thoughtfully over time and as time allows.
Pausing the logger will not pause the collation of
filter hit statistics, thus it is possible to lower
the logger overhead by pausing logger output without
losing filter hit collation.
The motivations for the re-arrangement:
- Reducing the number of entry points:
matchStringExactString() has been removed and
matchString() is simply reused with a modifier parameter
to enable matching variants.
- Presumption that most matches, if any, occur early with
the left-most tokens in a URL. This gives a very small
marginal performance gain as per built-in benchmark.
Related discussion:
- https://www.reddit.com/r/uBlockOrigin/comments/bqnsoa/
The `all` option is equivalent to specifying all
network-based types + `popup`, `document`,
`inline-font`, `inline-script`.
Example from discussion:
||bet365.com^$all
Above will block all network requests, block all popups,
prevent inline fonts/scripts from `bet365.com`. EasyList-
compatible syntax does not allow to accomplish that
semantic when using only `||bet365.com^`.
If using specific negated type options along with `all`,
the order in which the options appear is important. In
such case `all` should always be first, followed by
the negated type option(s).
Related issue:
- https://github.com/gorhill/uBlock/issues/2394
Additionally, I added a new advanced setting to control
how long after launch an auto-update session should be
started -- value is in seconds:
autoUpdateDelayAfterLaunch 180
Related issue:
- https://github.com/gorhill/uBlock/issues/3271
When navigating away by clicking another pane tab button,
there will be an embedded warning, which can be ignore
in order to proceed to the new pane, or dismissed by
either clicking on the "Stay" button or anywhere else
in the dashboard.
When navigating away by trying to close the tab, there will
be a built-in browser warning asking for confirmation.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/131
The new advanced setting and its default value is:
allowGenericProceduralFilters false
Whenever this setting is toggled, the user is responsible
of forcing a reload of all filter lists so as to allow uBO
to process differently any existing generic procedural
cosmetic filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/214
Built-in whitelist directives are now rendered differently
than user-defined whitelist directives. Also, removing a
built-in whitelist directive will only cause that directive
to be commented out, so that users do not have to remember
built-in directives should they want to bring them back.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/494
The built-in per-site switch rule
`no-scripting: behind-the-scene false` has been removed,
it should not ever be needed since there will always be a
valid root context for main- and sub-frames.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/551
The issue fixes previewing the hiding/unhiding of targeted
elements in the element picker.
However it does not address the case of previewing
`:style(...)` operators -- this would require a much
more complex fix, which I am not sure is worth the
amount of work and increased code complexity.
Related issue:
- https://github.com/gorhill/uBlock/issues/127
Additionally, the extended exception filters in the
logger will be rendered with a line-through to more
easily distinguish them from non-exception ones.
Also, opportunistically converted revisited code to
ES6 syntax.
This was a TODO item:
- 07cbae66a4/src/js/cosmetic-filtering.js (L375)
µBlock.staticExtFilteringEngine.HostnameBasedDB has been
re-factored to accomodate the storing of specific cosmetic
filters.
As a result of this refactoring:
- Memory usage has been further decreased
- Performance of selector retrieval marginally
improved
- New internal representation opens the door
to use a specialized version of HNTrie, which
should further improve performance/memory
usage
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/572
Wildcards are now allowed in the hostname part of redirect
filters. There will be an attempt to find the longest
right-hand portion of the hostname with no wildcard. If
no non-empty hostname can be extracted, `*` will be used.
To be used at the console, as an investigation tool for
development purpose.
Using it to verify the content of the largest
FilterHostnameDict instance, I spotted an all-uppercase
hostname in the HNTrieRef instance:
µBlock.staticNetFilteringEngine.categories.get(0).get(0x10000000).dict.dump();
Thus the changes to static-net-filtering.js are to fix
the erroneous insertion of filters with uppercase
characters. The single instance found was a hostname entry
in Malware Domain List (TRIANGLESERVICESLTD dot COM).
The `null` placeholder are not necessary, we can just use
default arguments instead, and add the HNTrieContainer
references if and only if they are instanciated.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/550
Related Chromium issue (I can't access it):
- https://bugs.chromium.org/p/chromium/issues/detail?id=957866
Findings so far: affects browsers based on Chromium 74.
I could not reproduce the issue with either Chromium 73 or
Google Chrome 75.
This commit is a mitigation: to prevent sites from using
uBO's internal WAR secret for tracking purpose. A secret
can be used for at most one second, after which a new secret
is generated.
The original issue related to the implementation of
secret-gated web accessible resources is:
- https://github.com/gorhill/uBlock/issues/2823
The purpose of this new `:nth-ancestor(n)` operator is to
lookup the nth ancestor relative to the currently selected
node.
It is essentially equivalent to `:xpath(..)`, where
ancestor distance is expressed as a number rather than a
sequence of slash-separated `..`.
The rationale to introduce this new procedural selector
is to have a low overhead way to accomplish ancestor
selection.
This commit implements the alphabetical ordering of HNTrie
nodes, so as to make it possible to bail out early at
HNTrie.matches() time.
Contrary to what I expected, there is no performance gain
observed to HNTrie.matches() as per benchmarks -- I find
the results perplexing.
Because of this I will revert this commit immediately.
The purpose of this commit is to record the changes so
that I can bring them back to life in the future whenever
I want to investigate further.
Consider the two following filters:
example.com
www.example.com
This commit make it so that if the first filter is
already present in a given HNTrie, the second filter
will not be stored, since HNTrie will _always_
return the first filter as a match whenever the
hostname to match is example.com or any subdomain
of example.com.
The detection of such pointless filters is
virtually free when adding a hostname to an HNTrie
instance (given how data is stored in the trie), so
in practice no overhead is incurred to detect such
pointless filters.
The ability to ignore impossible to match filters
in HNTrie instances will _especially_ benefit those
using large hosts files.
Examples of how this helps using real configurations:
- Default lists:
444 filters out of 100,382 were ignored as a result
of this commit.
- Default lists + "Energized Ultimate Protection":
283,669 filters out of 903,235 were ignored as a
result of this commit.
Side note: There was no measurable difference between
the two configurations above in the performance of
the matching algorithm as reported by the built-in
benchmark tool.
The staticNetFilteringEngine uses token hashes to store/lookup
filters into Map objects.
Before this commit, the tokens were encoded into token hashes
as JS numbers (not exceeding MAX_SAFE_INTEGER) using at most
the 8 first characters of the token.
With this commit, token hashes are now restricted to fit
into 32-bit integers, and are derived from at most the 7 first
characters. This improves filter look-up performance as per
built-in benchmark().
Related commit:
- 69a43e07c4
Using 32 bits of token hash rather than just the 16 lower
bits does help discard more unknown tokens.
Using the default filter lists, the known-token lookup
table is populated by 12,276 entries, out of 65,536, thus
making the case that theoretically there is a lot of
possible tokens which can be discarded.
In practice, running the built-in
staticNetFilteringEngine.benchmark() with default filter
lists, I find that 1,518,929 tokens were skipped out of
4,441,891 extracted tokens, or 34%.
Related commit:
- 3f3a1543ea
The regression was preventing uBO to find from which list a filter
originated. This affected only filters for which the `domain=`
option had multiple hostnames.
Given that all tokens extracted from one single URL are potentially
iterated multiple times in a single URL-matching cycle, it pays to
ignore extracted tokens which are known to not be used anywhere in
the static filtering engine.
The gain in processing a single network request in the static
filtering engine can become especially high when dealing with
long and random-looking URLs, which URLs have a high likelihood
of containing a majority of tokens which are known to not be in
use.
Related commit:
- 99390390fc
The token information available at compile time can be stored
in the filter to be used at match() time. This allows the use of
startsWith() rather than a more costly indexOf() call as a first
quick test to detect mismatches.
Due to how web pages typically load secondary resources and due
to how HNTrieContainer instances are used in uBO, there is a
great likelihood that the result of a previous call to
HNTrieRef.matches() can be reused in a subsequent call.
This has been confirmed by instrumenting HNTrieRef.matches().
Since uBO uses distinct HNTrieContainer instances to either
match against the request or the origin hostnames, this
means a high likelihood of repeated calls to HNTrieRef.matches()
with the same hostname as argument, hence a performance gain
when caching the argument+result -- as despite that
HNTrie.matches() is fast, comparing two short strings is even
faster if this allows to skip HNTrie.matches() altogether.
Performance- and memory-related work. Three more classes have
been created to avoid regex-based filters internally.
Purpose is to enforce filters which have only one single
wildcard in their pattern, a common occurrence. The filter
pattern is split in two literal string segments.
Similar as above, with the added condition that the filter is
hostname-anchored (`||`). The "Wildcard2" variant is a further
specialization to enforce filters where the only wildcard
is immediately preceded by the `^` special character, again
a very common occurrence.
Using two literal string segments in lieu of regexes allows to
quickly detect a mismatch by just testing the first segment.
Additionally, this reduces memory footprint as regexes are
much more expensive memory-wise than plain strings.
These three new filter classes allow to replace the use of
5276 regex-based filters internally with plain string-based
filters.
Often-called isHnAnchored() has been further fine-tuned to
avoid as much work as possible. I have also observed that
using an arrow function for closure-purpose helps measurably
performance, as per built-in benchmark.
The purpose of using a custom base128 encoder is to
convert array buffers into strings, to allow a direct
string-to-array buffer conversion at load time:
string => array buffer
Whereas a JSON array would require an extra step:
JSON array as string => JS array => array buffer
Turns out that the current use of a custom base128 encoding
results in a significantly larger selfie storage usage when
converting array buffers into strings.
Speculation: possibly the browser convert the strings to
save into JSON strings internally. Since the custom base128
encoder is likely to cause the resulting string to contain
a lot of unprintable ASCII characters, these will need to
be escaped when converted to JSON -- escaped characters
occupy more space than non-escaped ones.
Using a sequence of base 64 numbers means only printable
will be present in the output string, hence no escaping
necessary. I have observed significant reduction in
storage usage for selfie purpose.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/528#issuecomment-484408622
Following STrie-related work in above issue, I noticed that a large
number of filters in EasyList were filters which only had to match
against the document origin. For instance, among just the top 10
most populous buckets, there were four such buckets with over
hundreds of entries each:
- bits: 72, token: "http", 146 entries
- bits: 72, token: "https", 139 entries
- bits: 88, token: "http", 122 entries
- bits: 88, token: "https", 118 entries
These filters in these buckets have to be matched against all
the network requests.
In order to leverage HNTrie for these filters[1], they are now handled
in a special way so as to ensure they all end up in a single HNTrie
(per bucket), which means that instead of scanning hundreds of entries
per URL, there is now a single scan per bucket per URL for these
apply-everywhere filters.
Now, any filter which fulfill ALL the following condition will be
processed in a special manner internally:
- Is of the form `|https://` or `|http://` or `*`; and
- Does have a `domain=` option; and
- Does not have a negated domain in its `domain=` option; and
- Does not have `csp=` option; and
- Does not have a `redirect=` option
If a filter does not fulfill ALL the conditions above, no change
in behavior.
A filter which matches ALL of the above will be processed in a special
manner:
- The `domain=` option will be decomposed so as to create as many
distinct filter as there is distinct value in the `domain=` option
- This also apply to the `badfilter` version of the filter, which
means it now become possible to `badfilter` only one of the
distinct filter without having to `badfilter` all of them.
- The logger will always report these special filters with only a
single hostname in the `domain=` option.
***
[1] HNTrie is currently WASM-ed on Firefox.
As a development tool for investigation purpose. To use, enter the
following at uBO's dev console:
µBlock.staticNetFilteringEngine.filterClassHistogram()
In the static network filtering engine, `google` token is too
generic and probably leads to too many false positives, beside
causing too large filter bucket.
Implement a plain string trie container class: STrieContainer.
Make use of STrieContainer where beneficial
Some filter buckets can grow quite large, and in such case
coalescing "trieable" filter classes into a single trie reduces
lookup performance and memory usage.
For instance, at time of commit, the filter bucket for the
`ad` keyword contains 919 entries[1].
Coalescing trieable filters of the same class into a single plain
string trie reduced the size of the bucket into 50 entries + two
tries which are scanned only once each whenever the bucket is
visited.
[1] Enter the following code at uBO's dev console:
µBlock.staticNetFilteringEngine.categories.get(0).get(µBlock.urlTokenizer.tokenHashFromString('ad'))
Refactor static network filtering engine code to make use of
ES6's syntactic sugar `class`.
Change first auto-update run from 7 to 5 minutes.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/416
The Chromium version of uBO has declared `unlimitedStorage` since the
extension was first published in 2014. Declaring this permission in
Firefox brings uBO inline with the Chromium version. I suspect some
reported errors could be caused by IndexedDB eviction due to the lack
of `unlimitedStorage` permission.
Additionally, a timeout has been added when uBO tries to access its
indexedDB storage. It's unclear whether this will help with the
mentioned related issue though, the root cause is still to be
identified.
Relocate workaround to the code responsible to compute filtering context, such
that the workaround will also be applied in other code paths, for example also
for webRequest.onHeadersReceived.
With the new PSL implementation, benchmarks do not show benefit
from caching the domain extracted from a hostname for later
reuse -- the caching seems to even add an overhead instead with
the new publicSuffixList implementation.
Default behavior is to fall back to an alternative backend
if the uBO-selected one is not available. However there will be
no fall back when the `cacheStorageAPI` is set to one specific
backend by the user.
The value of `suspendTabsUntilReady` was disregarded in Firefox and
uBO defaulted to always defer tab loading until it was ready.
This commit allows to disable the deferring of tab loading in
Firefox. The new valid values for `suspendTabsUntilReady` are:
- `unset`: leave it to the platform to pick the optimal
behavior (default)
- `no`: do no suspend tab loading at launch time
- `yes`: suspend tab loading at launch time
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/409
By default `indexedDB` is used in Firefox for purpose of cache storage
backend.
This commit allows to force the use of `browser.storage.local` instead
as cache storage backend. For this to happen, set `cacheStorageAPI` to
`browser.storage.local` in advanced settings.
Additionally, should `indexedDB` not be available for whatever reason,
uBO will automatically fallback to `browser.storage.local`.
These filters are to be considered obsolete since they can't be
matched against network requests in the webRequest API.
They were probably meant to work when ABP was pre-webext, which
means they are quite probably obsolete and there is no longer
a point for uBO to conveniently translate them into CSP directives.
This removes the derivation of FilterOrigin flavors from
FilterOrigin itself and simplify code paths. FilterOrigin
flavors are small specialized classes, no need to
overcomplicate with derivation.
Specifically, this removes an indirect call to reach the
match() method.
As seen at:
https://whotracks.me/blog/adblockers_performance_study.html
The requests.json.gz file can be downloaded from:
https://cdn.cliqz.com/adblocking/requests_top500.json.gz
Copy the file into ./tmp/requests.json.gz
If the file is present when you build uBO using `make-[target].sh` from
the shell, the resulting package will contain `./assets/requests.json`,
which will be looked-up by the method below to launch a benchmark
session.
From uBO's dev console, launch the benchmark:
µBlock.staticNetFilteringEngine.benchmark();
The usual browser dev tools can be used to obtain useful profiling
data, i.e. start the profiler, call the benchmark method from the
console, then stop the profiler when it completes.
Keep in mind that the measurements at the blog post above where obtained
with ONLY EasyList. The CPU reportedly used was:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-6600U+%40+2.60GHz&id=2608
Rename ./tmp/requests.json.gz to something else if you no longer want
./assets/requests.json in the build.
The motivation is to address the higher peak memory usage at launch
time with 3rd-gen HNTrie when a selfie was present.
The selfie generation prior to this change was to collect all
filtering data into a single data structure, and then to serialize
that whole structure at once into storage (using JSON.stringify).
However, HNTrie serialization requires that a large UintArray32 be
converted into a plain JS array, which itslef would be indirectly
converted into a JSON string. This was the main reason why peak
memory usage would be higher at launch from selfie, since the JSON
string would need to be wholly unserialized into JS objects, which
themselves would need to be converted into more specialized data
structures (like that Uint32Array one).
The solution to lower peak memory usage at launch is to refactor
selfie generation to allow a more piecemeal approach: each filtering
component is given the ability to serialize itself rather than to be
forced to be embedded in the master selfie. With this approach, the
HNTrie buffer can now serialize to its own storage by converting the
buffer data directly into a string which can be directly sent to
storage. This avoiding expensive intermediate steps such as
converting into a JS array and then to a JSON string.
As part of the refactoring, there was also opportunistic code
upgrade to ES6 and Promise (eventually all of uBO's code will be
proper ES6).
Additionally, the polyfill to bring getBytesInUse() to Firefox has
been revisited to replace the rather expensive previous
implementation with an implementation with virtually no overhead.
The Promise chain was not properly designed for WASM module
loading. This became apparent when removing WASM modules
from Opera build[1].
The problem was that errors thrown by fetch() -- used to
load WASM modules -- were not properly handled.
[1] Opera refuses updating uBO if there are unrecognized file
types in the package, and `.wasm`/`.wat` files are not
recognized by Opera uploader.
Related issue:
- https://github.com/NanoAdblocker/NanoCore/issues/239
The erroneous behavior was to compute the URL of a sublist as
relative to the URL of the root list, which may differ from the
URL of a parent list.
Those spurious disconnections have been observed to occur at
uBO's launch time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/403
I have observed that this fixes an issue observed on Firefox 64
(current stable).
The reported Waterfox issue *may* be fixed as a result. If not,
the issue he still considered fixed as Waterfox is not
officially supported.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/399
The advanced setting `cacheStorageAPI` has been added to allow
a user to force the use of IndexedDB as cache storage. Set to
`IndexedDB` to force use of IndexedDB. Default to `unset`.
Performance-related work: the logger data has been decoupled
from the DOM -- inspired from CodeMirror's way of efficiently
handling large amout of text data.
This decoupling now makes the logger highly efficient CPU- and
memory-wise, and open the way to more possibilities.
Ability to configure some aspect of the logger behavior and
visuals:
- The hard-coded limit of 5000 entries has been
removed and is now replaced with a variety of
user-configurable settings to enforce the discarding of
logger entries.
- Some columns in the logger output can now be hidden.
The filter list look-up feature has been merged into the
existing overlay dialog used to create URL rules or static
filters, as an entry in a new "Details" pane.
Other issues addressed during refactoring:
- https://github.com/uBlockOrigin/uBlock-issues/issues/280
- https://github.com/gorhill/uBlock/issues/1999
The minimum version supported on Firefox has been bumped
up to 55.0.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/372
- https://github.com/gorhill/uBlock/issues/93
A new advanced settings has been added: `autoCommentFilterTemplate`.
Default value is `{{date}} {{origin}}`.
Placeholders are identified by `{{...}}`. There are currently
only three placeholders supported:
- `{{date}}`: will be replaced with current date
- `{{time}}`: will be replaced with current time
- `{{origin}}`: will be replaced with site information on which
the filter(s) was created
If no placeholder is found in `autoCommentFilterTemplate`, this
will disable auto-commenting. So one can use `-` to disable
auto-commenting.
Additionally, if auto-commenting is enabled, uBO will not emit a
comment if an emitted comment would be a duplicate of the last
one found in the user filter list.
The DOM surveyor will now use time-based logic to spread its work
over time. This allows the surveying to better scale down on
slower devices.
Additionally, the DOM surveyor code has been reworked to lower as
much as possible memory churning when collating nodes to survey.
This rework has been motivated after profiling the "monstrous DOM"
seen in the following page:
<https://doc.rust-lang.org/std/iter/trait.Iterator.html>
The idea is that making the DOM surveyor efficient on such
"monstrous DOM" case should make it efficient everywhere in
practice.
- Avoid concatenating with empty array: though the concatenated
array is empty, this still forces the creation of a whole new
array as per semantic of Array.prototype.concat().
<https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/concat>
- Do not convert arrays to strings when sending data to
main process in surveyPhase1(): I no longer see any benefit
doing so in profiling data (if I recall properly this was
benefiting Firefox, but I can't remember for sure anymore why
I chose to do so back then).
Related issue:
- https://github.com/gorhill/uBlock/issues/3683
This commit further increases uBO's procedural cosmetic filters
Adguard's cosmetic filter syntax -- specifically those procedural
cosmetic filters where plain CSS selectors appeared following
a procedural oeprator (this was rejected as invalid by uBO).
Also, experimental support for `:watch-attrs` procedural
operator, as discussed in <https://github.com/uBlockOrigin/uBlock-issues/issues/341#issuecomment-449765525>.
Support may be dropped before next release depending on whether
a better solution is suggested.
Additionally, the usual opportunistic refactoring toward ES6
syntax.
Related issue:
- https://github.com/gorhill/uBlock/issues/3708
This was brought into the issue above but I ended up forgotting
about it after I focused mostly on the second issue brought up
in there.
commit 7c6cacc59b27660fabacb55d668ef099b222a9e6
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sat Nov 3 08:52:51 2018 -0300
code review: finalize support for wasm-based hntrie
commit 8596ed80e3bdac2c36e3c860b51e7189f6bc8487
Merge: cbe1f2e 000eb82
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sat Nov 3 08:41:40 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit cbe1f2e2f38484d42af3204ec7f1b5decd30f99e
Merge: 270fc7f dbb7e80
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 17:43:20 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit 270fc7f9b3b73d79e6355522c1a42ce782fe7e5c
Merge: d2a89cf d693d4f
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 16:21:08 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit d2a89cf28f0816ffd4617c2c7b4ccfcdcc30e1b4
Merge: d7afc78 649f82f
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 14:54:58 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit d7afc78b5f5675d7d34c5a1d0ec3099a77caef49
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 13:56:11 2018 -0300
finalize wasm-based hntrie implementation
commit e7b9e043cf36ad055791713e34eb0322dec84627
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 08:14:02 2018 -0300
add first-pass implementation of wasm version of hntrie
commit 1015cb34624f3ef73ace58b58fe4e03dfc59897f
Author: Raymond Hill <rhill@raymondhill.net>
Date: Wed Oct 31 17:16:47 2018 -0300
back up draft work toward experimenting with wasm hntries
<https://github.com/gorhill/uBlock/issues/3436>: a new per-site switch
has been added, no-scripting, which purpose is to wholly disable/enable
javascript for a given site. This new switch has precedence over all
other ways javascript can be disabled, including precedence over dynamic
filtering rules.
The popup panel will report the number of script resources which have
been seen by uBO for the current page. There is a minor inaccuracy to
be fixed regarding the count, and which fix requires to extend request
journaling.
<https://github.com/gorhill/uBlock/issues/308>: the `noscript` tags will
now be respected when the new no-scripting switch is in effect on a given
site.
A default setting has been added to the _Settings_ pane to
disable/enable globally the new no-script switch, such that one can
work in default-deny mode regarding javascript execution.
<https://github.com/uBlockOrigin/uBlock-issues/issues/155>: a new
hidden setting, `requestJournalProcessPeriod`, has been added to
allow controlling the delay before uBO internally process it's
network request journal queue. Default to 1000 (milliseconds).
Squashed commit of the following:
commit 6a8473822537636ac54d5dabdb14472114bb730b
Author: Raymond Hill <rhill@raymondhill.net>
Date: Mon Aug 6 10:56:44 2018 -0400
remove remnant of snappyjs and spurious instruction
commit 9a4b709bee97d3cc2235fab602359fa5953bdb46
Author: Raymond Hill <rhill@raymondhill.net>
Date: Mon Aug 6 09:48:58 2018 -0400
make cache storage compression optionally available on all platforms
New advanced setting: `cacheStorageCompression`. Default is `false`.
commit 22ee6547f2f7c9c5aefe25dea1262a1b31612155
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sun Aug 5 19:16:26 2018 -0400
remove Chromium from lz4 experiment
commit ee3e201c45afe983508f70713a2d43af74737d8d
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sun Aug 5 18:52:43 2018 -0400
import lz4-block-codec.wasm library
commit 883a3118efcfd749c82356fde7134754d6ae371d
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sun Aug 5 18:50:46 2018 -0400
implement storage compression through lz4-wasm [draft]
commit 48d1ccaba407de447c2cd6747dc3a90839c260a7
Merge: 8ae77e6 b34c897
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sat Aug 4 08:56:51 2018 -0400
Merge branch 'master' of github.com:gorhill/uBlock into lz4
commit 8ae77e6aeeaa85af335e664c2560d2afd37288c6
Author: Raymond Hill <rhill@raymondhill.net>
Date: Wed Jul 25 18:17:45 2018 -0400
experiment with compression
potentially causing high-generic cosmetic filters to not be applied
because the MRU cache contains an empty list of high-generic filters
when there is a query from a content script for cosmetic filters
before they are fully loaded and ready.
- collate together specific filters with same base domain
- replace string-based hash to integer-based hash
- revisit code to benefit from ES6-specific syntax
- Default to being detached
- Default to "Current tab"
- Append current tab title to "Current tab" entry
- Avoid iterating through all tabs when no change
* Fix leftovers from old code.
* change changes.procedural.size to changes.procedural.length
changes.procedural is an array so it should be changes.procedural.length
the code works with changes.procedural.size because (undefined !== 0) is always true.
Only resources from within current directory will be allowed,
everything else will be silently rejected.
For example, this will forbid pulling lists from different repos
on GitHub, despite the lists being same origin.
A new filtering class has been created: "static extended filtering".
This new class is an umbrella class for more specialized filtering
engines:
- Cosmetic filtering
- Scriptlet filtering
- HTML filtering
HTML filtering is available only on platforms which support modifying
the response body on the fly, so only Firefox 57+ at the moment.
With the ability to modify the response body, HTML filtering has
been introduced: removing elements from the DOM before the source
data has been parsed by the browser.
A consequence of HTML filtering ability is to bring back script tag
filtering feature.
* dom-inspector: Improvments
- Fix race between userCSS injection and element highlight resulting in none or not all elements highlighted.
- Fix page being scanned twice resulting in unneeded slowdown.
* dom-inspector: Clear mutationTimer to allow more than one update.
* dom-inspector: Fix procedural filters shown as declarative with expando.
This is required as per Firefox extension reviewers. Mail exchange:
========
Reviewer:
> Do I read the code correctly that you are executing remote JS by
> downloading/updating from
> https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/resources.txt
> and injecting scripts in contentscripts.js?
Me:
> Yes, resources.txt contains scriptlets or other resources used to:
>
> - Minimize potential page breakage (e.g. google-analytics.com/ga.js);
> - Defuse anti-blockers (e.g. bab-defuser.js);
> - Defuse anti-blockers or minimize page breakage through redirection
> (e.g. 2x2-transparent.png)
>
> This is not a new feature -- this is also part of the legacy version,
> and I consider this is a major feature of uBO. Given how fast things can
> change out there, this allows me to quickly push fixes when a new issue
> is reported for a site without having to go through a full update of the
> extension.
Reviewer:
> I am aware that this is not a new feature. I am unclear why it has been
> allowed in the past, since it violates our policy about remote code
> execution. I assume it was missed due to the fairly complex codebase.
>
> I can approve this version so you are not blocked on the migration, but
> eventually, you cannot use functionality that executes remote code.
> Since we're moving to a more automated review process, you will be able
> to ship new versions without being blocked on a human review.
Me:
> Do I understand correctly that extensions such as TamperMonkey or
> ViolentMonkey won't be allowed on AMO?
>
> Those extensions are even more permissive than uBO given a user can
> import scripts from any source, while with uBO only scriptlets which are
> part of the project are allowed.
Reviewer:
> The key difference between add-ons like Tampermonkey and uBO is that in
> Tampermonkey, users are making an active and conscious decision to
> download and execute that specific code. In uBO, the user did not
> initiate that download/execution, nor are they even aware of it
> happening.
Me:
> So users of TamperMonkey -- tech-savvy or not -- can download & inject
> countless 3rd-party user scripts from countless authors, have them
> update on their own automatically at regular interval with no user
> intervention.
>
> On the other hand, it's not acceptable for me, the author of the
> extension, who users implicitly trusted when installing the extension,
> who is completely controlling and vouching for the content of
> "resources.txt", to have this one 1st-party resource file[1] to be
> updated at regular interval with no user intervention.
>
> So anyways, what is expected from me at this point? Do I need to remove
> scriptlet injection and resource redirection features? Do I need to
> remove only the updating part of resources.txt?
>
> [1] key to core features of uBO (counter anti-blockers + page breakage
> mitigations) and possibly an important factor in installing the
> extension.
========
Now about this commit: the purpose of the code change here is to
prevent "resources.txt" -- which is part of the package -- from being
updated -- this applies only to the Firefox webext[-hybrid] version
of uBO.
- badfilter option was no longer working following last refactoring
changes.
- performance work:
- reduce duplication of large strings.
- new lighter FilterBucket to use when only 2 filters: FilterPair.
Use class(es) whenever available instead of the id when selecting a
broad cosmetic filter (ctrl-click).
When asking for a broad cosmetic filter, using the id instead of
whatever available class(es) is limiting usefulness. The change
here address this.
Example of use case: open
<http://forums.mozillazine.org/viewtopic.php?f=38&t=3027325>.
Now how to remove all signature widgets from all posts?
Without the change here, this was not possible without opening the
browser's inspector, finding out and manually typing whatever class
is used to identify the signature's root element.
With this commit, ctrl-click will now use whatever class information
exist instead of the id.
* refactoring assets management code
* finalizing refactoring of assets management
* various code review of new assets management code
* fix#2281
* fix#1961
* fix#1293
* fix#1275
* fix update scheduler timing logic
* forward compatibility (to be removed once 1.11+ is widespread)
* more codereview; give admins ability to specify own assets.json
* "assetKey" is more accurate than "path"
* fix group count update when building dom incrementally
* reorganize content (order, added URLs, etc.)
* ability to customize updater through advanced settings
* better spinner icon
Using Google bigquery I found that your code was comparing the result of `typeof` with a variable named `undefined`.
As typeof returns a string it should compare to a string with the content `'undefined'`
