Implement a plain string trie container class: STrieContainer.
Make use of STrieContainer where beneficial
Some filter buckets can grow quite large, and in such case
coalescing "trieable" filter classes into a single trie reduces
lookup performance and memory usage.
For instance, at time of commit, the filter bucket for the
`ad` keyword contains 919 entries[1].
Coalescing trieable filters of the same class into a single plain
string trie reduced the size of the bucket into 50 entries + two
tries which are scanned only once each whenever the bucket is
visited.
[1] Enter the following code at uBO's dev console:
µBlock.staticNetFilteringEngine.categories.get(0).get(µBlock.urlTokenizer.tokenHashFromString('ad'))
Refactor static network filtering engine code to make use of
ES6's syntactic sugar `class`.
Change first auto-update run from 7 to 5 minutes.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/416
The Chromium version of uBO has declared `unlimitedStorage` since the
extension was first published in 2014. Declaring this permission in
Firefox brings uBO inline with the Chromium version. I suspect some
reported errors could be caused by IndexedDB eviction due to the lack
of `unlimitedStorage` permission.
Additionally, a timeout has been added when uBO tries to access its
indexedDB storage. It's unclear whether this will help with the
mentioned related issue though, the root cause is still to be
identified.
Relocate workaround to the code responsible to compute filtering context, such
that the workaround will also be applied in other code paths, for example also
for webRequest.onHeadersReceived.
With the new PSL implementation, benchmarks do not show benefit
from caching the domain extracted from a hostname for later
reuse -- the caching seems to even add an overhead instead with
the new publicSuffixList implementation.
Default behavior is to fall back to an alternative backend
if the uBO-selected one is not available. However there will be
no fall back when the `cacheStorageAPI` is set to one specific
backend by the user.
The value of `suspendTabsUntilReady` was disregarded in Firefox and
uBO defaulted to always defer tab loading until it was ready.
This commit allows to disable the deferring of tab loading in
Firefox. The new valid values for `suspendTabsUntilReady` are:
- `unset`: leave it to the platform to pick the optimal
behavior (default)
- `no`: do no suspend tab loading at launch time
- `yes`: suspend tab loading at launch time
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/409
By default `indexedDB` is used in Firefox for purpose of cache storage
backend.
This commit allows to force the use of `browser.storage.local` instead
as cache storage backend. For this to happen, set `cacheStorageAPI` to
`browser.storage.local` in advanced settings.
Additionally, should `indexedDB` not be available for whatever reason,
uBO will automatically fallback to `browser.storage.local`.
These filters are to be considered obsolete since they can't be
matched against network requests in the webRequest API.
They were probably meant to work when ABP was pre-webext, which
means they are quite probably obsolete and there is no longer
a point for uBO to conveniently translate them into CSP directives.
This removes the derivation of FilterOrigin flavors from
FilterOrigin itself and simplify code paths. FilterOrigin
flavors are small specialized classes, no need to
overcomplicate with derivation.
Specifically, this removes an indirect call to reach the
match() method.
As seen at:
https://whotracks.me/blog/adblockers_performance_study.html
The requests.json.gz file can be downloaded from:
https://cdn.cliqz.com/adblocking/requests_top500.json.gz
Copy the file into ./tmp/requests.json.gz
If the file is present when you build uBO using `make-[target].sh` from
the shell, the resulting package will contain `./assets/requests.json`,
which will be looked-up by the method below to launch a benchmark
session.
From uBO's dev console, launch the benchmark:
µBlock.staticNetFilteringEngine.benchmark();
The usual browser dev tools can be used to obtain useful profiling
data, i.e. start the profiler, call the benchmark method from the
console, then stop the profiler when it completes.
Keep in mind that the measurements at the blog post above where obtained
with ONLY EasyList. The CPU reportedly used was:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-6600U+%40+2.60GHz&id=2608
Rename ./tmp/requests.json.gz to something else if you no longer want
./assets/requests.json in the build.
The motivation is to address the higher peak memory usage at launch
time with 3rd-gen HNTrie when a selfie was present.
The selfie generation prior to this change was to collect all
filtering data into a single data structure, and then to serialize
that whole structure at once into storage (using JSON.stringify).
However, HNTrie serialization requires that a large UintArray32 be
converted into a plain JS array, which itslef would be indirectly
converted into a JSON string. This was the main reason why peak
memory usage would be higher at launch from selfie, since the JSON
string would need to be wholly unserialized into JS objects, which
themselves would need to be converted into more specialized data
structures (like that Uint32Array one).
The solution to lower peak memory usage at launch is to refactor
selfie generation to allow a more piecemeal approach: each filtering
component is given the ability to serialize itself rather than to be
forced to be embedded in the master selfie. With this approach, the
HNTrie buffer can now serialize to its own storage by converting the
buffer data directly into a string which can be directly sent to
storage. This avoiding expensive intermediate steps such as
converting into a JS array and then to a JSON string.
As part of the refactoring, there was also opportunistic code
upgrade to ES6 and Promise (eventually all of uBO's code will be
proper ES6).
Additionally, the polyfill to bring getBytesInUse() to Firefox has
been revisited to replace the rather expensive previous
implementation with an implementation with virtually no overhead.
The Promise chain was not properly designed for WASM module
loading. This became apparent when removing WASM modules
from Opera build[1].
The problem was that errors thrown by fetch() -- used to
load WASM modules -- were not properly handled.
[1] Opera refuses updating uBO if there are unrecognized file
types in the package, and `.wasm`/`.wat` files are not
recognized by Opera uploader.
Related issue:
- https://github.com/NanoAdblocker/NanoCore/issues/239
The erroneous behavior was to compute the URL of a sublist as
relative to the URL of the root list, which may differ from the
URL of a parent list.
Those spurious disconnections have been observed to occur at
uBO's launch time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/403
I have observed that this fixes an issue observed on Firefox 64
(current stable).
The reported Waterfox issue *may* be fixed as a result. If not,
the issue he still considered fixed as Waterfox is not
officially supported.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/399
The advanced setting `cacheStorageAPI` has been added to allow
a user to force the use of IndexedDB as cache storage. Set to
`IndexedDB` to force use of IndexedDB. Default to `unset`.
Performance-related work: the logger data has been decoupled
from the DOM -- inspired from CodeMirror's way of efficiently
handling large amout of text data.
This decoupling now makes the logger highly efficient CPU- and
memory-wise, and open the way to more possibilities.
Ability to configure some aspect of the logger behavior and
visuals:
- The hard-coded limit of 5000 entries has been
removed and is now replaced with a variety of
user-configurable settings to enforce the discarding of
logger entries.
- Some columns in the logger output can now be hidden.
The filter list look-up feature has been merged into the
existing overlay dialog used to create URL rules or static
filters, as an entry in a new "Details" pane.
Other issues addressed during refactoring:
- https://github.com/uBlockOrigin/uBlock-issues/issues/280
- https://github.com/gorhill/uBlock/issues/1999
The minimum version supported on Firefox has been bumped
up to 55.0.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/372
- https://github.com/gorhill/uBlock/issues/93
A new advanced settings has been added: `autoCommentFilterTemplate`.
Default value is `{{date}} {{origin}}`.
Placeholders are identified by `{{...}}`. There are currently
only three placeholders supported:
- `{{date}}`: will be replaced with current date
- `{{time}}`: will be replaced with current time
- `{{origin}}`: will be replaced with site information on which
the filter(s) was created
If no placeholder is found in `autoCommentFilterTemplate`, this
will disable auto-commenting. So one can use `-` to disable
auto-commenting.
Additionally, if auto-commenting is enabled, uBO will not emit a
comment if an emitted comment would be a duplicate of the last
one found in the user filter list.
The DOM surveyor will now use time-based logic to spread its work
over time. This allows the surveying to better scale down on
slower devices.
Additionally, the DOM surveyor code has been reworked to lower as
much as possible memory churning when collating nodes to survey.
This rework has been motivated after profiling the "monstrous DOM"
seen in the following page:
<https://doc.rust-lang.org/std/iter/trait.Iterator.html>
The idea is that making the DOM surveyor efficient on such
"monstrous DOM" case should make it efficient everywhere in
practice.
- Avoid concatenating with empty array: though the concatenated
array is empty, this still forces the creation of a whole new
array as per semantic of Array.prototype.concat().
<https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/concat>
- Do not convert arrays to strings when sending data to
main process in surveyPhase1(): I no longer see any benefit
doing so in profiling data (if I recall properly this was
benefiting Firefox, but I can't remember for sure anymore why
I chose to do so back then).
Hopefully this will fix the issue seen on Firefox for Android: for
some reasons, the popup panel's default size is now more zoomed
out than it used to be in latest stable release.
Additionally, I fixed the syntactically incorrect instances of
:before and :after.
Related issue:
- https://github.com/gorhill/uBlock/issues/3683
This commit further increases uBO's procedural cosmetic filters
Adguard's cosmetic filter syntax -- specifically those procedural
cosmetic filters where plain CSS selectors appeared following
a procedural oeprator (this was rejected as invalid by uBO).
Also, experimental support for `:watch-attrs` procedural
operator, as discussed in <https://github.com/uBlockOrigin/uBlock-issues/issues/341#issuecomment-449765525>.
Support may be dropped before next release depending on whether
a better solution is suggested.
Additionally, the usual opportunistic refactoring toward ES6
syntax.
- Make FA icons non-selectable, because of spurious selection
on Firefox for Android when merely tapping an icon
- Disable spellchecking in the static network filter editor
in the logger
Related issue:
- https://github.com/gorhill/uBlock/issues/3708
This was brought into the issue above but I ended up forgotting
about it after I focused mostly on the second issue brought up
in there.
commit 7c6cacc59b27660fabacb55d668ef099b222a9e6
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sat Nov 3 08:52:51 2018 -0300
code review: finalize support for wasm-based hntrie
commit 8596ed80e3bdac2c36e3c860b51e7189f6bc8487
Merge: cbe1f2e 000eb82
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sat Nov 3 08:41:40 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit cbe1f2e2f38484d42af3204ec7f1b5decd30f99e
Merge: 270fc7f dbb7e80
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 17:43:20 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit 270fc7f9b3b73d79e6355522c1a42ce782fe7e5c
Merge: d2a89cf d693d4f
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 16:21:08 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit d2a89cf28f0816ffd4617c2c7b4ccfcdcc30e1b4
Merge: d7afc78 649f82f
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 14:54:58 2018 -0300
Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm
commit d7afc78b5f5675d7d34c5a1d0ec3099a77caef49
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 13:56:11 2018 -0300
finalize wasm-based hntrie implementation
commit e7b9e043cf36ad055791713e34eb0322dec84627
Author: Raymond Hill <rhill@raymondhill.net>
Date: Fri Nov 2 08:14:02 2018 -0300
add first-pass implementation of wasm version of hntrie
commit 1015cb34624f3ef73ace58b58fe4e03dfc59897f
Author: Raymond Hill <rhill@raymondhill.net>
Date: Wed Oct 31 17:16:47 2018 -0300
back up draft work toward experimenting with wasm hntries
