Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
Reported internally:
> STR --
>
> Import https://cdn.statically.io/gh/uBlockOrigin/uAssets/master/filters/filters.txt
> as a Custom filter list.
>
> Observe the filter count at 24K instead of true count
> being 29K.
>
> Force updating is sometimes compiling 24K filters and
> on subsequent updates 29K, so I narrowed it down to
> this host.
Findings:
One of the sublists was erroring when being fetched on
this particular CDN server (reason unknown).
uBO was not properly handling network errors when
fetching a sublist. This commit make it so that if
a sublist can't be fetched, then the error is propagated
as if it affected the whole list, in which case uBO will
use an alternative URL if any.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1365
When compiled data format changes, do not rely on order
of operations at launch to assume deletion of storage
occurs before attempts to access it. It's unclear this
commit will fix the reported issue, as I could not
reproduce it except when outright commenting out the code
to prevent the storage deletion from occurring.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/763
Changes:
From now on, uBO will allow click-to-subscribe on only
a few select domains, currently:
- https://filterlists.com/
- https://github.com/
- https://github.io/
More domains can be added if and only the demonstration
is made that more than a marginal number of filter lists
can be subscribed from those domains.
The browser alert box is no longer used to confirm
subscription to a filter list. Instead, the asset
viewer has been expanded to serve that purpose. This
way, users can peruse at the content of a filter list
before subscribing to it.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
CodeMirror's code folding reference:
- https://codemirror.net/doc/manual.html#addon_foldcode
This commit adds support for code-folding to the filter
list editor/viewer.
The following blocks of code are foldable by clicking the
corresponding marker in the gutter:
- !#if/#endif blocks
- !#include blocks
Addtionally, the following changes:
- The `!#include` line is now preserved when importing a
sublist
- The `!#if` directives will be syntax-colored according
to whether they evaluate to true or false on the current
platform
- Double-clicking on a foldable line in the gutter will
select the content of the foldable block
- Minor visual improvement to matching brackets
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Invalid values for `!#if ...` will be highlighted as errors.
Auto completion is now supported for both the directives
themselves and the valid values for `!#if ...`.
For examples, when pressing ctrl-space:
- `!#e` will auto-complete to `!#endif`
- `!#i` will offer to choose between `!#if ` or `!#include `
- `!#if fir` will auto-complete to `!#if env_firefox`
Additionally, support for some of AdGuard preparsing
directives, i.e. `!#if adguard` is now a valid and will be
honoured -- it always evaluate to `false` in uBO.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/hbpo86/
For unknown reasons at this point, it appears some users
end up with a truncated version of EasyList, leading to
improper blocking in uBO.
This commit adds a heuristic to discard a downloaded list
when its new size is more than 25% smaller than the
currently cached version.
This commit add the ability to fetch from CDN servers
when an asset is fetched as a result of auto-update.
If an asset has a `cdnURLs` entry in `assets.json`,
the asset will be auto-updated using one of those
CDN URLs. When many CDN URLs are specified, those
URLs will be shuffled in order to spread the bandwidth
across all specified CDN servers. If all specified CDN
servers fail to respond, uBO will fall back to usual
`contentURLs` entry.
The `cdnURLs` are used only when an asset is
auto-updated, this ensures a user will get the more
recent available version of an asset when manually
updating.
The motivation of this new feature is to relieve
GitHub from acting as a CDN (which it is not) for
uBO -- an increasing concern with the growing adoption
of uBO along with the growing size of key uBO assets.
The readTime property is unused for compiled or selfie
assets, and thus writing back to storage to persist
this property is useless for those assets, and an undue
overhead when reading such assets, especially at launch
time.
Assets are always loaded from their compiled or selfie
counterparts at launch.
This commit will prevent those unnecessary storage
write operations.
Reported by:
- https://github.com/uBlock-user:
Imported custom list were incorrectly seen as out of
date immediately after import operation.
Regression from:
- e27328f931
A few lines of code were improperly removed during
refactoring.
Related commit:
- e27328f931
The regression was preventing the compiled filter
lists from being properly loaded by uBO, thus
always causing a full parsing/compiling at
launch time.
The ability to redirect xmlhttprequest to binary
resources was lost when redirectable/injectable
resources became immutable in commit
152cea2dfe.
This commit restores the ability to redirect a
xmlhttprequest to a binary resource by making
it possible to derive a data: URI from the
content of binary resources such as images.
Addtionally a redirect to a data: URI can be
forced by prefixing the resource token with `%`.
This is a non-official feature at this point,
i.e. it could be removed at any time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/682#issuecomment-515197130
The following advanced setting has been added:
updateAssetBypassBrowserCache
Default to `false`. If set to `true`, uBO will ensure the
browser cache is bypassed when fetching a remote resource.
This is for the convenience of filter list maintainers who
may want to test the latest version of their lists when
fetched from their remote location.
Related issue:
- https://github.com/gorhill/uBlock/issues/2394
Additionally, I added a new advanced setting to control
how long after launch an auto-update session should be
started -- value is in seconds:
autoUpdateDelayAfterLaunch 180
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/409
By default `indexedDB` is used in Firefox for purpose of cache storage
backend.
This commit allows to force the use of `browser.storage.local` instead
as cache storage backend. For this to happen, set `cacheStorageAPI` to
`browser.storage.local` in advanced settings.
Additionally, should `indexedDB` not be available for whatever reason,
uBO will automatically fallback to `browser.storage.local`.
The motivation is to address the higher peak memory usage at launch
time with 3rd-gen HNTrie when a selfie was present.
The selfie generation prior to this change was to collect all
filtering data into a single data structure, and then to serialize
that whole structure at once into storage (using JSON.stringify).
However, HNTrie serialization requires that a large UintArray32 be
converted into a plain JS array, which itslef would be indirectly
converted into a JSON string. This was the main reason why peak
memory usage would be higher at launch from selfie, since the JSON
string would need to be wholly unserialized into JS objects, which
themselves would need to be converted into more specialized data
structures (like that Uint32Array one).
The solution to lower peak memory usage at launch is to refactor
selfie generation to allow a more piecemeal approach: each filtering
component is given the ability to serialize itself rather than to be
forced to be embedded in the master selfie. With this approach, the
HNTrie buffer can now serialize to its own storage by converting the
buffer data directly into a string which can be directly sent to
storage. This avoiding expensive intermediate steps such as
converting into a JS array and then to a JSON string.
As part of the refactoring, there was also opportunistic code
upgrade to ES6 and Promise (eventually all of uBO's code will be
proper ES6).
Additionally, the polyfill to bring getBytesInUse() to Firefox has
been revisited to replace the rather expensive previous
implementation with an implementation with virtually no overhead.
Related issue:
- https://github.com/NanoAdblocker/NanoCore/issues/239
The erroneous behavior was to compute the URL of a sublist as
relative to the URL of the root list, which may differ from the
URL of a parent list.
Performance-related work: the logger data has been decoupled
from the DOM -- inspired from CodeMirror's way of efficiently
handling large amout of text data.
This decoupling now makes the logger highly efficient CPU- and
memory-wise, and open the way to more possibilities.
Ability to configure some aspect of the logger behavior and
visuals:
- The hard-coded limit of 5000 entries has been
removed and is now replaced with a variety of
user-configurable settings to enforce the discarding of
logger entries.
- Some columns in the logger output can now be hidden.
The filter list look-up feature has been merged into the
existing overlay dialog used to create URL rules or static
filters, as an entry in a new "Details" pane.
Other issues addressed during refactoring:
- https://github.com/uBlockOrigin/uBlock-issues/issues/280
- https://github.com/gorhill/uBlock/issues/1999
The minimum version supported on Firefox has been bumped
up to 55.0.
Squashed commit of the following:
commit 6a8473822537636ac54d5dabdb14472114bb730b
Author: Raymond Hill <rhill@raymondhill.net>
Date: Mon Aug 6 10:56:44 2018 -0400
remove remnant of snappyjs and spurious instruction
commit 9a4b709bee97d3cc2235fab602359fa5953bdb46
Author: Raymond Hill <rhill@raymondhill.net>
Date: Mon Aug 6 09:48:58 2018 -0400
make cache storage compression optionally available on all platforms
New advanced setting: `cacheStorageCompression`. Default is `false`.
commit 22ee6547f2f7c9c5aefe25dea1262a1b31612155
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sun Aug 5 19:16:26 2018 -0400
remove Chromium from lz4 experiment
commit ee3e201c45afe983508f70713a2d43af74737d8d
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sun Aug 5 18:52:43 2018 -0400
import lz4-block-codec.wasm library
commit 883a3118efcfd749c82356fde7134754d6ae371d
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sun Aug 5 18:50:46 2018 -0400
implement storage compression through lz4-wasm [draft]
commit 48d1ccaba407de447c2cd6747dc3a90839c260a7
Merge: 8ae77e6 b34c897
Author: Raymond Hill <rhill@raymondhill.net>
Date: Sat Aug 4 08:56:51 2018 -0400
Merge branch 'master' of github.com:gorhill/uBlock into lz4
commit 8ae77e6aeeaa85af335e664c2560d2afd37288c6
Author: Raymond Hill <rhill@raymondhill.net>
Date: Wed Jul 25 18:17:45 2018 -0400
experiment with compression
