1
0
mirror of https://github.com/gorhill/uBlock.git synced 2024-11-17 07:52:42 +01:00
Commit Graph

266 Commits

Author SHA1 Message Date
Raymond Hill
00b790ce72
Add support for more !#if pre-parser directive tokens
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1205
2020-08-13 09:32:34 -04:00
C0rn3j
3fed25a52d
Use ISO8061 dates in filter comments 2020-08-03 10:30:36 -04:00
Raymond Hill
e44a568278
Add CoreMirror's code-folding ability to list editor/viewer
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134

CodeMirror's code folding reference:
- https://codemirror.net/doc/manual.html#addon_foldcode

This commit adds support for code-folding to the filter
list editor/viewer.

The following blocks of code are foldable by clicking the
corresponding marker in the gutter:

- !#if/#endif blocks
- !#include blocks

Addtionally, the following changes:

- The `!#include` line is now preserved when importing a
  sublist
- The `!#if` directives will be syntax-colored according
  to whether they evaluate to true or false on the current
  platform
- Double-clicking on a foldable line in the gutter will
  select the content of the foldable block
- Minor visual improvement to matching brackets
2020-07-10 08:01:39 -04:00
Raymond Hill
ebf7fb145e
Fine tune auto-completion for !#if directives
Auto-completion will work only for uBO's own
tokens, compatibility-related tokens[1] will not be
taken into account for auto-completion.

The reason is to not have the compatibility-related
tokens get in the way of auto-completion in order
to not inconvenience uBO's filter list maintainers.

[1] `adguard_ext_chromium`, `adguard_ext_firefox`,
    etc.
2020-07-09 08:09:51 -04:00
Raymond Hill
83c01fb352
Add syntax highlighting/auto-completion for preparsing directives
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134

Invalid values for `!#if ...` will be highlighted as errors.

Auto completion is now supported for both the directives
themselves and the valid values for `!#if ...`.

For examples, when pressing ctrl-space:

- `!#e` will auto-complete to `!#endif`
- `!#i` will offer to choose between `!#if ` or `!#include `
- `!#if fir` will auto-complete to `!#if env_firefox`

Additionally, support for some of AdGuard preparsing
directives, i.e. `!#if adguard` is now a valid and will be
honoured -- it always evaluate to `false` in uBO.
2020-07-08 09:52:27 -04:00
Raymond Hill
18a5f41a04
Better processing of Expires directive in filter list
In case of invalid `Expires` value -- i.e. `NaN` -- do
not use `1` as default value, just let uBO pick the
value according to the global default (which is `5` as
of commit time).
2020-07-06 08:31:53 -04:00
Raymond Hill
0da34f7edf
Handle properly Unicode characters in static network filters
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/772

Unicode characters inside hostname part of a filter will
be converted to punycode.

Unicode characters anywhere else in the pattern will be
percent-encoded.

Unicode characters which cannot be encoded will cause a
filter to be invalid.
2020-07-04 14:47:33 -04:00
Raymond Hill
aab3812089
Ignore !#include directives within inactive !#if/!#endif blocks
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1113
2020-07-03 08:43:40 -04:00
Raymond Hill
bc7f149252
Minor code review of static parser code 2020-06-09 11:58:27 -04:00
Raymond Hill
01b1ed9a98
Add a new static filtering parser
A new standalone static filtering parser is introduced,
vAPI.StaticFilteringParser. It's purpose is to parse
line of text into representation suitable for
compiling filters. It can additionally serves for
syntax highlighting purpose.

As a side effect, this solves:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1038

This is a first draft, there are more work left to do
to further perfect the implementation and extend its
capabilities, especially those useful to assist filter
authors.

For the time being, this commits break line-continuation
syntax highlighting -- which was already flaky prior to
this commit anyway.
2020-06-04 07:18:54 -04:00
Raymond Hill
4687c60bf9
Support fetching assets from CDNs when auto-updating
This commit add the ability to fetch from CDN servers
when an asset is fetched as a result of auto-update.

If an asset has a `cdnURLs` entry in `assets.json`,
the asset will be auto-updated using one of those
CDN URLs. When many CDN URLs are specified, those
URLs will be shuffled in order to spread the bandwidth
across all specified CDN servers. If all specified CDN
servers fail to respond, uBO will fall back to usual
`contentURLs` entry.

The `cdnURLs` are used only when an asset is
auto-updated, this ensures a user will get the more
recent available version of an asset when manually
updating.

The motivation of this new feature is to relieve
GitHub from acting as a CDN (which it is not) for
uBO -- an increasing concern with the growing adoption
of uBO along with the growing size of key uBO assets.
2020-04-08 09:57:55 -04:00
Raymond Hill
11d24abea0
Move proxy-detection code to Firefox-specific code
Related commit:
- https://github.com/uBlockOrigin/uBlock-issues/issues/911

The motivation is to avoid executing code which is
unnecessary on platforms not supporting the browser.dns
API.
2020-03-23 13:31:43 -04:00
Raymond Hill
3f7ece9469
Do not cname-uncloak when a proxy is in use
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/911

Since cname-uncloaking is available only on Firefox
at the moment, the fix is relevant only to Firefox.

By default uBO will no longer cname-uncloak when it
detects that network requests are being being proxied.

This default behavior can be overriden by setting the
new advanced setting `cnameUncloakProxied` to `true`.
The new setting default to `false`, i.e. cname-uncloaking
is disabled when uBO detects that a proxy is in use.

This new advanced setting may disappear once the
following Firefox issue is fixed:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1618271
2020-03-22 14:52:58 -04:00
Raymond Hill
ca80d2826b
Add indentation requirement for line continuation
Related commit:
- https://github.com/gorhill/uBlock/commit/703c525b01aa

This adds an indentation requirement for line
continuation to take place. The conditions are now
as follow:
- Current line ends with ` \`: ASCII space + backslash
- Next line starts with `    `: four ASCII spaces
2020-03-15 08:15:17 -04:00
Raymond Hill
703c525b01
Support line continuation in filter lists
If a line in a filter list ends with a space
(ASCII code 32) followed by a backslash
(ASCII code 92), those two characters will be
removed, the line will be trimmed and the next
line will be trimmed and concatenated to form
a new, longer line.

The purpose is to give filter list authors
a way to visually break apart unduly long
filters and thus make maintenance easier.

When line continuation is used, it is suggested
that the extra lines are prepended with four
space so as to make it more visually obvious that
the extra line(s) are the continuation of a
previous line.

Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943

The filter referenced in the above issue was the
motivation to implement this feature:
- https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0

I verified and could not find any instance in major
filter lists of lines ending with ` \`, thus the
change should be safe.
2020-03-14 13:34:13 -04:00
Raymond Hill
3621792f16
Rework/remove remnant of code dependent on localStorage
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/899
2020-02-23 12:18:45 -05:00
Raymond Hill
15470bcbdc
Ensure disableWebAssembly setting is loaded before use
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/899

WASM modules are now loaded on demand rather than at
script evaluation time.
2020-02-22 13:36:22 -05:00
JustOff
a806dd4bd2
Add env_legacy to the pre-processor supported tokens (#3768)
This will allow specifically target uBlock Origin for Firefox legacy-based browsers in shared filter lists,
see https://github.com/gorhill/uBlock-for-firefox-legacy/pull/1.
2020-02-09 08:23:00 -05:00
Raymond Hill
651955b97c
Throw if mismatched size when unserializing an array buffer
An exception will be thrown if the length of an unserialized
array buffer does not match exactly the original size at
serialization time.
2020-02-04 09:55:02 -05:00
Raymond Hill
f8ec54c635
Fix compatibility issue with hosts files
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/847

IP address `0` is a valid synonym of `0.0.0.0`.
2020-01-10 11:07:31 -05:00
Raymond Hill
91e702cebb
Enable CNAME uncloaking by default
Advanced setting `cnameAliasList` has been removed.

New advanced settings:

cnameUncloak:
  Boolean
Default value:
  true
Description:
  Whether to CNAME-uncloak hostnames.

cnameIgnoreExceptions:
  Boolean
Default value:
  true
Description:
  Whether to bypass the uncloaking of network requests
  which were excepted by filters/rules. This is
  necessary so as to avoid undue breakage by having
  exception filters being rendered useless as a result
  of CNAME-uncloaking.
  For example, `google-analytics.com` uncloaks to
  `www-google-analytics.l.google.com` and both hostnames
  appear in Peter Lowe's list, which means exception
  filters for `google-analytics.com` (to fix site
  breakage) would be rendered useless as the uncloaking
  would cause the network request to be ultimately
  blocked.
2019-12-01 12:05:49 -05:00
Raymond Hill
e98a4b1ace
Discard :: from parsed hosts files
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/800
2019-12-01 09:15:25 -05:00
Raymond Hill
a16e4161de
Fine tune hostname uncloaking through CNAME-lookup
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780

Related commit:
- https://github.com/gorhill/uBlock/commit/3a564c199260

This adds two new advanced settings:

- cnameIgnoreRootDocument
  - Default to `true`
  - Tells uBO to skip CNAME-lookup for root document.

- cnameReplayFullURL
  - Default to `false`
  - Tells uBO whether to replay the whole URL or just
    the origin part of it.
    Replaying only the origin part is meant to lower
    undue breakage and improve performance by avoiding
    repeating the pattern-matching of the whole URL --
    which pattern-matching was most likely already
    accomplished with the original request.

This commit is meant to explore enabling CNAME-lookup
by default for the next stable release while:

- Eliminating a development burden by removing the
  need to create a new filtering syntax to deal with
  undesirable CNAME-cloaked hostnames

- Eliminating a filter list maintainer burden by
  removing the need to find/deal with all base
  domains which engage in undesirable CNAME-cloaked
  hostnames

The hope is that the approach implemented in this
commit should require at most a few unbreak rules
with no further need for special filtering syntax
or filter list maintance efforts.
2019-11-23 13:07:23 -05:00
Raymond Hill
68e1b58bb6
Trim trailing spaces from string values in advanced settings 2019-11-20 11:45:10 -05:00
Raymond Hill
3a564c1992
Add ability to uncloak CNAME records
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780

New webext permission added: `dns`, which purpose is
to allow an extension to fetch the DNS record of
specific hostnames, reference documentation:

https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/dns

The webext API `dns` is available in Firefox 60+ only.

The new API will enable uBO to "uncloak" the actual
hostname used in network requests. The ability is
currently disabled by default for now -- this is only
a first commit related to the above issue to allow
advanced users to immediately use the new ability.

Four advanced settings have been created to control the
uncloaking of actual hostnames:

cnameAliasList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to "uncloak" the  hostnames in the list will.

cnameIgnoreList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to NOT re-run the network request through uBO's
filtering engine with the CNAME hostname. This is
useful to exclude commonly used actual hostnames
from being re-run through uBO's filtering engine, so
as to avoid pointless overhead.

cnameIgnore1stParty: boolean.
Default value: true.
Whether uBO should ignore to re-run a network request
through the filtering engine when the CNAME hostname
is 1st-party to the alias hostname.

cnameMaxTTL: number of minutes.
Default value: 120.
This tells uBO to clear its CNAME cache after the
specified time. For efficiency purpose, uBO will
cache alias=>CNAME associations for reuse so as
to reduce calls to `browser.dns.resolve`. All the
associations will be cleared after the specified time
to ensure the map does not grow too large and too
ensure uBO uses up to date CNAME information.
2019-11-19 12:05:33 -05:00
Raymond Hill
a69b301d81
Fine-tune new bidi-trie code
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
2019-10-29 10:26:34 -04:00
Raymond Hill
d7b2d31180
Harden compiled/selfie format change detection at launch
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/759

This commit adds code to rely less on the state of the
cache storage to decide whether filter lists should be
re-compiled or whether the selfie is currently valid
at launch time when a change in compiled/selfie format
is detected.
2019-10-27 11:49:05 -04:00
Raymond Hill
7971b22385
Expand bidi-trie usage in static network filtering engine
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
- https://github.com/uBlockOrigin/uBlock-issues/issues/528

The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.

Example of filters that could be moved into a bidi-trie
data structure:

    &ad_box_
    /w/d/capu.php?z=$script,third-party
    ||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif

Examples of filters that could NOT be moved to a bidi-trie:

    -adap.$domain=~l-adap.org
    /tsc.php?*&ses=
    ||ibsrv.net/*forumsponsor$domain=[...]
    @@||imgspice.com/jquery.cookie.js|$script
    ||view.atdmt.com^*/iview/$third-party
    ||postimg.cc/image/$csp=[...]

Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).

Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.

This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.

- Added ability to associate a pattern with an integer
  in the bidi-trie [1].
    - The bidi-trie match code passes this externally
      provided integer when calling an externally
      provided method used for testing extra conditions
      that may be present for a plain pattern found to
      be matching in the bidi-trie.

- Decomposed existing filters into smaller logical units:
    - FilterPlainLeftAnchored =>
        FilterPatternPlain +
        FilterAnchorLeft
    - FilterPlainRightAnchored =>
        FilterPatternPlain +
        FilterAnchorRight
    - FilterExactMatch =>
        FilterPatternPlain +
        FilterAnchorLeft +
        FilterAnchorRight
    - FilterPlainHnAnchored =>
        FilterPatternPlain +
        FilterAnchorHn
    - FilterWildcard1 =>
        FilterPatternPlain + [
          FilterPatternLeft or
          FilterPatternRight
        ]
    - FilterWildcard1HnAnchored =>
        FilterPatternPlain + [
          FilterPatternLeft or
          FilterPatternRight
        ] +
        FilterAnchorHn
    - FilterGenericHnAnchored =>
        FilterPatternGeneric +
        FilterAnchorHn
    - FilterGenericHnAndRightAnchored =>
        FilterPatternGeneric +
        FilterAnchorRight +
        FilterAnchorHn
    - FilterOriginMixedSet =>
        FilterOriginMissSet +
        FilterOriginHitSet
    - Instances of FilterOrigin[...], FilterDataHolder
      can also be added to a composite filter to
      represent `domain=` and `csp=` options.

- Added a new filter class, FilterComposite, for
  filters which are a combination of two or more
  logical units. A FilterComposite instance is a
  match when *all* filters composing it are a
  match.

Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.

Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.

Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.

The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.

Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.

A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.

This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.

As a result of the above changes, the map of `token`
(integer)  => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.

Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.

`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.

New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.

All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.

Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.

FilterParser is now instanciated on demand and
released when no longer used.

***

[1] 135a45a878/src/js/strie.js (L120)
[2] e94024d350
[3] 135a45a878/src/js/static-net-filtering.js (L1630)
[4] 135a45a878/src/js/static-net-filtering.js (L1566)
2019-10-21 08:15:58 -04:00
Raymond Hill
8c47fa1a3e
Use async/await instead of chained thens 2019-09-21 19:48:02 -04:00
Raymond Hill
eb871ae558
Fix regression in selfie destruction code
Related commit:
- 915687fddb (diff-73ef8c4664f2ec8c02320d50b2908efdR1100-R1113)

Since selfie destruction is now deferred so as to
coallesce burst of call to destroy(), the selfie
load code must mind whether there is a pending
destruction in order to decide whether the
selfie can be safely loaded.

Related feedback:
- 23c4c80136 (commitcomment-35179834)
2019-09-21 19:24:47 -04:00
Raymond Hill
26235d80d0
Fix regression in importation of custom lists
Reported by:
- https://github.com/uBlock-user:

Imported custom list were incorrectly seen as out of
date immediately after import operation.

Regression from:
- e27328f931

A few lines of code were improperly removed during
refactoring.
2019-09-17 07:44:19 -04:00
Raymond Hill
0051f3b5c7
Work toward modernizing code base: promisification
Swathes of code have been converted to use
Promises/async/await. More left to do.

Related commits:
- eec53c0154
- 915687fddb
- 55cc0c6997
- e27328f931
2019-09-16 16:17:48 -04:00
Raymond Hill
915687fddb
Work toward modernizing code base: promisification
Also, coallesce calls to selfieManager.destroy() so as
to avoid undue repeated calls to storage deletion of
selfie assets.

Related commit:
- e27328f931
2019-09-15 09:36:50 -04:00
Raymond Hill
e27328f931
Work toward modernizing code base: promisification
Swathes of code have been converted to use
Promises/async/await. More left to do.

In the process, a regression affecting the fix to
<https://github.com/uBlockOrigin/uBlock-issues/issues/682>
has been fixed.
2019-09-15 07:58:28 -04:00
Raymond Hill
ca9b29c7ec
Distinguish between priviledge and unprivileged messages
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/710

Messages from unprivileged ports (i.e. from content scripts)
are no longer relayed to message handlers which are to be
strictly used to execute privileged code.

The last remaining case of unprivileged messages which
should be converted into a privileged ones will be taken
care of when the following issue is fixed:
- https://github.com/gorhill/uBlock/issues/3497
2019-09-01 12:43:12 -04:00
Raymond Hill
1d805fb9da
Add support for !#if false in list directive processor
To more easily comment out blocks of filters.
2019-08-25 09:08:41 -04:00
Raymond Hill
7ff750eaf6
Reflect blocking mode in badge color of toolbar icon
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/cmh910/

Additionally, the `3p` rule has been made distinct from
`3p-script`/`3p-frame` for the purpose of
"Relax blocking mode" command.

The badge color will hint at the current blocking mode.
There are four colors for the four following blocking
modes:
- JavaScript wholly disabled
- All 3rd parties blocked
- 3rd-party scripts and frames blocked
- None of the above

The default badge color will be used when JavaScript is not
wholly disabled and when there are no rules for `3p`,
`3p-script` or `3p-frame`.

A new advanced setting has been added to let the user choose
the badge colors for the various blocking modes,
`blockingProfileColors`. The value *must* be a sequence of
4 valid CSS color values that match 6 hexadecimal digits
prefixed with`#` -- anything else will be ignored.
2019-08-10 10:57:24 -04:00
Raymond Hill
5e1f4d7906
Invalidate browser's memory cache after using element picker
Related reports:
- https://www.reddit.com/r/uBlockOrigin/comments/cj7g7m/
- https://www.reddit.com/r/uBlockOrigin/comments/cnq0bi/

The browser cache will be invalidated only when creating
static network filter through the element picker.

As per @gwarser's arguments in favor of implementing
usage of webRequest.handlerBehaviorChanged():
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/handlerBehaviorChanged
2019-08-09 09:31:20 -04:00
Raymond Hill
48347897ad
Remove obsolete ublock-resources-related code
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/679
2019-07-21 11:00:14 -04:00
Raymond Hill
10fe9fe656
Allow setting assetsBootstrapLocation from admin settings
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/666
2019-07-18 10:53:08 -04:00
Raymond Hill
da4c4ded8d
Add a way to reload resources in dev build
Since resources are now immutable, by default they are
only compiled once each time uBO updates to a new
version. However I need a way to force a re-compiling
of the resource in the dev build. This commit adds code
to invalidate the resources selfie when forcing the
update of any filter list.
2019-07-08 08:41:28 -04:00
Raymond Hill
6f5aa947fb
Finalize converting resources.txt into immutable resources
With hindsight, I revised decisions made earlier during
this development cycle:

Un-redirectable scriptlets have been removed from
/web_accessible_resources and instead put in the new
/assets/resources/scriptlets.js, which contains all
scriptlets used for web page injection purpose.

uBO will no longer fetch a remote version of built-in
resources.

Advanced setting `userResourcesLocation` will still be
honoured by uBO, and if set, will be fetched every
time at least one asset is updated.
2019-07-06 12:36:28 -04:00
Raymond Hill
6220e1d3eb
Add missing newline 2019-07-05 08:22:26 -04:00
Raymond Hill
a992875c94
Save only modified immediate hidden settings 2019-07-05 07:33:09 -04:00
Raymond Hill
1fb9845c35
Remove useless code 2019-07-04 14:10:23 -04:00
Raymond Hill
152cea2dfe
Refactor management of injectable resources
This is a first step, the ultimate goal is to remove
the need for resources.txt, or at least to reduce to
only hotfixes or for trivial resources targeting very
specific websites.

Most resources will become immutable, i.e. they will
be part of uBO's code base. Advantages include easier
code maintenance (jshint, syntax highlight), and to
make scriptlets more easy to code review by external
parties (for example extension store reviewers).

TODO:

- More scriptlets need to be imported before next
  release.
- Need to make legacy versions of uBO use a legacy
  version of resources.txt, as all the now obsolete
  scriptlets will have to be removed once uBO's
  next release become widespread.
- Possibly need to add code to load binary
  resources so that they can be injected as
  data: URI. So far it's unclear whether this is
  really needed. For example, this would be needed
  if a xmlhttprequest is redirected to an image
  resource.
2019-07-03 09:47:56 -04:00
Raymond Hill
2bcf671dae
Put back erroneously removed line
Regression from 1dfdc40e09
2019-06-30 12:54:05 -04:00
Raymond Hill
1dfdc40e09
Add ability to suspend network request handler at will
This works only for platforms supporting the return of
Promise by network listeners, i.e. only Firefox at this
point.

When filter lists are reloaded[1], there is a small
time window in which some network requests which should
have normally been blocked are not being blocked
because the static network filtering engine may not
have yet loaded all the filters in memory

This is now addressed by suspending the network request
handler when filter lists are reloaded -- again, this
works only on supported platforms.

[1] Examples: when a filter list update session
    completes; when user filters change, when
    adding/removing filter lists.
2019-06-30 10:09:27 -04:00
Raymond Hill
9065bbdd48
Code review of whitelisting-related code
- Use `Map()` instead of `{}` for internal data
  structure
- Export as array of directives instead of as
  a string
2019-06-25 11:57:14 -04:00
Raymond Hill
de41c1bf53
Fix parsing of recursive !#if-`!#endif directives
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/270
2019-05-18 10:31:04 -04:00
Raymond Hill
fc109c8b7c
Revisit code to benefit from ES6 syntax 2019-05-15 14:49:12 -04:00
Raymond Hill
a52b07ff6e
Make userResourcesLocation able to support multiple URLs
The URLs must be space-separated.

Reminders:
- The additional resources will be updated at the same time
  the built-in resource file is updated
- Purging the cache of 'uBlock filters' will also purge the
  cache of the built-in resource file -- and hence force a
  reload of the user's custom resources if any

Related issues:
- https://github.com/gorhill/uBlock/issues/3307
- https://github.com/uBlockOrigin/uAssets/issues/5184#issuecomment-475875189

Addtionally:
- Opportunitically promisified assets.fetchText()
- Fixed https://github.com/gorhill/uBlock/issues/3586
2019-04-20 17:16:49 -04:00
Raymond Hill
fa83744b58
Use a sequence of base 64 numbers to encode array buffers
The purpose of using a custom base128 encoder is to
convert array buffers into strings, to allow a direct
string-to-array buffer conversion at load time:

  string => array buffer

Whereas a JSON array would require an extra step:

  JSON array as string => JS array => array buffer

Turns out that the current use of a custom base128 encoding
results in a significantly larger selfie storage usage when
converting array buffers into strings.

Speculation: possibly the browser convert the strings to
save into JSON strings internally. Since the custom base128
encoder is likely to cause the resulting string to contain
a lot of unprintable ASCII characters, these will need to
be escaped when converted to JSON -- escaped characters
occupy more space than non-escaped ones.

Using a sequence of base 64 numbers means only printable
will be present in the output string, hence no escaping
necessary. I have observed significant reduction in
storage usage for selfie purpose.
2019-04-20 09:06:54 -04:00
Raymond Hill
c9c21f9cbf
Add more languages for list selection at install/reset time
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/501

Also, the handling of 3-letter language codes has been fixed.
2019-04-14 18:20:57 -04:00
Raymond Hill
2fd587b7ae
Simplyfy code to gather storage used with StorageManager.estimate()
Documentation:
https://developer.mozilla.org/docs/Web/API/StorageManager
2019-03-22 22:09:27 -03:00
Raymond Hill
008370e4b9
Fix https://github.com/uBlockOrigin/uBlock-issues/issues/461
uBO will fallback using a JSON string when trying to encode an array
buffer in Chromium version 59 and earlier.
2019-03-16 09:00:31 -04:00
Raymond Hill
87feb47b51
Support disabling suspendTabsUntilReady in Firefox
The value of `suspendTabsUntilReady` was disregarded in Firefox and
uBO defaulted to always defer tab loading until it was ready.

This commit allows to disable the deferring of tab loading in
Firefox. The new valid values for `suspendTabsUntilReady` are:
- `unset`: leave it to the platform to pick the optimal
  behavior (default)
- `no`: do no suspend tab loading at launch time
- `yes`: suspend tab loading at launch time
2019-02-19 12:30:37 -05:00
Raymond Hill
426a6ea9a7
Fix spurious output at uBO's dev console
Regression from https://github.com/gorhill/uBlock/commit/0d369cda21bb
2019-02-18 14:41:04 -05:00
Raymond Hill
f7a632c94a
Fix regression preventing new install or reseting to factory default
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/431
2019-02-18 06:18:30 -05:00
Raymond Hill
0d369cda21
Allow use of browser.storage.local as cache storage backend in Firefox
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/409

By default `indexedDB` is used in Firefox for purpose of cache storage
backend.

This commit allows to force the use of `browser.storage.local` instead
as cache storage backend. For this to happen, set `cacheStorageAPI` to
`browser.storage.local` in advanced settings.

Additionally, should `indexedDB` not be available for whatever reason,
uBO will automatically fallback to `browser.storage.local`.
2019-02-17 15:40:09 -05:00
Raymond Hill
ed7e34fb07
Refactor selfie generation into a more flexible persistence mechanism
The motivation is to address the higher peak memory usage at launch
time with 3rd-gen HNTrie when a selfie was present.

The selfie generation prior to this change was to collect all
filtering data into a single data structure, and then to serialize
that whole structure at once into storage (using JSON.stringify).

However, HNTrie serialization requires that a large UintArray32 be
converted into a plain JS array, which itslef would be indirectly
converted into a JSON string. This was the main reason why peak
memory usage would be higher at launch from selfie, since the JSON
string would need to be wholly unserialized into JS objects, which
themselves would need to be converted into more specialized data
structures (like that Uint32Array one).

The solution to lower peak memory usage at launch is to refactor
selfie generation to allow a more piecemeal approach: each filtering
component is given the ability to serialize itself rather than to be
forced to be embedded in the master selfie. With this approach, the
HNTrie buffer can now serialize to its own storage by converting the
buffer data directly into a string which can be directly sent to
storage. This avoiding expensive intermediate steps such as
converting into a JS array and then to a JSON string.

As part of the refactoring, there was also opportunistic code
upgrade to ES6 and Promise (eventually all of uBO's code will be
proper ES6).

Additionally, the polyfill to bring getBytesInUse() to Firefox has
been revisited to replace the rather expensive previous
implementation with an implementation with virtually no overhead.
2019-02-14 13:33:55 -05:00
Raymond Hill
4da340384a
Update publicSuffixLibrary to latest (WASM-able) version
See https://github.com/gorhill/publicsuffixlist.js
2019-02-10 12:19:05 -05:00
Raymond Hill
a026e9ae54
Fix reverting use of IndexedDB as default cache storage on Chromium
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/399

The advanced setting `cacheStorageAPI` has been added to allow
a user to force the use of IndexedDB as cache storage. Set to
`IndexedDB` to force use of IndexedDB. Default to `unset`.
2019-01-25 18:49:30 -05:00
Raymond Hill
64bea27881
Add ability to control auto-commenting at filter creation time
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/372
- https://github.com/gorhill/uBlock/issues/93

A new advanced settings has been added: `autoCommentFilterTemplate`.

Default value is `{{date}} {{origin}}`.

Placeholders are identified by `{{...}}`. There are currently
only three placeholders supported:

- `{{date}}`: will be replaced with current date
- `{{time}}`: will be replaced with current time
- `{{origin}}`: will be replaced with site information on which
  the filter(s) was created

If no placeholder is found in `autoCommentFilterTemplate`, this
will disable auto-commenting. So one can use `-` to disable
auto-commenting.

Additionally, if auto-commenting is enabled, uBO will not emit a
comment if an emitted comment would be a duplicate of the last
one found in the user filter list.
2019-01-08 07:37:50 -05:00
Raymond Hill
261ef8c510
Add support for procedural :not to HTML filtering
Related issue: <https://github.com/gorhill/uBlock/issues/3683>

Additionally, improve compile-time error reporting in the logger
2018-12-15 10:46:17 -05:00
Raymond Hill
e7d4aff2a2
fix https://github.com/uBlockOrigin/uBlock-issues/issues/328 2018-12-05 19:18:20 -05:00
Raymond Hill
2189f020df
add new advanced setting to disable use of WASM for dev purpose 2018-11-16 10:19:06 -05:00
Raymond Hill
d7d544cda0
Squashed commit of the following:
commit 7c6cacc59b27660fabacb55d668ef099b222a9e6
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Sat Nov 3 08:52:51 2018 -0300

    code review: finalize support for wasm-based hntrie

commit 8596ed80e3bdac2c36e3c860b51e7189f6bc8487
Merge: cbe1f2e 000eb82
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Sat Nov 3 08:41:40 2018 -0300

    Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm

commit cbe1f2e2f38484d42af3204ec7f1b5decd30f99e
Merge: 270fc7f dbb7e80
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Fri Nov 2 17:43:20 2018 -0300

    Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm

commit 270fc7f9b3b73d79e6355522c1a42ce782fe7e5c
Merge: d2a89cf d693d4f
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Fri Nov 2 16:21:08 2018 -0300

    Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm

commit d2a89cf28f0816ffd4617c2c7b4ccfcdcc30e1b4
Merge: d7afc78 649f82f
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Fri Nov 2 14:54:58 2018 -0300

    Merge branch 'master' of github.com:gorhill/uBlock into trie-wasm

commit d7afc78b5f5675d7d34c5a1d0ec3099a77caef49
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Fri Nov 2 13:56:11 2018 -0300

    finalize wasm-based hntrie implementation

commit e7b9e043cf36ad055791713e34eb0322dec84627
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Fri Nov 2 08:14:02 2018 -0300

    add first-pass implementation of wasm version of hntrie

commit 1015cb34624f3ef73ace58b58fe4e03dfc59897f
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Wed Oct 31 17:16:47 2018 -0300

    back up draft work toward experimenting with wasm hntries
2018-11-03 08:58:46 -03:00
Raymond Hill
c0fc7dd869
fix https://github.com/uBlockOrigin/uBlock-issues/issues/248 2018-10-24 20:02:15 -03:00
Raymond Hill
cabb0d36b6
fix https://github.com/gorhill/uBlock/issues/3371 2018-10-23 14:01:08 -03:00
Raymond Hill
6d34a52179
fix #2859 2018-09-03 14:06:49 -04:00
Raymond Hill
8ca6239e64
fix https://github.com/NanoAdblocker/NanoCore/issues/202 2018-08-30 08:32:56 -04:00
Raymond Hill
38aabc937a
reorganize cache storage compression; workaround fix for #2812 2018-08-11 10:39:43 -04:00
Raymond Hill
e163080518
added optional lz4 compression for cache storage (https://github.com/uBlockOrigin/uBlock-issues/issues/141)
Squashed commit of the following:

commit 6a8473822537636ac54d5dabdb14472114bb730b
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Mon Aug 6 10:56:44 2018 -0400

    remove remnant of snappyjs and spurious instruction

commit 9a4b709bee97d3cc2235fab602359fa5953bdb46
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Mon Aug 6 09:48:58 2018 -0400

    make cache storage compression optionally available on all platforms

    New advanced setting: `cacheStorageCompression`. Default is `false`.

commit 22ee6547f2f7c9c5aefe25dea1262a1b31612155
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Sun Aug 5 19:16:26 2018 -0400

    remove Chromium from lz4 experiment

commit ee3e201c45afe983508f70713a2d43af74737d8d
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Sun Aug 5 18:52:43 2018 -0400

    import lz4-block-codec.wasm library

commit 883a3118efcfd749c82356fde7134754d6ae371d
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Sun Aug 5 18:50:46 2018 -0400

    implement storage compression through lz4-wasm [draft]

commit 48d1ccaba407de447c2cd6747dc3a90839c260a7
Merge: 8ae77e6 b34c897
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Sat Aug 4 08:56:51 2018 -0400

    Merge branch 'master' of github.com:gorhill/uBlock into lz4

commit 8ae77e6aeeaa85af335e664c2560d2afd37288c6
Author: Raymond Hill <rhill@raymondhill.net>
Date:   Wed Jul 25 18:17:45 2018 -0400

    experiment with compression
2018-08-06 12:34:41 -04:00
Raymond Hill
7702b1f4bc
minor code review 2018-07-25 18:24:14 -04:00
Raymond Hill
7ae68c8d7d
fix https://github.com/uBlockOrigin/uBlock-issues/issues/144 2018-07-24 09:17:18 -04:00
Raymond Hill
798f8dab9d
reduce baseline memory at selfie-load time 2018-06-01 07:54:31 -04:00
Hugo Xu
de6cf0e6aa Align header parsing to ABP (#3724) 2018-05-16 14:55:12 -04:00
Hugo Xu
25786c3257 Round hour up to nearest day (#3723)
* Update storage.js

* Update storage.js
2018-05-16 11:01:40 -04:00
Raymond Hill
f64d703ba0
fix quirks in parsing of Steven Black's hosts files 2018-05-01 18:29:58 -04:00
Raymond Hill
3923520b87
remove no longer needed platform-dependent polyfill.js 2018-04-27 08:36:38 -04:00
Hugo Xu
ae6b376a75 Fix RegExp escape (#3700) 2018-04-11 11:28:10 -04:00
Raymond Hill
c34326cf4e
code review: further tuning filter list directives 2018-04-11 06:34:13 -04:00
Raymond Hill
00cb02ec47
fix new hidden setting default value not being applied (https://github.com/orgs/uBlockOrigin/teams/volunteers/discussions/62) 2018-04-09 17:46:29 -04:00
Raymond Hill
47598d4499
further fine tune "Filter lists" pane (b60c06f3) 2018-04-09 15:45:25 -04:00
Raymond Hill
26c2320e46
code review: improve compatibility with StevenBlack hosts files 2018-04-09 15:40:29 -04:00
Raymond Hill
daf948b779
fix https://github.com/gorhill/uBlock/pull/3679#issuecomment-379092981 2018-04-06 16:15:10 -04:00
Raymond Hill
a015ad0a5c
Merge branch 'master' of github.com:gorhill/uBlock 2018-04-06 16:02:41 -04:00
Raymond Hill
4d8974fe80
code review: avoid redundant PSL selfie 2018-04-06 16:02:35 -04:00
Hugo Xu
fc4fccd8e3 Accept # Title: and # Expires:; optimize RegExp (#3679)
* Accept `# Title:` and `# Expires`; optimize RegExp

* Prevent confusion with network filter
2018-04-06 15:59:01 -04:00
Raymond Hill
c0387835fa
code review: modernize old code 2018-04-05 15:22:19 -04:00
Raymond Hill
93f49a61d7
add pre-processor directives to filter list compiler (https://github.com/AdguardTeam/AdguardBrowserExtension/issues/917) 2018-04-05 07:29:15 -04:00
Raymond Hill
9fb455d2df
fix #3406 2018-02-23 06:42:17 -05:00
Raymond Hill
a81d2a759b
fix #3318, #3387 2018-02-21 13:29:36 -05:00
Raymond Hill
17930cc778
fix #3474, #2823 2018-02-15 17:25:38 -05:00
Raymond Hill
636dcf7ee4
fix #3383 2018-01-01 07:52:03 -05:00
Raymond Hill
5c20182948
fix regression in per-list filter counts (reported by @mapx-) 2017-12-29 13:31:37 -05:00
Raymond Hill
a9f68fe02f
Fix #3069, and consequently #3374, #3378.
A new filtering class has been created: "static extended filtering".
This new class is an umbrella class for more specialized filtering
engines:
- Cosmetic filtering
- Scriptlet filtering
- HTML filtering

HTML filtering is available only on platforms which support modifying
the response body on the fly, so only Firefox 57+ at the moment.

With the ability to modify the response body, HTML filtering has
been introduced: removing elements from the DOM before the source
data has been parsed by the browser.

A consequence of HTML filtering ability is to bring back script tag
filtering feature.
2017-12-28 13:49:02 -05:00
Raymond Hill
36956cbc7a
remove obsolete resource caching code 2017-12-14 16:42:54 -05:00
gorhill
118240d36d
fix #3232 (regression from 386e8bee9c) 2017-11-15 16:32:52 -05:00