The code exported to nodejs package was revised to use modern
JavaScript syntax. A few issues were fixed at the same time.
The exported classes are:
- DynamicHostRuleFiltering
- DynamicURLRuleFiltering
- DynamicSwitchRuleFiltering
These related to the content the of "My rules" pane in the
uBlock Origin extension.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
The title of tabs in uBO is solely to have a better
presentation in the logger -- no other purpose.
This commit simplify keeping track of the titles, from
an active approach by directly querying it from tabs
whenever a change occurs, to a passive approach by
storing it when the title string become available in
some tab event handlers.
Related issue:
- https://github.com/gorhill/uBlock/issues/3212
The element picker will now properly work on sites where
cosmetic filtering is disabled, but will not allow the
creation of cosmetic filters when specific cosmetic filters
are not meant to be enforced in the current page.
When specific cosmetic filters are not meant to be enforced,
the element picker will still allow the creation of network
filters, that is unless the current page is trusted, in which
case using the element picker is pointless.
Ever since the `redirect` code was refactored:
157cef6034
This advanced setting is no longer needed, as the same
can be accomplished with a plain network filter:
@@*$redirect-rule
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/210
Additionally, a small (experimental) widget has been added
to emphasize/de-emphasize rows which have 3rd-party
scripts/frames, so as to more easily identify which rows
are "affected" by 3rd-party scripts and/or frames.
Tooltip localization for the new widget is not available
yet as I want wait for the feature to be fully settled.
Content scripts can't properly look up effective context
for sandboxed frames. This commit add ability to extract
effective context from already existing store of frames
used for each tab.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1241
uBO will not discard secondary requests fired before a root
frame is committed, by ensuring that if newly uncommitted
root frames are of the same origin as previous one(s), the
uncommited journal slot pointer is not updated.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/857
The recognized resources are:
- abp-resource:blank-mp3
- abp-resource:blank-js
ABP's tokens are excluded from auto-complete so as to not
get in the way of uBO's filter list maintainers.
Reported internally by @uBlock-user.
Also, fixed broken caching of `cname` exception, which forced
uBO to repeatedly evaluate whether a `cname` exception exists
when a block `cname`-cloaked request is encountered.
All matching `redirect-rule` directives will now be reported
in the logger, instead of just the effective one.
The highest-ranked redirect directive will be the one
effectively used for redirection. This way filter list
authors can see whether a lower priority redirect is
being overriden by a higher priority one.
The default priority has been changed to 10, so as to allow
more leeway to create lower ranked redirect directives.
Additonally, rendering of redirect directives with explicit
priority has been fixed in the logger, they will no longer
be rendered as unknown redirect tokens.
New filter options
==================
Strict partyness: `1P`, `3P`
----------------------------
The current options 1p/3p are meant to "weakly" match partyness, i.e. a
network request is considered 1st-party to its context as long as both the
context and the request share the same base domain.
The new partyness options are meant to check for strict partyness, i.e. a
network request will be considered 1st-party if and only if both the context
and the request share the same hostname.
For examples:
- context: `www.example.org`
- request: `www.example.org`
- `1p`: yes, `1P`: yes
- `3p`: no, `3P`: no
- context: `www.example.org`
- request: `subdomain.example.org`
- `1p`: yes, `1P`: no
- `3p`: no, `3P`: yes
- context: `www.example.org`
- request: `www.example.com`
- `1p`: no, `1P`: no
- `3p`: yes, `3P`: yes
The strict partyness options will be visually emphasized in the editor so as
to prevent mistakenly using `1P` or `3P` where weak partyness is meant to be
used.
Filter on response headers: `header=`
-------------------------------------
Currently experimental and under evaluation. Disabled by default, enable by
toggling `filterOnHeaders` to `true` in advanced settings.
Ability to filter network requests according to whether a specific response
header is present and whether it matches or does not match a specific value.
For example:
*$1p,3P,script,header=via:1\.1\s+google
The above filter is meant to block network requests which fullfill all the
following conditions:
- is weakly 1st-party to the context
- is not strictly 1st-party to the context
- is of type `script`
- has a response HTTP header named `via`, which value matches the regular
expression `1\.1\s+google`.
The matches are always performed in a case-insensitive manner.
The header value is assumed to be a literal regular expression, except for
the following special characters:
- to anchor to start of string, use leading `|`, not `^`
- to anchor to end of string, use trailing `|`, not `$`
- to invert the test, use a leading `!`
To block a network request if it merely contains a specific HTTP header is
just a matter of specifying the header name without a header value:
*$1p,3P,script,header=via
Generic exception filters can be used to disable specific block `header=`
filters, i.e. `@@*$1p,3P,script,header` will override the block `header=`
filters given as example above.
Dynamic filtering's `allow` rules override block `headers=` filters.
Important: It is key that filter authors use as many narrowing filter options
as possible when using the `header=` option, and the `header=` option should
be used ONLY when other filter options are not sufficient.
More documentation justifying the purpose of `header=` option will be
provided eventually if ever it is decided to move it from experimental to
stable status.
To be decided: to restrict usage of this filter option to only uBO's own
filter lists or "My filters".
Changes
=======
Fine tuning `queryprune=`
-------------------------
The following changes have been implemented:
The special value `*` (i.e. `queryprune=*`) means "remove all query
parameters".
If the `queryprune=` value is made only of alphanumeric characters
(including `_`), the value will be internally converted to regex equivalent
`^value=`. This ensures a better future compatibility with AdGuard's
`removeparam=`.
If the `queryprune=` value starts with `!`, the test will be inverted. This
can be used to remove all query parameters EXCEPT those who match the
specified value.
Other
-----
The legacy code to test for spurious CSP reports has been removed. This
is no longer an issue ever since uBO redirects to local resources through
web accessible resources.
Notes
=====
The following new and recently added filter options are not compatible with
Chromium's manifest v3 changes:
- `queryprune=`
- `1P`
- `3P`
- `header=`
Reported internally by @gwarser.
In rare occasion, a timing issue could cause uBO to redirect
to a web accessible resource meant to be used for another
network request. This is a regression introduced with the
following commit:
- 2e5d32e967
Additionally, I identified another issue which would cause
cached redirection to fail when a cache entry with redirection
to a web accessible resource was being reused, an issue which
could especially affect pages which are generated dynamically
(i.e. without full page reload).
This commit moves the parsing, compiling and enforcement
of the `redirect=` and `redirect-rule=` network filter
options into the static network filtering engine as
modifier options -- just like `csp=` and `queryprune=`.
This solves the two following issues:
- https://github.com/gorhill/uBlock/issues/3590
- https://github.com/uBlockOrigin/uBlock-issues/issues/1008#issuecomment-716164214
Additionally, `redirect=` option is not longer afflicted
by static network filtering syntax quirks, `redirect=`
filters can be used with any other static filtering
modifier options, can be excepted using `@@` and can be
badfilter-ed.
Since more than one `redirect=` directives could be found
to apply to a single network request, the concept of
redirect priority is introduced.
By default, `redirect=` directives have an implicit
priority of 0. Filter authors can declare an explicit
priority by appending `:[integer]` to the token of the
`redirect=` option, for example:
||example.com/*.js$1p,script,redirect=noopjs:100
The priority dictates which redirect token out of many
will be ultimately used. Cases of multiple `redirect=`
directives applying to a single blocked network request
are expected to be rather unlikely.
Explicit redirect priority should be used if and only if
there is a case of redirect ambiguity to solve.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/760
The purpose of this new network filter option is to remove
query parameters form the URL of network requests.
The name `queryprune` has been picked over `querystrip`
since the purpose of the option is to remove some
parameters from the URL rather than all parameters.
`queryprune` is a modifier option (like `csp`) in that it
does not cause a network request to be blocked but rather
modified before being emitted.
`queryprune` must be assigned a value, which value will
determine which parameters from a query string will be
removed. The syntax for the value is that of regular
expression *except* for the following rules:
- do not wrap the regex directive between `/`
- do not use regex special values `^` and `$`
- do not use literal comma character in the value,
though you can use hex-encoded version, `\x2c`
- to match the start of a query parameter, prepend `|`
- to match the end of a query parameter, append `|`
`queryprune` regex-like values will be tested against each
key-value parameter pair as `[key]=[value]` string. This
way you can prune according to either the key, the value,
or both.
This commit introduces the concept of modifier filter
options, which as of now are:
- `csp=`
- `queryprune=`
They both work in similar way when used with `important`
option or when used in exception filters. Modifier
options can apply to any network requests, hence the
logger reports the type of the network requests, and no
longer use the modifier as the type, i.e. `csp` filters
are no longer reported as requests of type `csp`.
Though modifier options can apply to any network requests,
for the time being the `csp=` modifier option still apply
only to top or embedded (frame) documents, just as before.
In some future we may want to apply `csp=` directives to
network requests of type script, to control the behavior
of service workers for example.
A new built-in filter expression has been added to the
logger: "modified", which allow to see all the network
requests which were modified before being emitted. The
translation work for this new option will be available
in a future commit.
Related issues:
- https://github.com/gorhill/uBlock/issues/1390
- https://github.com/gorhill/uBlock/issues/2334
The deadline to interactively load a specific media
element has been extended from 2sec to 5sec.
Clicking over a blocked large media element will cause
uBO to lookup and handle all potentially blocked large
elements at the cursor position. This should take care
of being able to unblock media elements hidden under
other DOM object.
The CSS style applied to blocked large media elements
has been fine tuned to improve interactivity.
uBO will now remember the specific media elements which
were unblocked and keep them exempted from being
further blocked. This would be an issue when unblocking
a video and then a bit later seeking to another point
in the video, in which case uBO would again block
network requests for that video.
The redirectable resource has been renamed
`click2load.html`, so as to avoid uses of dash
characters and to also allow for future different
click-to-load resources.
Additionally, as a requirement to support click-to-load
feature, redirected resources will from now on no
longer be collapsed.
Related issues:
- https://github.com/gorhill/uBlock/issues/2688
- https://github.com/gorhill/uBlock/issues/3619
- https://github.com/gorhill/uBlock/issues/1899
This new feature should considered in its draft
stage and it needs to be fine-tuned as per
feedback.
Important: Only embedded frames can be converted
into click-to-load widgets, as only these can be
properly shieded from access by page content.
Examples of usage:
||youtube.com/embed/$3p,frame,redirect=clicktoload
||scribd.com/embeds/$3p,frame,redirect=clicktoload
||player.vimeo.com/video/$3p,frame,redirect=clicktoload
Related discussion:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1652925
It's not clear the code here will fix the reported
issue, but I did identify that the subframe
dictionary of a very long-lived web page can
theoretically grow unbound.
This concerns the static network filtering engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943
* * *
New static network filter type: `cname`
By default, network requests which are result of
resolving a canonical name are subject to filtering.
This filtering can be bypassed by creating exception
filters using the `cname` option. For example:
@@*$cname
The filter above tells the network filtering engine
to except network requests which fulfill all the
following conditions:
- network request is blocked
- network request is that of an unaliased hostname
Filter list authors are discouraged from using
exception filters of `cname` type, unless there no
other practical solution such that maintenance
burden become the greater issue. Of course, such
exception filters should be as narrow as possible,
i.e. apply to specific domain, etc.
* * *
New static network filter option: `denyallow`
The purpose of `denyallow` is bring
default-deny/allow-exceptionally ability into static
network filtering arsenal. Example of usage:
*$3p,script, \
denyallow=x.com|y.com \
domain=a.com|b.com
The above filter tells the network filtering engine that
when the context is `a.com` or `b.com`, block all
3rd-party scripts except those from `x.com` and `y.com`.
Essentially, the new `denyallow` option makes it easier
to implement default-deny/allow-exceptionally in static
filter lists, whereas before this had to be done with
unwieldy regular expressions[1], or through the mix of
broadly blocking filters along with exception filters[2].
[1] https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0
[2] Typically filters which pattern are of the
form `|http*://`
While testing for the new `elemhide` option using ABP's
test page[1], I found out that the placeholder of the
blocked image on that page was not properly collapsed
by uBO.
The reason was because the page is very simple and
flushing the blocked-elements cache at
webNavigation.onCommitted time was causing the loss
of information collected between
webRequest.onBeforeRequest and
webNavigation.onCommitted, preventing uBO from
properly collapsing the placeholders.
The blocked-elements cache is now flushed
ONLY at webRequest.onBeforeRequest time.
[1] https://testpages.adblockplus.org/en/exceptions/elemhide
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/651
The `behnind-the-scene` context was wrongly used to
evaluate the whitelist status of the context of tabless
network requests. The document origin must be used
instead when it's available.
Additionally, much code has been revisited for better
ES6 syntax compliance.
Performance-related work: the logger data has been decoupled
from the DOM -- inspired from CodeMirror's way of efficiently
handling large amout of text data.
This decoupling now makes the logger highly efficient CPU- and
memory-wise, and open the way to more possibilities.
Ability to configure some aspect of the logger behavior and
visuals:
- The hard-coded limit of 5000 entries has been
removed and is now replaced with a variety of
user-configurable settings to enforce the discarding of
logger entries.
- Some columns in the logger output can now be hidden.
The filter list look-up feature has been merged into the
existing overlay dialog used to create URL rules or static
filters, as an entry in a new "Details" pane.
Other issues addressed during refactoring:
- https://github.com/uBlockOrigin/uBlock-issues/issues/280
- https://github.com/gorhill/uBlock/issues/1999
The minimum version supported on Firefox has been bumped
up to 55.0.
<https://github.com/gorhill/uBlock/issues/3436>: a new per-site switch
has been added, no-scripting, which purpose is to wholly disable/enable
javascript for a given site. This new switch has precedence over all
other ways javascript can be disabled, including precedence over dynamic
filtering rules.
The popup panel will report the number of script resources which have
been seen by uBO for the current page. There is a minor inaccuracy to
be fixed regarding the count, and which fix requires to extend request
journaling.
<https://github.com/gorhill/uBlock/issues/308>: the `noscript` tags will
now be respected when the new no-scripting switch is in effect on a given
site.
A default setting has been added to the _Settings_ pane to
disable/enable globally the new no-script switch, such that one can
work in default-deny mode regarding javascript execution.
<https://github.com/uBlockOrigin/uBlock-issues/issues/155>: a new
hidden setting, `requestJournalProcessPeriod`, has been added to
allow controlling the delay before uBO internally process it's
network request journal queue. Default to 1000 (milliseconds).
