The important bit is now considered an action bit
so that there is no more a need for the `important`
property in the parser. The modify bit is now
considered a realm bit.
When the modify bit is set, the action bits become
available to be used to further narrow the realm.
This could be useful in the future if we want to
spread the population of modifier filters across
different buckets.
Reusing the same iterator instance for all cases
of `domain=` option parsing should reduce memory
churning.
Additonally, fine tune regex used to extract
valid token from regex-based filters to increase
likelihood of being able to extract a valid
token.
Reported internally by @gwarser.
In rare occasion, a timing issue could cause uBO to redirect
to a web accessible resource meant to be used for another
network request. This is a regression introduced with the
following commit:
- 2e5d32e967
Additionally, I identified another issue which would cause
cached redirection to fail when a cache entry with redirection
to a web accessible resource was being reused, an issue which
could especially affect pages which are generated dynamically
(i.e. without full page reload).
filterUnits is now treated as a buffer which is
pre-allocated and which will grow in chunks so as
to minimize memory allocations. Entries are never
released, just null-ed.
Additionally, move urlTokenizer into the static
network filtering engine, since it's not used
anywhere else.
Notably, defer the post-load optimization operations
to a few seconds after the filters have been all
loaded in memory -- this is not a critical step for
the filtering engine to work properly, hence this
can be delayed in order to ensure readiness as soon
as possible.
Most notably, the `denyallow=` option now requires
the presence of a valid `domain=` option to not be
rejected.
Using `denyallow=` without narrowing down using the
`domain=` option leads to catastrophic blocking
behvior, hence the requirement for a valid `domain=`
option.
Related commit:
- b265f2644d
The optimization in the commit above was meant to
improve the performance of lookup operations of
modifier filters, but I forgot to enable the
optimisation for that class of filters.
This means this commit brings another significant
performance gain on top of the previous commit, as
shown by the built-in benchmark.
Additionally a few minor code rearrangements.
Performance-related work.
There is a fair number of filters which can't be tokenized
in uBO's own filter lists. Majority of those filters also
declare a `domain=` option, examples:
*$script,redirect-rule=noopjs,domain=...
*$script,3p,domain=...,denyallow=...
*$frame,3p,domain=...
Such filters can be found in uBO's asset viewer using the
following search expression:
/^\*?\$[^\n]*?domain=/
Some filter buckets will contain many of those filters, for
instance one of the bucket holding untokenizable `redirect=`
filters has over 170 entries, which must be all visited when
collating all matching `redirect=` filters.
When a bucket contains many such filters, I found that it's
worth to extract all the non-negated hostname values from
`domain=` options into a single hntrie and perform a pre-test
at match() time to find out whether the current origin of a
network request matches any one of the collected hostnames,
so as to avoid iterating through all the filters.
Since there is rarely a match() for vast majority of network
requests with `domain=` option, this pre-test saves a good
amount of work, and this is measurable with the built-in
benchmark.
This commit moves the parsing, compiling and enforcement
of the `redirect=` and `redirect-rule=` network filter
options into the static network filtering engine as
modifier options -- just like `csp=` and `queryprune=`.
This solves the two following issues:
- https://github.com/gorhill/uBlock/issues/3590
- https://github.com/uBlockOrigin/uBlock-issues/issues/1008#issuecomment-716164214
Additionally, `redirect=` option is not longer afflicted
by static network filtering syntax quirks, `redirect=`
filters can be used with any other static filtering
modifier options, can be excepted using `@@` and can be
badfilter-ed.
Since more than one `redirect=` directives could be found
to apply to a single network request, the concept of
redirect priority is introduced.
By default, `redirect=` directives have an implicit
priority of 0. Filter authors can declare an explicit
priority by appending `:[integer]` to the token of the
`redirect=` option, for example:
||example.com/*.js$1p,script,redirect=noopjs:100
The priority dictates which redirect token out of many
will be ultimately used. Cases of multiple `redirect=`
directives applying to a single blocked network request
are expected to be rather unlikely.
Explicit redirect priority should be used if and only if
there is a case of redirect ambiguity to solve.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/760
The purpose of this new network filter option is to remove
query parameters form the URL of network requests.
The name `queryprune` has been picked over `querystrip`
since the purpose of the option is to remove some
parameters from the URL rather than all parameters.
`queryprune` is a modifier option (like `csp`) in that it
does not cause a network request to be blocked but rather
modified before being emitted.
`queryprune` must be assigned a value, which value will
determine which parameters from a query string will be
removed. The syntax for the value is that of regular
expression *except* for the following rules:
- do not wrap the regex directive between `/`
- do not use regex special values `^` and `$`
- do not use literal comma character in the value,
though you can use hex-encoded version, `\x2c`
- to match the start of a query parameter, prepend `|`
- to match the end of a query parameter, append `|`
`queryprune` regex-like values will be tested against each
key-value parameter pair as `[key]=[value]` string. This
way you can prune according to either the key, the value,
or both.
This commit introduces the concept of modifier filter
options, which as of now are:
- `csp=`
- `queryprune=`
They both work in similar way when used with `important`
option or when used in exception filters. Modifier
options can apply to any network requests, hence the
logger reports the type of the network requests, and no
longer use the modifier as the type, i.e. `csp` filters
are no longer reported as requests of type `csp`.
Though modifier options can apply to any network requests,
for the time being the `csp=` modifier option still apply
only to top or embedded (frame) documents, just as before.
In some future we may want to apply `csp=` directives to
network requests of type script, to control the behavior
of service workers for example.
A new built-in filter expression has been added to the
logger: "modified", which allow to see all the network
requests which were modified before being emitted. The
translation work for this new option will be available
in a future commit.
Additionally, add a button in the About pane
to launch benchmark sessions. The button will
be available only when advanced setting
`benchmarkDatasetURL` is set and pointing to
a valid dataset.
Related issues:
- https://github.com/gorhill/uBlock/issues/1390
- https://github.com/gorhill/uBlock/issues/2334
The deadline to interactively load a specific media
element has been extended from 2sec to 5sec.
Clicking over a blocked large media element will cause
uBO to lookup and handle all potentially blocked large
elements at the cursor position. This should take care
of being able to unblock media elements hidden under
other DOM object.
The CSS style applied to blocked large media elements
has been fine tuned to improve interactivity.
uBO will now remember the specific media elements which
were unblocked and keep them exempted from being
further blocked. This would be an issue when unblocking
a video and then a bit later seeking to another point
in the video, in which case uBO would again block
network requests for that video.
The specificity slider will now be more intuitive
by ordering candidates by match count from highest
match count to the left to the lowest match count
to the right.
Candidates with same match counts will be discarded
and replaced with the shortest candidate.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Double-clicking on...
... a filter option will cause the option to be
wholly selected, including `=[value]` if present;
... a value assigned to a filter option will cause
the value to be wholly selected, except when the
value is a hostname/entity, in which case all the
labels from the cursor position to the right-most
label will be selected.
This allows to bring in all the benefits of
syntax highlighting and enhanced editing
features in the element picker, like auto-
completion, etc.
This is also a necessary step to possibly solve
the following issue:
- https://github.com/gorhill/uBlock/issues/2035
Additionally, incrementally improved the behavior
of uBO's custom CodeMirror static filtering syntax
mode when double-clicking somewhere in a static
extended filter:
- on a class/id string will cause the whole
class/id string to be selected, including the
prepending `.`/`#`.
- somewhere in a hostname/entity will cause all
the labels from the cursor position to the
right-most label to be selected (subject to
change/fine-tune as per feedback of filter
list maintainers).
Related feedback:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134#issuecomment-679421316
The redirectable resource has been renamed
`click2load.html`, so as to avoid uses of dash
characters and to also allow for future different
click-to-load resources.
Additionally, as a requirement to support click-to-load
feature, redirected resources will from now on no
longer be collapsed.
Related issues:
- https://github.com/gorhill/uBlock/issues/2688
- https://github.com/gorhill/uBlock/issues/3619
- https://github.com/gorhill/uBlock/issues/1899
This new feature should considered in its draft
stage and it needs to be fine-tuned as per
feedback.
Important: Only embedded frames can be converted
into click-to-load widgets, as only these can be
properly shieded from access by page content.
Examples of usage:
||youtube.com/embed/$3p,frame,redirect=clicktoload
||scribd.com/embeds/$3p,frame,redirect=clicktoload
||player.vimeo.com/video/$3p,frame,redirect=clicktoload
Related feedback:
- https://github.com/uBlockOrigin/uBlock-issues/issues/401#issuecomment-703075797
Name: `uiTheme`
Default: `unset`
Values:
- `unset`: uBO will pick the theme according to
browser's `prefers-color-scheme`
- `light`: force light scheme
- `dark`: force dark theme
This advanced setting is not to be documented yet as
it has not been decided this is a long term solution.
Double-cliking on a URL will cause the whole URL to be
selected, thus making it easier to navigate to this
URL (through your browser "Open in new tab" entry in
contextual menu).
Unrecognized scriptlet names will be highlighted so as
to warn that the filter is not going to be effective.
Injecting declarative CSS `:style()` selector could cause
the instatiation of the procedural filterer, even when
there was no actual procedural cosmetic filter to
enforce.
This commit ensure that the procedural cosmetic filterer
is instantiated only when there are actual procedural
filters to enforce.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/763
Changes:
From now on, uBO will allow click-to-subscribe on only
a few select domains, currently:
- https://filterlists.com/
- https://github.com/
- https://github.io/
More domains can be added if and only the demonstration
is made that more than a marginal number of filter lists
can be subscribed from those domains.
The browser alert box is no longer used to confirm
subscription to a filter list. Instead, the asset
viewer has been expanded to serve that purpose. This
way, users can peruse at the content of a filter list
before subscribing to it.
Now that the element picker user interface is isolated
frompage content world, there is no need to use the
`important` modifier for the element picker's own
CSS styles.
Further iterating on the work done in following commit:
- 1268f0ae43
This commit adds a new widget to the element picker to
control the depth of a cosmetic filter selector. The
new widget is essentially just another way of selecting
the depth, which is still controllable through picking
one of the cosmetic filters in the list of candidates.
In addition to what is deemed really bad lists by consensus,
some lists will also be labelled "really bad list"
temporarily so as to force-remove them from the set of
filter lists.
This will be the case for filter lists which are not
necessarily "bad lists" but which were once part of
uBO's stock filter lists and have been removed since
then for various reasons.
This will ensure that the majority of users who do not
modifies uBO's default listset will still have a
configuration which matches the official default listset.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/851
The ctrl key is no longer used to adjust specificity of
a candidate filter.
A new widget has been added to adjust the specificity of
a candidate filter to various level. The widget will be
visible as long as the candidate filter matches one entry
in the list of suggested candidate cosmetic filters.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1226
Related commit:
- 9eb455ab5e
In the previous commit, the element picker dialog was
isolated from the page content. This commit is to also
isolate the svg layers from the page content.
With this commit, there is no longer a need for an anonymous
iframe and the isolated world iframe is now directly
embedded in the page.
As a result, pages are now unable to interfere with any
of the element picker user interface. Pages can now only
see an iframe, but are unable to see the content of that
iframe. The styles applied to the iframe are from a user
stylesheet, so as to ensure pages can't override the
iframe's style properties set by uBO.
This commit adds concept of "really bad list" to the
badlists infrastructure. Really bad lists won't be
fetched from a remote server, while plain bad list
will be fetched but won't be compiled.
A really bad list is denoted by the `nofetch` token
following the URL.
Really bad lists can cause more serious issues such
as causing undue launch delays because the remote
server where a really bad list is hosted fails to
respond properly and times out.
Such an example of really bad list is hpHosts which
original server no longer exist.
Many filter lists are known to cause serious filtering
issues in uBO and are not meant to be used in uBO.
Unfortunately, unwitting users keep importing these
filter lists and as a result this ends up causing
filtering issues for which the resolution is always
to remove the incompatible filter list.
Example of inconpatible filter lists:
- Reek's Anti-Adblock Killer
- AdBlock Warning Removal List
- ABP anti-circumvention filter list
uBO will use the following resource to know
which filter lists are incompatible:
- https://github.com/uBlockOrigin/uAssets/blob/master/filters/badlists.txt
Incompatible filter lists can still be imported into
uBO, useful for asset-viewing purpose, but their content
will be discarded at compile time.
Cloud storage is a limited resource, and thus it
makes sense to support data compression before
sending the data to cloud storage.
A new hidden setting allows to toggle on
cloud storage compression:
name: cloudStorageCompression
default: false
By default, this hidden setting is `false`, and a
user must set it to `true` to enable compression
of cloud storage items.
This hidden setting will eventually be toggled
to `true` by default, when there is good confidence
a majority of users are using a version of uBO
which can properly handle compressed cloud storage
items.
A cursory assessment shows that compressed items
are roughly 40-50% smaller in size.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1196
For yet unknown reasons, in Chromium 84 the rendering of
`svg/use`-based icons in the dashboard causes page load
stall and undue high memory usage when the dashboard is
opened, especially the Filter list pane.
The fix in this commit is to avoid using `svg/use` and to
directly create and insert the `path` element defining an
icon.
Added a dotted box around found text occurrences,
as just pale yellow to highlight the text is not
enough to visually distinguish from surrounding text.
Iterating through found text occurrences will now
ensure they are vertically positioned in the middle
of the editor.
Code review following latest changes.
Also, move the input field to the left so that it
renders properly on smaller displays and does not
jump around when the result position/count numbers
change.
This also makes it easier to add more functionality
to the editor's toolbar in the future.
Related commit:
- 23332400f5
Since the search worker can go away after its time-to-live
elapsed, we may need to pass again the haystack on which
search operations are performed.
Before this commit, CodeMirror's add-on for search occurrences
was limited to find at most 1000 first occurrences, because of
performance considerations.
This commit removes this low limit by having the search
occurrences done in a dedicated worker. The limit is now
time-based, and highly unlikely to ever be hit under normal
condition.
With this change, all search occurrences are gathered,
and as a result:
- All occurrences are reported in the scrollbar instead of
just the 1,000 first
- The total count of all occurrences is now reported, instead
of capping at "1000+".
- The current occurrence rank at the cursor or selection
position is now reported -- this was not possible to report
this before.
The number of occurrences is line-based, it's not useful to
report finer-grained occurences in uBO.
Since it's possible to execute specific code paths according
to whether the context is that of a worker or not, it's possible
to keep the main/thread code in a single file. Keeping the
main/worker code paths into a single file is more convenient
for both code maintenance and code review.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1118
Usage of space in network filter patterns will now be
strictly interpreted as the filter being a hosts file
entry.
Usage of space in any other scenario will cause the
pattern of the network filter to be rejected as
erroneous.
The pseudo user styles code served only browsers based
on Chromium 65 and earlier -- Chromium 66 supports
native user styles and was first released more than two
years ago.
In Chromium-based browsers, the pseudo user styles code
is being unconditionally injected in every page/frame
just in case the browser is version 65 or earlier.
Removing pseudo user styles reduce uBO's main content
script in Chromium-based browsers by more than 20K.
Related thread:
- https://github.com/NanoAdblocker/NanoCore/issues/348#issuecomment-653646507
Related commit:
- aed850978e
No need to mind rejected promise after all,
vAPI.tabs.executeScript() is designed to
not fail -- I had forgotten about this.
Related discussion:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1652925
It's not clear the code here will fix the reported
issue, but I did identify that the subframe
dictionary of a very long-lived web page can
theoretically grow unbound.
Regression from:
- 16727d68c8
The issue was causing the element picker to being
unable to select elements with no valid `srcset`
property.
Test case -- trying to select one of the embedded
frames in the following page would fail:
- http://raymondhill.net/ublock/tiles1.html
Lines in AdGuard filter lists have trailing `\r`
characters, and these caused the redirect engine
compile code to reject as invalid the redirect
token.
This is trivially fixed by trimming the raw option
strings before parsing it in the redirect engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
CodeMirror's code folding reference:
- https://codemirror.net/doc/manual.html#addon_foldcode
This commit adds support for code-folding to the filter
list editor/viewer.
The following blocks of code are foldable by clicking the
corresponding marker in the gutter:
- !#if/#endif blocks
- !#include blocks
Addtionally, the following changes:
- The `!#include` line is now preserved when importing a
sublist
- The `!#if` directives will be syntax-colored according
to whether they evaluate to true or false on the current
platform
- Double-clicking on a foldable line in the gutter will
select the content of the foldable block
- Minor visual improvement to matching brackets
Auto-completion will work only for uBO's own
tokens, compatibility-related tokens[1] will not be
taken into account for auto-completion.
The reason is to not have the compatibility-related
tokens get in the way of auto-completion in order
to not inconvenience uBO's filter list maintainers.
[1] `adguard_ext_chromium`, `adguard_ext_firefox`,
etc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Invalid values for `!#if ...` will be highlighted as errors.
Auto completion is now supported for both the directives
themselves and the valid values for `!#if ...`.
For examples, when pressing ctrl-space:
- `!#e` will auto-complete to `!#endif`
- `!#i` will offer to choose between `!#if ` or `!#include `
- `!#if fir` will auto-complete to `!#if env_firefox`
Additionally, support for some of AdGuard preparsing
directives, i.e. `!#if adguard` is now a valid and will be
honoured -- it always evaluate to `false` in uBO.
In case of invalid `Expires` value -- i.e. `NaN` -- do
not use `1` as default value, just let uBO pick the
value according to the global default (which is `5` as
of commit time).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Specifically;
- `beacon`, `ping`, and `websocket` cannot be redirected;
- it's ok to not specify a type when redirecting to `empty`
resource;
- `csp=` option can't be mixed with other types, redirec
directives, and more `csp=` options.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1071
Additionally, match unconditionally against `srcset` attribute
when trying to find matching elements in the page. For example,
sometimes an img element may set both `src` and `srcset`
properties, they should not be deemed mutually exclusive.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/772
Unicode characters inside hostname part of a filter will
be converted to punycode.
Unicode characters anywhere else in the pattern will be
percent-encoded.
Unicode characters which cannot be encoded will cause a
filter to be invalid.
This fix the case of the following filter:
trk*.vidible.tv
Not matching:
https://trk.vidible.tv/trk/.vidible.tv
The wildcard is supposed to match any number of
characters, including zero characters. The issue
is that the code was not matching zero characters.
This is due to an incorrect comparison in
BidiTrieContainer.indexOf(), causing the code to
bail out before testing for the zero character
condition.
The regression broke filters of the form:
||trk*.vidible.tv^
The new parser will eventually interpret differently
wildcard characters when they are used in a manner
meant to represent only hostname-valid characters,
but this will come in a future version -- for now
the default meaning must be preserved until the
static network filtering engine is modified to
enforce the new interpretation.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/hbpo86/
For unknown reasons at this point, it appears some users
end up with a truncated version of EasyList, leading to
improper blocking in uBO.
This commit adds a heuristic to discard a downloaded list
when its new size is more than 25% smaller than the
currently cached version.
This commit adds CodeMirror's auto-completion capability
to the _My filters_ pane.
Currently, auto-completion is available for scriptlet
tokens: pressing ctrl-space while the text cursor is
positioned where a scriptlet token should appear will
cause auto-completion to kick-in. In case of ambiguity,
CodeMirror's widget to pick a specific scriptlet will
appear.
Rename `l` property to `len`, to avoid ambiguity as
`l` could mean _left_ or _length_. Typically `l` is
to be used for _left_ (whereas `r` is to be used for
_right_).
Additionally, add CodeMirror's bracket-matching and
bracket auto-closing to _My filters_ pane and and
bracket-matching to asset viewer page.
Specifically, do not invalidate valid hostnames when
there are extraneous separators: that sort of error
will be visually highlighted but will not otherwise
prevent a filter from being properly enforced.
This ensures proper garbage collection once the parser
is no longer referenced -- this is important now that
the parser is instantiated on-demand only.
Reported internally.
Potential regex flags are passed as is to RegExp contructor,
and in case of failure the query is deemed a plain text one.
Related commit:
- 8de67d22bd (diff-3f4aa453cefa49f6431f1bba3bb53a8e)
This commit moves some of the parsing logic of static
extended filtering into the static filtering parser; this
allows better syntax highlighting and creation-time
error-catching for cosmetic, HTML, and scriptlet filters.
Specifically:
There can't be more than one `redirect` or `csp`
option.
There can be no type specified when there is a
`csp` option.
There must be one single network-related type when
there is a `redirect` option. Since `empty` and
`mp4` imply a `redirect` and a network-related
type, these can't be used along another `redirect`
or network-related type.
Related commit:
- 01b1ed9a98
Reported internally. The issue could cause the logger
to be unable to successfully reverse-lookup a filter
list for a filter which had tokens longer than 6
characters followed by wildcard.
Regression from:
- 01b1ed9a98
Reported internally.
Regression from:
- 01b1ed9a98
The regression was made apparent be the fact that
AdGuard filter lists have lines ending with the
CR character.
Computing storage-used figures can take longer than
usual sometimes when a lot of filter lists are enabled,
and this can cause the Settings pane to take longer to
be filled the first time it is opened. Fetching
settings and storage-used figures separately removes
that potential delay (they were fetched together in
a single Promise.all() call).
Regression from:
- 01b1ed9a98
The new parser needs to be able to deal with trailing
newline characters, which if present will be interpreted
as trailing spaces.
A new standalone static filtering parser is introduced,
vAPI.StaticFilteringParser. It's purpose is to parse
line of text into representation suitable for
compiling filters. It can additionally serves for
syntax highlighting purpose.
As a side effect, this solves:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1038
This is a first draft, there are more work left to do
to further perfect the implementation and extend its
capabilities, especially those useful to assist filter
authors.
For the time being, this commits break line-continuation
syntax highlighting -- which was already flaky prior to
this commit anyway.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1008
This commit adds support entity-matching in the filter
option `domain=`. Example:
pattern$domain=google.*
The `*` above is meant to match any suffix from the Public
Suffix List. The semantic is exactly the same as the
already existing entity-matching support in static
extended filtering:
- https://github.com/gorhill/uBlock/wiki/Static-filter-syntax#entity
Additionally, in this commit:
Fix cases where "just-origin" filters of the form `|http*://`
were erroneously normalized to `|http://`. The proper
normalization of `|http*://` is `*`.
Add support to store hostname strings into the character
buffer of a hntrie container. As of commit time, there are
5,544 instances of FilterOriginHit, and 732 instances of
FilterOriginMiss, which filters require storing/matching a
single hostname string. Those strings are now stored in the
character buffer of the already existing origin-related
hntrie container. (The same approach is used for plain
patterns which are not part of a bidi-trie.)
Pressing spacebar will toggle god mode in popup
panel.
This capability shouldn't encourage people to
misuse _allow_ rules as having to press the
space bar act as an extra necessary step which
purpose is to make the creation of _allow_ rules
more mindful.
