(In addition to in already supported single- and double-quote).
The parsing of (optionally) quoted arguments from an argument
list has been spinned off into a standalone helper in order to
be reused in other parts of the parser eventually.
The `urltransform` option allows to redirect a non-blocked network
request to another URL. There are restrictions on its usage:
- require a trusted source -- thus uBO-maintained lists or user
filters
- the `urltransform` value must start with a `/`
If at least one of these conditions is not fulfilled, the filter
will be invalid and rejected.
The requirement to start with `/` is to enforce that only the path
part of a URL can be modified, thus ensuring the network request
is redirected to the same scheme and authority (as defined at
https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax).
Usage example (redirect requests for CSS resources to a non-existing
resource, for demonstration purpose):
||iana.org^$css,urltransform=/notfound.css
Name of this option is inspired from DNR API:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/URLTransform
This commit required to bring the concept of "trusted source" to
the static network filtering engine.
As per discussion with uBO volunteers.
Volunteers offering support for uBO will be able to craft links with
specially formed URLs, which once clicked will cause uBO to automatically
force an update of specified filter lists.
The URL must be crafted as shown in the example below:
https://ublockorigin.github.io/uAssets/update-lists.html?listkeys=ublock-filters,easylist
Where the `listkeys` parameter is a comma-separated list of tokens
corresponding to filter lists. If a token does not match an enabled
filter list, it will be ignored.
The ability to update filter lists through a specially crafted link
is available only on uBO's own support sites:
- https://github.com/uBlockOrigin/
- https://reddit.com/r/uBlockOrigin/
- https://ublockorigin.github.io/
Additionally, a visual cue has been added in the "Filter lists" pane
to easily spot the filter lists which have been recently updated, where
"recently" is currently defined as less than an hour ago.
Additionally, finalize versioning scheme for uBOL. Since most updates
will be simply related to update rulesets, the version will from now
on reflects the date at which the extension package was created:
year.month.day.minutes
So for example:
2023.8.19.690
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2778
Regression from:
bb41d9594f
The regression occurred because the modified code made the assumption
that a leading combinator would never be preceded by whitespace, while
the parser didn't prevent this.
The parser has been fixed to ensure there is never a leading
whitespace in a selector.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2773
The `randomize` paramater introduced in https://github.com/gorhill/uBlock/commit/418087de9c
is now named `directive`, and beside the `true` value which is meant
to respond with a random 10-character string, it can now take the
following value:
war:[web_accessible_resource name]
In order to mock the XHR response with a web accessible resource. For
example:
piquark6046.github.io##+js(no-xhr-if, adsbygoogle.js, war:googlesyndication_adsbygoogle.js)
Will cause the XHR performed by the webpage to resolve to the content
of `/web_accessible_resources/googlesyndication_adsbygoogle.js`.
Should the resource not exist, the empty string will be returned.
Additionally:
Use `export UBO_VERSION=local` at the console to build MV3 extension using
current version of uBO code base. By default, the version is taken from
`./platform/mv3/ubo-version' and usually set to last stable release.
Prepend pattern with `!` to test for unmatched patterns in
stack trace. This applies to sctiplet parameters which purpose
is to test against the stack, i.e. `aost` and `json-prune`.
Additionally, dropped support for JSON notation in favor of
optional variable arguments notation.
Related discussion:
- https://github.com/uBlockOrigin/uBlock-discussions/discussions/789#discussioncomment-6520330
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/2730
CSS selectors used in cosmetic filtering are normalized in order
to ignore non-functional differences. For instance:
example.org##body p
example.org#@#body p
The first cosmetic filter should be excepted by the second one,
but this was not the case because the fast path use to compile
common CSS selectors was not causing normalization to take
place.
The fix is to ensure that the fast path used to compile most
common CSS selectors is taken only when in presence of already
normalized CSS selectors.
The change allows to better parse AdGuard filters with `replace=`
option when the value to the `replace=` option contains dollar
sign character `$`. uBO will still reject these filters but will
better identify which dollar sign `$` is the real filter option
anchor.
Reference:
- https://adguard.com/kb/general/ad-filtering/create-own-filters/#noop-modifier
uBO already supported the noop filter option `_` to allow filter
authors to resolve possible ambiguities arising when crafting network
filters with many options.
AdGuard extended the semantic of the `_` option to also resolve
readability issues by supporting multiple instances of the `_` option
in a single filter, and also by supporting any number of consecutive
`_` in a single noop filter option.
Reference:
https://adguard.com/kb/general/ad-filtering/create-own-filters/#conditions-directive
This commit should make uBO fully compatible with the `!#if`
directives found throughout AdGuard's filter lists.
Additionally, added the new `!#else` directive for convenience
to filter list authors:
!#if cap_html_filtering
example.com##^script:has-text(fakeAd)
!#else
example.com##+js(rmnt, script, fakeAd)
!#endif
Related issue:
- https://github.com/AdguardTeam/Scriptlets/issues/332
Additionally, uBO's own scriplet syntax now also accept quoting
the parameters with either `'` or `"`. This can be used to avoid
having to escape commas when they are present in a parameter.
The syntax highlighter could throw with some invalid static
network filter patterns. This was caused by the syntax
highlighter still drilling down the pattern parts after
having told codemirror to style the whole pattern as an
error, thus causing the codemirror stream position to go
backward.
Since in uBOL filter lists from various sources are combined into
a single list, there must be a way to turn on/off trust level
inside the resulting combined filter list so as to be able to
validate the trust level of filters requiring trust.
This commit adds new parser directives understood only by MV3
compiler to turn on/off trust flag internally.
New official name: `no-window-open-if`.
The pattern will now be matched against all arguments passed
to `window.open()`: all the arguments are joined as a single
space-spearated string, and the result is used as the target
for matching the pattern.
To enable logging, used the extra parameters approach, i.e.
`log, 1`, which should come after the positional arguments
`pattern`, `delay`, and `decoy`.
Source code of scriplets is now fetched directly from uBO
project, so there is no longer the need to keep duplicate
versions of scriplet code.
All scriplet filters are now supported.
Some filters with entity-based domain option can be salvaged
when there are non-entity-based domain option, but since we are
throwing away the entity-based entries, we are only partially
converting to DNR. This commit will log a warning about this
in log.txt. Before this commit, only non-salvageable filters
were logged.
This reflects the _world_ of the MV3 scripting API:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/scripting/ExecutionWorld
MAIN: page's world
ISOLATED: extension's content script world
Some scriptlets are best executed in either world, so this
commit allows to pick in which world a scriptlet should execute
(default to MAIN).
For instance, the new sed.js scriptlet will now execute in
the ISOLATED world.
At the moment, the only filter lists deemed from a "trusted source"
are uBO-specific filter lists (i.e. "uBlock filters -- ..."), and
the user's own filters from "My filters".
A new scriptlet which can only be used by filter lists from trusted
sources has been introduced: `sed.js`.
The new `sed.js` scriptlet provides the ability to perform
text-level substitutions. Usage:
example.org##+js(sed, nodeName, pattern, replacement, ...)
`nodeName`
The name of the node for which the text content must be substituted.
Valid node names can be found at:
https://developer.mozilla.org/en-US/docs/Web/API/Node/nodeName
`pattern`
A string or regex to find in the text content of the node as the target of
substitution.
`replacement`
The replacement text. Can be omitted if the goal is to delete the text which
matches the pattern. Cannot be omitted if extra pairs of parameters have to be
used (see below).
Optionally, extra pairs of parameters to modify the behavior of the scriptlet:
`condition, pattern`
A string or regex which must be found in the text content of the node
in order for the substitution to occur.
`sedCount, n`
This will cause the scriptlet to stop after n instances of substitution. Since
a mutation oberver is used by the scriptlet, it's advised to stop it whenever
it becomes pointless. Default to zero, which means the scriptlet never stops.
`tryCount, n`
This will cause the scriptlet to stop after n instances of mutation observer
run (regardless of whether a substitution occurred). Default to zero, which
means the scriptlet never stops.
`log, 1`
This will cause the scriptlet to output information at the console, useful as
a debugging tool for filter authors. The logging ability is supported only
in the dev build of uBO.
Examples of usage:
example.com##+js(sed, script, /devtoolsDetector\.launch\(\)\;/, , sedCount, 1)
example.com##+js(sed, #text, /^Advertisement$/)
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/13enzvv/
When assessing which default lists to disable/enable after
updating from 1.48.x to 1.49.x, uBO has to ignore imported
lists, as these do not have a `off` property -- the
non-existence of this property was used to determine whether
a list was default or not. There needs to be an extra test for
whether the list is imported or not.
As discussed internally with filter list maintainers.
Additionally, added a search field to filter out lists. This
is still a work in progress, no need to open issues about this,
I am aware of what is missing (i18n, more tags, etc.)
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2582
If there exist any built-in filter list which last update time
is older than 2 hours, the "Report a filter issue" page will ask
the user to update their filter lists then verify that the issue
still exists.
Once filter lists are updated, the troubleshooting information
will reflect the change in update time.
This allows to reduce the horizontal size of the gutter; more efficient
management of folding since we already need to parse each line for the
linter; and eventually this will also allows to detect unbalanced
!#if-!#endif directives and report such cases as errors.
Additionally, keep incrementally improving error reporting details
in the linter.
maintainers.
To enable logging, use the JSON approach to pass parameters to
`acs` scriptlet. Example:
..##+js(acs, { "target": "document.oncontextmenu", "log": true })
Whereas "target", "needle", and "context" correspond to their
respective positional argument. Using JSON form to pass parameters
allows to specify extra paramters to facilitate debugging of that
scriptlet:
- `"log": true` => output useful information at the dev console.
- `"debug": true` => break at key locations in the scriptlet.
The added logging/debugging capabilities work only in the dev build
of uBO or if the advanced setting `filterAuthorMode` is set to
`true`.
As per CodeMirror's documentation, eachLine() iterator is
faster, so use this. Also no need to keep track or marked
lines, we can just find them on demand, this makes the code
simpler.
uBO will now verify that at least one unprocessed network requests
at launch should have been blocked in order to warn users of
unprocessed network requests through the `!` toolbar icon badge.
For example, with default filter lists, there is nothing to block
on `wikipedia.org`, and hence in this case it's not useful to
present the user with the `!` badge.
Therefore uBO will not show the badge *only* when at least one
unprocessed network requests should have been blocked had uBO been
ready when it was fired by the browser.
Related commit:
- https://github.com/gorhill/uBlock/commit/769b8da664be
Emergency update of assets could be repeatedly launched
every 15 seconds if a resource could not be fetched from
a server. A cooldown period has been added to prevent
repeatedly launching emergency updates.
Before this commit, a network error condition was not reported
when the filter list was imported, it was only reported afterward
when the imported filter list was updated.
Caused by the fact that external filter lists do not have an
`off` property even when they are not enabled.
Additionally, set the default update cycle check period to 2hr.
Builtin scriptlets are no longer parsed as text-based resources,
they are imported as JS functions, and `toString()` is used to
obtain text-based representation of a scriptlet.
Scriptlet parameters are now passed as function call arguments
rather than by replacing text-based occurrences of `{{i}}`. The
arguments are always string values (see below for exception).
Support for argument as Object has been added. This opens the
door to have scriptlets using named arguments rather than
positional arguments, and hence easier to extend functionality
of existing scriptlets. Example:
example.com##+js(scriplet, { "prop": "adblock", "value": false, "log": true })
Compatibility with user-provided scriptlets has been preserved.
User-provided scriptlets can benefit some of the changes:
Use the form `function(..){..}` instead of `(function(..){..})();`
in order to received scriptlet arguments as part of function call
-- instead of using `{{i}}`.
If using the form `function(..){..}`, you can choose to receive
an Object as argument -- just be sure that your scriptlet's
parameter is valid JSON notation.
This commit fix properly handling toggling off the default
status of a list such that the list will be automatically
turned off when its status change from default to non-default.
Additionally, imported lists which become stock lists will
be properly migrated from imported lists section.
Related discussion:
- https://github.com/uBlockOrigin/uAssets/discussions/16939
Various feedback of people trying to interact with uBO's dashboard
at browser launch, before uBO's main process is fully initialized,
causing confusion, and potential loss of data.
This new context menu entry will be available only when the
advanced setting `filterAuthorMode` is set to `true`. See:
https://github.com/gorhill/uBlock/wiki/Advanced-settings#filterauthormode
The purpose is for filter list maintainers to easily access
the source code of web pages when investigating filter issues,
without having to necessarily go through the logger.
Additionally an input field to enter URL directly has been
added to the code viewer for convenience.
Related discussion:
- https://github.com/uBlockOrigin/uAssets/discussions/16939
Until uBO's filtering engines are properly initialized, there will
be a distinct toolbar icon to help users understand that uBO may
not be fully initialized when a webpage is loaded -- often the
cause of improper filtering of trackers/ads at browser launch.
DNR API does not support block-then-redirect concept, hence no
point in generating a DNR block rule for `redirect=` filters.
Additionally, better handle orphaned filter anchors (`||`, `|`).
This commits add the ability to open html/css/script
resources from the logger, as a formatted and syntax-
highligthed document.
The goal is to make it easier for filter list authors to
investigate filter-related issues.
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2234
Example of usage:
/img[a-z]{3,5}\.buzz/##+js(nowoif)
Use sparingly, when no other solution is practical from a maintenance point
of view -- keeping in mind that uBO has to iterate through all the regex-based
values, unlike plain hosyname or entity-based values which are mere lookups.
Related commit:
- b1de8d3fe4
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2234
Example of usage:
@@*$ghide,domain=/img[a-z]{3,5}\.buzz/
Regex-based domain values can be negated just like plain or
entity-based values:
*$domain=~/regex.../
This new syntax does not apply to static extended filters.
This commit is a rewrite of the static filtering parser into a
tree-based data structure, for easier maintenance and better
abstraction of parsed filters.
This simplifies greatly syntax coloring of filters and also
simplify extending filter syntax.
The minimum version of Chromium-based browsers has been raised
to version 73 because of usage of String.matchAll().
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/2442
Cosmetic filters with unknown plain CSS pseudo-classes or
unknown plain CSS pseudo-elements will be rejected, except
for pseudo-classes/pseudo-elements which start with a `-`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/896
Additionally, added the keyboard shortcuts to reload the current
tab to the logger. This changes the prior behavior of reloading
the logger content itself.
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2412#discussioncomment-4421741
The new option is `to=` and the value is a list of domain list with
similar syntax as `domain=` option. Entity-based syntax is supported,
and also negated hostname.
The main motivation is to give uBO's static network filtering engine
with an equivalent of DNR's `requestDomains` and `excludedRequestDomains`.
Essentially `to=` is a superset of `denyallow=`, but for now I decided
against deprecating `denyallow=`, which still does not support entity-
based syntax and for which negated domains are not allowed.
This commit also introduces the `from=` option, which is just an alias
for the `domain=` option. The logger will render network filters using
the `from=` version.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1861
The "exceptor" feature has been rewritten, with the following
changes as a result:
- The excepted filters cease to exist when closing the logger
- It's now possible to temporary except network filters
When toggling on/off a temporary exception, filter lists are now
fully reloaded. This simplified managing temporary exceptions, and
made it easy to implement temporary exception for network filters,
but this also means there might be a perceptible delay when
adding/removing temporary exceptions. At this point I consider
this an acceptable side-effect just to bring the ability to easily
create temporary exception for network filters, while this
simplified the existing temporary exception code throughout.
Bring latest changes to procedural cosmetic filtering to uBOL.
Fix procedural filtering used in HTML filters.
Standardize quick hash algorithm used throughout to DJB2
(except that initialization step is skipped):
- http://www.cse.yorku.ca/~oz/hash.html#djb2
These two new pseudo selectors are _action_ operators, and thus can
only be used at the end of a selector. They both take as argument
a string or regex literal.
For `:remove-class()`, when the argument matches a class name, that
class name is removed.
For `:remove-attr()`, when the argument matches an attribute name,
that attribute is removed.
These operators are meant to replace `+js(remove-attr, ...)` and
`+js(remove-class, ...)`, which from now on are candidate for
deprecation in some future.
Once the next stable release is widespread, filter authors must use
these two new operators instead of their `+js()` counterparts.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/2329
The supported syntax is exactly as per AdGuard's documentation:
- https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters#extended-css-matches-attr
Though recommended, the quotes are not mandatory in uBO if
the argument does not cause the parser to fail and if there
are no ambiguities.
Additionally, improved the code to better unquote pseudo-operator
arguments, and to bring it closer to how AdGuard does it as per
documentation. When using quotes, `"` and `\` should be escaped
to preserve these characters in the unquoted version of the
argument.
Additionally, it is now possible to have `:has-text()` match the
empty string by just quoting the empty string:
...##foo:has-text("")
