This commit adds concept of "really bad list" to the
badlists infrastructure. Really bad lists won't be
fetched from a remote server, while plain bad list
will be fetched but won't be compiled.
A really bad list is denoted by the `nofetch` token
following the URL.
Really bad lists can cause more serious issues such
as causing undue launch delays because the remote
server where a really bad list is hosted fails to
respond properly and times out.
Such an example of really bad list is hpHosts which
original server no longer exist.
Many filter lists are known to cause serious filtering
issues in uBO and are not meant to be used in uBO.
Unfortunately, unwitting users keep importing these
filter lists and as a result this ends up causing
filtering issues for which the resolution is always
to remove the incompatible filter list.
Example of inconpatible filter lists:
- Reek's Anti-Adblock Killer
- AdBlock Warning Removal List
- ABP anti-circumvention filter list
uBO will use the following resource to know
which filter lists are incompatible:
- https://github.com/uBlockOrigin/uAssets/blob/master/filters/badlists.txt
Incompatible filter lists can still be imported into
uBO, useful for asset-viewing purpose, but their content
will be discarded at compile time.
Cloud storage is a limited resource, and thus it
makes sense to support data compression before
sending the data to cloud storage.
A new hidden setting allows to toggle on
cloud storage compression:
name: cloudStorageCompression
default: false
By default, this hidden setting is `false`, and a
user must set it to `true` to enable compression
of cloud storage items.
This hidden setting will eventually be toggled
to `true` by default, when there is good confidence
a majority of users are using a version of uBO
which can properly handle compressed cloud storage
items.
A cursory assessment shows that compressed items
are roughly 40-50% smaller in size.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1196
For yet unknown reasons, in Chromium 84 the rendering of
`svg/use`-based icons in the dashboard causes page load
stall and undue high memory usage when the dashboard is
opened, especially the Filter list pane.
The fix in this commit is to avoid using `svg/use` and to
directly create and insert the `path` element defining an
icon.
Added a dotted box around found text occurrences,
as just pale yellow to highlight the text is not
enough to visually distinguish from surrounding text.
Iterating through found text occurrences will now
ensure they are vertically positioned in the middle
of the editor.
Code review following latest changes.
Also, move the input field to the left so that it
renders properly on smaller displays and does not
jump around when the result position/count numbers
change.
This also makes it easier to add more functionality
to the editor's toolbar in the future.
Related commit:
- 23332400f5
Since the search worker can go away after its time-to-live
elapsed, we may need to pass again the haystack on which
search operations are performed.
Before this commit, CodeMirror's add-on for search occurrences
was limited to find at most 1000 first occurrences, because of
performance considerations.
This commit removes this low limit by having the search
occurrences done in a dedicated worker. The limit is now
time-based, and highly unlikely to ever be hit under normal
condition.
With this change, all search occurrences are gathered,
and as a result:
- All occurrences are reported in the scrollbar instead of
just the 1,000 first
- The total count of all occurrences is now reported, instead
of capping at "1000+".
- The current occurrence rank at the cursor or selection
position is now reported -- this was not possible to report
this before.
The number of occurrences is line-based, it's not useful to
report finer-grained occurences in uBO.
Since it's possible to execute specific code paths according
to whether the context is that of a worker or not, it's possible
to keep the main/thread code in a single file. Keeping the
main/worker code paths into a single file is more convenient
for both code maintenance and code review.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1118
Usage of space in network filter patterns will now be
strictly interpreted as the filter being a hosts file
entry.
Usage of space in any other scenario will cause the
pattern of the network filter to be rejected as
erroneous.
The pseudo user styles code served only browsers based
on Chromium 65 and earlier -- Chromium 66 supports
native user styles and was first released more than two
years ago.
In Chromium-based browsers, the pseudo user styles code
is being unconditionally injected in every page/frame
just in case the browser is version 65 or earlier.
Removing pseudo user styles reduce uBO's main content
script in Chromium-based browsers by more than 20K.
Related thread:
- https://github.com/NanoAdblocker/NanoCore/issues/348#issuecomment-653646507
Related commit:
- aed850978e
No need to mind rejected promise after all,
vAPI.tabs.executeScript() is designed to
not fail -- I had forgotten about this.
Related discussion:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1652925
It's not clear the code here will fix the reported
issue, but I did identify that the subframe
dictionary of a very long-lived web page can
theoretically grow unbound.
Regression from:
- 16727d68c8
The issue was causing the element picker to being
unable to select elements with no valid `srcset`
property.
Test case -- trying to select one of the embedded
frames in the following page would fail:
- http://raymondhill.net/ublock/tiles1.html
Lines in AdGuard filter lists have trailing `\r`
characters, and these caused the redirect engine
compile code to reject as invalid the redirect
token.
This is trivially fixed by trimming the raw option
strings before parsing it in the redirect engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
CodeMirror's code folding reference:
- https://codemirror.net/doc/manual.html#addon_foldcode
This commit adds support for code-folding to the filter
list editor/viewer.
The following blocks of code are foldable by clicking the
corresponding marker in the gutter:
- !#if/#endif blocks
- !#include blocks
Addtionally, the following changes:
- The `!#include` line is now preserved when importing a
sublist
- The `!#if` directives will be syntax-colored according
to whether they evaluate to true or false on the current
platform
- Double-clicking on a foldable line in the gutter will
select the content of the foldable block
- Minor visual improvement to matching brackets
Auto-completion will work only for uBO's own
tokens, compatibility-related tokens[1] will not be
taken into account for auto-completion.
The reason is to not have the compatibility-related
tokens get in the way of auto-completion in order
to not inconvenience uBO's filter list maintainers.
[1] `adguard_ext_chromium`, `adguard_ext_firefox`,
etc.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Invalid values for `!#if ...` will be highlighted as errors.
Auto completion is now supported for both the directives
themselves and the valid values for `!#if ...`.
For examples, when pressing ctrl-space:
- `!#e` will auto-complete to `!#endif`
- `!#i` will offer to choose between `!#if ` or `!#include `
- `!#if fir` will auto-complete to `!#if env_firefox`
Additionally, support for some of AdGuard preparsing
directives, i.e. `!#if adguard` is now a valid and will be
honoured -- it always evaluate to `false` in uBO.
In case of invalid `Expires` value -- i.e. `NaN` -- do
not use `1` as default value, just let uBO pick the
value according to the global default (which is `5` as
of commit time).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1134
Specifically;
- `beacon`, `ping`, and `websocket` cannot be redirected;
- it's ok to not specify a type when redirecting to `empty`
resource;
- `csp=` option can't be mixed with other types, redirec
directives, and more `csp=` options.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1071
Additionally, match unconditionally against `srcset` attribute
when trying to find matching elements in the page. For example,
sometimes an img element may set both `src` and `srcset`
properties, they should not be deemed mutually exclusive.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/772
Unicode characters inside hostname part of a filter will
be converted to punycode.
Unicode characters anywhere else in the pattern will be
percent-encoded.
Unicode characters which cannot be encoded will cause a
filter to be invalid.
This fix the case of the following filter:
trk*.vidible.tv
Not matching:
https://trk.vidible.tv/trk/.vidible.tv
The wildcard is supposed to match any number of
characters, including zero characters. The issue
is that the code was not matching zero characters.
This is due to an incorrect comparison in
BidiTrieContainer.indexOf(), causing the code to
bail out before testing for the zero character
condition.
The regression broke filters of the form:
||trk*.vidible.tv^
The new parser will eventually interpret differently
wildcard characters when they are used in a manner
meant to represent only hostname-valid characters,
but this will come in a future version -- for now
the default meaning must be preserved until the
static network filtering engine is modified to
enforce the new interpretation.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/hbpo86/
For unknown reasons at this point, it appears some users
end up with a truncated version of EasyList, leading to
improper blocking in uBO.
This commit adds a heuristic to discard a downloaded list
when its new size is more than 25% smaller than the
currently cached version.
This commit adds CodeMirror's auto-completion capability
to the _My filters_ pane.
Currently, auto-completion is available for scriptlet
tokens: pressing ctrl-space while the text cursor is
positioned where a scriptlet token should appear will
cause auto-completion to kick-in. In case of ambiguity,
CodeMirror's widget to pick a specific scriptlet will
appear.
Rename `l` property to `len`, to avoid ambiguity as
`l` could mean _left_ or _length_. Typically `l` is
to be used for _left_ (whereas `r` is to be used for
_right_).
Additionally, add CodeMirror's bracket-matching and
bracket auto-closing to _My filters_ pane and and
bracket-matching to asset viewer page.
Specifically, do not invalidate valid hostnames when
there are extraneous separators: that sort of error
will be visually highlighted but will not otherwise
prevent a filter from being properly enforced.
This ensures proper garbage collection once the parser
is no longer referenced -- this is important now that
the parser is instantiated on-demand only.
Reported internally.
Potential regex flags are passed as is to RegExp contructor,
and in case of failure the query is deemed a plain text one.
Related commit:
- 8de67d22bd (diff-3f4aa453cefa49f6431f1bba3bb53a8e)
This commit moves some of the parsing logic of static
extended filtering into the static filtering parser; this
allows better syntax highlighting and creation-time
error-catching for cosmetic, HTML, and scriptlet filters.
Specifically:
There can't be more than one `redirect` or `csp`
option.
There can be no type specified when there is a
`csp` option.
There must be one single network-related type when
there is a `redirect` option. Since `empty` and
`mp4` imply a `redirect` and a network-related
type, these can't be used along another `redirect`
or network-related type.
Related commit:
- 01b1ed9a98
Reported internally. The issue could cause the logger
to be unable to successfully reverse-lookup a filter
list for a filter which had tokens longer than 6
characters followed by wildcard.
Regression from:
- 01b1ed9a98
Reported internally.
Regression from:
- 01b1ed9a98
The regression was made apparent be the fact that
AdGuard filter lists have lines ending with the
CR character.
Computing storage-used figures can take longer than
usual sometimes when a lot of filter lists are enabled,
and this can cause the Settings pane to take longer to
be filled the first time it is opened. Fetching
settings and storage-used figures separately removes
that potential delay (they were fetched together in
a single Promise.all() call).
Regression from:
- 01b1ed9a98
The new parser needs to be able to deal with trailing
newline characters, which if present will be interpreted
as trailing spaces.
A new standalone static filtering parser is introduced,
vAPI.StaticFilteringParser. It's purpose is to parse
line of text into representation suitable for
compiling filters. It can additionally serves for
syntax highlighting purpose.
As a side effect, this solves:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1038
This is a first draft, there are more work left to do
to further perfect the implementation and extend its
capabilities, especially those useful to assist filter
authors.
For the time being, this commits break line-continuation
syntax highlighting -- which was already flaky prior to
this commit anyway.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1008
This commit adds support entity-matching in the filter
option `domain=`. Example:
pattern$domain=google.*
The `*` above is meant to match any suffix from the Public
Suffix List. The semantic is exactly the same as the
already existing entity-matching support in static
extended filtering:
- https://github.com/gorhill/uBlock/wiki/Static-filter-syntax#entity
Additionally, in this commit:
Fix cases where "just-origin" filters of the form `|http*://`
were erroneously normalized to `|http://`. The proper
normalization of `|http*://` is `*`.
Add support to store hostname strings into the character
buffer of a hntrie container. As of commit time, there are
5,544 instances of FilterOriginHit, and 732 instances of
FilterOriginMiss, which filters require storing/matching a
single hostname string. Those strings are now stored in the
character buffer of the already existing origin-related
hntrie container. (The same approach is used for plain
patterns which are not part of a bidi-trie.)
Pressing spacebar will toggle god mode in popup
panel.
This capability shouldn't encourage people to
misuse _allow_ rules as having to press the
space bar act as an extra necessary step which
purpose is to make the creation of _allow_ rules
more mindful.
There have been too many examples out there of users
opting-in to "I am an advanced user" and yet still misusing
dynamic filtering by creating _allow_ rules where _noop_
rules should be used.
Creating _allow_ rules has serious consequences as these
override blocking static filters and can potentially
disable other advanced filtering ability such as
HTML filtering and scriptlet injection -- often used
to deal with anti-blocker mechanisms.
The ability to point-and-click to create _allow_ rules
from the popup panel is no longer allowed by default.
An new advanced setting has been added to enable
the ability to create _allow_ rules from the popup
panel, `popupPanelGodMode`, which default to `false`.
Set to `true` to restore ability to set _allow_ rules
from popup panel.
Since the creation of _allow_ rules is especially useful
to filter list authors, to diagnose and narrow down site
breakage as a result of problematic blocking filter,
the creation of _allow_ rules will still be available
when the advanced setting `filterAuthorMode` is `true`.
This change is probably going to be problematic to all
those users who were misusing dynamic filtering by
creating _allow_ rules instead of _noop_ rules -- but
the breakage is going to bring their misusing to their
attention, a positive outcome.
Default to `unset`.
To allow users to bypass uBO's default CSS styles in
case they are causing issues to specific users. It is
the responsibility of the user to ensure the value of
`uiStyles` contains valid CSS property declarations.
uBO will assign the value to `document.body.style.cssText`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1044
For example, in the case of the issue above, one could
set `uiStyles` to `font-family: sans-serif` to force uBO
to the system font for its user interface.
FilterPlainHostname, an atomic filter unit, has been
removed and is being replaced with a composite filter
made of a pattern filter and a filter which test
hostname boundaries.
Doing so enables filters formerly being represented
by FilterPlainHostname to be now represented as a
plain pattern, and thus to be potentially stored in
a bidi-trie.
Comparing the new filter histogram with the previous
one:
FilterPatternPlain 24612 26432 1820
FilterComposite 17656 17125 -531
FilterPlainTrie Content 12977 13519 542
FilterPlainHostname 2904 0 -2904
FilterBucket 2121 1961 -160
FilterPlainTrie 1418 1578 160
Which means:
- An extra 542 patterns could be stored in bidi-tries
- There are 531 less composite filters needed
- An extra 160 buckets could be aggregated into 160
bidi-trie
Memory-wise, it's a marginal gain (as per Chromium's
Javascript VM instance figure) -- i.e. not worth
talking about). CPU-wise, no measurable difference.
The benefit is that I consider this conceptually
simplifies slightly the static network filtering
code base.
Specifically, the sticky controls box is re-parented
pre-emptively instead of waiting for the decision as
to whether the panel must be toggled into a vertical
layout mode.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1004
The fix is trivial enough that I feel confident to make
it part of current RC cycle. There are many reports of
block page at launch on Chromium.
The fix is to let the platform pick it's natural suspend
state rather than force it. Once advanced settings are
read, the suspend state will for toggled only if
`suspendTabsUntilReady` is either `yes` or `no`. This
there should no longer be blocked page on Chromium at
launch when using default settings.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/gac2h8/
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1005
There is probably more work to do regarding the
shown URL -- it might be better to truncate it
on small screen display since it can be quite
long sometimes.
In rare instances -- though it definitely happens
eventually -- the popup panel viewport width is seen as
insufficiently wide enough and as a result the popup panel
is toggled into vertical-layout mode.
The added code uses animation frames to delay the
code testing the viewport width. Hopefully this will
work.
The old "classic" popup panel will still be used
when at least one of the following is true:
- advanced setting `uiFlavor` is set to `classic`; or
- the browser is Chromium 65 or older; or
- the browser is Firefox 67 or older
The default configuration of the new popup panel
at installation time is to show the power button,
statistics and the basic tool icons, i.e. access
to dashboard, logger, pickers.
For existing installations, the new popup panel
will be configured by respecting the existing
configuration of the classic one.
The new popup panel is currently already in use
on Firefox for Android, and the visual redesign
was made according to suggestions and feedback
from <https://github.com/brampitoyo> to be
optimal for Firefox for Android.
The new popup panel will allow closing the following
pending issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/255
- https://github.com/uBlockOrigin/uBlock-issues/issues/178
The blocking-stuff button icons have been brought
closer to the power button.
The go-somewhere button icons have been moved at
the bottom.
The hostname of the current site is now shown below
the power button. The base domain is bolded.
This re-ordering is for a few reasons:
- The blocking-stuff button icons are now closer
to the revert/persist/reload transient buttons
which show up when modifying the blocking
behavior of uBO.
- The makes it easier to potentially bring in a
very basic view where only the power button and
the statistics are shown and all the button
icons are hidden from view.
This commit specifically address bringing the
desktop version of the new popup panel's look
and feel more inline with the classic one:
- Hide tool captions on desktop
- Bring back no-popups switch on desktop
- Bring back tooltips on desktop (though they
are now rendered natively by the browser)
- Use the Photon icons suggested by @brampitoyo
for the no-popups and no-remote-fonts
switches
Noteworthy:
- Make new popup panel better resize in desktop
environment
- Fix button visuals in "My rules" pane
- Keep "Update now" button visible until update
is completed
- Renaming CSS variables for consistency
Better constrast for warning color. Related feedback:
- 5bee33253f (commitcomment-38700085)
Revisit how large numbers are rendered in a
compact form in the new popup panel. Feedback
from https://crowdin.com/project/ublock pointed
out that the string to translate was flawed for
Chinese locale (and possibly in some other
locales as well).
The rendering of the total number of blocked
requests will now be abbreviated using `M` and
`G` when the block count is respectively above
1 million and 1 billion.
The storage used figure in the Settings pane
will be rendered using KB, MB or GB.
As per email feedback from Mozilla's
https://github.com/brampitoyo
This is yet another incremental step toward
redesigning the UI, much more is left to do.
The idea is to align uBO's UI to that of
Firefox Preview.
Additionally, code has been added to reset
the new popup panel to vertical layout should
the viewport be not wide enough to
accomodate the horizontal layout.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/g4ufvi/
Related commit:
- 2ac288397c
Instead of having the `localStorage` data being accessed
from different locations, all accesses are now funnelled
to the main process.
Doing so simplifies the code in auxiliary processes and
also remove the need for browser.storage.local.onChanged()
listeners.
No longer using an onChanged() listener also happens to
remove spurious warnings from the Firefox console.
Related commits:
- 0662767dd6
- 99b2a0a761
- a8e3118fea
- 453f5450b6
- a45a21f337
This commit focuses on implementing the use of
CSS `var(...)` throughout so as to make it easy
to create themes -- a requirement for uBO to
support a dark theme.
There is still work to do regarding converting
uBO's CSS to completely support `var(...)` but
being able to start using theming will help
complete and fine tune CSS `var(...)` support.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/401
Related commits:
- a8e3118fea
- 453f5450b6
- a45a21f337
Roughly, the changes in this commit:
- Make it easier to implement a dark theme eventually:
- Flatten the look of buttons
- Remove the use of `opacity` style property as a
color modifier
- Some work toward removing dependency on FontAwesome
font
Related commits:
- 453f5450b6
- a45a21f337
This commit fixes parts of the re-design not
rendering as intended as per <https://github.com/brampitoyo>'s
feedback screenshots.
The dashboard tab buttons have been converted to
`span` tags (from `a` tags), thus eliminating
the unwanted side effect of the text being
rendered as unvisited links (blueish).
The font size intended for touch screens was not
taking effect due to bad cut & paste.
Other minor adjustments to improve consistency
in spacing.
This commit add the ability to fetch from CDN servers
when an asset is fetched as a result of auto-update.
If an asset has a `cdnURLs` entry in `assets.json`,
the asset will be auto-updated using one of those
CDN URLs. When many CDN URLs are specified, those
URLs will be shuffled in order to spread the bandwidth
across all specified CDN servers. If all specified CDN
servers fail to respond, uBO will fall back to usual
`contentURLs` entry.
The `cdnURLs` are used only when an asset is
auto-updated, this ensures a user will get the more
recent available version of an asset when manually
updating.
The motivation of this new feature is to relieve
GitHub from acting as a CDN (which it is not) for
uBO -- an increasing concern with the growing adoption
of uBO along with the growing size of key uBO assets.
Related commit:
- 453f5450b6
The changes in this commit are mostly related
to the "Filter lists" pane.
An eye icon has been added, which purpose is
to view the content of a list. Clicking on the
name of the list will toggle the list's
checkbox.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/974
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/fuscia/
The race condition was that a content script could
query the main process to retrieve cosmetic filters
while the cosmetic filters had not been yet fully
loaded into memory. The fix ensure that an already
injected content script will re-query once the
cosmetic filters are fully loaded in memory at
browser launch time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/911
Since cname-uncloaking is available only on Firefox
at the moment, the fix is relevant only to Firefox.
By default uBO will no longer cname-uncloak when it
detects that network requests are being being proxied.
This default behavior can be overriden by setting the
new advanced setting `cnameUncloakProxied` to `true`.
The new setting default to `false`, i.e. cname-uncloaking
is disabled when uBO detects that a proxy is in use.
This new advanced setting may disappear once the
following Firefox issue is fixed:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1618271
Provide a way to optionally deduplicate filter
instances, useful for filter instances with:
- high likelihood of duplication; and
- non-trivial memory footprint per instance
- For examples, filter instances to implement
`domain=`, `denyallow=`, `csp=`.
Cursory tests show this helps further reduce
uBO's memory footprint.
This concerns the static network filtering engine.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943
* * *
New static network filter type: `cname`
By default, network requests which are result of
resolving a canonical name are subject to filtering.
This filtering can be bypassed by creating exception
filters using the `cname` option. For example:
@@*$cname
The filter above tells the network filtering engine
to except network requests which fulfill all the
following conditions:
- network request is blocked
- network request is that of an unaliased hostname
Filter list authors are discouraged from using
exception filters of `cname` type, unless there no
other practical solution such that maintenance
burden become the greater issue. Of course, such
exception filters should be as narrow as possible,
i.e. apply to specific domain, etc.
* * *
New static network filter option: `denyallow`
The purpose of `denyallow` is bring
default-deny/allow-exceptionally ability into static
network filtering arsenal. Example of usage:
*$3p,script, \
denyallow=x.com|y.com \
domain=a.com|b.com
The above filter tells the network filtering engine that
when the context is `a.com` or `b.com`, block all
3rd-party scripts except those from `x.com` and `y.com`.
Essentially, the new `denyallow` option makes it easier
to implement default-deny/allow-exceptionally in static
filter lists, whereas before this had to be done with
unwieldy regular expressions[1], or through the mix of
broadly blocking filters along with exception filters[2].
[1] https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0
[2] Typically filters which pattern are of the
form `|http*://`
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/923
Use current page's hostname instead of that of
image URLs to decide whether to reset pattern
union with previous picker sessions.
The fixed issue arose from the fact that the
page uses URLs from different origins in a
single srcset attribute.
Related commit:
- https://github.com/gorhill/uBlock/commit/703c525b01aa
This adds an indentation requirement for line
continuation to take place. The conditions are now
as follow:
- Current line ends with ` \`: ASCII space + backslash
- Next line starts with ` `: four ASCII spaces
If a line in a filter list ends with a space
(ASCII code 32) followed by a backslash
(ASCII code 92), those two characters will be
removed, the line will be trimmed and the next
line will be trimmed and concatenated to form
a new, longer line.
The purpose is to give filter list authors
a way to visually break apart unduly long
filters and thus make maintenance easier.
When line continuation is used, it is suggested
that the extra lines are prepended with four
space so as to make it more visually obvious that
the extra line(s) are the continuation of a
previous line.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/943
The filter referenced in the above issue was the
motivation to implement this feature:
- https://hg.adblockplus.org/ruadlist/rev/f362910bc9a0
I verified and could not find any instance in major
filter lists of lines ending with ` \`, thus the
change should be safe.
Reported internally. The following invalid filter was not
discarded by uBO:
123tvnow.com##.123tv-ads
The correct form should be:
123tvnow.com##.\31 23tv-ads
Not discarding invalid CSS selector-based cosmetic
filter may break CSS selector-based cosmetic
filtering.
***
New procedural cosmetic operator: `:remove()`
Related issue:
- https://github.com/gorhill/uBlock/issues/2252
The purpose is to outright remove elements from the
DOM tree. Since `:remove()` is an "action" operator,
it must only be used as a trailing operator (just
like the `:style()` operator).
AdGuard's cosmetic filter syntax `{ remove: true; }`
will be converted to uBO's `:remove()` operator
internally.
***
New procedural cosmetic operator: `:upward(...)`
The purpose is to lookup an ancestor element.
When used with an integer argument, it is synonym of
`:nth-ancestor()`, which will be deprecated and which
will no longer be supported once no longer used in
mainstream filter lists.
Filter lists maintainers must only use `:upward(int)`
instead of `:nth-ancestor(int)` once the new operator
become available in all stable releases of uBO.
`:upward()` can also accept a CSS selector as argument,
in which case the nearest ancestor which matches the
CSS selector will be selected.
Override the Backspace key, not just the Delete key, as Mac keyboards have Backspace as the only delete key and label it Delete.
Source of key value: https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/key/Key_Values#Editing_keys
More background on Mac keyboard shortcuts: Mac keyboards can emulate Delete using fn+Delete, meaning Forward Delete, but Mac software does not use it except in text editing. When deletion is dangerous, Mac software requires holding a modifier key in conjuction with Delete, but I think it’s better to make deletion easy in this case.
This new binding has a potential downside: if the user Backspace key normally goes Back in history (which can differ across OSs and browsers), this will change the behavior to delete the selected element instead. If the user really wants to go back in history, they will have to press Escape to leave the mode and then press Backspace, or they will have to press an alternative keyboard shortcuts such as Alt+Left. I think the user will rarely want to go back in history in the middle of picking an element, though.
That downside could be mitigated by conditioning the key check on `runtime.PlatformOs` (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/PlatformOs). But that would complicate the code a lot compared to the importance of this feature, and such detection would still fail to capture the user’s intent accurately in all cases. I think it’s better to unconditionally accept both Backspace (Delete) and Delete (Forward Delete).
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/899
browser.storage.local is now used to store non-critical
local settings.
These settings are all collated under the key
`localStorage`, and vAPI.localStorage is an API to
handle access to these values stored under this key.
vAPI.localStorage.getItem() is still synchronous but
its purpose is to return internally cached values --
this minimizes code changes throughout uBO.
New advanced setting: `benchmarkDatasetURL`
Default value: `unset`
To specify a URL from where the benchmark dataset will be
fetched. This allows to launch benchmark operations from
within published versions of uBO, rather than from just
a locally built version.
A specific cosmetic filter of the following form...
*##.selector
... will be unconditionally injected into all web pages,
whereas a cosmetic filter of the form...
##.selector
... would be injected only when uBO's DOM surveyor finds
at least one matching element in a web page.
The new specific-generic form will also be disabled when a
web page is subject to a `generichide` exception filter,
since the filter is essentially a generic one -- the only
difference from the usual generic form is that the filter
is injected unconditionally instead of through the DOM
surveyor.
Specific-generic cosmetic filters will NOT be discarded
when checking the "Ignore generic cosmetic filters"
option in the "Filter lists" pane -- since the purpose
of this option is primarily to disable the DOM surveyor.
Specific-generic cosmetic filters should be used
parcimoniously and only when using a normal specific
filter is really impractical.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/803
Hostnames which have unpunycoded IDN or canonical
name information will have this information reported
as an extra line under the hostname name in the popup
panel, rather than as a tooltip.
For the fenix UI flavor, the save/revert tools are
now shown regardless of whether the overview area is
visible or not.
More specifically, this commit makes it so that
the new default UI for Firefox Preview will be
automatically selected.
A default platform-specific UI can be cancelled
by setting the advanced setting `uiFlavor` to `-`.
The toolip of a canonical name entry (blue)
in the popup panel will provide the alias
hostname related to the canonical hostname
in order to help with rule-making decisions.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/882
Related commits:
- https://github.com/gorhill/uBlock/commit/a95ef16e064a
- https://github.com/gorhill/uBlock/commit/7971b223855d
Leading wildcards before valid token characters need to
be kept in order to respect the semantic of the filter.
A leading wildcard in such case changes the semantic of
a filter, i.e. two following filters are semantically
different:
example/abc
*example/abc
As a result, µBlock.BidiTrieContainer.indexOf() is now
able to deal with a needle of length zero -- which is
what happens in FilterPatternLeft(Ex) with filter
patterns starting with `*` (or `^*`) and followed by
valid token characters (0-9, a-z and %).
First draft of changes as discussed with Firefox
Preview people.
In order to allow testing/evaluating these changes,
the new advanced setting `uiFlavor` has been added.
Default to `unset`; and can currently only be set
to `fenix`.
The new setting takes effect at launch only. This
new setting is not to be mentioned in official
documentation for now.
This is ongoing work, not open to external feedback.
This commit make it so that if an *unexpected* error is
thrown when trying to load the selfie at launch, the filter
lists will still be loaded as a result.
Tokens which are as long or longer than the max token
length possible do not need to have the prefix part
evaluated against special regex characters. This will
help increase the likelihood of extracting a valid
token from regex-based filters.
Actual case found in EasyPrivacy:
/^https?:\/\/eulerian..*\/[a-z0-9]{2,12}\.js/$script
Before this commit, uBO was not able to extract a
valid token, while now uBO is able to extract `eulerian`
as a valid token (consequently the regex-based filter
will now be evaluated only when the token `euleria` is
found in a URL).
This is related to the list of domains/subdomains in
the dynamic filtering pane of the popup panel.
Related issue:
- https://github.com/gorhill/uBlock/issues/284
Clicking on the empty space of a row will toggle
the visibility of the subdomains.
Additionally, the root context will always be visible
regardless of the expand/collspase state, along with
a visual indicator that a specific domain or subdomain
is the actual root context. (the root context is the
hostname in which local rules are created).
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/issues/805#issuecomment-561500819
uBO was testing for regex-like plain patterns after
prepending `@@` in the case of exception filters, thus
preventing proper detection of regex-like plain
patterns. The filtering engine was not affected, only
the proper rendering of the filter in the logger was
affected.
Advanced setting `cnameAliasList` has been removed.
New advanced settings:
cnameUncloak:
Boolean
Default value:
true
Description:
Whether to CNAME-uncloak hostnames.
cnameIgnoreExceptions:
Boolean
Default value:
true
Description:
Whether to bypass the uncloaking of network requests
which were excepted by filters/rules. This is
necessary so as to avoid undue breakage by having
exception filters being rendered useless as a result
of CNAME-uncloaking.
For example, `google-analytics.com` uncloaks to
`www-google-analytics.l.google.com` and both hostnames
appear in Peter Lowe's list, which means exception
filters for `google-analytics.com` (to fix site
breakage) would be rendered useless as the uncloaking
would cause the network request to be ultimately
blocked.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
Related commit:
- https://github.com/gorhill/uBlock/commit/3a564c199260
This adds two new advanced settings:
- cnameIgnoreRootDocument
- Default to `true`
- Tells uBO to skip CNAME-lookup for root document.
- cnameReplayFullURL
- Default to `false`
- Tells uBO whether to replay the whole URL or just
the origin part of it.
Replaying only the origin part is meant to lower
undue breakage and improve performance by avoiding
repeating the pattern-matching of the whole URL --
which pattern-matching was most likely already
accomplished with the original request.
This commit is meant to explore enabling CNAME-lookup
by default for the next stable release while:
- Eliminating a development burden by removing the
need to create a new filtering syntax to deal with
undesirable CNAME-cloaked hostnames
- Eliminating a filter list maintainer burden by
removing the need to find/deal with all base
domains which engage in undesirable CNAME-cloaked
hostnames
The hope is that the approach implemented in this
commit should require at most a few unbreak rules
with no further need for special filtering syntax
or filter list maintance efforts.
Related feedback:
- https://www.reddit.com/r/uBlockOrigin/comments/dzw57l/
Each token requires two slots in the token indices
array. This commit fixes uBO breaking when dealing
with very long URLs with lot of distinct tokens in
them.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
- Handle DNS lookup failure
- Skip DNS lookup for non network-based URLs
- Benchmark code to be able to provide an estimate
based on objective measurements regarding added
overhead when DNS lookup is enabled
(quick answer: a complete non-issue)
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/780
New webext permission added: `dns`, which purpose is
to allow an extension to fetch the DNS record of
specific hostnames, reference documentation:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/dns
The webext API `dns` is available in Firefox 60+ only.
The new API will enable uBO to "uncloak" the actual
hostname used in network requests. The ability is
currently disabled by default for now -- this is only
a first commit related to the above issue to allow
advanced users to immediately use the new ability.
Four advanced settings have been created to control the
uncloaking of actual hostnames:
cnameAliasList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to "uncloak" the hostnames in the list will.
cnameIgnoreList: a space-separated list of hostnames.
Default value: unset => empty list.
Special value: * => all hostnames.
A space-separated list of hostnames => this tells uBO
to NOT re-run the network request through uBO's
filtering engine with the CNAME hostname. This is
useful to exclude commonly used actual hostnames
from being re-run through uBO's filtering engine, so
as to avoid pointless overhead.
cnameIgnore1stParty: boolean.
Default value: true.
Whether uBO should ignore to re-run a network request
through the filtering engine when the CNAME hostname
is 1st-party to the alias hostname.
cnameMaxTTL: number of minutes.
Default value: 120.
This tells uBO to clear its CNAME cache after the
specified time. For efficiency purpose, uBO will
cache alias=>CNAME associations for reuse so as
to reduce calls to `browser.dns.resolve`. All the
associations will be cleared after the specified time
to ensure the map does not grow too large and too
ensure uBO uses up to date CNAME information.
Related issue:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1588916
Due to browser behavior changes during development,
uBO can end up using obsolete cached compiled filter
lists or selfie.
Saving magic format values in the cache storage
itself ensure uBO will properly detect actually
obsolete compiled lists/selfie.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
As per various feedbacks:
Added an advanced setting to keep the original behavior,
which can be potentially costly CPU-wise on some sites:
popupCosmeticFilterBadgeSlow
Default to `false`. Set to `true` to restore original
method of surveying the number of elements hidden as
a result of applying cosmetic filtering.
As suggested by <https://github.com/gwarser>, skip
descendant of nodes which have been found to be a
match in order to potentially increase the number
of nodes which can be surveyed in the alloted time.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
When trying the number of hidden elements as a result of
cosmetic filtering for popup panel badge purpose, the
code will bail out if this takes too long, and in such
case the badge will be set to `?`, meaning the number
of hidden elements is undetermined.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The code used to count expensive-to-survey particular
inline javascript occurrences will be executed if and
only if no other script tags have been found in the
DOM. This is acceptable since the badge count is to be
deemed approximate (as per documentation).
Related feedback:
- c090d2fde4 (commitcomment-35767596)
Mind that there might not be selectors to match as a
result of `generichide` or `no-cosmetic-filtering`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/756
The badge value for the no-cosmetic-filtering
switch will be evaluated on-demand only, when
the user hover over the switch with the mouse
cursor.
For touch screen displays, a tap on the switch
will cause the badge to be rendered if not
already done, otherwise this will toggle the
switch as usual.
