2017-11-02 20:49:11 +01:00
|
|
|
/*******************************************************************************
|
|
|
|
|
|
|
|
uBlock Origin - a browser extension to block requests.
|
2018-11-03 12:58:46 +01:00
|
|
|
Copyright (C) 2017-present Raymond Hill
|
2017-11-02 20:49:11 +01:00
|
|
|
|
|
|
|
This program is free software: you can redistribute it and/or modify
|
|
|
|
it under the terms of the GNU General Public License as published by
|
|
|
|
the Free Software Foundation, either version 3 of the License, or
|
|
|
|
(at your option) any later version.
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
GNU General Public License for more details.
|
|
|
|
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
|
|
along with this program. If not, see {http://www.gnu.org/licenses/}.
|
|
|
|
|
|
|
|
Home: https://github.com/gorhill/uBlock
|
|
|
|
*/
|
|
|
|
|
2018-11-03 12:58:46 +01:00
|
|
|
/* globals WebAssembly */
|
|
|
|
|
2017-11-02 20:49:11 +01:00
|
|
|
'use strict';
|
|
|
|
|
|
|
|
/*******************************************************************************
|
|
|
|
|
|
|
|
The original prototype was to develop an idea I had about using jump indices
|
|
|
|
in a TypedArray for quickly matching hostnames (or more generally strings)[1].
|
|
|
|
Once I had a working, un-optimized prototype, I realized I had ended up
|
|
|
|
with something formally named a "trie": <https://en.wikipedia.org/wiki/Trie>,
|
|
|
|
hence the name. I have no idea whether the implementation here or one
|
|
|
|
resembling it has been done elsewhere.
|
|
|
|
|
2018-12-04 19:02:09 +01:00
|
|
|
"HN" in HNTrieContainer stands for "HostName", because the trie is
|
|
|
|
specialized to deal with matching hostnames -- which is a bit more
|
|
|
|
complicated than matching plain strings.
|
2017-11-02 20:49:11 +01:00
|
|
|
|
|
|
|
For example, `www.abc.com` is deemed matching `abc.com`, because the former
|
|
|
|
is a subdomain of the latter. The opposite is of course not true.
|
|
|
|
|
2019-05-14 15:31:51 +02:00
|
|
|
The resulting read-only tries created as a result of using HNTrieContainer
|
|
|
|
are simply just typed arrays filled with integers. The matching algorithm is
|
2017-11-02 20:49:11 +01:00
|
|
|
just a matter of reading/comparing these integers, and further using them as
|
|
|
|
indices in the array as a way to move around in the trie.
|
|
|
|
|
|
|
|
[1] To solve <https://github.com/gorhill/uBlock/issues/3193>
|
|
|
|
|
2018-12-04 19:02:09 +01:00
|
|
|
Since this trie is specialized for matching hostnames, the stored
|
|
|
|
strings are reversed internally, because of hostname comparison logic:
|
|
|
|
|
|
|
|
Correct matching:
|
|
|
|
index 0123456
|
|
|
|
abc.com
|
|
|
|
|
|
|
|
|
www.abc.com
|
|
|
|
index 01234567890
|
|
|
|
|
|
|
|
Incorrect matching (typically used for plain strings):
|
|
|
|
index 0123456
|
|
|
|
abc.com
|
|
|
|
|
|
|
|
|
www.abc.com
|
|
|
|
index 01234567890
|
|
|
|
|
|
|
|
------------------------------------------------------------------------------
|
|
|
|
|
|
|
|
1st iteration:
|
|
|
|
- https://github.com/gorhill/uBlock/blob/ff58107dac3a32607f8113e39ed5015584506813/src/js/hntrie.js
|
|
|
|
- Suitable for small to medium set of hostnames
|
|
|
|
- One buffer per trie
|
|
|
|
|
|
|
|
2nd iteration: goal was to make matches() method wasm-able
|
|
|
|
- https://github.com/gorhill/uBlock/blob/c3b0fd31f64bd7ffecdd282fb1208fe07aac3eb0/src/js/hntrie.js
|
|
|
|
- Suitable for small to medium set of hostnames
|
|
|
|
- Distinct tries all share same buffer:
|
|
|
|
- Reduced memory footprint
|
|
|
|
- https://stackoverflow.com/questions/45803829/memory-overhead-of-typed-arrays-vs-strings/45808835#45808835
|
|
|
|
- Reusing needle character lookups for all tries
|
|
|
|
- This significantly reduce the number of String.charCodeAt() calls
|
|
|
|
- Slightly improved creation time
|
|
|
|
|
|
|
|
This is the 3rd iteration: goal was to make add() method wasm-able and
|
|
|
|
further improve memory/CPU efficiency.
|
|
|
|
|
|
|
|
This 3rd iteration has the following new traits:
|
|
|
|
- Suitable for small to large set of hostnames
|
|
|
|
- Support multiple trie containers (instanciable)
|
|
|
|
- Designed to hold large number of hostnames
|
|
|
|
- Hostnames can be added at any time (instead of all at once)
|
|
|
|
- This means pre-sorting is no longer a requirement
|
|
|
|
- The trie is always compact
|
|
|
|
- There is no longer a need for a `vacuum` method
|
|
|
|
- This makes the add() method wasm-able
|
|
|
|
- It can return the exact hostname which caused the match
|
|
|
|
- serializable/unserializable available for fast loading
|
|
|
|
- Distinct trie reference support the iteration protocol, thus allowing
|
|
|
|
to extract all the hostnames in the trie
|
|
|
|
|
|
|
|
Its primary purpose is to replace the use of Set() as a mean to hold
|
|
|
|
large number of hostnames (ex. FilterHostnameDict in static filtering
|
|
|
|
engine).
|
|
|
|
|
|
|
|
A HNTrieContainer is mostly a large buffer in which distinct but related
|
|
|
|
tries are stored. The memory layout of the buffer is as follow:
|
|
|
|
|
|
|
|
0-254: needle being processed
|
|
|
|
255: length of needle
|
|
|
|
256-259: offset to start of trie data section (=> trie0)
|
|
|
|
260-263: offset to end of trie data section (=> trie1)
|
|
|
|
264-267: offset to start of character data section (=> char0)
|
|
|
|
268-271: offset to end of character data section (=> char1)
|
|
|
|
272: start of trie data section
|
|
|
|
|
2017-11-02 20:49:11 +01:00
|
|
|
*/
|
|
|
|
|
2019-06-19 16:00:19 +02:00
|
|
|
const PAGE_SIZE = 65536;
|
|
|
|
// i32 / i8
|
|
|
|
const TRIE0_SLOT = 256 >>> 2; // 64 / 256
|
|
|
|
const TRIE1_SLOT = TRIE0_SLOT + 1; // 65 / 260
|
|
|
|
const CHAR0_SLOT = TRIE0_SLOT + 2; // 66 / 264
|
|
|
|
const CHAR1_SLOT = TRIE0_SLOT + 3; // 67 / 268
|
|
|
|
const TRIE0_START = TRIE0_SLOT + 4 << 2; // 272
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2020-02-23 18:18:45 +01:00
|
|
|
const roundToPageSize = v => (v + PAGE_SIZE-1) & ~(PAGE_SIZE-1);
|
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
const HNTrieContainer = class {
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2020-02-23 18:18:45 +01:00
|
|
|
constructor() {
|
|
|
|
const len = PAGE_SIZE * 2;
|
|
|
|
this.buf = new Uint8Array(len);
|
2019-04-26 01:38:07 +02:00
|
|
|
this.buf32 = new Uint32Array(this.buf.buffer);
|
|
|
|
this.needle = '';
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[TRIE0_SLOT] = TRIE0_START;
|
|
|
|
this.buf32[TRIE1_SLOT] = this.buf32[TRIE0_SLOT];
|
2020-02-23 18:18:45 +01:00
|
|
|
this.buf32[CHAR0_SLOT] = len >>> 1;
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR1_SLOT] = this.buf32[CHAR0_SLOT];
|
2019-04-26 01:38:07 +02:00
|
|
|
this.wasmMemory = null;
|
2020-12-04 13:53:01 +01:00
|
|
|
|
|
|
|
this.lastStored = '';
|
|
|
|
this.lastStoredLen = this.lastStoredIndex = 0;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
|
|
|
|
//--------------------------------------------------------------------------
|
|
|
|
// Public methods
|
|
|
|
//--------------------------------------------------------------------------
|
2018-11-03 12:58:46 +01:00
|
|
|
|
2020-02-23 18:18:45 +01:00
|
|
|
reset(details) {
|
|
|
|
if (
|
|
|
|
details instanceof Object &&
|
|
|
|
typeof details.byteLength === 'number' &&
|
|
|
|
typeof details.char0 === 'number'
|
|
|
|
) {
|
|
|
|
if ( details.byteLength > this.buf.byteLength ) {
|
|
|
|
this.reallocateBuf(details.byteLength);
|
|
|
|
}
|
|
|
|
this.buf32[CHAR0_SLOT] = details.char0;
|
|
|
|
}
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[TRIE1_SLOT] = this.buf32[TRIE0_SLOT];
|
|
|
|
this.buf32[CHAR1_SLOT] = this.buf32[CHAR0_SLOT];
|
2020-12-04 13:53:01 +01:00
|
|
|
|
|
|
|
this.lastStored = '';
|
|
|
|
this.lastStoredLen = this.lastStoredIndex = 0;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
setNeedle(needle) {
|
2018-11-03 12:58:46 +01:00
|
|
|
if ( needle !== this.needle ) {
|
2018-12-04 19:02:09 +01:00
|
|
|
const buf = this.buf;
|
2018-11-03 12:58:46 +01:00
|
|
|
let i = needle.length;
|
2019-06-19 16:00:19 +02:00
|
|
|
if ( i > 255 ) { i = 255; }
|
2018-11-03 12:58:46 +01:00
|
|
|
buf[255] = i;
|
|
|
|
while ( i-- ) {
|
|
|
|
buf[i] = needle.charCodeAt(i);
|
|
|
|
}
|
|
|
|
this.needle = needle;
|
2017-11-02 20:49:11 +01:00
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
return this;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2017-11-02 20:49:11 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
matchesJS(iroot) {
|
2019-06-19 16:00:19 +02:00
|
|
|
const buf32 = this.buf32;
|
|
|
|
const buf8 = this.buf;
|
|
|
|
const char0 = buf32[CHAR0_SLOT];
|
|
|
|
let ineedle = buf8[255];
|
|
|
|
let icell = buf32[iroot+0];
|
2019-04-29 19:15:16 +02:00
|
|
|
if ( icell === 0 ) { return -1; }
|
2018-11-03 12:58:46 +01:00
|
|
|
for (;;) {
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( ineedle === 0 ) { return -1; }
|
2018-11-03 12:58:46 +01:00
|
|
|
ineedle -= 1;
|
2019-06-19 16:00:19 +02:00
|
|
|
let c = buf8[ineedle];
|
2019-04-30 13:00:52 +02:00
|
|
|
let v, i0;
|
2018-12-04 19:02:09 +01:00
|
|
|
// find first segment with a first-character match
|
2018-11-03 12:58:46 +01:00
|
|
|
for (;;) {
|
2019-06-19 16:00:19 +02:00
|
|
|
v = buf32[icell+2];
|
2018-12-04 19:02:09 +01:00
|
|
|
i0 = char0 + (v & 0x00FFFFFF);
|
2019-06-19 16:00:19 +02:00
|
|
|
if ( buf8[i0] === c ) { break; }
|
|
|
|
icell = buf32[icell+0];
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( icell === 0 ) { return -1; }
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
// all characters in segment must match
|
|
|
|
let n = v >>> 24;
|
|
|
|
if ( n > 1 ) {
|
|
|
|
n -= 1;
|
|
|
|
if ( n > ineedle ) { return -1; }
|
|
|
|
i0 += 1;
|
|
|
|
const i1 = i0 + n;
|
2018-11-03 12:58:46 +01:00
|
|
|
do {
|
|
|
|
ineedle -= 1;
|
2019-06-19 16:00:19 +02:00
|
|
|
if ( buf8[i0] !== buf8[ineedle] ) { return -1; }
|
2018-12-04 19:02:09 +01:00
|
|
|
i0 += 1;
|
|
|
|
} while ( i0 < i1 );
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
// next segment
|
2019-06-19 16:00:19 +02:00
|
|
|
icell = buf32[icell+1];
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( icell === 0 ) { break; }
|
2019-06-19 16:00:19 +02:00
|
|
|
if ( buf32[icell+2] === 0 ) {
|
|
|
|
if ( ineedle === 0 || buf8[ineedle-1] === 0x2E ) {
|
2018-12-04 19:02:09 +01:00
|
|
|
return ineedle;
|
|
|
|
}
|
2019-06-19 16:00:19 +02:00
|
|
|
icell = buf32[icell+1];
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2017-11-02 20:49:11 +01:00
|
|
|
}
|
2019-06-19 16:00:19 +02:00
|
|
|
return ineedle === 0 || buf8[ineedle-1] === 0x2E ? ineedle : -1;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2017-11-02 20:49:11 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
createOne(args) {
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( Array.isArray(args) ) {
|
|
|
|
return new this.HNTrieRef(this, args[0], args[1]);
|
2017-11-02 20:49:11 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
// grow buffer if needed
|
2019-06-19 16:00:19 +02:00
|
|
|
if ( (this.buf32[CHAR0_SLOT] - this.buf32[TRIE1_SLOT]) < 12 ) {
|
2018-12-04 19:02:09 +01:00
|
|
|
this.growBuf(12, 0);
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2019-06-19 16:00:19 +02:00
|
|
|
const iroot = this.buf32[TRIE1_SLOT] >>> 2;
|
|
|
|
this.buf32[TRIE1_SLOT] += 12;
|
2018-12-04 19:02:09 +01:00
|
|
|
this.buf32[iroot+0] = 0;
|
|
|
|
this.buf32[iroot+1] = 0;
|
|
|
|
this.buf32[iroot+2] = 0;
|
|
|
|
return new this.HNTrieRef(this, iroot, 0);
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2017-11-02 20:49:11 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
compileOne(trieRef) {
|
2018-12-04 19:02:09 +01:00
|
|
|
return [ trieRef.iroot, trieRef.size ];
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2017-11-02 20:49:11 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
addJS(iroot) {
|
2018-12-04 19:02:09 +01:00
|
|
|
let lhnchar = this.buf[255];
|
|
|
|
if ( lhnchar === 0 ) { return 0; }
|
|
|
|
// grow buffer if needed
|
|
|
|
if (
|
2019-06-19 16:00:19 +02:00
|
|
|
(this.buf32[CHAR0_SLOT] - this.buf32[TRIE1_SLOT]) < 24 ||
|
|
|
|
(this.buf.length - this.buf32[CHAR1_SLOT]) < 256
|
2018-12-04 19:02:09 +01:00
|
|
|
) {
|
|
|
|
this.growBuf(24, 256);
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2019-04-29 19:15:16 +02:00
|
|
|
let icell = this.buf32[iroot+0];
|
|
|
|
// special case: first node in trie
|
|
|
|
if ( icell === 0 ) {
|
|
|
|
this.buf32[iroot+0] = this.addCell(0, 0, this.addSegment(lhnchar));
|
|
|
|
return 1;
|
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
//
|
2019-06-19 16:00:19 +02:00
|
|
|
const char0 = this.buf32[CHAR0_SLOT];
|
2019-04-30 13:00:52 +02:00
|
|
|
let inext;
|
2018-12-04 19:02:09 +01:00
|
|
|
// find a matching cell: move down
|
2018-11-03 12:58:46 +01:00
|
|
|
for (;;) {
|
2018-12-04 19:02:09 +01:00
|
|
|
const vseg = this.buf32[icell+2];
|
|
|
|
// skip boundary cells
|
|
|
|
if ( vseg === 0 ) {
|
2019-04-29 19:15:16 +02:00
|
|
|
// remainder is at label boundary? if yes, no need to add
|
|
|
|
// the rest since the shortest match is always reported
|
|
|
|
if ( this.buf[lhnchar-1] === 0x2E /* '.' */ ) { return -1; }
|
2018-12-04 19:02:09 +01:00
|
|
|
icell = this.buf32[icell+1];
|
|
|
|
continue;
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
let isegchar0 = char0 + (vseg & 0x00FFFFFF);
|
|
|
|
// if first character is no match, move to next descendant
|
2019-04-30 13:00:52 +02:00
|
|
|
if ( this.buf[isegchar0] !== this.buf[lhnchar-1] ) {
|
2018-12-04 19:02:09 +01:00
|
|
|
inext = this.buf32[icell+0];
|
|
|
|
if ( inext === 0 ) {
|
|
|
|
this.buf32[icell+0] = this.addCell(0, 0, this.addSegment(lhnchar));
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
icell = inext;
|
|
|
|
continue;
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2019-04-30 13:00:52 +02:00
|
|
|
// 1st character was tested
|
2018-12-04 19:02:09 +01:00
|
|
|
let isegchar = 1;
|
|
|
|
lhnchar -= 1;
|
|
|
|
// find 1st mismatch in rest of segment
|
|
|
|
const lsegchar = vseg >>> 24;
|
|
|
|
if ( lsegchar !== 1 ) {
|
|
|
|
for (;;) {
|
|
|
|
if ( isegchar === lsegchar ) { break; }
|
|
|
|
if ( lhnchar === 0 ) { break; }
|
2019-04-30 13:00:52 +02:00
|
|
|
if ( this.buf[isegchar0+isegchar] !== this.buf[lhnchar-1] ) { break; }
|
2018-12-04 19:02:09 +01:00
|
|
|
isegchar += 1;
|
|
|
|
lhnchar -= 1;
|
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
// all segment characters matched
|
|
|
|
if ( isegchar === lsegchar ) {
|
|
|
|
inext = this.buf32[icell+1];
|
|
|
|
// needle remainder: no
|
|
|
|
if ( lhnchar === 0 ) {
|
|
|
|
// boundary cell already present
|
|
|
|
if ( inext === 0 || this.buf32[inext+2] === 0 ) { return 0; }
|
|
|
|
// need boundary cell
|
|
|
|
this.buf32[icell+1] = this.addCell(0, inext, 0);
|
|
|
|
}
|
|
|
|
// needle remainder: yes
|
|
|
|
else {
|
|
|
|
if ( inext !== 0 ) {
|
|
|
|
icell = inext;
|
|
|
|
continue;
|
|
|
|
}
|
2019-04-29 19:15:16 +02:00
|
|
|
// remainder is at label boundary? if yes, no need to add
|
|
|
|
// the rest since the shortest match is always reported
|
|
|
|
if ( this.buf[lhnchar-1] === 0x2E /* '.' */ ) { return -1; }
|
2018-12-04 19:02:09 +01:00
|
|
|
// boundary cell + needle remainder
|
|
|
|
inext = this.addCell(0, 0, 0);
|
|
|
|
this.buf32[icell+1] = inext;
|
|
|
|
this.buf32[inext+1] = this.addCell(0, 0, this.addSegment(lhnchar));
|
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
// some segment characters matched
|
|
|
|
else {
|
|
|
|
// split current cell
|
|
|
|
isegchar0 -= char0;
|
|
|
|
this.buf32[icell+2] = isegchar << 24 | isegchar0;
|
|
|
|
inext = this.addCell(
|
|
|
|
0,
|
|
|
|
this.buf32[icell+1],
|
|
|
|
lsegchar - isegchar << 24 | isegchar0 + isegchar
|
|
|
|
);
|
2019-04-30 13:00:52 +02:00
|
|
|
this.buf32[icell+1] = inext;
|
2018-12-04 19:02:09 +01:00
|
|
|
// needle remainder: no = need boundary cell
|
|
|
|
if ( lhnchar === 0 ) {
|
|
|
|
this.buf32[icell+1] = this.addCell(0, inext, 0);
|
|
|
|
}
|
|
|
|
// needle remainder: yes = need new cell for remaining characters
|
|
|
|
else {
|
2019-04-30 13:00:52 +02:00
|
|
|
this.buf32[inext+0] = this.addCell(0, 0, this.addSegment(lhnchar));
|
2018-12-04 19:02:09 +01:00
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
return 1;
|
2017-11-02 20:49:11 +01:00
|
|
|
}
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
optimize() {
|
2018-12-04 19:02:09 +01:00
|
|
|
this.shrinkBuf();
|
|
|
|
return {
|
|
|
|
byteLength: this.buf.byteLength,
|
2019-06-19 16:00:19 +02:00
|
|
|
char0: this.buf32[CHAR0_SLOT],
|
2018-12-04 19:02:09 +01:00
|
|
|
};
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
fromIterable(hostnames, add) {
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( add === undefined ) { add = 'add'; }
|
|
|
|
const trieRef = this.createOne();
|
|
|
|
for ( const hn of hostnames ) {
|
|
|
|
trieRef[add](hn);
|
2017-11-02 20:49:11 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
return trieRef;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
serialize(encoder) {
|
2019-02-14 19:33:55 +01:00
|
|
|
if ( encoder instanceof Object ) {
|
|
|
|
return encoder.encode(
|
|
|
|
this.buf32.buffer,
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR1_SLOT]
|
2019-02-14 19:33:55 +01:00
|
|
|
);
|
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
return Array.from(
|
|
|
|
new Uint32Array(
|
|
|
|
this.buf32.buffer,
|
|
|
|
0,
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR1_SLOT] + 3 >>> 2
|
2018-12-04 19:02:09 +01:00
|
|
|
)
|
|
|
|
);
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
unserialize(selfie, decoder) {
|
2019-03-16 14:00:31 +01:00
|
|
|
this.needle = '';
|
2019-02-14 19:33:55 +01:00
|
|
|
const shouldDecode = typeof selfie === 'string';
|
|
|
|
let byteLength = shouldDecode
|
|
|
|
? decoder.decodeSize(selfie)
|
|
|
|
: selfie.length << 2;
|
2019-04-20 15:06:54 +02:00
|
|
|
if ( byteLength === 0 ) { return false; }
|
2020-02-23 18:18:45 +01:00
|
|
|
byteLength = roundToPageSize(byteLength);
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( this.wasmMemory !== null ) {
|
|
|
|
const pageCountBefore = this.buf.length >>> 16;
|
2019-02-14 19:33:55 +01:00
|
|
|
const pageCountAfter = byteLength >>> 16;
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( pageCountAfter > pageCountBefore ) {
|
|
|
|
this.wasmMemory.grow(pageCountAfter - pageCountBefore);
|
|
|
|
this.buf = new Uint8Array(this.wasmMemory.buffer);
|
|
|
|
this.buf32 = new Uint32Array(this.buf.buffer);
|
|
|
|
}
|
2019-02-14 19:33:55 +01:00
|
|
|
} else if ( byteLength > this.buf.length ) {
|
|
|
|
this.buf = new Uint8Array(byteLength);
|
|
|
|
this.buf32 = new Uint32Array(this.buf.buffer);
|
|
|
|
}
|
|
|
|
if ( shouldDecode ) {
|
|
|
|
decoder.decode(selfie, this.buf.buffer);
|
2018-11-03 12:58:46 +01:00
|
|
|
} else {
|
2019-02-14 19:33:55 +01:00
|
|
|
this.buf32.set(selfie);
|
2017-11-02 20:49:11 +01:00
|
|
|
}
|
2019-04-20 15:06:54 +02:00
|
|
|
return true;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2020-05-24 16:46:16 +02:00
|
|
|
// The following *Hostname() methods can be used to store hostname strings
|
|
|
|
// outside the trie. This is useful to store/match hostnames which are
|
|
|
|
// not part of a collection, and yet still benefit from storing the strings
|
|
|
|
// into a trie container's character buffer.
|
|
|
|
// TODO: WASM version of matchesHostname()
|
|
|
|
|
|
|
|
storeHostname(hn) {
|
|
|
|
let n = hn.length;
|
|
|
|
if ( n > 255 ) {
|
|
|
|
hn = hn.slice(-255);
|
|
|
|
n = 255;
|
|
|
|
}
|
2020-12-04 13:53:01 +01:00
|
|
|
if ( n === this.lastStoredLen && hn === this.lastStored ) {
|
|
|
|
return this.lastStoredIndex;
|
|
|
|
}
|
|
|
|
this.lastStored = hn;
|
|
|
|
this.lastStoredLen = n;
|
2020-05-24 16:46:16 +02:00
|
|
|
if ( (this.buf.length - this.buf32[CHAR1_SLOT]) < n ) {
|
|
|
|
this.growBuf(0, n);
|
|
|
|
}
|
|
|
|
const offset = this.buf32[CHAR1_SLOT];
|
|
|
|
this.buf32[CHAR1_SLOT] = offset + n;
|
|
|
|
const buf8 = this.buf;
|
|
|
|
for ( let i = 0; i < n; i++ ) {
|
|
|
|
buf8[offset+i] = hn.charCodeAt(i);
|
|
|
|
}
|
2020-12-04 13:53:01 +01:00
|
|
|
return (this.lastStoredIndex = offset - this.buf32[CHAR0_SLOT]);
|
2020-05-24 16:46:16 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
extractHostname(i, n) {
|
|
|
|
const textDecoder = new TextDecoder();
|
|
|
|
const offset = this.buf32[CHAR0_SLOT] + i;
|
|
|
|
return textDecoder.decode(this.buf.subarray(offset, offset + n));
|
|
|
|
}
|
|
|
|
|
|
|
|
matchesHostname(hn, i, n) {
|
|
|
|
this.setNeedle(hn);
|
|
|
|
const buf8 = this.buf;
|
|
|
|
const hr = buf8[255];
|
|
|
|
if ( n > hr ) { return false; }
|
|
|
|
const hl = hr - n;
|
|
|
|
const nl = this.buf32[CHAR0_SLOT] + i;
|
|
|
|
for ( let j = 0; j < n; j++ ) {
|
|
|
|
if ( buf8[nl+j] !== buf8[hl+j] ) { return false; }
|
|
|
|
}
|
|
|
|
return n === hr || hn.charCodeAt(hl-1) === 0x2E /* '.' */;
|
|
|
|
}
|
|
|
|
|
2021-07-25 16:55:35 +02:00
|
|
|
async enableWASM(modulePath) {
|
2020-02-22 19:36:22 +01:00
|
|
|
if ( typeof WebAssembly !== 'object' ) { return false; }
|
|
|
|
if ( this.wasmMemory instanceof WebAssembly.Memory ) { return true; }
|
2021-07-25 16:55:35 +02:00
|
|
|
const module = await getWasmModule(modulePath);
|
2020-05-15 18:03:05 +02:00
|
|
|
if ( module instanceof WebAssembly.Module === false ) { return false; }
|
2020-02-22 19:36:22 +01:00
|
|
|
const memory = new WebAssembly.Memory({ initial: 2 });
|
2020-05-15 18:03:05 +02:00
|
|
|
const instance = await WebAssembly.instantiate(module, {
|
|
|
|
imports: {
|
|
|
|
memory,
|
|
|
|
growBuf: this.growBuf.bind(this, 24, 256)
|
2020-02-22 19:36:22 +01:00
|
|
|
}
|
2020-05-15 18:03:05 +02:00
|
|
|
});
|
2020-02-22 19:36:22 +01:00
|
|
|
if ( instance instanceof WebAssembly.Instance === false ) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
this.wasmMemory = memory;
|
|
|
|
const curPageCount = memory.buffer.byteLength >>> 16;
|
2020-05-15 18:03:05 +02:00
|
|
|
const newPageCount = roundToPageSize(this.buf.byteLength) >>> 16;
|
2020-02-22 19:36:22 +01:00
|
|
|
if ( newPageCount > curPageCount ) {
|
|
|
|
memory.grow(newPageCount - curPageCount);
|
|
|
|
}
|
|
|
|
const buf = new Uint8Array(memory.buffer);
|
|
|
|
buf.set(this.buf);
|
|
|
|
this.buf = buf;
|
|
|
|
this.buf32 = new Uint32Array(this.buf.buffer);
|
|
|
|
this.matches = this.matchesWASM = instance.exports.matches;
|
|
|
|
this.add = this.addWASM = instance.exports.add;
|
|
|
|
}
|
|
|
|
|
2018-12-04 19:02:09 +01:00
|
|
|
//--------------------------------------------------------------------------
|
|
|
|
// Private methods
|
|
|
|
//--------------------------------------------------------------------------
|
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
addCell(idown, iright, v) {
|
2019-06-19 16:00:19 +02:00
|
|
|
let icell = this.buf32[TRIE1_SLOT];
|
|
|
|
this.buf32[TRIE1_SLOT] = icell + 12;
|
2018-12-04 19:02:09 +01:00
|
|
|
icell >>>= 2;
|
|
|
|
this.buf32[icell+0] = idown;
|
|
|
|
this.buf32[icell+1] = iright;
|
|
|
|
this.buf32[icell+2] = v;
|
|
|
|
return icell;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
addSegment(lsegchar) {
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( lsegchar === 0 ) { return 0; }
|
2019-06-19 16:00:19 +02:00
|
|
|
let char1 = this.buf32[CHAR1_SLOT];
|
|
|
|
const isegchar = char1 - this.buf32[CHAR0_SLOT];
|
2018-12-04 19:02:09 +01:00
|
|
|
let i = lsegchar;
|
|
|
|
do {
|
|
|
|
this.buf[char1++] = this.buf[--i];
|
|
|
|
} while ( i !== 0 );
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR1_SLOT] = char1;
|
2018-12-04 19:02:09 +01:00
|
|
|
return (lsegchar << 24) | isegchar;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-11-03 12:58:46 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
growBuf(trieGrow, charGrow) {
|
2018-12-04 19:02:09 +01:00
|
|
|
const char0 = Math.max(
|
2020-02-23 18:18:45 +01:00
|
|
|
roundToPageSize(this.buf32[TRIE1_SLOT] + trieGrow),
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR0_SLOT]
|
2018-12-04 19:02:09 +01:00
|
|
|
);
|
2019-06-19 16:00:19 +02:00
|
|
|
const char1 = char0 + this.buf32[CHAR1_SLOT] - this.buf32[CHAR0_SLOT];
|
2018-12-04 19:02:09 +01:00
|
|
|
const bufLen = Math.max(
|
2020-02-23 18:18:45 +01:00
|
|
|
roundToPageSize(char1 + charGrow),
|
2018-12-04 19:02:09 +01:00
|
|
|
this.buf.length
|
|
|
|
);
|
|
|
|
this.resizeBuf(bufLen, char0);
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
shrinkBuf() {
|
2018-12-04 19:02:09 +01:00
|
|
|
// Can't shrink WebAssembly.Memory
|
|
|
|
if ( this.wasmMemory !== null ) { return; }
|
2019-06-19 16:00:19 +02:00
|
|
|
const char0 = this.buf32[TRIE1_SLOT] + 24;
|
|
|
|
const char1 = char0 + this.buf32[CHAR1_SLOT] - this.buf32[CHAR0_SLOT];
|
2018-12-04 19:02:09 +01:00
|
|
|
const bufLen = char1 + 256;
|
|
|
|
this.resizeBuf(bufLen, char0);
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
resizeBuf(bufLen, char0) {
|
2020-02-23 18:18:45 +01:00
|
|
|
bufLen = roundToPageSize(bufLen);
|
2018-12-04 19:02:09 +01:00
|
|
|
if (
|
|
|
|
bufLen === this.buf.length &&
|
2019-06-19 16:00:19 +02:00
|
|
|
char0 === this.buf32[CHAR0_SLOT]
|
2018-12-04 19:02:09 +01:00
|
|
|
) {
|
|
|
|
return;
|
|
|
|
}
|
2019-06-19 16:00:19 +02:00
|
|
|
const charDataLen = this.buf32[CHAR1_SLOT] - this.buf32[CHAR0_SLOT];
|
2018-12-04 19:02:09 +01:00
|
|
|
if ( this.wasmMemory !== null ) {
|
|
|
|
const pageCount = (bufLen >>> 16) - (this.buf.byteLength >>> 16);
|
|
|
|
if ( pageCount > 0 ) {
|
|
|
|
this.wasmMemory.grow(pageCount);
|
|
|
|
this.buf = new Uint8Array(this.wasmMemory.buffer);
|
|
|
|
this.buf32 = new Uint32Array(this.wasmMemory.buffer);
|
|
|
|
}
|
|
|
|
} else if ( bufLen !== this.buf.length ) {
|
|
|
|
const newBuf = new Uint8Array(bufLen);
|
|
|
|
newBuf.set(
|
|
|
|
new Uint8Array(
|
|
|
|
this.buf.buffer,
|
|
|
|
0,
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[TRIE1_SLOT]
|
2018-12-04 19:02:09 +01:00
|
|
|
),
|
|
|
|
0
|
|
|
|
);
|
|
|
|
newBuf.set(
|
|
|
|
new Uint8Array(
|
|
|
|
this.buf.buffer,
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR0_SLOT],
|
2018-12-04 19:02:09 +01:00
|
|
|
charDataLen
|
|
|
|
),
|
|
|
|
char0
|
|
|
|
);
|
|
|
|
this.buf = newBuf;
|
|
|
|
this.buf32 = new Uint32Array(this.buf.buffer);
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR0_SLOT] = char0;
|
|
|
|
this.buf32[CHAR1_SLOT] = char0 + charDataLen;
|
2018-12-04 19:02:09 +01:00
|
|
|
}
|
2019-06-19 16:00:19 +02:00
|
|
|
if ( char0 !== this.buf32[CHAR0_SLOT] ) {
|
2018-12-04 19:02:09 +01:00
|
|
|
this.buf.set(
|
|
|
|
new Uint8Array(
|
|
|
|
this.buf.buffer,
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR0_SLOT],
|
2018-12-04 19:02:09 +01:00
|
|
|
charDataLen
|
|
|
|
),
|
|
|
|
char0
|
|
|
|
);
|
2019-06-19 16:00:19 +02:00
|
|
|
this.buf32[CHAR0_SLOT] = char0;
|
|
|
|
this.buf32[CHAR1_SLOT] = char0 + charDataLen;
|
2018-12-04 19:02:09 +01:00
|
|
|
}
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2020-02-23 18:18:45 +01:00
|
|
|
|
|
|
|
reallocateBuf(newSize) {
|
|
|
|
newSize = roundToPageSize(newSize);
|
|
|
|
if ( newSize === this.buf.length ) { return; }
|
|
|
|
if ( this.wasmMemory === null ) {
|
|
|
|
const newBuf = new Uint8Array(newSize);
|
|
|
|
newBuf.set(
|
|
|
|
newBuf.length < this.buf.length
|
|
|
|
? this.buf.subarray(0, newBuf.length)
|
|
|
|
: this.buf
|
|
|
|
);
|
|
|
|
this.buf = newBuf;
|
|
|
|
} else {
|
|
|
|
const growBy =
|
|
|
|
((newSize + 0xFFFF) >>> 16) - (this.buf.length >>> 16);
|
|
|
|
if ( growBy <= 0 ) { return; }
|
|
|
|
this.wasmMemory.grow(growBy);
|
|
|
|
this.buf = new Uint8Array(this.wasmMemory.buffer);
|
|
|
|
}
|
|
|
|
this.buf32 = new Uint32Array(this.buf.buffer);
|
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
};
|
2018-11-03 12:58:46 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
HNTrieContainer.prototype.matches = HNTrieContainer.prototype.matchesJS;
|
|
|
|
HNTrieContainer.prototype.matchesWASM = null;
|
|
|
|
|
|
|
|
HNTrieContainer.prototype.add = HNTrieContainer.prototype.addJS;
|
|
|
|
HNTrieContainer.prototype.addWASM = null;
|
|
|
|
|
|
|
|
/*******************************************************************************
|
|
|
|
|
|
|
|
Class to hold reference to a specific trie
|
|
|
|
|
|
|
|
*/
|
2018-12-04 19:02:09 +01:00
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
HNTrieContainer.prototype.HNTrieRef = class {
|
|
|
|
|
|
|
|
constructor(container, iroot, size) {
|
|
|
|
this.container = container;
|
|
|
|
this.iroot = iroot;
|
|
|
|
this.size = size;
|
Expand bidi-trie usage in static network filtering engine
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
- https://github.com/uBlockOrigin/uBlock-issues/issues/528
The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.
Example of filters that could be moved into a bidi-trie
data structure:
&ad_box_
/w/d/capu.php?z=$script,third-party
||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif
Examples of filters that could NOT be moved to a bidi-trie:
-adap.$domain=~l-adap.org
/tsc.php?*&ses=
||ibsrv.net/*forumsponsor$domain=[...]
@@||imgspice.com/jquery.cookie.js|$script
||view.atdmt.com^*/iview/$third-party
||postimg.cc/image/$csp=[...]
Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).
Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.
This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.
- Added ability to associate a pattern with an integer
in the bidi-trie [1].
- The bidi-trie match code passes this externally
provided integer when calling an externally
provided method used for testing extra conditions
that may be present for a plain pattern found to
be matching in the bidi-trie.
- Decomposed existing filters into smaller logical units:
- FilterPlainLeftAnchored =>
FilterPatternPlain +
FilterAnchorLeft
- FilterPlainRightAnchored =>
FilterPatternPlain +
FilterAnchorRight
- FilterExactMatch =>
FilterPatternPlain +
FilterAnchorLeft +
FilterAnchorRight
- FilterPlainHnAnchored =>
FilterPatternPlain +
FilterAnchorHn
- FilterWildcard1 =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
]
- FilterWildcard1HnAnchored =>
FilterPatternPlain + [
FilterPatternLeft or
FilterPatternRight
] +
FilterAnchorHn
- FilterGenericHnAnchored =>
FilterPatternGeneric +
FilterAnchorHn
- FilterGenericHnAndRightAnchored =>
FilterPatternGeneric +
FilterAnchorRight +
FilterAnchorHn
- FilterOriginMixedSet =>
FilterOriginMissSet +
FilterOriginHitSet
- Instances of FilterOrigin[...], FilterDataHolder
can also be added to a composite filter to
represent `domain=` and `csp=` options.
- Added a new filter class, FilterComposite, for
filters which are a combination of two or more
logical units. A FilterComposite instance is a
match when *all* filters composing it are a
match.
Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.
Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.
Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.
The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.
Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.
A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.
This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.
As a result of the above changes, the map of `token`
(integer) => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.
Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.
`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.
New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.
All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.
Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.
FilterParser is now instanciated on demand and
released when no longer used.
***
[1] https://github.com/gorhill/uBlock/blob/135a45a878f5b93bc538f822981e3a42b1e9073f/src/js/strie.js#L120
[2] https://github.com/gorhill/uBlock/commit/e94024d350b066e4e04a772b0a3dbc69daab3fb7
[3] https://github.com/gorhill/uBlock/blob/135a45a878f5b93bc538f822981e3a42b1e9073f/src/js/static-net-filtering.js#L1630
[4] https://github.com/gorhill/uBlock/blob/135a45a878f5b93bc538f822981e3a42b1e9073f/src/js/static-net-filtering.js#L1566
2019-10-21 14:15:58 +02:00
|
|
|
this.needle = '';
|
|
|
|
this.last = -1;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
add(hn) {
|
2019-04-29 19:15:16 +02:00
|
|
|
if ( this.container.setNeedle(hn).add(this.iroot) > 0 ) {
|
2019-04-26 00:36:03 +02:00
|
|
|
this.last = -1;
|
|
|
|
this.needle = '';
|
2018-12-04 19:02:09 +01:00
|
|
|
this.size += 1;
|
|
|
|
return true;
|
2018-11-03 12:58:46 +01:00
|
|
|
}
|
2018-12-04 19:02:09 +01:00
|
|
|
return false;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
addJS(hn) {
|
2019-04-29 19:15:16 +02:00
|
|
|
if ( this.container.setNeedle(hn).addJS(this.iroot) > 0 ) {
|
2019-04-26 00:36:03 +02:00
|
|
|
this.last = -1;
|
|
|
|
this.needle = '';
|
2018-12-04 19:02:09 +01:00
|
|
|
this.size += 1;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return false;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
addWASM(hn) {
|
2019-04-29 19:15:16 +02:00
|
|
|
if ( this.container.setNeedle(hn).addWASM(this.iroot) > 0 ) {
|
2019-04-26 00:36:03 +02:00
|
|
|
this.last = -1;
|
|
|
|
this.needle = '';
|
2018-12-04 19:02:09 +01:00
|
|
|
this.size += 1;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return false;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
matches(needle) {
|
2019-04-26 00:36:03 +02:00
|
|
|
if ( needle !== this.needle ) {
|
|
|
|
this.needle = needle;
|
|
|
|
this.last = this.container.setNeedle(needle).matches(this.iroot);
|
|
|
|
}
|
|
|
|
return this.last;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
matchesJS(needle) {
|
2019-04-26 00:36:03 +02:00
|
|
|
if ( needle !== this.needle ) {
|
|
|
|
this.needle = needle;
|
|
|
|
this.last = this.container.setNeedle(needle).matchesJS(this.iroot);
|
|
|
|
}
|
|
|
|
return this.last;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
matchesWASM(needle) {
|
2019-04-26 00:36:03 +02:00
|
|
|
if ( needle !== this.needle ) {
|
|
|
|
this.needle = needle;
|
|
|
|
this.last = this.container.setNeedle(needle).matchesWASM(this.iroot);
|
|
|
|
}
|
|
|
|
return this.last;
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
|
|
|
|
2019-05-06 17:12:39 +02:00
|
|
|
dump() {
|
|
|
|
let hostnames = Array.from(this);
|
|
|
|
if ( String.prototype.padStart instanceof Function ) {
|
|
|
|
const maxlen = Math.min(
|
|
|
|
hostnames.reduce((maxlen, hn) => Math.max(maxlen, hn.length), 0),
|
|
|
|
64
|
|
|
|
);
|
|
|
|
hostnames = hostnames.map(hn => hn.padStart(maxlen));
|
|
|
|
}
|
|
|
|
for ( const hn of hostnames ) {
|
|
|
|
console.log(hn);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
[Symbol.iterator]() {
|
2018-12-04 19:02:09 +01:00
|
|
|
return {
|
|
|
|
value: undefined,
|
|
|
|
done: false,
|
|
|
|
next: function() {
|
|
|
|
if ( this.icell === 0 ) {
|
|
|
|
if ( this.forks.length === 0 ) {
|
|
|
|
this.value = undefined;
|
|
|
|
this.done = true;
|
|
|
|
return this;
|
|
|
|
}
|
|
|
|
this.charPtr = this.forks.pop();
|
|
|
|
this.icell = this.forks.pop();
|
|
|
|
}
|
|
|
|
for (;;) {
|
|
|
|
const idown = this.container.buf32[this.icell+0];
|
|
|
|
if ( idown !== 0 ) {
|
|
|
|
this.forks.push(idown, this.charPtr);
|
|
|
|
}
|
|
|
|
const v = this.container.buf32[this.icell+2];
|
2019-06-19 16:00:19 +02:00
|
|
|
let i0 = this.container.buf32[CHAR0_SLOT] + (v & 0x00FFFFFF);
|
2018-12-04 19:02:09 +01:00
|
|
|
const i1 = i0 + (v >>> 24);
|
|
|
|
while ( i0 < i1 ) {
|
|
|
|
this.charPtr -= 1;
|
|
|
|
this.charBuf[this.charPtr] = this.container.buf[i0];
|
|
|
|
i0 += 1;
|
|
|
|
}
|
|
|
|
this.icell = this.container.buf32[this.icell+1];
|
|
|
|
if ( this.icell === 0 ) {
|
|
|
|
return this.toHostname();
|
|
|
|
}
|
|
|
|
if ( this.container.buf32[this.icell+2] === 0 ) {
|
|
|
|
this.icell = this.container.buf32[this.icell+1];
|
|
|
|
return this.toHostname();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
},
|
|
|
|
toHostname: function() {
|
|
|
|
this.value = this.textDecoder.decode(
|
|
|
|
new Uint8Array(this.charBuf.buffer, this.charPtr)
|
|
|
|
);
|
|
|
|
return this;
|
|
|
|
},
|
|
|
|
container: this.container,
|
|
|
|
icell: this.iroot,
|
|
|
|
charBuf: new Uint8Array(256),
|
|
|
|
charPtr: 256,
|
|
|
|
forks: [],
|
|
|
|
textDecoder: new TextDecoder()
|
|
|
|
};
|
2019-04-26 01:38:07 +02:00
|
|
|
}
|
2017-11-02 20:49:11 +01:00
|
|
|
};
|
|
|
|
|
2019-04-26 01:38:07 +02:00
|
|
|
HNTrieContainer.prototype.HNTrieRef.prototype.last = -1;
|
|
|
|
HNTrieContainer.prototype.HNTrieRef.prototype.needle = '';
|
|
|
|
|
2018-11-03 12:58:46 +01:00
|
|
|
/******************************************************************************/
|
2017-11-02 20:49:11 +01:00
|
|
|
|
2018-12-04 19:02:09 +01:00
|
|
|
// Code below is to attempt to load a WASM module which implements:
|
|
|
|
//
|
|
|
|
// - HNTrieContainer.add()
|
|
|
|
// - HNTrieContainer.matches()
|
|
|
|
//
|
|
|
|
// The WASM module is entirely optional, the JS implementations will be
|
|
|
|
// used should the WASM module be unavailable for whatever reason.
|
|
|
|
|
2020-02-22 19:36:22 +01:00
|
|
|
const getWasmModule = (( ) => {
|
|
|
|
let wasmModulePromise;
|
2017-11-02 20:49:11 +01:00
|
|
|
|
2021-07-25 16:55:35 +02:00
|
|
|
return async function(modulePath) {
|
2020-02-22 19:36:22 +01:00
|
|
|
if ( wasmModulePromise instanceof Promise ) {
|
|
|
|
return wasmModulePromise;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (
|
|
|
|
typeof WebAssembly !== 'object' ||
|
|
|
|
typeof WebAssembly.compileStreaming !== 'function'
|
|
|
|
) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Soft-dependency on vAPI so that the code here can be used outside of
|
|
|
|
// uBO (i.e. tests, benchmarks)
|
|
|
|
if ( typeof vAPI === 'object' && vAPI.canWASM !== true ) { return; }
|
|
|
|
|
|
|
|
// The wasm module will work only if CPU is natively little-endian,
|
|
|
|
// as we use native uint32 array in our js code.
|
|
|
|
const uint32s = new Uint32Array(1);
|
|
|
|
const uint8s = new Uint8Array(uint32s.buffer);
|
|
|
|
uint32s[0] = 1;
|
|
|
|
if ( uint8s[0] !== 1 ) { return; }
|
|
|
|
|
|
|
|
wasmModulePromise = fetch(
|
2021-07-25 16:55:35 +02:00
|
|
|
`${modulePath}/wasm/hntrie.wasm`,
|
2020-02-22 19:36:22 +01:00
|
|
|
{ mode: 'same-origin' }
|
|
|
|
).then(
|
|
|
|
WebAssembly.compileStreaming
|
|
|
|
).catch(reason => {
|
2021-07-29 01:48:38 +02:00
|
|
|
console.info(reason);
|
2020-02-22 19:36:22 +01:00
|
|
|
});
|
|
|
|
|
|
|
|
return wasmModulePromise;
|
|
|
|
};
|
2018-11-03 12:58:46 +01:00
|
|
|
})();
|
2019-06-19 16:00:19 +02:00
|
|
|
|
|
|
|
/******************************************************************************/
|
|
|
|
|
2021-07-25 16:55:35 +02:00
|
|
|
export { HNTrieContainer };
|