Currently a CHECK-NOT directive succeeds whenever the corresponding
match fails. However match can fail due to an error rather than a lack
of match, for instance if a variable is undefined. This commit makes match
error a failure for CHECK-NOT.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D86222
In input dump annotations, `check:2'1` indicates diagnostic 1 for the
`CHECK` directive on check file line 2. Without this patch,
`-dump-input` computes the diagnostic index with the assumption that
FileCheck *consecutively* produces all diagnostics for the same
pattern. Already, that can be a false assumption, as in the examples
below. Moreover, it seems like a brittle assumption as FileCheck
evolves. Finally, it actually complicates the implementation even if
it makes it slightly more efficient.
This patch avoids that assumption. Examples below show results after
applying this patch. Before applying this patch, `'N` is omitted
throughout these examples because the implementation doesn't notice
there's more than one diagnostic per pattern.
First, `CHECK-LABEL` violates the assumption because `CHECK-LABEL`
tries to match twice, and other directives can match in between:
```
$ cat check
CHECK: foobar
CHECK-LABEL: foobar
$ FileCheck -vv check < input |& tail -8
<<<<<<
1: text
2: foobar
label:2'0 ^~~~~~
check:1 ^~~~~~
label:2'1 X error: no match found
3: text
>>>>>>
```
Second, `--implicit-check-not` is obviously processed many times among
other directives:
```
$ cat check
CHECK: foo
CHECK: foo
$ FileCheck -vv -dump-input=always -implicit-check-not=foo \
check < input |& tail -16
<<<<<<
1: text
not:imp1'0 X~~~~
2: foo
check:1 ^~~
not:imp1'1 X
3: text
not:imp1'1 ~~~~~
4: foo
check:2 ^~~
not:imp1'2 X
5: text
not:imp1'2 ~~~~~
6:
eof:2 ^
>>>>>>
```
Reviewed By: thopre, jhenderson
Differential Revision: https://reviews.llvm.org/D97813
FileCheck string substitution block parsing code only report an invalid
variable name in a string variable use if it starts with a forbidden
character. It does not report anything if there are unparsed characters
after the variable name, i.e. [[X-Y]] is parsed as [[X]] and no error is
returned. This commit fixes that.
Reviewed By: jdenny, jhenderson
Differential Revision: https://reviews.llvm.org/D98691
Fixed substitution printing not to produce an empty diagnostic for
errors handled elsewhere.
Reviewed By: thopre
Differential Revision: https://reviews.llvm.org/D98088
A more general name might be match-time error propagation. That is,
it's conceivable we'll one day have non-numeric errors that require
the handling fixed by this patch.
Without this patch, FileCheck behaves as follows:
```
$ cat check
CHECK-NOT: [[#0x8000000000000000+0x8000000000000000]]
$ FileCheck -vv -dump-input=never check < input
check:1:54: remark: implicit EOF: expected string found in input
CHECK-NOT: [[#0x8000000000000000+0x8000000000000000]]
^
<stdin>:2:1: note: found here
^
check:1:15: error: unable to substitute variable or numeric expression: overflow error
CHECK-NOT: [[#0x8000000000000000+0x8000000000000000]]
^
$ echo $?
0
```
Notice that the exit status is 0 even though there's an error.
Moreover, FileCheck doesn't print the error diagnostic unless both
`-dump-input=never` and `-vv` are specified.
The same problem occurs when `CHECK-NOT` does have a match but a
capture fails due to overflow: exit status is 0, and no diagnostic is
printed unless both `-dump-input=never` and `-vv` are specified. The
usefulness of capturing from `CHECK-NOT` is questionable, but this
case should certainly produce an error.
With this patch, FileCheck always includes the error diagnostic and
has non-zero exit status for the above examples. It's conceivable
that this change will cause some existing tests to fail, but my
assumption is that they should fail. Moreover, with nearly every
project enabled, this patch didn't produce additional `check-all`
failures for me.
This patch also extends input dumps to include such numeric error
diagnostics for both expected and excluded patterns.
As noted in fixmes in some of the tests added by this patch, this
patch worsens an existing issue with redundant diagnostics. I'll fix
that bug in a subsequent patch.
Reviewed By: thopre, jhenderson
Differential Revision: https://reviews.llvm.org/D98086
Add printf-style alternate form flag to prefix hex number with 0x when
present. This works on both empty numeric expression (e.g. variable
definition from input) and when matching a numeric expression. The
syntax is as follows:
[[#%#<precision specifier><format specifier>, ...]
where <precision specifier> and <format specifier> are optional and ...
can be a variable definition or not with an empty expression or not.
This feature was requested in https://reviews.llvm.org/D81144#2075532
for llvm/test/MC/ELF/gen-dwarf64.s
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D97845
When commit da108b4ed4e6e7267701e76d5fd3b87609c9ab77 introduced
the CHECK-NEXT directive, it added logic to skip to the next line when
printing a diagnostic if the current matching position is at the end of
a line. This was fine while FileCheck did not support regular expression
but since it does now it can be confusing when the pattern to match
starts with the expectation of a newline (e.g. CHECK-NEXT: {{\n}}foo).
It is also inconsistent with the column information in the diagnostic
which does point to the end of line.
This commit removes this logic altogether, such that failure to match
diagnostic for such cases would show the end of line and be consistent
with the column information. The commit also adapts all existing
testcases accordingly.
Note to reviewers: An alternative approach would be to restrict the code
to only skip to the next line if the first character of the pattern is
known not to match a whitespace-like character. This would respect the
original intent but keep the inconsistency in terms of column info and
requires more code. I've only chosen this current approach by laziness
and would be happy to restrict the logic instead.
Reviewed By: jdenny, jhenderson
Differential Revision: https://reviews.llvm.org/D93341
Link: https://lists.llvm.org/pipermail/llvm-dev/2020-October/146162.html "[RFC] FileCheck: (dis)allowing unused prefixes"
If a downstream project using lit needs time for transition,
add the following to `lit.local.cfg`:
```
from lit.llvm.subst import ToolSubst
fc = ToolSubst('FileCheck', unresolved='fatal')
config.substitutions.insert(0, (fc.regex, 'FileCheck --allow-unused-prefixes'))
```
Differential Revision: https://reviews.llvm.org/D95849
cl::ZeroOrMore allows the option to be specified multiple times, which makes
downstream projects possible to specify a default value in lit configuration
while some tests can override the value.
This patch sets the default for llvm tests, with the exception of tests
under Reduce, because quite a few of them use 'FileCheck' as parameter
to a tool, and including a flag as that parameter would complicate
matters.
The rest of the patch undo-es the lit.local.cfg changes we progressively
introduced as temporary measure to avoid regressions under various
directories.
Differential Revision: https://reviews.llvm.org/D95111
The test related to directives with prefix NUMEXPR-CONSTRAINT-NOMATCH
refers to the numeric variable UNSI which is defined by a directive with
prefix CHECK not enabled on the lit command-line. Rather than adding the
prefix CHECK to the lit command-line, this commit defined variable UNSI
in a new NUMEXPR-CONSTRAINT-NOMATCH prefixed directive. The directive
needs to contain more than just the variable otherwise the error message
from FileCheck is about the wrong line being matched.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D93346
Introduce CHECK modifiers that change the behavior of the CHECK
directive. Also add a LITERAL modifier for cases where matching could
end requiring escaping strings interpreted as regex where only
literal/fixed string matching is desired (making the CHECK's more
difficult to write/fragile and difficult to interpret).
If more than a prefix is provided - e.g. --check-prefixes=CHECK,FOO - we
don't report if (say) FOO is never used. This may lead to a gap in our
test coverage.
This patch introduces a new option, --allow-unused-prefixes. It
currently is set to true, keeping today's behavior. After we explicitly
set it in tests where this behavior was actually intentional, we will
switch it to false by default.
Differential Revision: https://reviews.llvm.org/D90281
Add printf-style precision specifier to pad numbers to a given number of
digits when matching them if the value is smaller than the given
precision. This works on both empty numeric expression (e.g. variable
definition from input) and when matching a numeric expression. The
syntax is as follows:
[[#%.<precision><format specifier>, ...]
where <format specifier> is optional and ... can be a variable
definition or not with an empty expression or not. In the absence of a
precision specifier, a variable definition will accept leading zeros.
Reviewed By: jhenderson, grimar
Differential Revision: https://reviews.llvm.org/D81667
This commit makes FileCheck print all CHECK-NOT directive failure in a
CHECK-NOT block even if one fails. Prior to that, it would stop trying
to match CHECK-NOT directive as soon as one in the block fails.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D86315
Substitutions are already reported in the diagnostics appearing before
the input dump in the case of failed directives, and they're reported
in traces (produced by `-vv -dump-input=never`) in the case of
successful directives. However, those reports are not always
convenient to view while investigating the input dump, so this patch
adds the substitution report to the input dump too. For example:
```
$ cat check
CHECK: hello [[WHAT:[a-z]+]]
CHECK: [[VERB]] [[WHAT]]
$ FileCheck -vv -DVERB=goodbye check < input |& tail -8
<<<<<<
1: hello world
check:1 ^~~~~~~~~~~
2: goodbye word
check:2'0 X~~~~~~~~~~~ error: no match found
check:2'1 with "VERB" equal to "goodbye"
check:2'2 with "WHAT" equal to "world"
>>>>>>
```
Without this patch, the location reported for a substitution for a
directive match is the directive's full match range. This location is
misleading as it implies the substitution itself matches that range.
This patch changes the reported location to just the match range start
to suggest the substitution is known at the start of the match. (As
in the above example, input dumps don't mark any range for
substitutions. The location info in that case simply identifies the
right line for the annotation.)
Reviewed By: mehdi_amini, thopre
Differential Revision: https://reviews.llvm.org/D83650
This makes the input dump filtering implemented by D82203 more
configurable. D82203 enables filtering out everything but the initial
input lines of error diagnostics (plus some context). This patch
enables including any line with any kind of annotation.
Reviewed By: mehdi_amini
Differential Revision: https://reviews.llvm.org/D83097
For example, given `-dump-input-context=3 -vv`, the following now
shows more leading context for the error than requested because a
leading ellipsis would occupy the same number of lines as it would
elide:
```
<<<<<<
1: foo6
2: foo5
3: foo4
4: foo3
5: foo2
6: foo1
7: hello world
check:1 ^~~~~
check:2 X~~~~ error: no match found
8: foo1
check:2 ~~~~
9: foo2
check:2 ~~~~
10: foo3
check:2 ~~~~
.
.
.
>>>>>>
```
Reviewed By: mehdi_amini
Differential Revision: https://reviews.llvm.org/D83526
This patch is motivated by discussions at each of:
* <https://reviews.llvm.org/D81422>
* <http://lists.llvm.org/pipermail/llvm-dev/2020-June/142369.html>
When input is dumped as specified by `-dump-input=fail`, this patch
filters the dump to show only input lines that are the starting lines
of error diagnostics plus the number of contextual lines specified
`-dump-input-context` (defaults to 5).
When `-dump-input=always`, there might be not be any errors, so all
input lines are printed, as without this patch.
Here's some sample output with `-dump-input-context=3 -vv`:
```
<<<<<<
.
.
.
13: foo
14: foo
15: hello world
check:1 ^~~~~~~~~~~
16: foo
check:2'0 X~~ error: no match found
17: foo
check:2'0 ~~~
18: foo
check:2'0 ~~~
19: foo
check:2'0 ~~~
.
.
.
27: foo
check:2'0 ~~~
28: foo
check:2'0 ~~~
29: foo
check:2'0 ~~~
30: goodbye word
check:2'0 ~~~~~~~~~~~~
check:2'1 ? possible intended match
31: foo
check:2'0 ~~~
32: foo
check:2'0 ~~~
33: foo
check:2'0 ~~~
.
.
.
>>>>>>
```
Reviewed By: mehdi_amini, arsenm, jhenderson, rsmith, SjoerdMeijer, Meinersbur, lattner
Differential Revision: https://reviews.llvm.org/D82203
Document the default of `fail` in `-help`. Extend `-dump-input=help`
to help users find related command-line options, but let `-help`
provide their full documentation.
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D83091
`FILECHECK_OPTS` was implemented so that a test runner, such as CI,
can specify FileCheck debugging options, such as `-v` and `-vv`.
However, if a test suite has a FileCheck call that already specifies
`-v` or `-vv`, then that call will fail if `FILECHECK_OPTS` also
specifies it.
For `-vv`, this problem already exists:
`clang/test/CodeGen/aarch64-v8.2a-fp16-intrinsics-constrained.c`
It's not yet clear if the `-vv` in that test was intentional, but this
usage shouldn't fail anyway. It's already true that FileCheck permits
`-vv` and `-v` together even though `-vv` implies `-v`.
Compare D70784, which fixed the same problem for `-dump-input`.
Reviewed By: jhenderson, thopre
Differential Revision: https://reviews.llvm.org/D82601
This patch is part of a patch series to add support for FileCheck
numeric expressions. This specific patch adds support for specifying the
matching constraint for a numeric expression, ie. how the value being
matched should relate to the numeric expression.
This commit only adds the equality constraint where the numeric value
matched must be equal to the numeric expression. It is the default
matching constraint used when not specified. It is added to provision
other matching constraint (e.g. inequality relations).
Copyright:
- Linaro (changes up to diff 183612 of revision D55940)
- GraphCore (changes in later versions of revision D55940 and
in new revision created off D55940)
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D60391
This patch extends numerical expressions to allow calls to
predefined functions. These calls can be combined with the
existing numerical operators, which includes nesting calls.
The call syntax is:
<func>(<args>)
Where <func> is a predefined string literal, currently limited to
one of add, max, min and sub. <arg> is a comma seperated list of
numerical expressions.
Subscribers: arichardson, hiraditya, thopre, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D79936
Having the input dumped on failure seems like a better
default: I debugged FileCheck tests for a while without knowing
about this option, which really helps to understand failures.
Remove `-dump-input-on-failure` and the environment variable
FILECHECK_DUMP_INPUT_ON_FAILURE which are now obsolete.
Differential Revision: https://reviews.llvm.org/D81422
Summary:
This patch is part of a patch series to add support for FileCheck
numeric expressions. This specific patch adds support signed numeric
values, thus allowing negative numeric values.
As such, the patch adds a new class to represent a signed or unsigned
value and add the logic for type promotion and type conversion in
numeric expression mixing signed and unsigned values. It also adds
the %d format specifier to represent signed value.
Finally, it also adds underflow and overflow detection when performing a
binary operation.
Copyright:
- Linaro (changes up to diff 183612 of revision D55940)
- GraphCore (changes in later versions of revision D55940 and
in new revision created off D55940)
Reviewers: jhenderson, chandlerc, jdenny, probinson, grimar, arichardson
Reviewed By: jhenderson, arichardson
Subscribers: MaskRay, hiraditya, llvm-commits, probinson, dblaikie, grimar, arichardson, kristina, hfinkel, rogfer01, JonChesterfield
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D60390
With this change it is be possible to write FileCheck expressions such
as [[#(VAR+1)-2]]. Currently, the only supported arithmetic operators are
plus and minus, so this is not particularly useful yet. However, it our
CHERI fork we have tests that benefit from having multiplication in
FileCheck expressions. Allowing parenthesized expressions is the simplest
way for us to work around the current lack of operator precedence in
FileCheck expressions.
Reviewed By: thopre, jhenderson
Differential Revision: https://reviews.llvm.org/D77383
Summary:
Various improvement for FileCheck's numeric-expression.txt test:
- remove unused values in USE DEF FMT IMPL MATCH section
- replace 14 literal for 0xe and 0xE to have example of hex literals
- rename variable to be more self-descriptive
- move CHECK as comment of the values being matched to help readability
- add conversion tests
- simplify test for use of several numeric variables by using existing
variable
- adjust position of error message check to match the alignment of the
error message wrt. the output matched by the previous check
Reviewed By: jhenderson, jdenny
Differential Revision: https://reviews.llvm.org/D79820
D79276 caused the following builder to fail:
http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/23489
Specifically, FileCheck dumped stack in the following tests:
LLVM :: MC/Mips/micromips-jump-pc-region.s
LLVM :: MC/Mips/mips-jump-pc-region.s
Those tests contained characters encoded as 160 but that render (at
least for me in vim) like a single space (32). Those characters
appeared between the `#` and `RUN:` on several lines, and D79276
caused FileCheck to process those lines differently: `RUN:` is a
comment directive. As a result, D79276 caused FileCheck to start
calling is `isalnum` on those characters.
The problem is that FileCheck calls `isalnum` on type `char` without
casting to `unsigned char` first, so it sign-extends 160 beyond what
`unsigned char` or `EOF` can represent. C says that has undefined
behavior. This problem is general to FileCheck's prefix parsing and
so exists independently of D79276.
524457edbc3d fixed the above tests. This patch changes FileCheck to
use LLVM's replacements for `ctype.h` functions, and it adds tests for
cases that are representative with or without D79276.
Reviewed By: jhenderson, thopre, efriedma
Differential Revision: https://reviews.llvm.org/D79810
Sometimes you want to disable a FileCheck directive without removing
it entirely, or you want to write comments that mention a directive by
name. The `COM:` directive makes it easy to do this. For example,
you might have:
```
; X32: pinsrd_1:
; X32: pinsrd $1, 4(%esp), %xmm0
; COM: FIXME: X64 isn't working correctly yet for this part of codegen, but
; COM: X64 will have something similar to X32:
; COM:
; COM: X64: pinsrd_1:
; COM: X64: pinsrd $1, %edi, %xmm0
```
Without this patch, you need to use some combination of rewording and
directive syntax mangling to prevent FileCheck from recognizing the
commented occurrences of `X32:` and `X64:` above as directives.
Moreover, FileCheck diagnostics have been proposed that might complain
about the occurrences of `X64` that don't have the trailing `:`
because they look like directive typos:
<http://lists.llvm.org/pipermail/llvm-dev/2020-April/140610.html>
I think dodging all these problems can prove tedious for test authors,
and directive syntax mangling already makes the purpose of existing
test code unclear. `COM:` can avoid all these problems.
This patch also updates the small set of existing tests that define
`COM` as a check prefix:
- clang/test/CodeGen/default-address-space.c
- clang/test/CodeGenOpenCL/addr-space-struct-arg.cl
- clang/test/Driver/hip-device-libs.hip
- llvm/test/Assembler/drop-debug-info-nonzero-alloca.ll
I think lit should support `COM:` as well. Perhaps `clang -verify`
should too.
Reviewed By: jhenderson, thopre
Differential Revision: https://reviews.llvm.org/D79276
This will prove especially helpful after D79276, which introduces
comment prefixes. Specifically, identifying whether there's a
uniqueness violation will be helpful as prefixes will be required to
be unique across both check prefixes and comment prefixes.
Also, remove a related comment about `cl::list` that no longer seems
relevant now that FileCheck is also a library.
Reviewed By: jhenderson, thopre
Differential Revision: https://reviews.llvm.org/D79375
Sometimes you want to disable a FileCheck directive without removing
it entirely, or you want to write comments that mention a directive by
name. The `COM:` directive makes it easy to do this. For example,
you might have:
```
; X32: pinsrd_1:
; X32: pinsrd $1, 4(%esp), %xmm0
; COM: FIXME: X64 isn't working correctly yet for this part of codegen, but
; COM: X64 will have something similar to X32:
; COM:
; COM: X64: pinsrd_1:
; COM: X64: pinsrd $1, %edi, %xmm0
```
Without this patch, you need to use some combination of rewording and
directive syntax mangling to prevent FileCheck from recognizing the
commented occurrences of `X32:` and `X64:` above as directives.
Moreover, FileCheck diagnostics have been proposed that might complain
about the occurrences of `X64` that don't have the trailing `:`
because they look like directive typos:
<http://lists.llvm.org/pipermail/llvm-dev/2020-April/140610.html>
I think dodging all these problems can prove tedious for test authors,
and directive syntax mangling already makes the purpose of existing
test code unclear. `COM:` can avoid all these problems.
This patch also updates the small set of existing tests that define
`COM` as a check prefix:
- clang/test/CodeGen/default-address-space.c
- clang/test/CodeGenOpenCL/addr-space-struct-arg.cl
- clang/test/Driver/hip-device-libs.hip
- llvm/test/Assembler/drop-debug-info-nonzero-alloca.ll
I think lit should support `COM:` as well. Perhaps `clang -verify`
should too.
Reviewed By: jhenderson, thopre
Differential Revision: https://reviews.llvm.org/D79276
This will prove especially helpful after D79276, which introduces
comment prefixes. Specifically, identifying whether there's a
uniqueness violation will be helpful as prefixes will be required to
be unique across both check prefixes and comment prefixes.
Also, remove a related comment about `cl::list` that no longer seems
relevant now that FileCheck is also a library.
Reviewed By: jhenderson, thopre
Differential Revision: https://reviews.llvm.org/D79375
Without this patch, `--dump-input` annotations on a single input line
are sorted by the associated directive's check-file line. That seemed
fine because that's often identical to the order in which FileCheck
looks for matches for those directives.
The first problem is that an `--implicit-check-not` pattern has no
check-file line. The logical equivalent is sorting in command-line
order, but that's not implemented.
The second problem is that, unlike a directive, an
`--implicit-check-not` pattern applies at many points, between many
different pairs of directives. However, sorting in command-line order
gathers all its associated diagnostics together at one point in an
input line's list of annotations.
In general, it seems to be easier to understand FileCheck's logic when
annotations on a single input line are sorted in the order FileCheck
produced the associated diagnostics, so this patch makes that change.
As documented in the patch, the annotation sort order is also
especially relevant to `CHECK-LABEL`, `CHECK-NOT`, and `CHECK-DAG`, so
this patch updates or extends tests to check the sort makes sense for
them. (However, the sort for `CHECK-DAG` annotations should not
actually be altered by this patch.)
Reviewed By: thopre
Differential Revision: https://reviews.llvm.org/D77607
Currently, `--dump-input` implies that all `--implicit-check-not`
patterns appear on line 1 by printing annotations like:
```
1: foo bar baz
not:1 !~~ error: no match expected
```
This patch changes that to:
```
1: foo bar baz
not:imp1 !~~ error: no match expected
```
`imp1` indicates the first `--implicit-check-not` pattern.
Reviewed By: thopre
Differential Revision: https://reviews.llvm.org/D77605
Imagine we have the following invocation:
`FileCheck -check-prefix=UNKNOWN-PREFIX -implicit-check-not=something`
When the check prefix does not exist it does not fail.
This patch fixes the issue.
Differential revision: https://reviews.llvm.org/D78024
Summary:
Improve error message in case of conflict between several implicit
format to mention the operand that conflict.
Reviewers: jhenderson, jdenny, probinson, grimar, arichardson, rnk
Reviewed By: jdenny
Subscribers: hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D77741
