The shtest-inject test relied on being executed in "single process" mode
and started to fail with a `PicklingError` after it was removed:
```
Can't pickle <class 'lit.TestingConfig.CustomFormat'>: attribute
lookup lit.TestingConfig.CustomFormat failed
```
This happened because the test config has to be serialized to the worker
process, but apparently the `CustomFormat` class defined inline is not
serializable.
This change allows passing the tested functionality (preamble_commands)
directly to `lit.formats.ShTest` so we can use it directly in the test.
Remove the "serial run" abstraction which bypasses Python's
`multiprocessing.Pool` and instead directly runs tests without spawning
worker processes. This abstraction has not offered the benefits I hoped
it would and therefore does not carry its weight.
In previous commits [1,2] I changed worker.py to only send back the test
result from the worker process instead of the whole test object. This
was a mistake. lit.Test contains fields (e.g., xfials, requires,
unsupported) that are only populated when we actually execute the test,
but are queried when we report the results in the parent process. This
commit essentially reverts the following changes:
[1] a3d2f9b53ac006cb972b61b0dbfcb5babe4356bf
[2] 17bb660fb83e869652ac87b145b0e26b708aab60
A previous attempt to cleanup module imports broke installing via
pip/setup.py [1]. This should be fixed now.
[1] cf252240e8819d0c90a5e10f773078bdeba33e44
Reviewed By: paquette
Differential Revision: https://reviews.llvm.org/D76940
This minor refactoring allows reducing the amount of processing that
is duplicated when we re-run a flaky test. It also has the nice
side effect that libc++'s current test format supports flaky .sh.cpp
tests, because those are built on top of _runShTest, not executeShTest.
This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cxx %s -o %t.exe' and the script would be properly
expanded.
Differential Revision: https://reviews.llvm.org/D76178
This allows creating custom test formats on top of `executeShTest` that
inject commands at the beginning of the file being parsed, without
requiring these commands to physically appear in the test file itself.
For example, one could define a test format that prints out additional
debug information at the beginning of each test. More realistically,
this has been used to define custom test formats like one that supports
compilation failure tests (e.g. with the extension `compile.fail.cpp`)
by injecting a command that calls the compiler on the file itself and
expects it to fail.
Without this change, the only alternative is to create a temporary file
with the same content as the original test, then prepend the desired
`// RUN:` lines to that file, and call `executeShTest` on that file
instead. This is both slow and cumbersome to do.
Differential Revision: https://reviews.llvm.org/D76290
This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.
This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.
Differential Revision: https://reviews.llvm.org/D76288
And add llvm-go back to the test dependencies.
No longer necessary now that llvm-go has been brought back.
This reverts commit e8f8873da5eaad187f82dad78ebdb3ab3df22b36.
Basically change the layout to please `go build` and remove references to
`llvm-go`.
Update llvm/test/Bindings/Go/ to use the system go compiler
Differential Revision: https://reviews.llvm.org/D74540
While looking at cycle time graphs of some of my bots, I noticed
that 327894859cc made check-llvm noticeably slower on macOS and
Windows.
As it turns out, the 5 substitutions added in that change were
enough to cause lit to thrash the build-in cache in re.compile()
(re.sub() is implemented as re.compile().sub()), and apparently
applySubstitutions() is on the cricital path and slow when all
regexes need to compile all the time.
(See `_MAXCACHE = 512` in cpython/Lib/re.py)
Supporting full regexes for lit substitutions seems a bit like
overkill, but for now add a simple unbounded cache to recover
the lost performance.
No intended behavior change.
A refactoring commit (cf252240) removed this line. Removing it broke installing
lit with pip and setup.py.
This adds the line back in so that we can install lit again.
For an example of how this appeared, see:
http://green.lab.llvm.org/green/job/LNT_Tests/5853/
File "/Users/buildslave/jenkins/...s/__init__.py", line 2453, in resolve
raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'main'
It would appear that the removal of this lit feature was incomplete
and there is a test case that still tests for this. This patch removes
the remaining tests to bring the bots back to green. I would encourage the
author to do a post-commit review on this in case there is a more desirable fix.
Summary:
Remove REQUIRES-ANY alias lit directive since it is hardly used and can
be easily implemented using an OR expression using REQUIRES. Fixup
remaining testcases still using REQUIRES-ANY.
Reviewers: probinson, jdenny, gparker42
Reviewed By: gparker42
Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits
Tags: #llvm, #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D71408
E.g. the mingw python distributed in msys2 (the mingw one, which is a
normal win32 application and doesn't use the msys2 runtime itself),
despite being a normal win32 python, still uses forward slashes. This
works fine for other cases (many, but not all), but when constructing a
raw NT path, all path separators must be backslashes.
Differential Revision: https://reviews.llvm.org/D71490
Jenkins sometimes starts a new working directory by appending @2 (or
incrementing the number if the @n suffix is already there). This causes
several clang tests to fail as:
s@INPUT_DIR@%/S/Inputs@g
gets expanded to the invalid:
s@INPUT_DIR@/path/to/workdir@2/Inputs@g
~~~~~~~~~~
where the part marked with ~'s is interpreted as the flags. These are
invalid and the test fails.
Previous fixes simply exchanged the @ character for another like | but
that's just moving the problem. Address it by adding an expansion that
escapes the @ character we're using as a delimiter as well as other magic
characters in the replacement of sed's s@@@.
There's still room for expansions to cause trouble though. One I ran into
while testing this was that having a directory called foo@bar causes lots
of `CHECK-NOT: foo` directives to match. There's also things like
directories containing `\1`
Tests go through the following stages:
*) discovered
*) filtered
*) executed
Only executed tests have a result (e.g., PASS, FAIL, XFAIL, etc.). See
"result codes" in Test.py.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D70612
Fail early, when we discover no tests at all, or filter out all of them.
There is also `--allow-empty-runs` to disable test to allow workflows
like `LIT_FILTER=abc ninja check-all`. Apparently `check-all` invokes
lit multiple times if certain projects are enabled, which would produce
unwanted "empty runs". Specify via `LIT_OPTS=--allow-empty-runs`.
There are 3 causes for empty runs:
1) No tests discovered. This is always an error. Fix test suite config
or command line.
2) All tests filtered out. This is an error by default, but can be
suppressed via `--alow-empty-runs`. Should prevent accidentally
passing empty runs, but allow the workflow above.
3) The number of shards is greater than the number of tests. Currently,
this is never an error. Personally, I think we should consider
making this an error by default; if this happens, you are doing
something wrong. I added a warning but did not change the behavior,
since this warrants more discussion.
Reviewed By: atrick, jdenny
Differential Revision: https://reviews.llvm.org/D70105
lit's test suite calls lit multiple times for various sample test
suites. `FILECHECK_OPTS` is safe for FileCheck calls in lit's test
suite. It's not safe for FileCheck calls in the sample test suites,
whose output affects the results of lit's test suite.
Without this patch, only one such sample test suite is protected from
`FILECHECK_OPTS`, and currently `shtest-shell.py` breaks with
`FILECHECK_OPTS=-vv`. Moreover, it's hard to predict the future,
especially false passes. Thus, this patch protects all existing and
future sample test suites from `FILECHECK_OPTS` (and the deprecated
`FILECHECK_DUMP_INPUT_ON_FAILURE`).
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D65156
Without this patch, when using lit's internal shell, if `not` on a lit
RUN line calls `env`, `diff`, or any of the other in-process shell
builtins that lit implements, lit accidentally searches for the latter
as an external executable. What's worse is that works fine when a
developer is testing on a platform where those executables are
available and behave as expected, but it then breaks on other
platforms.
`not` seems useful for some builtins, such as `diff`, so this patch
supports such uses. `not --crash` does not seem useful for builtins,
so this patch diagnoses such uses. In all cases, this patch ensures
shell builtins are found behind any sequence of `env` and `not`
commands.
`not` calling `env` calling an external command appears useful when
the `env` and external command are part of a lit substitution, as in
D65156. This patch supports that by looking through any sequence of
`env` and `not` commands, building the environment from the `env`s,
and storing the `not`s. The `not`s are then added back to the command
line without the `env`s to execute externally. This avoids the need
to replicate the `not` implementation, in particular the `--crash`
option, in lit.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D66531
The static analyzer's scan-build script is critical infrastructure but
is not well tested. To start to address this, add a new test directory under
tests/Analysis for scan-build lit tests and seed it with several tests. The
goal is that future scan-build changes will be accompanied by corresponding
tests.
Differential Revision: https://reviews.llvm.org/D69781
Without this patch, when using lit's internal shell, if `env` on a lit
RUN line calls `env`, lit accidentally searches for the latter as an
external executable. What's worse is that works fine when a developer
is testing on a platform where `env` is available and behaves as
expected, but it then breaks on other platforms.
`env` calling `env` can make sense if one such `env` is within a lit
substitution, as in D65156 and D65121. This patch ensures that lit
executes both as internal commands.
Reviewed By: probinson, mgorny, rnk
Differential Revision: https://reviews.llvm.org/D65697
Without this patch, when using lit's internal shell, if `env` on a lit
RUN line calls `cd`, `mkdir`, or any of the other in-process shell
builtins that lit implements, lit accidentally searches for the latter
as an external executable.
This patch puts such builtins in a map so that boilerplate for them
need be implemented only once. This patch moves that handling after
processing of `env` so that `env` calling such a builtin can be
detected. Finally, because such calls appear to be useless, this
patch takes the safe approach of diagnosing them rather than
supporting them.
Reviewed By: probinson, mgorny, rnk
Differential Revision: https://reviews.llvm.org/D66506
