I did some profiling of lit while trying to optimize the libc++ test
startup for remote hosts and it turns out that there is a realpath() call
for every message printed and this shows up in the profile.
The inspect.getframeinfo() function calls realpath() internally and
moreover we don't need most of the other information returned from it.
This patch uses inspect.getsourcefile() and os.path.abspath() to remove
../ from the path instead. Not resolving symlinks reduces the startup time
for running a single test with lit by about 50ms for me.
Reviewed By: ldionne, yln
Differential Revision: https://reviews.llvm.org/D89186
I recently had to run the test suite with a debug Python which got started
warning about some invalid escape sequences in LLDB's Python code. They all
attempt to add a backslash by doing a single backslash instead of a double
backslash in a normal string. This seems to work fine for now, but Python says
this behaviour is deprecated, so this patch turns all those strings into raw
strings (where a single backslash is actually a single backslash)
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D88289
The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine within a short time window (1 or 2
seconds). While this "seems to work" it can fail on resource constrained
machines. We could bump the timeout a little bit (bumping it too
much would mean the test would take a long time to execute) but it wouldn't
really solve the problem of the test being prone to failures.
This patch tries to remove this flakeyness by separating testing into
two separate parts:
1. Testing if a test can hit a timeout.
2. Testing if a test can run to completion in the presence of a
timeout.
This way we can give (1.) a really short timeout (to make the test run
as fast as possible) and (2.) a really long timeout. This means for (2.)
we are no longer trying to rely on the "short" test executing within
some short time window. Instead the window is now 3600 seconds which
should be long enough even for a heavily resource constrained machine to
execute the "short" test.
Thanks to Julian Lettner for suggesting this approach. This superseeds
my original approach in https://reviews.llvm.org/D88807.
This patch also changes the command line override test to run the quick
test rather than the slow one to make the test run faster.
Differential Revision: https://reviews.llvm.org/D89020
This reverts b3418cb4eb1456c41606f4621dcfa362fe54183c and d12ae042e17b27ebc8d2b5ae3d8dd5f88384d093
This breaks some external bots, see discussion in https://reviews.llvm.org/D84380
In the meanwhile, please use `cmake -DLLVM_LIT_TOOLS_DIR="C:/Program Files/Git/usr/bin"` or add it to %PATH%.
Summary:
This patch adds an error to Clang that detects if OpenMP offloading is
used between two architectures with incompatible pointer sizes. This
ensures that the data mapping can be done correctly and solves an issue
in code generation generating the wrong size pointer. This patch adds a
new lit substitution, %omp_powerpc_triple that, if the system is 32-bit or
64-bit, sets the powerpc triple accordingly. This was required to fix
some OpenMP tests that automatically populated the target architecture.
Reviewers: jdoerfert
Subscribers: cfe-commits guansong sstefan1 yaxunl delcypher
Tags: OpenMP clang LLVM
Differential Revision: https://reviews.llvm.org/D88594
This should fix the test failures on the clang win64 bot:
http://lab.llvm.org:8011/builders/clang-x64-windows-msvc/builds/18830
It has been red since Sept 23-ish.
This was subtle to debug. Windows has 'find' and 'sort' utilities in
C:\Windows\system32, but they don't support all the same flags as the
coreutils programs. I configured the buildbot above with Python 2.7
64-bit (hey, it was set up in 2016). When I installed git for Windows, I
opted to add all the Unix utilities that come with git to the system
PATH. This is *almost* enough to make the LLVM tests pass, but not
quite, because if you use the system PATH, the Windows version of find
and sort come first, but the tests that use diff, cmp, etc, will all
pass. So only a handful of tests will fail, and with cryptic error
messages.
The code changed in this CL doesn't work with Python 2. Before
Python 3.2, the winreg.OpenKey function did not accept the `access=`
keyword argument, the caller was required to pass an unused `reserved`
positional argument of 0. The try/except/pass around the OpenKey
operation masked this usage error in Python 2.
Further, the result of the registry operation has to be converted from
unicode to add it to the environment, but that was incidental.
Historically, we have told contributors that GnuWin32 is a pre-requisite
because our tests depend on utilities such as sed, grep, diff, and more.
However, Git on Windows includes versions of these utilities in its
installation. Furthermore, GnuWin32 has not been updated in many years.
For these reasons, it makes sense to have the ability to run llvm tests
in a way that is both:
a) Easier on the user (less stuff to install)
b) More up-to-date (The verions that ship with git are at least as
new, if not newer, than the versions in GnuWin32.
We add support for this here by attempting to detect where Git is
installed using the Windows registry, confirming the existence of
several common Unix tools, and then adding this location to lit's PATH
environment.
Differential Revision: https://reviews.llvm.org/D84380
`clang` currently requires the native linker on Solaris:
- It passes `-C` to `ld` which GNU `ld` doesn't understand.
- To use `gld`, one needs to pass the correct `-m EMU` option to select
the right emulation. Solaris `ld` cannot handle that option.
So far I've worked around this by passing `-DCLANG_DEFAULT_LINKER=/usr/bin/ld`
to `cmake`. However, if someone forgets this, it depends on the user's
`PATH` whether or not `clang` finds the correct linker, which doesn't make
for a good user experience.
While it would be nice to detect the linker flavor at runtime, this is more
involved. Instead, this patch defaults to `/usr/bin/ld` on Solaris. This
doesn't work on its own, however: a link fails with
clang-12: error: unable to execute command: Executable "x86_64-pc-solaris2.11-/usr/bin/ld" doesn't exist!
I avoid this by leaving absolute paths alone in `ToolChain::GetLinkerPath`.
Tested on `amd64-pc-solaris2.11`, `sparcv9-sun-solaris2.11`, and
`x86_64-pc-linux-gnu`.
Differential Revision: https://reviews.llvm.org/D84029
Failing test output sometimes contains control characters like \x1b (e.g.
if there was some -fcolor-diagnostics output) which are not allowed inside
XML files. This causes problems with CI systems: for example, the Jenkins
JUnit XML will throw an exception when ecountering those characters and
similar problems also occur with GitLab CI.
Reviewed By: yln, jdenny
Differential Revision: https://reviews.llvm.org/D84233
This produces a chrome://tracing compatible trace file in the same way
as -ftime-trace.
This can be useful in optimising test time where one long test is causing
long overall test time on a wide machine.
This also helped in finding tests which have side effects on others
(e.g. https://reviews.llvm.org/D84885).
Differential Revision: https://reviews.llvm.org/D84931
Otherwise, if a Lit script contains escaped substitutions (like %%p in this test https://github.com/llvm/llvm-project/blob/master/compiler-rt/test/asan/TestCases/Darwin/asan-symbolize-partial-report-with-module-map.cpp#L10), they are unescaped during recursive application of substitutions, and the results are unexpected.
We solve it using the fact that double percent signs are first replaced with #_MARKER_#, and only after all the other substitutions have been applied, #_MARKER_# is replaced with a single percent sign. The only change is that instead of replacing #_MARKER_# at each recursion step, we replace it once after the last recursion step.
Differential Revision: https://reviews.llvm.org/D83894
When running multiple shards, don't include skipped tests in the xunit
output since merging the files will result in duplicates.
In our CHERI Jenkins CI, I configured the libc++ tests to run using sharding
(since we are testing using a single-CPU QEMU). We then merge the generated
XUnit xml files to produce a final result, but if the individual XMLs
report tests excluded due to sharding each test is included N times in the
final result. This also makes it difficult to find the tests that were
skipped due to missing REQUIRES: etc.
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D84235
Summary:
As a corrollary, these tests are now run as part of the check-flang
target.
Reviewers: sscalpone
Subscribers: mgorny, delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D83946
Summary: In Python3 SubstituteCaptures are no longer converted to String implicitly behind the scenes. Converting explicitly makes the TestRunner to work in Python3.
Reviewers: gribozavr2, compnerd
Reviewed By: gribozavr2
Subscribers: tbkka, delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D81361
As per discussion in D69207, have lit ignore UnicodeDecodeErrors
when running with python 2 in an ASCII shell.
Differential Revision: https://reviews.llvm.org/D82754
Provide `--show-xxx` flags for all non-failure result codes, just as we
already do for `--show-xfail` and `--show-unsupported`.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D82233
Summary: This is a complementary patch to D82100 since the aix builbot is still running the unsupported test shtest-format-argv0. Add system-aix to the sub llvm-lit config.
Reviewers: daltenty, hubert.reinterpretcast
Reviewed By: hubert.reinterpretcast
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D82905
Summary: lit test `shtest-format.py` fails on AIX because one of the subtest of shtest-format requires the tool `[` to be installed under the system PATH. For AIX, `[` is only available as a shell builtin and does not present as an executable file under PATH. Hence, split the original shtest-format into two separate test files and added AIX as UNSUPPORTED for the test using `[` .
Reviewers: daltenty, hubert.reinterpretcast
Reviewed By: hubert.reinterpretcast
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D82100
Let's have one canonical place to define ResultCode instances and their
labels.
Also make ResultCode's `__init__` function self-registering to better
support custom ResultCodes.
Before this change we showed all result groups with a code that was not
explicitly hard-coded set. This set missed the FLAKYPASS result code.
Let's generalize the code to always show failures and the additionally
requested result codes.
MSVC uses lit for STL testing to run both the libcxx tests and our "native" suite of tests which has feature requirements that are not parsed from the test content. For consistency, the change treats the `unsupported` and `xfails` `Test` properties similarly to `requires`.
Differential Revision: https://reviews.llvm.org/D81782
Pass in all discovered tests to report generators.
The XunitReport generator now creates testcase items for unexecuted
tests and documents why they have been skipped. This makes it easier
to compare test runs with different filters or configurations, or across
platforms.
I don't know who is using the JsonReport generator and what the
expectations there are (it doesn't have tests), so decided to preserve
the old behavior by filtering out the unexecuted tests.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D81316
This allows running Lit tests that run ssh without having to manually
enter a password (which is inconvenient), by just having ssh-agent
setup properly when running the test suite.
In TestRunner.py, D78589 extracts a `_parseKeywords` function from
`parseIntegratedTestScript`, which then expects `_parseKeywords` to
always return a list of keyword/value pairs. However, the extracted
code sometimes returns an unresolved `lit.Test.Result` on a keyword
parsing error, which then produces a stack dump instead of the
expected diagnostic.
This patch fixes that, makes the style of those diagnostics more
consistent, and extends the lit test suite to cover them.
Reviewed By: ldionne
Differential Revision: https://reviews.llvm.org/D81665
Having the input dumped on failure seems like a better
default: I debugged FileCheck tests for a while without knowing
about this option, which really helps to understand failures.
Remove `-dump-input-on-failure` and the environment variable
FILECHECK_DUMP_INPUT_ON_FAILURE which are now obsolete.
Differential Revision: https://reviews.llvm.org/D81422
Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing tests) and summary count lines. For
example, "Failing Tests"/"Unexpected Failures". This commit changes lit
to label things consistently.
Improve wording of labels:
When talking about individual test results, the first word in
"Unexpected Failures", "Expected Passes", and "Individual Timeouts" is
superfluous. Some labels contain the word "Tests" and some don't.
Let's simplify the names.
Before:
```
Failing Tests (1):
...
Expected Passes : 3
Unexpected Failures: 1
```
After:
```
Failed Tests (1):
...
Passed: 3
Failed: 1
```
Reviewed By: ldionne
Differential Revision: https://reviews.llvm.org/D77708
Lit test suites can tend to accumulate annotations that are not necessarily
relevant as time goes by, for example XFAILS on old compilers or platforms.
To help spot old annotations that can be cleaned up, it can be useful to
look at all features used inside a test suite.
This commit adds a new Lit option '--show-used-features' that prints all
the features used in XFAIL, REQUIRES and UNSUPPORTED of all tests that
are discovered.
Differential Revision: https://reviews.llvm.org/D78589
This reverts commit 78dea0e8fb6cc5c0fae64d65b7f40560cab7b329.
The offending lldb test (which is a real bug exposed by this patch)
has been disabled on windows (see a67b2faa7c4cfbceffb4213f46769c45a5a9291a)
and lldb is queued for inclusion into precommit testing, which would
have caught this.
Differential Revision: https://reviews.llvm.org/D80389
lit runs a gtest executable multiple times. First it runs it to
discover tests, then later it runs the executable again for each test.
However, if the discovery fails (perhaps because of a broken
executable), then no tests were previously run and no failures were
reported. This patch creates a dummy test if discovery fails, which
will later fail when test are run and be reported as a failure.
Differential Revision: https://reviews.llvm.org/D80096
Summary:
Before making this change, whenever I ran "check-flang", I'd get an
error message like:
llvm-lit: /mnt/c/GitHub/f18/c751/flang/build/bin/../../../llvm/utils/lit/lit/main.py:252: warning: Failed to delete temp directory '/tmp/lit_tmp_gOKUIh'
With this change, there's no such message in the output, and the temp
directory is successfully removed.
Note that my working environment is on Windows 10 running Windows
Subsystem for Linux using the Ubuntu app. I'm running Python version
2.7.1.
Earlier versions of Python do not contain `shutil`. It may be that this
module was available on Windows systems later than other platforms.
Upgrading my version of Python made the problem go away
I don't believe that timing was a problem since inserting a long delay
didn't fix things.
So I added some text to the error message recommending that the user
upgrade their version of Python if they run into this problem.
Reviewers: yln, DavidTruby
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D79861
The argparse 'append' action concatenates multiple occurrences of an
argument (even when we specify `nargs=1` or `nargs='?'`). This means
that we create multiple identical output files if the `--output`
argument is given more than once. This isn't useful and we instead want
this to behave like a standard optional argument: last occurrence wins.
abhinavgaba reported that that the custom-result-category.py test hangs
on a Windows build bot [1]. Disable it for now.
[1] https://reviews.llvm.org/D78164#2018178
Factor out the report generators from main.py into reports.py.
I verified that we generate the exact same output by running `check-all`
and comparing the new and old output for both report flavors.
Don't update whole test object from the remote (pickled) finished test
object. Doing so also changes the config and suite members, which we
want to avoid.
Track and print the number of tests that were discovered but not
executed due to test selection options:
* --filter (regex filter)
* --max-tests (limits number of tests)
* sharding feature
With this change all discovered tests are accounted for: every
discovered test is included in one of the counts printed in the summary.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D78078
The lnt test suite defines custom result codes [1]. Support those via
an extension API instead of "by accident", which should offer the
advantage of properly handling them when we print test results.
[1] https://reviews.llvm.org/D77986
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D78164
This reverts commit bc3f54de1827e58655c34477d09211cbc42589bd.
The patch breaks in the following two scenarios:
1. When manually passing an absolute path to llvm-lit with a lower-case
drive letter: `python bin\llvm-lit.py -sv c:\llvm-project\clang\test\PCH`
2. When the PWD has a lower-case drive letter, like after running
`cd c:\` with a lower-case "c:" (cmd's default is upper-case, but
it takes case-ness from what's passed to `cd` apparently).
