1
0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-10-19 11:02:59 +02:00
Commit Graph

619 Commits

Author SHA1 Message Date
Justin Bogner
586bd2098d [lit] Move the shtest-xunit-output check lines into shtest-format
These two tests are operating on the same test suite, which causes
them to be racy about writing temporary files and can cause spurious
failures. Merge them into one test to avoid the issue.

llvm-svn: 337718
2018-07-23 18:08:36 +00:00
George Karpenkov
56b2870853 [FileCheck] Provide an option for FileCheck to dump original input to stderr on failure
The option can be either set using environment variable (e.g. env
FILECHECK_DUMP_INPUT_ON_FAILURE=1 ninja check-fuzzer) or with a
FileCheck flag.

This can be extremely useful for debugging, cf.
https://groups.google.com/forum/#!topic/llvm-dev/kLrzg8OM_h8 for
discussion.

Differential Revision: https://reviews.llvm.org/D49328

llvm-svn: 337609
2018-07-20 20:21:57 +00:00
Hans Wennborg
f9ea32fcfd Relax shtest-run-at-line.py
The test was failing on Windows machines which had bash.exe on PATH (but
not in the so called lit tools dir, containing cmp.exe, grep.exe etc.).

The problem was that the outer lit invocation would load LLVMConfig
from utils/lit/lit/llvm/config.py, which looks up the tools path with
getToolsPath(). That has a surprising side effect of also setting
bashPath, in our case setting it to empty.

The outer lit invocation would thus configure the pdbg0 and pdbg1
substitutions based on not running with bash.

But the inner lit invocation would not load LLVMConfig, so bash
would be found on PATH, that would be used as external shell,
and so the output wouldn't match pdbg0 and pdbg1.

It seems weird to me that getBashPath() will return different results
depending on whether getToolsPath() has been called before, but I
also don't know how to fix it properly.

This commit just relaxes the test case, because there doesn't seem
to be much point in testing for the exact syntax of the run file
as long as it works.

(See https://crbug.com/850023)

llvm-svn: 334100
2018-06-06 14:53:03 +00:00
Vlad Tsyrklevich
48f4d3ae20 [Analyzer] Fix the Z3 lit test config
Summary:
The '%analyze' extra_args config argument seems to have been erroneously
deleted in r315627 disabling Z3 tests for the clang analyzer. Add the
flag back.

Reviewers: george.karpenkov, NoQ, ddcc

Reviewed By: george.karpenkov

Subscribers: xazax.hun, szepet, delcypher, a.sidorin, llvm-commits

Differential Revision: https://reviews.llvm.org/D47722

llvm-svn: 334066
2018-06-06 06:25:37 +00:00
Joel E. Denny
a04841e104 [lit] Fix windows cmd.exe test config for r333620
llvm-svn: 333630
2018-05-31 05:48:33 +00:00
Joel E. Denny
bd389c40fd [lit] Terminate ": RUN at line N" with ";" not "&&"
This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was
broken by r333614.  The trouble was that "&&" changes the command for
which "$!" gives the pid.

llvm-svn: 333620
2018-05-31 03:40:37 +00:00
Joel E. Denny
3204e36001 [lit] Report line number for failed RUN command
(Relands r333584, reverted in 333592.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.  (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher,	asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333614
2018-05-31 00:55:32 +00:00
Joel E. Denny
664a77c58b Revert r333584: [lit] Report line number for failed RUN command
It breaks test-suite.

llvm-svn: 333592
2018-05-30 21:07:27 +00:00
Joel E. Denny
f5a15f4aa3 [lit] Report line number for failed RUN command
(Relands r330755 (reverted in r330848) with fix for PR37239.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.  (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher,	asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333584
2018-05-30 19:42:27 +00:00
Chris Matthews
0b61b91c62 Use quoteattr to ensure we make well formed attributes
We were making malformed XML on tests with ' in the name.  Switch to
using saxutils to set all of our attributes, so it can handle quotes
etc correctly.

llvm-svn: 333249
2018-05-25 00:31:36 +00:00
Dan Liew
d7e3a3632a [lit] Try to make shtest-timeout.py test more reliable by using a
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.

While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.

rdar://problem/40221572

llvm-svn: 332987
2018-05-22 15:06:29 +00:00
Dan Liew
bae1119a38 [lit] Don't run slow.py in shtest-timeout.py test.
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.

rdar://problem/40221572

llvm-svn: 332986
2018-05-22 15:06:24 +00:00
Dan Liew
b2006c8dcb [lit] Don't check output of commands used in shtest-timeout.py test.
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test failures.
What matters is that the right test cases reach a timeout.

rdar://problem/40221572

llvm-svn: 332985
2018-05-22 15:06:20 +00:00
Douglas Yung
d711b42b49 Mark test with "REQUIRES: shell" since it directly invokes "sh" and was failing on Windows.
llvm-svn: 332563
2018-05-17 01:36:25 +00:00
Alexander Richardson
7857762e4c Escape ]]> in xunit xml output
Summary:
This sequence ends the CDATA block so any characters after that are no
longer escaped. This can be fixed by replacing "]]>" with "]]]]><![CDATA[>".

Reviewers: cmatthews

Reviewed By: cmatthews

Differential Revision: https://reviews.llvm.org/D46886

llvm-svn: 332440
2018-05-16 09:00:28 +00:00
Chris Matthews
e0665809b0 remove output xml incase it is leftover from another run
llvm-svn: 332424
2018-05-16 00:37:00 +00:00
Chris Matthews
30e6528953 Use not to catch unexpected pass as well as remove old test results
As per review feedback, make sure we rm temp files, and make the return
code checking for lit more specific.

llvm-svn: 332423
2018-05-16 00:33:29 +00:00
Chris Matthews
c01e6f9ecb Requirements can have & in them!
Lets escape those so the XML is valid!

llvm-svn: 332161
2018-05-12 00:13:54 +00:00
Chris Matthews
5e42577a9e Add the message attribute to skipped
JUnit xml allows for a message attribute to be displayed on skips. Lets
populate that with an analysis of why we skipped the test.

llvm-svn: 332156
2018-05-11 23:15:11 +00:00
Chris Matthews
267c8ad156 Overhaul unicode handling in xunit output
I have seen a lot of errors where the xunit does not encode unicode
test output correctly.  Handle that explicitly now.

llvm-svn: 332148
2018-05-11 22:18:22 +00:00
Chris Matthews
c499ebbb98 [LIT] replace output escapes wit a cdata block
CDATA blocks don't need to have XML stuff escaped. Makes sense to wrap
output in them instead of escaping.

llvm-svn: 332116
2018-05-11 18:38:02 +00:00
Chris Matthews
4826565393 Support Unsupported Tests in xunit output
We were reporting  "Unsupported" tests in xunit as passes, however since
they are not run, it make more sense to mark them as skipped. The Junit
xml standard has support for that, so lets use it.

llvm-svn: 332065
2018-05-11 00:25:43 +00:00
Chris Matthews
f5b7f9eb76 Refactor xunit test case builder to not use as much str addition
String concatenation in python is slow.  Refactor to not concatenate the
possibly large strings of test output and instead write them directly
to the output file.

llvm-svn: 332064
2018-05-11 00:25:42 +00:00
Chris Matthews
3dee71508e [LIT] Move xunit tests tests into their own location, and and add failures
Failures will increase coverage.

llvm-svn: 332056
2018-05-10 22:51:28 +00:00
Chris Matthews
c5f2366160 [LIT] Add the missing file
I forgot to commit this file.

llvm-svn: 331946
2018-05-10 00:08:39 +00:00
Chris Matthews
0620816737 Refactor test incase results are backwards
Looks like results can come in either way in this file.  Loosen the ordering constraints.

llvm-svn: 331945
2018-05-10 00:06:17 +00:00
Chris Matthews
a7a8db71f5 [LIT] Handle xml characters in test names
Lit creates malformed xml when the test case has an & in the name.

Escape those correctly.

This also adds a test case which I will add other nasty encoding issues to in some followup commits.

llvm-svn: 331942
2018-05-09 23:48:32 +00:00
Nico Weber
bbbb3edf87 Remove 'abi-breaking-checks' lit feature.
Its only two uses were removed in r311730.
Effectively reverts r304851 (but that code has removed around a bit since then).
https://reviews.llvm.org/D46619

clang side done in r331871.

llvm-svn: 331872
2018-05-09 12:39:39 +00:00
Saleem Abdulrasool
7dafb8329b lit: flesh out SubsituteCaptures further
Add overloads for `__len__` and `__getitem__` to allow use of this class
on Linux as well as Windows.  With these overloads, lit can be used on
both hosts for the swift testsuite.

llvm-svn: 331431
2018-05-03 04:45:43 +00:00
Reid Kleckner
c5a26bc96b Revert r330755 "[lit] Report line number for failed RUN command"
It is causing many tests to fail on Windows buildbots:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211

llvm-svn: 330848
2018-04-25 17:30:00 +00:00
Joel E. Denny
f315d6c63e [lit] Report line number for failed RUN command
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

Reviewed By: asmith, delcypher

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 330755
2018-04-24 18:43:25 +00:00
Dan Liew
a348f9c134 [lit] Remove spurious - in invocation of lit in
`shtest-xunit-output.py` test.

Although there is no `-` file Jeremy Morse has reported to me that it
causes problems in their setup because lit tries to find it and ends up
loading an out of tree lit configuration file.

llvm-svn: 330728
2018-04-24 15:42:00 +00:00
Nico Weber
e478018779 Mostly revert r330672.
The test is apparently needed e.g. for check-cfi on Windows where we get
  'C:/b/slave/sanitizer-windows/build/./bin/clang.exe': command not found
without it.  Try to fix the problem that was fixed by r330672 by also checking
for isabs() instead.

llvm-svn: 330673
2018-04-24 01:24:42 +00:00
Nico Weber
00c5136820 Remove code that's almost always dead, and harmful if not.
lit's util.which() would check if the passed-in path existed directly,
and if so return it as-is.  This is never the case when running llvm's, clang's,
or lld's tests normally.  But when running `./llvm-lit path/to/clang/test`
with a cwd of llvm-build/bin, this if would detect that clang exists at path
'clang' and return 'clang' as the discovered clang binary -- and then lit would
use the " clang " -> "*** Do not use 'clang' in tests, use '%clang'. ***"
substitution to replace that with a broken test.  By removing this early
return, lit ends up with the usual absolute path and everything works even
in this uncommon case.

llvm-svn: 330672
2018-04-24 01:05:04 +00:00
Dan Liew
63cbec945c [lit] Fix a bug where UNRESOLVED tests were not handled in the XUnit
XML printer.

A test has been added that tries to comprehensively test emitting
XUnit XML output for shell tests.

Differential Revision: https://reviews.llvm.org/D45567

llvm-svn: 330409
2018-04-20 10:11:41 +00:00
Aaron Smith
9d297de1c9 [lit] Remove duplicate to_string method
There are two versions of to_string used by TestRunner.py. The one defined 
in TestRunner.py and the one defined in utils/lit/lit/util.py. The util.py
version is superior to the TestRunner.py version.

This change removes the duplicate to_string in TestRunner.py in favor of
always using the version from util.py. Beside removing duplicate code, this
makes it easier to debug TestRunner.py since only one version of to_string
is used.

Patch by Stella Stamenova!

llvm-svn: 329972
2018-04-12 23:45:15 +00:00
Aaron Smith
55218b8d96 [lit] Fix several Python 2/3 compatibility issues and tests
- In Python 2.x, basestring is the base string type, but in 
  Python 3.x basestring is not defined and instead str includes 
  unicode strings.

- When Python is in a path that includes spaces, it needs to 
  be specified with quotes in the test files for it to run.

- The cache.ll test relies on files of a specific size being 
  created by Python, but on some versions of Windows the 
  files that are created by the current code are one byte 
  larger than expected. To fix the test, update file creation 
  to always make files of the expected size.

Patch by Stella Stamenova!

llvm-svn: 329466
2018-04-07 00:21:28 +00:00
Reid Kleckner
d2dd25f14b 'cat' command for internal shell - Support Python 3
LLVM Bug Id : 36449

Revision 328563 caused tests to fail under python 3.

This patch modified cat.py file to support both python 2 and 3.
This patch also fixes CRLF issues on Windows.

Patch by Chamal de Silva

Differential Revision: https://reviews.llvm.org/D45077

llvm-svn: 329123
2018-04-03 22:38:25 +00:00
Aaron Smith
7a1394d5d7 [lit] Prefer opening files with open (Python 2) rather than io.open which requires io.
Only rely on Python 3 (io.open) when necessary. This puts TestRunnyer.py closer to how it behaved 
before the changes introduced in D43165 and silences a few Windows build bot failures.

Thanks to Stella Stamenova for the patch!

llvm-svn: 329037
2018-04-03 00:22:12 +00:00
Aaron Smith
e5a6e6bd67 [lit] One more try at fixing TestRunner.py for D43165
llvm-svn: 329026
2018-04-02 22:34:35 +00:00
Reid Kleckner
8a826a7e76 [lit] Attempt to fix builtin diff code for Python 2
llvm-svn: 329024
2018-04-02 22:19:42 +00:00
Aaron Smith
b62a4d5437 [lit] Fix problem in how Python versions open files with different encodings
Reapply D43165 which was reverted because of different versions of python failing. 
The one line fix for the different python versions was commited at the same time
that D43165 was reverted. If this change is giving you issues then get in touch
with your python version and we will fix it. 

llvm-svn: 329022
2018-04-02 22:08:56 +00:00
Aaron Smith
4964c3831d [lit] Use io.open to compare two files since it supports different encodings while older versions of open do not
llvm-svn: 329020
2018-04-02 21:44:51 +00:00
Reid Kleckner
86303cb33d Revert r329012 "[lit] Fix problem in how Python versions open files with different encodings"
This doesn't work with Python 2. See this build-in-progress:
http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/11105

llvm-svn: 329017
2018-04-02 21:33:46 +00:00
Aaron Smith
da573c6edd [lit] Fix problem in how Python versions open files with different encodings
Summary:
This issue was found when running the clang unit test on Windows. Python 3.x cannot open some of the files that the tests are using with a simple open because of their encoding. Python 2.7+ and Python 3.x both support io.open which allows for an encoding to be specified. 

This change will determine whether two files being compared should be opened (and then compared) as text or binary and whether to use utf-8 or the default encoding before proceeding with a line-by-line comparison.

Patch by Stella Stamenova!

Reviewers: zturner, llvm-commits, rnk, MaggieYi

Reviewed By: zturner

Subscribers: asmith, MatzeB, stella.stamenova, delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D43165

llvm-svn: 329012
2018-04-02 20:57:06 +00:00
Dan Liew
2a0a84a46f Revert "[lit] Temporarily disable shtest-timeout.py on darwin"
This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598)

Hopefully the test will now pass on the bots.

rdar://problem/38774530

llvm-svn: 328703
2018-03-28 13:55:13 +00:00
Dan Liew
a120c804c6 [lit] Remove a timing senstive part of shtest-timeout.py
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.

The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.

We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.

rdar://problem/38774530

llvm-svn: 328702
2018-03-28 13:55:08 +00:00
Mircea Trofin
b836fe81fc Revert "Revert "[lit] Generalized /dev/null support on Windows.""
Summary:
This reverts commit r328596.

Checking if the arguments are strings before testing if they contain "/dev/null".

Reviewers: rnk

Reviewed By: rnk

Subscribers: delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D44914

llvm-svn: 328603
2018-03-27 01:39:17 +00:00
Jan Korous
a54a2fbeca [lit] Temporarily disable shtest-timeout.py on darwin
Disabled until fixed in order to avoid random failures on green dragon.

rdar://problem/38774530

llvm-svn: 328598
2018-03-27 00:16:28 +00:00
Mircea Trofin
8c42530a18 Revert "[lit] Generalized /dev/null support on Windows."
This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92.

llvm-svn: 328596
2018-03-26 23:59:39 +00:00