mirror of
https://github.com/RPCS3/llvm-mirror.git
synced 2024-11-23 03:02:36 +01:00
907 lines
36 KiB
HTML
907 lines
36 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
|
|
"http://www.w3.org/TR/html4/strict.dtd">
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
|
|
<title>LLVM Testing Infrastructure Guide</title>
|
|
<link rel="stylesheet" href="llvm.css" type="text/css">
|
|
</head>
|
|
<body>
|
|
|
|
<h1>
|
|
LLVM Testing Infrastructure Guide
|
|
</h1>
|
|
|
|
<ol>
|
|
<li><a href="#overview">Overview</a></li>
|
|
<li><a href="#requirements">Requirements</a></li>
|
|
<li><a href="#org">LLVM testing infrastructure organization</a>
|
|
<ul>
|
|
<li><a href="#regressiontests">Regression tests</a></li>
|
|
<li><a href="#testsuite"><tt>test-suite</tt></a></li>
|
|
<li><a href="#debuginfotests">Debugging Information tests</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a href="#quick">Quick start</a>
|
|
<ul>
|
|
<li><a href="#quickregressiontests">Regression tests</a></li>
|
|
<li><a href="#quickdebuginfotests">Debugging Information tests</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a href="#rtstructure">Regression test structure</a>
|
|
<ul>
|
|
<li><a href="#rtcustom">Writing new regression tests</a></li>
|
|
<li><a href="#FileCheck">The FileCheck utility</a></li>
|
|
<li><a href="#rtvars">Variables and substitutions</a></li>
|
|
<li><a href="#rtfeatures">Other features</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a href="#testsuiteoverview"><tt>test-suite</tt> Overview</a>
|
|
<ul>
|
|
<li><a href="#testsuitequickstart"><tt>test-suite</tt> Quickstart</a></li>
|
|
<li><a href="#testsuitemakefiles"><tt>test-suite</tt> Makefiles</a></li>
|
|
</ul>
|
|
</li>
|
|
</ol>
|
|
|
|
<div class="doc_author">
|
|
<p>Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner</p>
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="overview">Overview</a></h2>
|
|
<!--=========================================================================-->
|
|
|
|
<div>
|
|
|
|
<p>This document is the reference manual for the LLVM testing infrastructure. It
|
|
documents the structure of the LLVM testing infrastructure, the tools needed to
|
|
use it, and how to add and run tests.</p>
|
|
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="requirements">Requirements</a></h2>
|
|
<!--=========================================================================-->
|
|
|
|
<div>
|
|
|
|
<p>In order to use the LLVM testing infrastructure, you will need all of the
|
|
software required to build LLVM, as well
|
|
as <a href="http://python.org">Python</a> 2.4 or later.</p>
|
|
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="org">LLVM testing infrastructure organization</a></h2>
|
|
<!--=========================================================================-->
|
|
|
|
<div>
|
|
|
|
<p>The LLVM testing infrastructure contains two major categories of tests:
|
|
regression tests and whole programs. The regression tests are contained inside
|
|
the LLVM repository itself under <tt>llvm/test</tt> and are expected to always
|
|
pass -- they should be run before every commit.</p>
|
|
|
|
<p>The whole programs tests are referred to as the "LLVM test suite" (or
|
|
"test-suite") and are in the <tt>test-suite</tt> module in subversion. For
|
|
historical reasons, these tests are also referred to as the "nightly tests" in
|
|
places, which is less ambiguous than "test-suite" and remains in use although we
|
|
run them much more often than nightly.</p>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="regressiontests">Regression tests</a></h3>
|
|
<!-- _______________________________________________________________________ -->
|
|
|
|
<div>
|
|
|
|
<p>The regression tests are small pieces of code that test a specific feature of
|
|
LLVM or trigger a specific bug in LLVM. They are usually written in LLVM
|
|
assembly language, but can be written in other languages if the test targets a
|
|
particular language front end (and the appropriate <tt>--with-llvmgcc</tt>
|
|
options were used at <tt>configure</tt> time of the <tt>llvm</tt> module). These
|
|
tests are driven by the 'lit' testing tool, which is part of LLVM.</p>
|
|
|
|
<p>These code fragments are not complete programs. The code generated
|
|
from them is never executed to determine correct behavior.</p>
|
|
|
|
<p>These code fragment tests are located in the <tt>llvm/test</tt>
|
|
directory.</p>
|
|
|
|
<p>Typically when a bug is found in LLVM, a regression test containing
|
|
just enough code to reproduce the problem should be written and placed
|
|
somewhere underneath this directory. In most cases, this will be a small
|
|
piece of LLVM assembly language code, often distilled from an actual
|
|
application or benchmark.</p>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="testsuite"><tt>test-suite</tt></a></h3>
|
|
<!-- _______________________________________________________________________ -->
|
|
|
|
<div>
|
|
|
|
<p>The test suite contains whole programs, which are pieces of code which can be
|
|
compiled and linked into a stand-alone program that can be executed. These
|
|
programs are generally written in high level languages such as C or C++.</p>
|
|
|
|
<p>These programs are compiled using a user specified compiler and set of flags,
|
|
and then executed to capture the program output and timing information. The
|
|
output of these programs is compared to a reference output to ensure that the
|
|
program is being compiled correctly.</p>
|
|
|
|
<p>In addition to compiling and executing programs, whole program tests serve as
|
|
a way of benchmarking LLVM performance, both in terms of the efficiency of the
|
|
programs generated as well as the speed with which LLVM compiles, optimizes, and
|
|
generates code.</p>
|
|
|
|
<p>The test-suite is located in the <tt>test-suite</tt> Subversion module.</p>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="debuginfotests">Debugging Information tests</a></h3>
|
|
<!-- _______________________________________________________________________ -->
|
|
|
|
<div>
|
|
|
|
<p>The test suite contains tests to check quality of debugging information.
|
|
The test are written in C based languages or in LLVM assembly language. </p>
|
|
|
|
<p>These tests are compiled and run under a debugger. The debugger output
|
|
is checked to validate of debugging information. See README.txt in the
|
|
test suite for more information . This test suite is located in the
|
|
<tt>debuginfo-tests</tt> Subversion module. </p>
|
|
|
|
</div>
|
|
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="quick">Quick start</a></h2>
|
|
<!--=========================================================================-->
|
|
|
|
<div>
|
|
|
|
<p>The tests are located in two separate Subversion modules. The regressions
|
|
tests are in the main "llvm" module under the directory
|
|
<tt>llvm/test</tt> (so you get these tests for free with the main llvm
|
|
tree). Use "make check-all" to run the regression tests after building
|
|
LLVM.</p>
|
|
|
|
<p>The more comprehensive test suite that includes whole programs in C and C++
|
|
is in the <tt>test-suite</tt>
|
|
module. See <a href="#testsuitequickstart"><tt>test-suite</tt> Quickstart</a>
|
|
for more information on running these tests.</p>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="quickregressiontests">Regression tests</a></h3>
|
|
<div>
|
|
<!-- _______________________________________________________________________ -->
|
|
<p>To run all of the LLVM regression tests, use master Makefile in
|
|
the <tt>llvm/test</tt> directory:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
% gmake -C llvm/test
|
|
</pre>
|
|
</div>
|
|
|
|
<p>or</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
% gmake check
|
|
</pre>
|
|
</div>
|
|
|
|
<p>If you have <a href="http://clang.llvm.org/">Clang</a> checked out and built,
|
|
you can run the LLVM and Clang tests simultaneously using:</p>
|
|
|
|
<p>or</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
% gmake check-all
|
|
</pre>
|
|
</div>
|
|
|
|
<p>To run the tests with Valgrind (Memcheck by default), just append
|
|
<tt>VG=1</tt> to the commands above, e.g.:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
% gmake check VG=1
|
|
</pre>
|
|
</div>
|
|
|
|
<p>To run individual tests or subsets of tests, you can use the 'llvm-lit'
|
|
script which is built as part of LLVM. For example, to run the
|
|
'Integer/BitCast.ll' test by itself you can run:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
% llvm-lit ~/llvm/test/Integer/BitCast.ll
|
|
</pre>
|
|
</div>
|
|
|
|
<p>or to run all of the ARM CodeGen tests:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
% llvm-lit ~/llvm/test/CodeGen/ARM
|
|
</pre>
|
|
</div>
|
|
|
|
<p>For more information on using the 'lit' tool, see 'llvm-lit --help' or the
|
|
'lit' man page.</p>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="quickdebuginfotests">Debugging Information tests</a></h3>
|
|
<div>
|
|
<!-- _______________________________________________________________________ -->
|
|
<div>
|
|
|
|
<p> To run debugging information tests simply checkout the tests inside
|
|
clang/test directory. </p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
%cd clang/test
|
|
% svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
|
|
</pre>
|
|
</div>
|
|
|
|
<p> These tests are already set up to run as part of clang regression tests.</p>
|
|
|
|
</div>
|
|
|
|
</div>
|
|
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="rtstructure">Regression test structure</a></h2>
|
|
<!--=========================================================================-->
|
|
<div>
|
|
<p>The LLVM regression tests are driven by 'lit' and are located in
|
|
the <tt>llvm/test</tt> directory.
|
|
|
|
<p>This directory contains a large array of small tests
|
|
that exercise various features of LLVM and to ensure that regressions do not
|
|
occur. The directory is broken into several sub-directories, each focused on
|
|
a particular area of LLVM. A few of the important ones are:</p>
|
|
|
|
<ul>
|
|
<li><tt>Analysis</tt>: checks Analysis passes.</li>
|
|
<li><tt>Archive</tt>: checks the Archive library.</li>
|
|
<li><tt>Assembler</tt>: checks Assembly reader/writer functionality.</li>
|
|
<li><tt>Bitcode</tt>: checks Bitcode reader/writer functionality.</li>
|
|
<li><tt>CodeGen</tt>: checks code generation and each target.</li>
|
|
<li><tt>Features</tt>: checks various features of the LLVM language.</li>
|
|
<li><tt>Linker</tt>: tests bitcode linking.</li>
|
|
<li><tt>Transforms</tt>: tests each of the scalar, IPO, and utility
|
|
transforms to ensure they make the right transformations.</li>
|
|
<li><tt>Verifier</tt>: tests the IR verifier.</li>
|
|
</ul>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="rtcustom">Writing new regression tests</a></h3>
|
|
<!-- _______________________________________________________________________ -->
|
|
<div>
|
|
<p>The regression test structure is very simple, but does require some
|
|
information to be set. This information is gathered via <tt>configure</tt> and
|
|
is written to a file, <tt>lit.site.cfg</tt>
|
|
in <tt>llvm/test</tt>. The <tt>llvm/test</tt> Makefile does this work for
|
|
you.</p>
|
|
|
|
<p>In order for the regression tests to work, each directory of tests must
|
|
have a <tt>lit.local.cfg</tt> file. Lit looks for this file to determine how
|
|
to run the tests. This file is just Python code and thus is very flexible,
|
|
but we've standardized it for the LLVM regression tests. If you're adding a
|
|
directory of tests, just copy <tt>lit.local.cfg</tt> from another directory to
|
|
get running. The standard <tt>lit.local.cfg</tt> simply specifies which files
|
|
to look in for tests. Any directory that contains only directories does not
|
|
need the <tt>lit.local.cfg</tt> file. Read the
|
|
<a href="http://llvm.org/cmds/lit.html">Lit documentation</a> for more
|
|
information. </p>
|
|
|
|
<p>The <tt>llvm-runtests</tt> function looks at each file that is passed to
|
|
it and gathers any lines together that match "RUN:". These are the "RUN" lines
|
|
that specify how the test is to be run. So, each test script must contain
|
|
RUN lines if it is to do anything. If there are no RUN lines, the
|
|
<tt>llvm-runtests</tt> function will issue an error and the test will
|
|
fail.</p>
|
|
|
|
<p>RUN lines are specified in the comments of the test program using the
|
|
keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline)
|
|
to execute. Together, these lines form the "script" that
|
|
<tt>llvm-runtests</tt> executes to run the test case. The syntax of the
|
|
RUN lines is similar to a shell's syntax for pipelines including I/O
|
|
redirection and variable substitution. However, even though these lines
|
|
may <i>look</i> like a shell script, they are not. RUN lines are interpreted
|
|
directly by the Tcl <tt>exec</tt> command. They are never executed by a
|
|
shell. Consequently the syntax differs from normal shell script syntax in a
|
|
few ways. You can specify as many RUN lines as needed.</p>
|
|
|
|
<p>lit performs substitution on each RUN line to replace LLVM tool
|
|
names with the full paths to the executable built for each tool (in
|
|
$(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit does not
|
|
invoke any stray LLVM tools in the user's path during testing.</p>
|
|
|
|
<p>Each RUN line is executed on its own, distinct from other lines unless
|
|
its last character is <tt>\</tt>. This continuation character causes the RUN
|
|
line to be concatenated with the next one. In this way you can build up long
|
|
pipelines of commands without making huge line lengths. The lines ending in
|
|
<tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is
|
|
found. This concatenated set of RUN lines then constitutes one execution.
|
|
Tcl will substitute variables and arrange for the pipeline to be executed. If
|
|
any process in the pipeline fails, the entire line (and test case) fails too.
|
|
</p>
|
|
|
|
<p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
; RUN: llvm-as < %s | llvm-dis > %t1
|
|
; RUN: llvm-dis < %s.bc-13 > %t2
|
|
; RUN: diff %t1 %t2
|
|
</pre>
|
|
</div>
|
|
|
|
<p>As with a Unix shell, the RUN: lines permit pipelines and I/O redirection
|
|
to be used. However, the usage is slightly different than for Bash. To check
|
|
what's legal, see the documentation for the
|
|
<a href="http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2">Tcl exec</a>
|
|
command and the
|
|
<a href="http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html">tutorial</a>.
|
|
The major differences are:</p>
|
|
<ul>
|
|
<li>You can't do <tt>2>&1</tt>. That will cause Tcl to write to a
|
|
file named <tt>&1</tt>. Usually this is done to get stderr to go through
|
|
a pipe. You can do that in tcl with <tt>|&</tt> so replace this idiom:
|
|
<tt>... 2>&1 | grep</tt> with <tt>... |& grep</tt></li>
|
|
<li>You can only redirect to a file, not to another descriptor and not from
|
|
a here document.</li>
|
|
<li>tcl supports redirecting to open files with the @ syntax but you
|
|
shouldn't use that here.</li>
|
|
</ul>
|
|
|
|
<p>There are some quoting rules that you must pay attention to when writing
|
|
your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any
|
|
quote characters so they will get passed to the invoked program. For
|
|
example:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
... | grep 'find this string'
|
|
</pre>
|
|
</div>
|
|
|
|
<p>This will fail because the ' characters are passed to grep. This would
|
|
instruction grep to look for <tt>'find</tt> in the files <tt>this</tt> and
|
|
<tt>string'</tt>. To avoid this use curly braces to tell Tcl that it should
|
|
treat everything enclosed as one value. So our example would become:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
... | grep {find this string}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>Additionally, the characters <tt>[</tt> and <tt>]</tt> are treated
|
|
specially by Tcl. They tell Tcl to interpret the content as a command to
|
|
execute. Since these characters are often used in regular expressions this can
|
|
have disastrous results and cause the entire test run in a directory to fail.
|
|
For example, a common idiom is to look for some basicblock number:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
... | grep bb[2-8]
|
|
</pre>
|
|
</div>
|
|
|
|
<p>This, however, will cause Tcl to fail because its going to try to execute
|
|
a program named "2-8". Instead, what you want is this:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
... | grep {bb\[2-8\]}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>Finally, if you need to pass the <tt>\</tt> character down to a program,
|
|
then it must be doubled. This is another Tcl special character. So, suppose
|
|
you had:
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
... | grep 'i32\*'
|
|
</pre>
|
|
</div>
|
|
|
|
<p>This will fail to match what you want (a pointer to i32). First, the
|
|
<tt>'</tt> do not get stripped off. Second, the <tt>\</tt> gets stripped off
|
|
by Tcl so what grep sees is: <tt>'i32*'</tt>. That's not likely to match
|
|
anything. To resolve this you must use <tt>\\</tt> and the <tt>{}</tt>, like
|
|
this:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
... | grep {i32\\*}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>If your system includes GNU <tt>grep</tt>, make sure
|
|
that <tt>GREP_OPTIONS</tt> is not set in your environment. Otherwise,
|
|
you may get invalid results (both false positives and false
|
|
negatives).</p>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="FileCheck">The FileCheck utility</a></h3>
|
|
<!-- _______________________________________________________________________ -->
|
|
|
|
<div>
|
|
|
|
<p>A powerful feature of the RUN: lines is that it allows any arbitrary commands
|
|
to be executed as part of the test harness. While standard (portable) unix
|
|
tools like 'grep' work fine on run lines, as you see above, there are a lot
|
|
of caveats due to interaction with Tcl syntax, and we want to make sure the
|
|
run lines are portable to a wide range of systems. Another major problem is
|
|
that grep is not very good at checking to verify that the output of a tools
|
|
contains a series of different output in a specific order. The FileCheck
|
|
tool was designed to help with these problems.</p>
|
|
|
|
<p>FileCheck (whose basic command line arguments are described in <a
|
|
href="http://llvm.org/cmds/FileCheck.html">the FileCheck man page</a> is
|
|
designed to read a file to check from standard input, and the set of things
|
|
to verify from a file specified as a command line argument. A simple example
|
|
of using FileCheck from a RUN line looks like this:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
; RUN: llvm-as < %s | llc -march=x86-64 | <b>FileCheck %s</b>
|
|
</pre>
|
|
</div>
|
|
|
|
<p>This syntax says to pipe the current file ("%s") into llvm-as, pipe that into
|
|
llc, then pipe the output of llc into FileCheck. This means that FileCheck will
|
|
be verifying its standard input (the llc output) against the filename argument
|
|
specified (the original .ll file specified by "%s"). To see how this works,
|
|
let's look at the rest of the .ll file (after the RUN line):</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
define void @sub1(i32* %p, i32 %v) {
|
|
entry:
|
|
; <b>CHECK: sub1:</b>
|
|
; <b>CHECK: subl</b>
|
|
%0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
|
|
ret void
|
|
}
|
|
|
|
define void @inc4(i64* %p) {
|
|
entry:
|
|
; <b>CHECK: inc4:</b>
|
|
; <b>CHECK: incq</b>
|
|
%0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
|
|
ret void
|
|
}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>Here you can see some "CHECK:" lines specified in comments. Now you can see
|
|
how the file is piped into llvm-as, then llc, and the machine code output is
|
|
what we are verifying. FileCheck checks the machine code output to verify that
|
|
it matches what the "CHECK:" lines specify.</p>
|
|
|
|
<p>The syntax of the CHECK: lines is very simple: they are fixed strings that
|
|
must occur in order. FileCheck defaults to ignoring horizontal whitespace
|
|
differences (e.g. a space is allowed to match a tab) but otherwise, the contents
|
|
of the CHECK: line is required to match some thing in the test file exactly.</p>
|
|
|
|
<p>One nice thing about FileCheck (compared to grep) is that it allows merging
|
|
test cases together into logical groups. For example, because the test above
|
|
is checking for the "sub1:" and "inc4:" labels, it will not match unless there
|
|
is a "subl" in between those labels. If it existed somewhere else in the file,
|
|
that would not count: "grep subl" matches if subl exists anywhere in the
|
|
file.</p>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h4>
|
|
<a name="FileCheck-check-prefix">The FileCheck -check-prefix option</a>
|
|
</h4>
|
|
|
|
<div>
|
|
|
|
<p>The FileCheck -check-prefix option allows multiple test configurations to be
|
|
driven from one .ll file. This is useful in many circumstances, for example,
|
|
testing different architectural variants with llc. Here's a simple example:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
|
|
; RUN: | <b>FileCheck %s -check-prefix=X32</b>
|
|
; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
|
|
; RUN: | <b>FileCheck %s -check-prefix=X64</b>
|
|
|
|
define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
|
|
%tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
|
|
ret <4 x i32> %tmp1
|
|
; <b>X32:</b> pinsrd_1:
|
|
; <b>X32:</b> pinsrd $1, 4(%esp), %xmm0
|
|
|
|
; <b>X64:</b> pinsrd_1:
|
|
; <b>X64:</b> pinsrd $1, %edi, %xmm0
|
|
}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>In this case, we're testing that we get the expected code generation with
|
|
both 32-bit and 64-bit code generation.</p>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h4>
|
|
<a name="FileCheck-CHECK-NEXT">The "CHECK-NEXT:" directive</a>
|
|
</h4>
|
|
|
|
<div>
|
|
|
|
<p>Sometimes you want to match lines and would like to verify that matches
|
|
happen on exactly consecutive lines with no other lines in between them. In
|
|
this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If
|
|
you specified a custom check prefix, just use "<PREFIX>-NEXT:". For
|
|
example, something like this works as you'd expect:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) {
|
|
%tmp3 = load <2 x double>* %A, align 16
|
|
%tmp7 = insertelement <2 x double> undef, double %B, i32 0
|
|
%tmp9 = shufflevector <2 x double> %tmp3,
|
|
<2 x double> %tmp7,
|
|
<2 x i32> < i32 0, i32 2 >
|
|
store <2 x double> %tmp9, <2 x double>* %r, align 16
|
|
ret void
|
|
|
|
; <b>CHECK:</b> t2:
|
|
; <b>CHECK:</b> movl 8(%esp), %eax
|
|
; <b>CHECK-NEXT:</b> movapd (%eax), %xmm0
|
|
; <b>CHECK-NEXT:</b> movhpd 12(%esp), %xmm0
|
|
; <b>CHECK-NEXT:</b> movl 4(%esp), %eax
|
|
; <b>CHECK-NEXT:</b> movapd %xmm0, (%eax)
|
|
; <b>CHECK-NEXT:</b> ret
|
|
}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>CHECK-NEXT: directives reject the input unless there is exactly one newline
|
|
between it an the previous directive. A CHECK-NEXT cannot be the first
|
|
directive in a file.</p>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h4>
|
|
<a name="FileCheck-CHECK-NOT">The "CHECK-NOT:" directive</a>
|
|
</h4>
|
|
|
|
<div>
|
|
|
|
<p>The CHECK-NOT: directive is used to verify that a string doesn't occur
|
|
between two matches (or the first match and the beginning of the file). For
|
|
example, to verify that a load is removed by a transformation, a test like this
|
|
can be used:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
define i8 @coerce_offset0(i32 %V, i32* %P) {
|
|
store i32 %V, i32* %P
|
|
|
|
%P2 = bitcast i32* %P to i8*
|
|
%P3 = getelementptr i8* %P2, i32 2
|
|
|
|
%A = load i8* %P3
|
|
ret i8 %A
|
|
; <b>CHECK:</b> @coerce_offset0
|
|
; <b>CHECK-NOT:</b> load
|
|
; <b>CHECK:</b> ret i8
|
|
}
|
|
</pre>
|
|
</div>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h4>
|
|
<a name="FileCheck-Matching">FileCheck Pattern Matching Syntax</a>
|
|
</h4>
|
|
|
|
<div>
|
|
|
|
<p>The CHECK: and CHECK-NOT: directives both take a pattern to match. For most
|
|
uses of FileCheck, fixed string matching is perfectly sufficient. For some
|
|
things, a more flexible form of matching is desired. To support this, FileCheck
|
|
allows you to specify regular expressions in matching strings, surrounded by
|
|
double braces: <b>{{yourregex}}</b>. Because we want to use fixed string
|
|
matching for a majority of what we do, FileCheck has been designed to support
|
|
mixing and matching fixed string matching with regular expressions. This allows
|
|
you to write things like this:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
; CHECK: movhpd <b>{{[0-9]+}}</b>(%esp), <b>{{%xmm[0-7]}}</b>
|
|
</pre>
|
|
</div>
|
|
|
|
<p>In this case, any offset from the ESP register will be allowed, and any xmm
|
|
register will be allowed.</p>
|
|
|
|
<p>Because regular expressions are enclosed with double braces, they are
|
|
visually distinct, and you don't need to use escape characters within the double
|
|
braces like you would in C. In the rare case that you want to match double
|
|
braces explicitly from the input, you can use something ugly like
|
|
<b>{{[{][{]}}</b> as your pattern.</p>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h4>
|
|
<a name="FileCheck-Variables">FileCheck Variables</a>
|
|
</h4>
|
|
|
|
<div>
|
|
|
|
<p>It is often useful to match a pattern and then verify that it occurs again
|
|
later in the file. For codegen tests, this can be useful to allow any register,
|
|
but verify that that register is used consistently later. To do this, FileCheck
|
|
allows named variables to be defined and substituted into patterns. Here is a
|
|
simple example:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
; CHECK: test5:
|
|
; CHECK: notw <b>[[REGISTER:%[a-z]+]]</b>
|
|
; CHECK: andw {{.*}}<b>[[REGISTER]]</b>
|
|
</pre>
|
|
</div>
|
|
|
|
<p>The first check line matches a regex (<tt>%[a-z]+</tt>) and captures it into
|
|
the variables "REGISTER". The second line verifies that whatever is in REGISTER
|
|
occurs later in the file after an "andw". FileCheck variable references are
|
|
always contained in <tt>[[ ]]</tt> pairs, are named, and their names can be
|
|
formed with the regex "<tt>[a-zA-Z][a-zA-Z0-9]*</tt>". If a colon follows the
|
|
name, then it is a definition of the variable, if not, it is a use.</p>
|
|
|
|
<p>FileCheck variables can be defined multiple times, and uses always get the
|
|
latest value. Note that variables are all read at the start of a "CHECK" line
|
|
and are all defined at the end. This means that if you have something like
|
|
"<tt>CHECK: [[XYZ:.*]]x[[XYZ]]</tt>" that the check line will read the previous
|
|
value of the XYZ variable and define a new one after the match is performed. If
|
|
you need to do something like this you can probably take advantage of the fact
|
|
that FileCheck is not actually line-oriented when it matches, this allows you to
|
|
define two separate CHECK lines that match on the same line.
|
|
</p>
|
|
|
|
</div>
|
|
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="rtvars">Variables and substitutions</a></h3>
|
|
<!-- _______________________________________________________________________ -->
|
|
<div>
|
|
<p>With a RUN line there are a number of substitutions that are permitted. In
|
|
general, any Tcl variable that is available in the <tt>substitute</tt>
|
|
function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
|
|
To make a substitution just write the variable's name preceded by a $.
|
|
Additionally, for compatibility reasons with previous versions of the test
|
|
library, certain names can be accessed with an alternate syntax: a % prefix.
|
|
These alternates are deprecated and may go away in a future version.
|
|
</p>
|
|
<p>Here are the available variable names. The alternate syntax is listed in
|
|
parentheses.</p>
|
|
|
|
<dl style="margin-left: 25px">
|
|
<dt><b>$test</b> (%s)</dt>
|
|
<dd>The full path to the test case's source. This is suitable for passing
|
|
on the command line as the input to an llvm tool.</dd>
|
|
|
|
<dt><b>$srcdir</b></dt>
|
|
<dd>The source directory from where the "<tt>make check</tt>" was run.</dd>
|
|
|
|
<dt><b>objdir</b></dt>
|
|
<dd>The object directory that corresponds to the <tt>$srcdir</tt>.</dd>
|
|
|
|
<dt><b>subdir</b></dt>
|
|
<dd>A partial path from the <tt>test</tt> directory that contains the
|
|
sub-directory that contains the test source being executed.</dd>
|
|
|
|
<dt><b>srcroot</b></dt>
|
|
<dd>The root directory of the LLVM src tree.</dd>
|
|
|
|
<dt><b>objroot</b></dt>
|
|
<dd>The root directory of the LLVM object tree. This could be the same
|
|
as the srcroot.</dd>
|
|
|
|
<dt><b>path</b><dt>
|
|
<dd>The path to the directory that contains the test case source. This is
|
|
for locating any supporting files that are not generated by the test, but
|
|
used by the test.</dd>
|
|
|
|
<dt><b>tmp</b></dt>
|
|
<dd>The path to a temporary file name that could be used for this test case.
|
|
The file name won't conflict with other test cases. You can append to it if
|
|
you need multiple temporaries. This is useful as the destination of some
|
|
redirected output.</dd>
|
|
|
|
<dt><b>target_triplet</b> (%target_triplet)</dt>
|
|
<dd>The target triplet that corresponds to the current host machine (the one
|
|
running the test cases). This should probably be called "host".<dd>
|
|
|
|
<dt><b>link</b> (%link)</dt>
|
|
<dd>This full link command used to link LLVM executables. This has all the
|
|
configured -I, -L and -l options.</dd>
|
|
|
|
<dt><b>shlibext</b> (%shlibext)</dt>
|
|
<dd>The suffix for the host platforms share library (dll) files. This
|
|
includes the period as the first character.</dd>
|
|
</dl>
|
|
<p>To add more variables, two things need to be changed. First, add a line in
|
|
the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will
|
|
"set" the variable as a global in the site.exp file. Second, in the
|
|
<tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name
|
|
to the list of "global" declarations at the beginning of the proc. That's it,
|
|
the variable can then be used in test scripts.</p>
|
|
</div>
|
|
|
|
<!-- _______________________________________________________________________ -->
|
|
<h3><a name="rtfeatures">Other Features</a></h3>
|
|
<!-- _______________________________________________________________________ -->
|
|
<div>
|
|
<p>To make RUN line writing easier, there are several shell scripts located
|
|
in the <tt>llvm/test/Scripts</tt> directory. This directory is in the PATH
|
|
when running tests, so you can just call these scripts using their name. For
|
|
example:</p>
|
|
<dl>
|
|
<dt><b>ignore</b></dt>
|
|
<dd>This script runs its arguments and then always returns 0. This is useful
|
|
in cases where the test needs to cause a tool to generate an error (e.g. to
|
|
check the error output). However, any program in a pipeline that returns a
|
|
non-zero result will cause the test to fail. This script overcomes that
|
|
issue and nicely documents that the test case is purposefully ignoring the
|
|
result code of the tool</dd>
|
|
|
|
<dt><b>not</b></dt>
|
|
<dd>This script runs its arguments and then inverts the result code from
|
|
it. Zero result codes become 1. Non-zero result codes become 0. This is
|
|
useful to invert the result of a grep. For example "not grep X" means
|
|
succeed only if you don't find X in the input.</dd>
|
|
</dl>
|
|
|
|
<p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
|
|
You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a
|
|
line near the top of the file. This signals that the test case should succeed
|
|
if the test fails. Such test cases are counted separately by the testing tool. To
|
|
specify an expected fail, use the XFAIL keyword in the comments of the test
|
|
program followed by a colon and one or more regular expressions (separated by
|
|
a comma). The regular expressions allow you to XFAIL the test conditionally by
|
|
host platform. The regular expressions following the : are matched against the
|
|
target triplet for the host machine. If there is a match, the test is expected
|
|
to fail. If not, the test is expected to succeed. To XFAIL everywhere just
|
|
specify <tt>XFAIL: *</tt>. Here is an example of an <tt>XFAIL</tt> line:</p>
|
|
|
|
<div class="doc_code">
|
|
<pre>
|
|
; XFAIL: darwin,sun
|
|
</pre>
|
|
</div>
|
|
|
|
<p>To make the output more useful, the <tt>llvm_runtest</tt> function wil
|
|
scan the lines of the test case for ones that contain a pattern that matches
|
|
PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
|
|
is related to the test case. The number after "PR" specifies the LLVM bugzilla
|
|
number. When a PR number is specified, it will be used in the pass/fail
|
|
reporting. This is useful to quickly get some context when a test fails.</p>
|
|
|
|
<p>Finally, any line that contains "END." will cause the special
|
|
interpretation of lines to terminate. This is generally done right after the
|
|
last RUN: line. This has two side effects: (a) it prevents special
|
|
interpretation of lines that are part of the test program, not the
|
|
instructions to the test case, and (b) it speeds things up for really big test
|
|
cases by avoiding interpretation of the remainder of the file.</p>
|
|
|
|
</div>
|
|
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="testsuiteoverview"><tt>test-suite</tt> Overview</a></h2>
|
|
<!--=========================================================================-->
|
|
|
|
<div>
|
|
|
|
<p>The <tt>test-suite</tt> module contains a number of programs that can be
|
|
compiled and executed. The <tt>test-suite</tt> includes reference outputs for
|
|
all of the programs, so that the output of the executed program can be checked
|
|
for correctness.</p>
|
|
|
|
<p><tt>test-suite</tt> tests are divided into three types of tests: MultiSource,
|
|
SingleSource, and External.</p>
|
|
|
|
<ul>
|
|
<li><tt>test-suite/SingleSource</tt>
|
|
<p>The SingleSource directory contains test programs that are only a single
|
|
source file in size. These are usually small benchmark programs or small
|
|
programs that calculate a particular value. Several such programs are grouped
|
|
together in each directory.</p></li>
|
|
|
|
<li><tt>test-suite/MultiSource</tt>
|
|
<p>The MultiSource directory contains subdirectories which contain entire
|
|
programs with multiple source files. Large benchmarks and whole applications
|
|
go here.</p></li>
|
|
|
|
<li><tt>test-suite/External</tt>
|
|
<p>The External directory contains Makefiles for building code that is external
|
|
to (i.e., not distributed with) LLVM. The most prominent members of this
|
|
directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt>
|
|
directory does not contain these actual tests, but only the Makefiles that know
|
|
how to properly compile these programs from somewhere else. When
|
|
using <tt>LNT</tt>, use the <tt>--test-externals</tt> option to include these
|
|
tests in the results.</p></li>
|
|
</ul>
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="testsuitequickstart"><tt>test-suite</tt> Quickstart</a></h2>
|
|
<!--=========================================================================-->
|
|
|
|
<div>
|
|
<p>The modern way of running the <tt>test-suite</tt> is focused on testing and
|
|
benchmarking complete compilers using
|
|
the <a href="http://llvm.org/docs/lnt">LNT</a> testing infrastructure.</p>
|
|
|
|
<p>For more information on using LNT to execute the <tt>test-suite</tt>, please
|
|
see the <a href="http://llvm.org/docs/lnt/quickstart.html">LNT Quickstart</a>
|
|
documentation.</p>
|
|
</div>
|
|
|
|
<!--=========================================================================-->
|
|
<h2><a name="testsuitemakefiles"><tt>test-suite</tt> Makefiles</a></h2>
|
|
<!--=========================================================================-->
|
|
|
|
<div>
|
|
<p>Historically, the <tt>test-suite</tt> was executed using a complicated setup
|
|
of Makefiles. The LNT based approach above is recommended for most users, but
|
|
there are some testing scenarios which are not supported by the LNT approach. In
|
|
addition, LNT currently uses the Makefile setup under the covers and so
|
|
developers who are interested in how LNT works under the hood may want to
|
|
understand the Makefile based setup.</p>
|
|
|
|
<p>For more information on the <tt>test-suite</tt> Makefile setup, please see
|
|
the <a href="TestSuiteMakefileGuide.html">Test Suite Makefile Guide.</a></p>
|
|
</div>
|
|
|
|
<!-- *********************************************************************** -->
|
|
|
|
<hr>
|
|
<address>
|
|
<a href="http://jigsaw.w3.org/css-validator/check/referer"><img
|
|
src="http://jigsaw.w3.org/css-validator/images/vcss-blue" alt="Valid CSS"></a>
|
|
<a href="http://validator.w3.org/check/referer"><img
|
|
src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a>
|
|
|
|
John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner<br>
|
|
<a href="http://llvm.org/">The LLVM Compiler Infrastructure</a><br>
|
|
Last modified: $Date$
|
|
</address>
|
|
</body>
|
|
</html>
|