mirror of
https://github.com/RPCS3/llvm-mirror.git
synced 2024-11-24 11:42:57 +01:00
a50a440b7c
The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
83 lines
3.5 KiB
Python
83 lines
3.5 KiB
Python
# REQUIRES: lit-max-individual-test-time
|
|
|
|
# llvm.org/PR33944
|
|
# UNSUPPORTED: system-windows
|
|
|
|
###############################################################################
|
|
# Check tests can hit timeout when set
|
|
###############################################################################
|
|
|
|
# Test per test timeout using external shell
|
|
# RUN: not %{lit} \
|
|
# RUN: %{inputs}/shtest-timeout/infinite_loop.py \
|
|
# RUN: -j 1 -v --debug --timeout 1 --param external=1 > %t.extsh.out 2> %t.extsh.err
|
|
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON < %t.extsh.out %s
|
|
# RUN: FileCheck --check-prefix=CHECK-EXTSH-ERR < %t.extsh.err %s
|
|
#
|
|
# CHECK-EXTSH-ERR: Using external shell
|
|
|
|
# Test per test timeout using internal shell
|
|
# RUN: not %{lit} \
|
|
# RUN: %{inputs}/shtest-timeout/infinite_loop.py \
|
|
# RUN: -j 1 -v --debug --timeout 1 --param external=0 > %t.intsh.out 2> %t.intsh.err
|
|
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON < %t.intsh.out %s
|
|
# RUN: FileCheck --check-prefix=CHECK-INTSH-OUT < %t.intsh.out %s
|
|
# RUN: FileCheck --check-prefix=CHECK-INTSH-ERR < %t.intsh.err %s
|
|
|
|
# CHECK-INTSH-OUT: TIMEOUT: per_test_timeout :: infinite_loop.py
|
|
# CHECK-INTSH-OUT: command output:
|
|
# CHECK-INTSH-OUT: command reached timeout: True
|
|
|
|
# CHECK-INTSH-ERR: Using internal shell
|
|
|
|
# Test per test timeout set via a config file rather than on the command line
|
|
# RUN: not %{lit} \
|
|
# RUN: %{inputs}/shtest-timeout/infinite_loop.py \
|
|
# RUN: -j 1 -v --debug --param external=0 \
|
|
# RUN: --param set_timeout=1 > %t.cfgset.out 2> %t.cfgset.err
|
|
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON < %t.cfgset.out %s
|
|
# RUN: FileCheck --check-prefix=CHECK-CFGSET-ERR < %t.cfgset.err %s
|
|
#
|
|
# CHECK-CFGSET-ERR: Using internal shell
|
|
|
|
# CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: infinite_loop.py
|
|
# CHECK-OUT-COMMON: Timeout: Reached timeout of 1 seconds
|
|
# CHECK-OUT-COMMON: Command {{([0-9]+ )?}}Output
|
|
# CHECK-OUT-COMMON: Timed Out: 1
|
|
|
|
|
|
###############################################################################
|
|
# Check tests can complete in with a timeout set
|
|
#
|
|
# `short.py` should execute quickly so we shouldn't wait anywhere near the
|
|
# 3600 second timeout.
|
|
###############################################################################
|
|
|
|
# Test per test timeout using external shell
|
|
# RUN: %{lit} \
|
|
# RUN: %{inputs}/shtest-timeout/short.py \
|
|
# RUN: -j 1 -v --debug --timeout 3600 --param external=1 > %t.pass.extsh.out 2> %t.pass.extsh.err
|
|
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON-SHORT < %t.pass.extsh.out %s
|
|
# RUN: FileCheck --check-prefix=CHECK-EXTSH-ERR < %t.pass.extsh.err %s
|
|
|
|
# Test per test timeout using internal shell
|
|
# RUN: %{lit} \
|
|
# RUN: %{inputs}/shtest-timeout/short.py \
|
|
# RUN: -j 1 -v --debug --timeout 3600 --param external=0 > %t.pass.intsh.out 2> %t.pass.intsh.err
|
|
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON-SHORT < %t.pass.intsh.out %s
|
|
# RUN: FileCheck --check-prefix=CHECK-INTSH-ERR < %t.pass.intsh.err %s
|
|
|
|
# CHECK-OUT-COMMON-SHORT: PASS: per_test_timeout :: short.py
|
|
# CHECK-OUT-COMMON-SHORT: Passed: 1
|
|
|
|
# Test per test timeout via a config file and on the command line.
|
|
# The value set on the command line should override the config file.
|
|
# RUN: %{lit} \
|
|
# RUN: %{inputs}/shtest-timeout/short.py \
|
|
# RUN: -j 1 -v --debug --param external=0 \
|
|
# RUN: --param set_timeout=1 --timeout=3600 > %t.pass.cmdover.out 2> %t.pass.cmdover.err
|
|
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON-SHORT < %t.pass.cmdover.out %s
|
|
# RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-ERR < %t.pass.cmdover.err %s
|
|
|
|
# CHECK-CMDLINE-OVERRIDE-ERR: Forcing timeout to be 3600 seconds
|