RPCS3/llvm-mirror
1
0
Fork 0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-11-23 11:13:28 +01:00
Code Issues Projects Releases Wiki Activity
4f02bdce75
llvm-mirror/utils/lit/tests/shtest-timeout.py

81 lines
3.4 KiB
Python
Raw Normal View History

[AIX][lit] Don't depend on psutil on AIX Summary: On AIX psutil can run into problems with permissions to read the process tree, which causes problems for python timeout tests which need to kill off a test and it's children. This patch adds a workaround by invoking shell via subprocess and using a platform specific option to ps to list all the descendant processes so we can kill them. We add some checks so lit can tell whether timeout tests are supported with out exposing whether we are utilizing the psutil implementation or the alternative. Reviewers: hubert.reinterpretcast, andusy, davide, delcypher Reviewed By: delcypher Subscribers: davide, delcypher, christof, lldb-commits, libcxx-commits, llvm-commits Tags: #lldb, #libc, #llvm Differential Revision: https://reviews.llvm.org/D64251 llvm-svn: 366912
2019-07-24 17:04:27 +02:00
# REQUIRES: lit-max-individual-test-time
Revert "[lit] Temporarily disable shtest-timeout.py on darwin" This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598) Hopefully the test will now pass on the bots. rdar://problem/38774530 llvm-svn: 328703
2018-03-28 15:55:13 +02:00
[lit] Disable shtest-timeout on Windows Summary: This is the only test that is still failing on Windows - or rather, it is expected to fail on the bots, but passes on the new bot that we're preparing causing a failure, so I'm going to disable it. Since the test has rarely, if ever, passed on the bots, this should have the same effect and it will unblock the creation of the new bot. Reviewers: asmith, delcypher, zturner Subscribers: stella.stamenova, llvm-commits Differential Revision: https://reviews.llvm.org/D51871 llvm-svn: 341856
2018-09-10 22:24:05 +02:00
# llvm.org/PR33944
lit: modernize the lit configuration for the lit tests Summary: This also normalizes the config feature that represents the windows platform to "system-windows" as opposed to having both "windows" and "system-windows" Reviewers: asmith, probinson Subscribers: delcypher, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D61798 llvm-svn: 361998
2019-05-29 20:07:39 +02:00
# UNSUPPORTED: system-windows
[lit] Disable shtest-timeout on Windows This test passes on Windows when using Python 3 but fails when using Python 2, so it needs more investigation before it can be enabled as the bots use Python 2. llvm-svn: 339184
2018-08-07 23:21:30 +02:00
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
###############################################################################
# Check tests can hit timeout when set
###############################################################################
[lit] Try to make `shtest-timeout.py` test more reliable by using a larger timeout value. This really isn't very good because it will still be susceptible to machine performance. While we are here also fix a bug in validation of `maxIndividualTestTime` where previously it wasn't checked if the type was an int. rdar://problem/40221572 llvm-svn: 332987
2018-05-22 17:06:29 +02:00
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
# Test per test timeout using external shell
# RUN: not %{lit} \
# RUN: %{inputs}/shtest-timeout/infinite_loop.py \
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# RUN: -j 1 -v --debug --timeout 1 --param external=1 > %t.extsh.out 2> %t.extsh.err
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON < %t.extsh.out %s
# RUN: FileCheck --check-prefix=CHECK-EXTSH-ERR < %t.extsh.err %s
#
# CHECK-EXTSH-ERR: Using external shell
# Test per test timeout using internal shell
# RUN: not %{lit} \
# RUN: %{inputs}/shtest-timeout/infinite_loop.py \
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# RUN: -j 1 -v --debug --timeout 1 --param external=0 > %t.intsh.out 2> %t.intsh.err
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON < %t.intsh.out %s
# RUN: FileCheck --check-prefix=CHECK-INTSH-OUT < %t.intsh.out %s
# RUN: FileCheck --check-prefix=CHECK-INTSH-ERR < %t.intsh.err %s
[lit] Fix test shtest-timeout.py for modern output Update the CHECK lines in the shtest-timeout.py lit test to account for the current output. The output has been changed in r271610 without adjusting the tests. Differential Revision: https://reviews.llvm.org/D25236 llvm-svn: 284057
2016-10-12 23:40:08 +02:00
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
# CHECK-INTSH-OUT: TIMEOUT: per_test_timeout :: infinite_loop.py
[lit] Fix test shtest-timeout.py for modern output Update the CHECK lines in the shtest-timeout.py lit test to account for the current output. The output has been changed in r271610 without adjusting the tests. Differential Revision: https://reviews.llvm.org/D25236 llvm-svn: 284057
2016-10-12 23:40:08 +02:00
# CHECK-INTSH-OUT: command reached timeout: True
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
# CHECK-INTSH-ERR: Using internal shell
# Test per test timeout set via a config file rather than on the command line
# RUN: not %{lit} \
# RUN: %{inputs}/shtest-timeout/infinite_loop.py \
# RUN: -j 1 -v --debug --param external=0 \
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# RUN: --param set_timeout=1 > %t.cfgset.out 2> %t.cfgset.err
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON < %t.cfgset.out %s
# RUN: FileCheck --check-prefix=CHECK-CFGSET-ERR < %t.cfgset.err %s
#
# CHECK-CFGSET-ERR: Using internal shell
# CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: infinite_loop.py
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# CHECK-OUT-COMMON: Timeout: Reached timeout of 1 seconds
# CHECK-OUT-COMMON: Timed Out: 1
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
###############################################################################
# Check tests can complete in with a timeout set
#
# `short.py` should execute quickly so we shouldn't wait anywhere near the
# 3600 second timeout.
###############################################################################
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# Test per test timeout using external shell
# RUN: %{lit} \
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
# RUN: %{inputs}/shtest-timeout/short.py \
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# RUN: -j 1 -v --debug --timeout 3600 --param external=1 > %t.pass.extsh.out 2> %t.pass.extsh.err
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON-SHORT < %t.pass.extsh.out %s
# RUN: FileCheck --check-prefix=CHECK-EXTSH-ERR < %t.pass.extsh.err %s
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# Test per test timeout using internal shell
# RUN: %{lit} \
# RUN: %{inputs}/shtest-timeout/short.py \
# RUN: -j 1 -v --debug --timeout 3600 --param external=0 > %t.pass.intsh.out 2> %t.pass.intsh.err
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON-SHORT < %t.pass.intsh.out %s
# RUN: FileCheck --check-prefix=CHECK-INTSH-ERR < %t.pass.intsh.err %s
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# CHECK-OUT-COMMON-SHORT: PASS: per_test_timeout :: short.py
# CHECK-OUT-COMMON-SHORT: Passed: 1
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# Test per test timeout via a config file and on the command line.
# The value set on the command line should override the config file.
# RUN: %{lit} \
# RUN: %{inputs}/shtest-timeout/short.py \
# RUN: -j 1 -v --debug --param external=0 \
# RUN: --param set_timeout=1 --timeout=3600 > %t.pass.cmdover.out 2> %t.pass.cmdover.err
# RUN: FileCheck --check-prefix=CHECK-OUT-COMMON-SHORT < %t.pass.cmdover.out %s
# RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-ERR < %t.pass.cmdover.err %s
[lit] Implement support of per test timeout in lit. This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-27 15:03:49 +01:00
[lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`. The tests previously relied on the `short.py` and `FirstTest.subTestA` script being executed on a machine within a short time window (1 or 2 seconds). While this "seems to work" it can fail on resource constrained machines. We could bump the timeout a little bit (bumping it too much would mean the test would take a long time to execute) but it wouldn't really solve the problem of the test being prone to failures. This patch tries to remove this flakeyness by separating testing into two separate parts: 1. Testing if a test can hit a timeout. 2. Testing if a test can run to completion in the presence of a timeout. This way we can give (1.) a really short timeout (to make the test run as fast as possible) and (2.) a really long timeout. This means for (2.) we are no longer trying to rely on the "short" test executing within some short time window. Instead the window is now 3600 seconds which should be long enough even for a heavily resource constrained machine to execute the "short" test. Thanks to Julian Lettner for suggesting this approach. This superseeds my original approach in https://reviews.llvm.org/D88807. This patch also changes the command line override test to run the quick test rather than the slow one to make the test run faster. Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 01:52:19 +02:00
# CHECK-CMDLINE-OVERRIDE-ERR: Forcing timeout to be 3600 seconds
Reference in New Issue Copy Permalink