PyCon 2019 in Cleveland, Ohio

Sunday 10 a.m.–1 p.m. in Expo Hall

Automatic Detection of Pseudo-tested Methods using Python and Pytest

Nicholas Tocci, Gregory M. Kapfhammer

Description

Test suites are a critical part of the development process. They help developers ensure that the behaviors that they have coded are working the way that they have intended. Test suites can help guide development towards a working system if implemented in the early stages of development, such as using Test-Driven-Development. If test suites are a measure for how well a system is working, how can you tell if the test suite is working? One solution is a developer can look to coverage to see how much of their system is being executed during the tests. One major problem can cause this type of benchmark to not give an accurate depiction of the fault detection effectiveness of the system. This issue is called a pseudo-tested method. A pseudo-tested method is a method that is being called in a test case that will never fail, thus pseudo-testing it. It is "pseudo-tested" because the developer thinks that the method is being tested because it appears in a test case, but the test is not accurately detecting if there is an error and is providing a pass every time it is run. Pseudo-tested methods are very difficult to detect. In fact, they are hard to detect because of the nature of them. They always pass, which begs the question, "Why investigate something that is not broken?" The only real way is to find it manually. Which includes two ways: a developer is looking for a problem in the test suite, or if they know that they made an error, but notice that the test never fails. There is a way to detect them automatically for the Java programming language, but none for Python. Python also has the potential to contain more pseudo-tested methods because of the loosely-typed nature of the language. For this reason, pseudo-tested methods could be present in many test suites, including ones with a high coverage percentage. Which led to the creation of Function-Fiasco. Function-Fiasco is an automatic detection system that finds pseudo-tested methods in Python based systems that are being tested under the Pytest framework. It finds methods in the system that is being observed and checks the return type. Once it has this, it randomizes an output that matches the return type to see if test cases that use this method still pass. If tests do still pass, the method is considered pseudo-tested. Pseudo-testing is an issue that has the potential to be found in a vast amount of systems, that are both of large and small scale. Function-Fiasco was created in Python for Python to help mitigate this risk.