Test Case Relevancy

Motivation

Sometimes a test case is only relevant for specific environment. Test Case Relevancy allows to filter irrelevant test cases out.

Implementation

Test case relevancy is function which takes environment parameters and returns True, False or a list of environment variables.

Syntax

Test case relevancy is defined by one or more condition: decision rules.

Allowed operators are: = == != < <= > >= &&. Anything beyond a # sign is considered to be a comment and will be ignored.

Everything is case insensitive. First matching rule wins (terminates immediately the relevancy evaluation, the rest of the rules will be ignored).

Defaults

If there is no rule specified, test case relevancy defaults to True, that is test case is relevant and should be executed.

Environment

Acceptable parameters defining the environment are:

product: product name (rhel rhel-5 rhel-5.6 rhdts rhscl)
distro: distribution (rhel-6 rhel-6.3 rhscl-1.0.0 f-28)
variant: distro variant (Client Desktop Server Workstation)
arch: architecture (i386 ppc64 s390x x86_64)
collection: software collection (python27 python33 perl516…)
component: component to be tested (php, apache, …)

While distro is always used to define the operating system the software is supposed to run on, product is used to describe the target product subscribed and consumed by the customer (can be a layered product on top of RHEL such as RHSCL, or RHEL itself if the component is included directly in the operating system).

Distro comparison operates in two modes:

Major mode: When comparing against a major version such as distro < rhel-6 other major versions are considered (matches any of rhel-3 rhel-4 rhel-5).
Minor mode: If minor version is provided as well, for example distro < rhel-6.3, comparison is performed only within the given major (matches rhel-6.0 rhel-6.1 rhel-6.2).

Rules which contain environment parameters which are not known at the time of evaluation will be skipped.

Decision

The decision part of the rule can contain following values:

True: test case is relevant for given environment
False: test case is not relevant for this environment
A=X B=Y C=Z: test case is relevant for the modified environment

The last option above allows to adjust the environment in which the test case is to be executed by providing the list of environment variables which will be passed to the test.

The decision value may be omitted. In such case True is used by default. So these two lines define identical relevancy:

distro > rhel-7: True
distro > rhel-7

Examples

mod_wsgi relevant for RHEL6 and newer:

distro < rhel-6: False

busybox not available for s390x on RHEL6:

arch = s390x && distro = rhel-6: False

perl-Config-General not present in the Client variant:

variant = Client: False

Run python unit tests under valgrind on suitable archs only:

arch = ia64: PHASES=novalgrind
arch = s390x && distro < rhel-6: PHASES=novalgrind

New component python-ctypes added in rhel-5.8:

distro < rhel-5:   False
distro < rhel-5.8: False

Exim present solely in RHEL5:

distro != rhel-5: False

Want to help? Learn how to contribute to Fedora Docs ›