This page describes all test types which can be defined in Lightning XML config file:

<?xml version="1.0" encoding="UTF-8"?>
<testSet>

    <!-- tests go here -->

</testSet>

Average Response Time Test

Verify if average response time is not higher than given value.

Sample configuration:

<avgRespTimeTest>
    <testName>Test #1</testName>
    <description>Verify average login times</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <maxAvgRespTime>4000</maxAvgRespTime>
</avgRespTimeTest>

Sample output:

Test name:            Test #1
Test type:            avgRespTimeTest
Test description:     Verify average login times
Transaction name:     Login
Expected result:      Average response time <= 4000
Actual result:        Average response time = 3583
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Maximum Response Time Test

Verify if maximum response time is not higher than given value.

Sample configuration:

<maxRespTimeTest>
    <testName>Test #1</testName>
    <description>Verify maximum login times</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <maxAllowedRespTime>4000</maxAllowedRespTime>
</maxRespTimeTest>

Sample output:

Test name:            Test #1
Test type:            maxRespTimeTest
Test description:     Verify maximum login times
Transaction name:     Login
Expected result:      Max response time <= 4000
Actual result:        Max response time = 3780
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Response Time Standard Deviation Test

Verify if standard deviation of response time is not higher than given value.

Sample configuration:

<respTimeStdDevTest>
    <testName>Test #2</testName>
    <description>Verify standard deviation</description>
    <transactionName>Search</transactionName>
    <regexp/>
    <maxRespTimeStdDev>500</maxRespTimeStdDev>
</respTimeStdDevTest>

Sample output:

Test name:            Test #2
Test type:            respTimeStdDevTest
Test description:     Verify standard deviation
Transaction name:     Search
Expected result:      Average standard deviance time <= 500
Actual result:        Average standard deviance time = 279
Transaction count:    5
Test result:          Pass

Note: There is more than one way to calculate standard deviation. Lightning calculates the sample standard deviation using org.apache.commons.math3.stat.descriptive.DescriptiveStatistics (which gives same results as STDEV.S in MS Excel), not the population standard deviation (STDEV.P in MS Excel).

Passed Transactions Test

Verify if number of failed transactions is not higher than given value.

Absolute Number

Sample configuration:

<passedTransactionsTest>
    <testName>Test #3</testName>
    <description>Verify number of passed tests</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <allowedNumberOfFailedTransactions>0</allowedNumberOfFailedTransactions>
</passedTransactionsTest>

Sample output:

Test name:            Test #3
Test type:            passedTransactionsTest
Test description:     Verify number of passed tests
Transaction name:     Login
Expected result:      Number of failed transactions <= 0
Actual result:        Number of failed transactions = 0
Transaction count:    5
Test result:          Pass

Percentage

Sample configuration:

<passedTransactionsTest>
    <testName>Test #3</testName>
    <description>Verify number of passed tests</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <allowedPercentOfFailedTransactions>0</allowedPercentOfFailedTransactions>
</passedTransactionsTest>

Sample output:

Test name:            Test #3
Test type:            passedTransactionsTest
Test description:     Verify number of passed tests
Transaction name:     Login
Expected result:      Percentage of failed transactions <= 0%
Actual result:        Percentage of failed transactions = 0%
Transaction count:    5
Test result:          Pass

n-th Percentile Response Time Test

Verify if response time for n-th percentile of transactions is not higher than given value.

Sample configuration:

<nthPercRespTimeTest>
    <testName>Test #4</testName>
    <description>Verify nth percentile</description>
    <transactionName>Search</transactionName>
    <regexp/>
    <percentile>80</percentile>
    <maxRespTime>11244</maxRespTime>
</nthPercRespTimeTest>

Sample output:

Test name:            Test #4
Test type:            nthPercRespTimeTest
Test description:     Verify nth percentile
Transaction name:     Search
Expected result:      80th percentile of transactions have response time <= 11244
Actual result:        80th percentile of transactions have response time = 11244
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Note: There is more than one way to calculate percentile. Lightning calculates percentile using org.apache.commons.math3.stat.descriptive.DescriptiveStatistics R_3 method , which is consistent with JMeter.

Throughput Test

Verify if throughput (transactions per second) is not lower than given value.

Sample configuration:

<throughputTest>
    <testName>Test #5</testName>
    <description>Verify throughput</description>
    <transactionName>Search</transactionName>
    <regexp/>
    <minThroughput>2</minThroughput>
</throughputTest>

Sample output:

Test name:            Test #5
Test type:            throughputTest
Test description:     Verify throughput
Transaction name:     Search
Expected result:      Throughput >= 2 / second
Actual result:        Throughput = 6 / second
Transaction count:    5
Test result:          Pass

Median Response Time Test

Verify if median response time is not higher than given value.

Sample configuration:

<medianRespTimeTest>
   <testName>Test #4</testName>
   <description>Verify median response time</description>
   <transactionName>Search</transactionName>
   <regexp/>
   <maxRespTime>11244</maxRespTime>
</medianRespTimeTest>

Sample output:

Test name:            Test #4
Test type:            medianRespTimeTest
Test description:     Verify median response time
Transaction name:     Search
Expected result:      median response time <= 11244
Actual result:        median response time = 11244
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Server Side Test

Verify server-side metrics.

Less or greater than

Sample configuration:

<serverSideTest>
    <testName>Test #2</testName>
    <subType>LESS_THAN</subType>
    <description>Verify server-side resource utilisation</description>
    <hostAndMetric>192.168.0.12 CPU</hostAndMetric>
    <metricValueA>80000</metricValueA>
</serverSideTest>

Sample output:

Test name:            Test #3

Test type:            serverSideTest

Test description:     Verify server-side resource utilisation

Host and metric:      192.168.0.12 CPU

Expected result:      Average value < 60000

Actual result:        Average value = 15189
Entries count:        2

Test result:          Pass

Between

Sample configuration:

<serverSideTest>
    <testName>Test #2</testName>
    <subType>BETWEEN</subType>
    <description>Verify server-side resource utilisation</description>
    <hostAndMetric>192.168.0.12 CPU</hostAndMetric>
    <metricValueA>40000</metricValueA>
    <metricValueB>80000</metricValueB>
</serverSideTest>

Sample output:

Test name:            Test #2

Test type:            serverSideTest

Test description:     Verify server-side resource utilisation

Host and metric:      192.168.0.12 CPU

Expected result:      Average value between 10000 and 40000

Actual result:        Average value = 15189
Entries count:        2

Test result:          Pass