Lightning

This page describes all test types which can be defined in Lightning XML config file:

<?xml version="1.0" encoding="UTF-8"?>
<testSet>

    <!-- tests go here -->

</testSet>

Average Response Time Test

Verify if average response time is not higher than given value.

Sample configuration:

<avgRespTimeTest>
    <testName>Test #1</testName>
    <description>Verify average login times</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <maxAvgRespTime>4000</maxAvgRespTime>
</avgRespTimeTest>

testName - included in test results, required, non-empty string
description - included in test results, optional, string
transactionName - name of transaction, optional, if provided must match value in JMeter CSV file in label column, if not provided, test applies to all records in JMeter CSV file
regexp - if name of transaction should be considered to be a regular expression, optional
maxAvgRespTime - maximum average response value for given transaction name, which will not fail the test, required, positive integer

Sample output:

Test name:            Test #1
Test type:            avgRespTimeTest
Test description:     Verify average login times
Transaction name:     Login
Expected result:      Average response time <= 4000
Actual result:        Average response time = 3583
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Maximum Response Time Test

Verify if maximum response time is not higher than given value.

Sample configuration:

<maxRespTimeTest>
    <testName>Test #1</testName>
    <description>Verify maximum login times</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <maxAllowedRespTime>4000</maxAllowedRespTime>
</maxRespTimeTest>

testName - included in test results, required, non-empty string
description - included in test results, optional, string
transactionName - name of transaction, optional, if provided must match value in JMeter CSV file in label column, if not provided, test applies to all records in JMeter CSV file
regexp - if name of transaction should be considered to be a regular expression, optional
maxAllowedRespTime - maximum response value for given transaction name, which will not fail the test, required, positive integer

Sample output:

Test name:            Test #1
Test type:            maxRespTimeTest
Test description:     Verify maximum login times
Transaction name:     Login
Expected result:      Max response time <= 4000
Actual result:        Max response time = 3780
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Response Time Standard Deviation Test

Verify if standard deviation of response time is not higher than given value.

Sample configuration:

<respTimeStdDevTest>
    <testName>Test #2</testName>
    <description>Verify standard deviation</description>
    <transactionName>Search</transactionName>
    <regexp/>
    <maxRespTimeStdDev>500</maxRespTimeStdDev>
</respTimeStdDevTest>

testName - included in test results, required, non-empty string
description - included in test results, optional, string
transactionName - name of transaction, optional, if provided must match value in JMeter CSV file in label column, if not provided, test applies to all records in JMeter CSV file
regexp - if name of transaction should be considered to be a regular expression, optional
maxRespTimeStdDev - maximum value of standard deviation for given transaction name, which will not fail the test, required, positive integer

Sample output:

Test name:            Test #2
Test type:            respTimeStdDevTest
Test description:     Verify standard deviation
Transaction name:     Search
Expected result:      Average standard deviance time <= 500
Actual result:        Average standard deviance time = 279
Transaction count:    5
Test result:          Pass

Note: There is more than one way to calculate standard deviation. Lightning calculates the sample standard deviation using org.apache.commons.math3.stat.descriptive.DescriptiveStatistics (which gives same results as STDEV.S in MS Excel), not the population standard deviation (STDEV.P in MS Excel).

Passed Transactions Test

Verify if number of failed transactions is not higher than given value.

Absolute Number

Sample configuration:

<passedTransactionsTest>
    <testName>Test #3</testName>
    <description>Verify number of passed tests</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <allowedNumberOfFailedTransactions>0</allowedNumberOfFailedTransactions>
</passedTransactionsTest>

testName - included in test results, required, non-empty string
description - included in test results, optional, string
transactionName - name of transaction, optional, if provided must match value in JMeter CSV file in label column, if not provided, test applies to all records in JMeter CSV file
regexp - if name of transaction should be considered to be a regular expression, optional
allowedNumberOfFailedTransactions - maximum number of failed transactions for given transaction name, which will not fail the test, required, non-negative integer

Sample output:

Test name:            Test #3
Test type:            passedTransactionsTest
Test description:     Verify number of passed tests
Transaction name:     Login
Expected result:      Number of failed transactions <= 0
Actual result:        Number of failed transactions = 0
Transaction count:    5
Test result:          Pass

Percentage

Sample configuration:

<passedTransactionsTest>
    <testName>Test #3</testName>
    <description>Verify number of passed tests</description>
    <transactionName>Login</transactionName>
    <regexp/>
    <allowedPercentOfFailedTransactions>0</allowedPercentOfFailedTransactions>
</passedTransactionsTest>

allowedPercentOfFailedTransactions - maximum percentage of failed transactions for given transaction name, which will not fail the test, required, integer in range 0-100
other elements same as above

Sample output:

Test name:            Test #3
Test type:            passedTransactionsTest
Test description:     Verify number of passed tests
Transaction name:     Login
Expected result:      Percentage of failed transactions <= 0%
Actual result:        Percentage of failed transactions = 0%
Transaction count:    5
Test result:          Pass

n-th Percentile Response Time Test

Verify if response time for n-th percentile of transactions is not higher than given value.

Sample configuration:

<nthPercRespTimeTest>
    <testName>Test #4</testName>
    <description>Verify nth percentile</description>
    <transactionName>Search</transactionName>
    <regexp/>
    <percentile>80</percentile>
    <maxRespTime>11244</maxRespTime>
</nthPercRespTimeTest>

testName - included in test results, required, non-empty string
description - included in test results, optional, string
transactionName - name of transaction, optional, if provided must match value in JMeter CSV file in label column, if not provided, test applies to all records in JMeter CSV file
regexp - if name of transaction should be considered to be a regular expression, optional
percentile - percentile, required, integer in range 1-100
maxRespTime - maximum percentile response time for given transaction name, which will not fail the test, required, integer

Sample output:

Test name:            Test #4
Test type:            nthPercRespTimeTest
Test description:     Verify nth percentile
Transaction name:     Search
Expected result:      80th percentile of transactions have response time <= 11244
Actual result:        80th percentile of transactions have response time = 11244
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Note: There is more than one way to calculate percentile. Lightning calculates percentile using org.apache.commons.math3.stat.descriptive.DescriptiveStatistics R_3 method , which is consistent with JMeter.

Throughput Test

Verify if throughput (transactions per second) is not lower than given value.

Sample configuration:

<throughputTest>
    <testName>Test #5</testName>
    <description>Verify throughput</description>
    <transactionName>Search</transactionName>
    <regexp/>
    <minThroughput>2</minThroughput>
</throughputTest>

testName - included in test results, required, non-empty string
description - included in test results, optional, string
transactionName - name of transaction, optional, if provided must match value in JMeter CSV file in label column, if not provided, test applies to all records in JMeter CSV file
regexp - if name of transaction should be considered to be a regular expression, optional
minThroughput - minimum throughput, which will not fail the test, required, double

Sample output:

Test name:            Test #5
Test type:            throughputTest
Test description:     Verify throughput
Transaction name:     Search
Expected result:      Throughput >= 2 / second
Actual result:        Throughput = 6 / second
Transaction count:    5
Test result:          Pass

Median Response Time Test

Verify if median response time is not higher than given value.

Sample configuration:

<medianRespTimeTest>
   <testName>Test #4</testName>
   <description>Verify median response time</description>
   <transactionName>Search</transactionName>
   <regexp/>
   <maxRespTime>11244</maxRespTime>
</medianRespTimeTest>

testName - included in test results, required, non-empty string
description - included in test results, optional, string
transactionName - name of transaction, optional, if provided must match value in JMeter CSV file in label column, if not provided, test applies to all records in JMeter CSV file
regexp - if name of transaction should be considered to be a regular expression, optional
maxRespTime - maximum response time, which will not fail the test, required, integer

Sample output:

Test name:            Test #4
Test type:            medianRespTimeTest
Test description:     Verify median response time
Transaction name:     Search
Expected result:      median response time <= 11244
Actual result:        median response time = 11244
Transaction count:    5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result:          Pass

Server Side Test

Verify server-side metrics.

Less or greater than

Sample configuration:

<serverSideTest>
    <testName>Test #2</testName>
    <subType>LESS_THAN</subType>
    <description>Verify server-side resource utilisation</description>
    <hostAndMetric>192.168.0.12 CPU</hostAndMetric>
    <metricValueA>80000</metricValueA>
</serverSideTest>

testName - included in test results, required, non-empty string
subType - specifies if average value needs to be above or below threshold given in metricValueA, required, LESS_THAN or GREATER_THAN
description - included in test results, optional, string
hostAndMetric - name of host and metric, required, if provided must match value in PerfMon Server Agent CSV file collected by JMeter PerfMon Metrics Collector
metricValueA - maximum or minimum average value of given metric, which will not fail the test, required, integer

Sample output:

Test name:            Test #3 
Test type:            serverSideTest 
Test description:     Verify server-side resource utilisation 
Host and metric:      192.168.0.12 CPU 
Expected result:      Average value < 60000 
Actual result:        Average value = 15189
Entries count:        2 
Test result:          Pass

Between

Sample configuration:

<serverSideTest>
    <testName>Test #2</testName>
    <subType>BETWEEN</subType>
    <description>Verify server-side resource utilisation</description>
    <hostAndMetric>192.168.0.12 CPU</hostAndMetric>
    <metricValueA>40000</metricValueA>
    <metricValueB>80000</metricValueB>
</serverSideTest>

subType - specifies if average value, required, BETWEEN
metricValueA - minimum average value of given metric, which will not fail the test, required, integer
metricValueB - maximum average value of given metric, which will not fail the test, required, integer
other elements same as above

Sample output:

Test name:            Test #2 
Test type:            serverSideTest 
Test description:     Verify server-side resource utilisation 
Host and metric:      192.168.0.12 CPU 
Expected result:      Average value between 10000 and 40000 
Actual result:        Average value = 15189
Entries count:        2 
Test result:          Pass

Lightning

Test types

Average Response Time Test

Maximum Response Time Test

Response Time Standard Deviation Test

Passed Transactions Test

Absolute Number

Percentage

n-th Percentile Response Time Test

Throughput Test

Median Response Time Test

Server Side Test

Less or greater than

Between