This page describes all test types which can be defined in Lightning XML config file:
<?xml version="1.0" encoding="UTF-8"?>
<testSet>
<!-- tests go here -->
</testSet>
Average Response Time Test
Verify if average response time is not higher than given value.
Sample configuration:
<avgRespTimeTest>
<testName>Test #1</testName>
<description>Verify average login times</description>
<transactionName>Login</transactionName>
<regexp/>
<maxAvgRespTime>4000</maxAvgRespTime>
</avgRespTimeTest>
- testName - included in test results,
required
,non-empty string
- description - included in test results,
optional
,string
- transactionName - name of transaction,
optional
, if provided must match value in JMeter CSV file inlabel
column, if not provided, test applies to all records in JMeter CSV file - regexp - if name of transaction should be considered to be a regular expression,
optional
- maxAvgRespTime - maximum average response value for given transaction name, which will not fail the test,
required
,positive integer
Sample output:
Test name: Test #1
Test type: avgRespTimeTest
Test description: Verify average login times
Transaction name: Login
Expected result: Average response time <= 4000
Actual result: Average response time = 3583
Transaction count: 5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result: Pass
Maximum Response Time Test
Verify if maximum response time is not higher than given value.
Sample configuration:
<maxRespTimeTest>
<testName>Test #1</testName>
<description>Verify maximum login times</description>
<transactionName>Login</transactionName>
<regexp/>
<maxAllowedRespTime>4000</maxAllowedRespTime>
</maxRespTimeTest>
- testName - included in test results,
required
,non-empty string
- description - included in test results,
optional
,string
- transactionName - name of transaction,
optional
, if provided must match value in JMeter CSV file inlabel
column, if not provided, test applies to all records in JMeter CSV file - regexp - if name of transaction should be considered to be a regular expression,
optional
- maxAllowedRespTime - maximum response value for given transaction name, which will not fail the test,
required
,positive integer
Sample output:
Test name: Test #1
Test type: maxRespTimeTest
Test description: Verify maximum login times
Transaction name: Login
Expected result: Max response time <= 4000
Actual result: Max response time = 3780
Transaction count: 5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result: Pass
Response Time Standard Deviation Test
Verify if standard deviation of response time is not higher than given value.
Sample configuration:
<respTimeStdDevTest>
<testName>Test #2</testName>
<description>Verify standard deviation</description>
<transactionName>Search</transactionName>
<regexp/>
<maxRespTimeStdDev>500</maxRespTimeStdDev>
</respTimeStdDevTest>
- testName - included in test results,
required
,non-empty string
- description - included in test results,
optional
,string
- transactionName - name of transaction,
optional
, if provided must match value in JMeter CSV file inlabel
column, if not provided, test applies to all records in JMeter CSV file - regexp - if name of transaction should be considered to be a regular expression,
optional
- maxRespTimeStdDev - maximum value of standard deviation for given transaction name, which will not fail the
test,
required
,positive integer
Sample output:
Test name: Test #2
Test type: respTimeStdDevTest
Test description: Verify standard deviation
Transaction name: Search
Expected result: Average standard deviance time <= 500
Actual result: Average standard deviance time = 279
Transaction count: 5
Test result: Pass
Note: There is more than one way to calculate standard deviation. Lightning calculates the sample standard deviation
using org.apache.commons.math3.stat.descriptive.DescriptiveStatistics
(which gives same results as STDEV.S in MS Excel),
not the population standard deviation (STDEV.P in MS Excel).
Passed Transactions Test
Verify if number of failed transactions is not higher than given value.
Absolute Number
Sample configuration:
<passedTransactionsTest>
<testName>Test #3</testName>
<description>Verify number of passed tests</description>
<transactionName>Login</transactionName>
<regexp/>
<allowedNumberOfFailedTransactions>0</allowedNumberOfFailedTransactions>
</passedTransactionsTest>
- testName - included in test results,
required
,non-empty string
- description - included in test results,
optional
,string
- transactionName - name of transaction,
optional
, if provided must match value in JMeter CSV file inlabel
column, if not provided, test applies to all records in JMeter CSV file - regexp - if name of transaction should be considered to be a regular expression,
optional
- allowedNumberOfFailedTransactions - maximum number of failed transactions for given transaction name, which
will not fail the test,
required
,non-negative integer
Sample output:
Test name: Test #3
Test type: passedTransactionsTest
Test description: Verify number of passed tests
Transaction name: Login
Expected result: Number of failed transactions <= 0
Actual result: Number of failed transactions = 0
Transaction count: 5
Test result: Pass
Percentage
Sample configuration:
<passedTransactionsTest>
<testName>Test #3</testName>
<description>Verify number of passed tests</description>
<transactionName>Login</transactionName>
<regexp/>
<allowedPercentOfFailedTransactions>0</allowedPercentOfFailedTransactions>
</passedTransactionsTest>
- allowedPercentOfFailedTransactions - maximum percentage of failed transactions for given transaction name,
which will not fail the test,
required
,integer
in range 0-100 - other elements same as above
Sample output:
Test name: Test #3
Test type: passedTransactionsTest
Test description: Verify number of passed tests
Transaction name: Login
Expected result: Percentage of failed transactions <= 0%
Actual result: Percentage of failed transactions = 0%
Transaction count: 5
Test result: Pass
n-th Percentile Response Time Test
Verify if response time for n-th percentile of transactions is not higher than given value.
Sample configuration:
<nthPercRespTimeTest>
<testName>Test #4</testName>
<description>Verify nth percentile</description>
<transactionName>Search</transactionName>
<regexp/>
<percentile>80</percentile>
<maxRespTime>11244</maxRespTime>
</nthPercRespTimeTest>
- testName - included in test results,
required
,non-empty string
- description - included in test results,
optional
,string
- transactionName - name of transaction,
optional
, if provided must match value in JMeter CSV file inlabel
column, if not provided, test applies to all records in JMeter CSV file - regexp - if name of transaction should be considered to be a regular expression,
optional
- percentile - percentile,
required
, integer in range 1-100 - maxRespTime - maximum percentile response time for given transaction name, which will not fail the
test,
required
,integer
Sample output:
Test name: Test #4
Test type: nthPercRespTimeTest
Test description: Verify nth percentile
Transaction name: Search
Expected result: 80th percentile of transactions have response time <= 11244
Actual result: 80th percentile of transactions have response time = 11244
Transaction count: 5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result: Pass
Note: There is more than one way to calculate percentile.
Lightning calculates percentile using org.apache.commons.math3.stat.descriptive.DescriptiveStatistics
R_3 method
, which is consistent with JMeter.
Throughput Test
Verify if throughput (transactions per second) is not lower than given value.
Sample configuration:
<throughputTest>
<testName>Test #5</testName>
<description>Verify throughput</description>
<transactionName>Search</transactionName>
<regexp/>
<minThroughput>2</minThroughput>
</throughputTest>
- testName - included in test results,
required
,non-empty string
- description - included in test results,
optional
,string
- transactionName - name of transaction,
optional
, if provided must match value in JMeter CSV file inlabel
column, if not provided, test applies to all records in JMeter CSV file - regexp - if name of transaction should be considered to be a regular expression,
optional
- minThroughput - minimum throughput, which will not fail the test,
required
,double
Sample output:
Test name: Test #5
Test type: throughputTest
Test description: Verify throughput
Transaction name: Search
Expected result: Throughput >= 2 / second
Actual result: Throughput = 6 / second
Transaction count: 5
Test result: Pass
Median Response Time Test
Verify if median response time is not higher than given value.
Sample configuration:
<medianRespTimeTest>
<testName>Test #4</testName>
<description>Verify median response time</description>
<transactionName>Search</transactionName>
<regexp/>
<maxRespTime>11244</maxRespTime>
</medianRespTimeTest>
- testName - included in test results,
required
,non-empty string
- description - included in test results,
optional
,string
- transactionName - name of transaction,
optional
, if provided must match value in JMeter CSV file inlabel
column, if not provided, test applies to all records in JMeter CSV file - regexp - if name of transaction should be considered to be a regular expression,
optional
- maxRespTime - maximum response time, which will not fail the test,
required
, integer
Sample output:
Test name: Test #4
Test type: medianRespTimeTest
Test description: Verify median response time
Transaction name: Search
Expected result: median response time <= 11244
Actual result: median response time = 11244
Transaction count: 5
Longest transactions: [3780, 3700, 3514, 3474, 3448]
Test result: Pass
Server Side Test
Verify server-side metrics.
Less or greater than
Sample configuration:
<serverSideTest>
<testName>Test #2</testName>
<subType>LESS_THAN</subType>
<description>Verify server-side resource utilisation</description>
<hostAndMetric>192.168.0.12 CPU</hostAndMetric>
<metricValueA>80000</metricValueA>
</serverSideTest>
- testName - included in test results,
required
,non-empty string
- subType - specifies if average value needs to be above or below threshold given in
metricValueA
,required
,LESS_THAN
orGREATER_THAN
- description - included in test results,
optional
,string
- hostAndMetric - name of host and metric,
required
, if provided must match value in PerfMon Server Agent CSV file collected by JMeter PerfMon Metrics Collector - metricValueA - maximum or minimum average value of given metric, which will not fail the test,
required
, integer
Sample output:
Test name: Test #3
Test type: serverSideTest
Test description: Verify server-side resource utilisation
Host and metric: 192.168.0.12 CPU
Expected result: Average value < 60000
Actual result: Average value = 15189
Entries count: 2
Test result: Pass
Between
Sample configuration:
<serverSideTest>
<testName>Test #2</testName>
<subType>BETWEEN</subType>
<description>Verify server-side resource utilisation</description>
<hostAndMetric>192.168.0.12 CPU</hostAndMetric>
<metricValueA>40000</metricValueA>
<metricValueB>80000</metricValueB>
</serverSideTest>
- subType - specifies if average value,
required
,BETWEEN
- metricValueA - minimum average value of given metric, which will not fail the test,
required
, integer - metricValueB - maximum average value of given metric, which will not fail the test,
required
, integer - other elements same as above
Sample output:
Test name: Test #2
Test type: serverSideTest
Test description: Verify server-side resource utilisation
Host and metric: 192.168.0.12 CPU
Expected result: Average value between 10000 and 40000
Actual result: Average value = 15189
Entries count: 2
Test result: Pass