ElasticSearch and Kibana for Test Automation Reporting Dashboard

A Way to Capture and Analyze Historic Automation Test report to support decisions

Author: Vikas Thange

Problem Statement:

Test reporting is crucial part in any automation test framework. Most of the test automation framework generate test report for last run including test name, status, failure messages, snaps. Few framework tools also helps in storing report for all automation test execution cycles including historic one. for example Klove (aka ExtentX). The biggest problem is these tools are only storing the predefined meta data about test case e.g. test name, class name, test status, duration etc. In practical scenarios just capturing this predefined data do not help in taking decisions.

This white paper is answer to the problem mentioned above. 

Proposal:

I am proposing a generic reporting system which not only captures historic test execution data including custom fields but also helps in depicting in graphical dashboard to support decisions.

Solutions:

Example:

Sample Dashboard

Architecture:  

High Level Architecture

It is a one time process that includes 3 major steps as below.

Step 1: Set up ElasticSearch Database, Define indexes with schema. Setup Kibana and set ElasticSearch as source

Step 2: Save Test result statistics to Elastic Search

In Automation framework, develop a plugin for testing framework which captures all data including test result meta data and custom fields values from test framework and save all data to ElasticSearch database using REST APIs. 

This can be done with any of 2 options

1. Direct JSON object to Elastic Search
1. Create POJO with test result meta data and custom fields. 
2. On test execution, Create instances of POJO classes with assigning values to all fields. 
3. Convert object to JSON and class elastic search REST API to save in database cluster.
2. LogStash
1. Modify framework logger to follow LogStash format
2. Feed log to LogStash and then ElasticSeach with indexes.

Step 3: Create UI Visualization for any meaningful data in with you are interested. and add it to Kibana dashboard. Analyze the Dashboard graphs, trends to take decisions.

The meaning full data can be any filter query based on the parameters stored in Elastic Search.

Below are the few sample examples scenarios to create visualization

1. Test execution trend with test status, test count and environment in given specific time
2. Which test case is getting maximum times skipped? or failed?
3. Which is the most occurring execution / failure message?
4. Which machine execute most of the test cases?
5. Filter test trend for one specific platform, test case, failure message etc.
6. Test execution trend on one particular machine

References: https://www.elastic.co/