What is Performance Testing?

What is Performance Testing?

Definition: 

                   In software engineering, performance test is a process to measure system's (Application, Network, Database or Devices) speed (responsiveness), stability and scalability under particular work load.

Core Performance Testing Activities: 

                                      

Core Performance Testing Activities

1. Identify the Test Environment

2. Identify Performance Acceptance Criteria

3. Plan and Design Tests

4. Configure the Test Environment

5. Implement the Test Design

6. Execute the Test

7. Analyze Results, Report, and Retest

1. Identify the Test Environment: 

                     Identify the physical test environment. The physical environment includes hardware, software, and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient test design and planning and helps you identify testing challenges early in the project.

2. Identify Performance Acceptance Criteria: 

                     

   Identify the response time, throughput, and resource utilization goals and constraints. In general,

·         Response time is a user concern, 

·         Throughput is a business concern, 

·         Resource utilization is a system concern. 

   Additionally, identify project success criteria that may not be captured by those goals and constraints; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics. 

3. Plan and Design Tests: 

 Identify key scenarios, determine variability among representative users and how to simulate that variability, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analyzed.    

4. Configure the Test Environment: 

 Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as necessary. 

5. Implement the Test Design: 

 Develop the performance tests in accordance with the test design. 

6. Execute the Test: 

Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment. 

7. Analyze Results, Report, and Retest:

 Consolidate and share results data. Analyze the data both individually and as a cross-functional team. Re-prioritize the remaining tests and re-execute them as needed. When all of the metric values are within accepted limits, none of the set thresholds have been violated, and all of the desired information has been collected, you have finished testing that particular scenario on that particular configuration.

Few example of why performance test is conducted:

Validating that the application performs properly

Validating that the application conforms to the performance needs of the business

Finding, Analysing, and helping fix performance problems

Validating the hardware for the application is adequate

Doing capacity planning for future demand of the application

Popular Performance Testing Tools

There are many different performance testing tools. Everyone claims their tools are the best. Here are my picks and this is what I think.

Commercial Performance Testing Tools:    HP Performance Center:
                                                                                     HP LoadRunner:
                                                                                    Neotys Neoload:
Open Source Performance Testing Tools:
                                                                             Apache JMeter

Performance tuning

Performance tuning

                       Let us have some information about tuning concept to be introduced during performance testing. In my scenario, the number of users were supported by the application was only 18. Beyond this number all users were failing to logging with the "Database Related Error". The error that I was getting was "Couldn't create Database Instance". [looks like it is a Database related error]

When I started looking into this issue, we had few areas that we could look for the solution.

 They were:    1. Connection Pools
                         2. Tomcat (if tomcat is the Web Server)
                         3. OS system file
                         4. Hardware resources

I encountered this issue on Solaris box. I started finding the solution for this issue and below are areas I looked into:

1. I started looking at the Application Server Connection pools issue. We had few parameters in the web.xml file under the middle-tier where min_connex and max_connex parameter was defined related to the Database. These parameters were set to 2 and 5 respectively. The best way to start finding the solutions for any issue is the log that is being written. In the middle-tier log, out of 4, 3 connection pools were always idle. So, there was no problem at the middle-tier side.

2. I tried increasing the  tomcat memory  from 512MB to 1024MB in catalina.sh file. This didn't resolve the issue.

3. Looked into the  /etc/System file on the Solaris OS  and saw some parameters like maxusers, maxuproc (maximum user processes). Setting maxusers to a value will control max_nprocs and maxuprc. The algorithms are:
max_nprocs = 10 + (16 x maxusers)
maxuprc = max_nprocs – reserved_procs (default is 5)
Even this setting didn't resolve the issue.

4. last but not the least, I looked into the  System resources  and found that the root path (i.e. "/") doesn't have much disk space to allocate DB resources any more beyond 18 users. Since the Oracle was setup at "/" path and the Oracle folder wasn't assigned any more system resources, Oracle was utilizing the space allocated to "/". Due to in-sufficient disk space on "/" path, users were not able to make connection with the Oracle DB and hence was failing.
Increasing the root disk space eliminated the issue with the user login.

LoadRunner Key Components

LoadRunner Key Components:

Vugen - Virtual User Generator (VuGen) –records Vuser scripts that emulate the steps of real users using the application.

Controller - The Controller is an administrative center for creating, maintaining, and executingScenarios. The Controller assigns Vusers and load generators to Scenarios, starts and stops load tests, and performs other administrative tasks.

Analysis - LoadRunner Analysis uses the load test results to creategraphsandreportsthat are used to correlate system information, identify bottlenecks, and performance issues.

Load Generator - Load generators (also known as hosts) are used to run the Vusers that generate load on the application under test.

Agent Process –  It Establish a Communication b/w the LG and Controller.

What is correlation ?

What is correlation? Difference between automatic correlation and manual correlation?

Correlation :   It’s a Concept to Capture the dynamic values Which are generated from ServerSide.   

                                                          ( OR )

The values that are generated by the server/application at the run time and are used by the application to process the requests are correlated,we capture these values at run time and used them in the script.

Example –        Session ID, transaction id etc, these values are always generated by the server at run time. If we knew them before hand, we could have parameterised them in the script but as these are generated by an algorithm/random number at the run time, they have to be handled at run time only.
LoadRunner provides a function to capture these values, either you do manually or auto correlate, same function is used i.e. web_reg_save_param().

Correlation is 2-types There are 1) Auto Correlation  &  2) Manual Correlation

Auto Correlation –

Loadrunner handles the process to capture the dynamic value in each run of script execution.  

If Auto Correlation is enabled, LoadRunner identifies the values that are generated at run time and correlate them. In this method all the handling will be done by Loadrunner itself, leaving almost no control for the user (if required). Also it correlates a lot other values that are not required to be correlated.

We’ve 2-ways of achieving Automatic Correlation.

a.       Enabled Correlation during recording.

b.      Scan for Correlation.

a. Enabled Correlation during recording :

·         Built in Correlation.

·         User Define Correlation

        Loadrunner maintains set of predefined rules called AutoCorrelation Library. If the dynamic values found among these rules,Loadrunner automatically inserts the  web-reg-save-param("Parameter Name”,”LB”,”RB”,LAST); in the script pane.

               1. Select  “Enabled Correlation during recording” . We can enable or disable a specific rule by selecting or clearing the Check box adjacent to the rule.

               2. Click on the new rule, we’ll pas LB, RB and Parameter name.

               3. We can also specify where to look for the dynamic value such as Header, Body, etc.

           web-reg-save-param();  will be written in the script wherever the dynamic value occurs within specified boundaries.

Scan for Correlation :Once you record the script, you’ve to replay the script than we need to use the “Scan for Correlation”

Once you reply the script, Replay Log will be created.

When we do scan for Correlation the Vugen will compare the Recording Log with Replay Log and highlight the differences b/w them.

 We click on the correlation button on the right side of the log window to correlate the differences.

Automatic correlation cannot ensure that we’ve captured all the dynamic values (especially when may miss user specific data ). This is the main reason we cannot depend on Automatic Correlation.

Manual Correlation –

          To ensure the Script is fully Correlated, we need to go for Mannual Correalion.

In the manual correlation, we’ve to identify the dynamic data that is generated by the server and correlate them by using function  web-reg-save-param("Parameter Name”,”LB”,”RB”,LAST);

We can identify the dynamic values by recording the script in 3-times.

                   1. Recording with user1 (business flow)

                   2. Recording with user2 (same as above)

                   3. Recording with user3 ( same step as above but different i/p values).

Compare the script …………..

 We can run the script after recording. Check the point of failure and if we find any value which is generated during run time. If we find any then we will check for the same step in Tree view. Select the step where the value has been generated. Go to Server response tab and search for that value.Now select the left and right boundaries for that value..Meaning some text from immediate left side and some from immediate right side.

Example:1 –  part_the-value;sessionid=543218;newvalue;

Here left boundary = part_the-value;sessionid=Right boundary = ;newvalue; So any value between these boundaries will be captured.

Example: 2  

web_reg_save_param(”SessionID”, “LB part_the-value;sessionid=”, “RB=;newvalue;”,LAST);
Now replace the value on the script with the parameter {SessionID}
This way we will have more control over the correlation.

What is 90th Percentile? Calculation of 90%?

90^th Percentile :

      

The 90% tells you the values for which 90% of the data points are smaller and 10% are bigger.

                                (OR)

The 90th percentile is the value for which 90% of the samples were less than that value

    

90the percentile means the highest response time among the best passed 90% of the attempts of the transaction.

 For an example, the transaction  passed for 20 times. So 90% of 20 values would be 18 i.e. LR will take 18^th value is the best response times for that transaction and will report highest among those. This in turn means (90 percentile means) 90% of the response times are less than or equal to this particular number

This 90th percentile value is considered as the benchmark

Statistical Steps to calculate the 90th percentile value:-

The 90th percentile is a measure of statistical distribution.The 90^th percentile tells you the value for which 90% of the samples were

less(Smaller) than that value and 10% are bigger.

Steps

1. Sort the transaction instances by ascending order.

2. Count the total number of transactions Say T.

3. Calculate 90/100 * T = Position (P)

4. This calculated position (P) contains the 90th percentile value (V).

Example 1:

There are certain instances of transaction "t" with the values 1,3,2,4,5,21,7,8,12,6,28,8,17,22,13,2,11,15,14,3 (in sec).

1.Sort by value -- 1,2,2,3,3,4,5,6,7,8,8,11,12,13,14,15,17,21,22,28

2. Count the total number of transactions;(T)= 20.

3. Calculate  Position (P) = (90/100) * T = 18;

4. Position 18 contains the 90th percentile value(V) =21;.

Simple Explanation :

         90% Response Time (RT) is the one factor we should always look in  once the analysis report gets generated to calculate the 90% RT.

•    Sort the transaction RT by their values.
•    Remove the top 10% instances
•    The higher value left is the 90% RT.

Ex:  

Consider we’ve a Script with transaction name “Trans(Ti)-testing” and there are 10 instances of this transaction i.e we run this transaction for 10 times.

•     Values of transaction 10 instances are;

         T1    T2    T3   T4  T5   T6   T7   T8   T9  T10

3

5

1

3

9

6

7

2

4

5

                     Sec

•     Sort them by their values

         T3    T8   T1   T4    T9   T2   T10  T6   T7  T5

1

2

3

3

4

5

5

6

7

9

                Sec

•     Remove the top 10% value i.e here  9 sec (T5)

          T3    T8    T1   T4   T9   T2   T10  T6   T7    T5

1

2

3

3

4

5

5

6

7

9 Remove

   Here the 90% RT value is    7Sec (T7)

What is Bottlenecks?How to identify bottlenecks in Performance Testing?

What is Bottlenecks?

Bottlenecks

 It’s an issue which can degrades the Application Performance.

  By Using Monitors we can find the Performance Bottlenecks.

  TheseMonitors might be  App Server Monitors

                                      Web Server Monitors

                                      DB Server Monitors

                                      N/w Monitors

  

They help finding out the troubled area in our Scenario which          causes increased Response time.

 The measurements made are Usually Performance Response            Time, Throughput, Hits/Sec, N/W delay Graphs etc….

How to identify bottlenecks in Performance Testing?

EXAMPLE

      

   we will see how to identify the bottleneck issues by comparing Average Transaction Response Time and Transaction Performance Summary

                 

Consider we have following three transactions

1.   Login

2.   Book Ticket

3.   Logout

 Below is the Transaction Performance Summary graph for the above   transactions.In this example, the average response time for BookTicket transaction is 11 seconds.

Below is the Average Transaction Response Time graph which shows Book Ticket transaction’s response time is very high (15 seconds) at 1^st and 7^th minute of the scenario.

Our aim is to identify the pinpoint of the issue and to analyze the root cause of high response time for the Book Ticket transaction. To achieve above said objective, we need diagnostics tool, by default HP provides following diagnostics modules: J2EE/.NET, Siebel, Oracle, SAP R/3.

Now, we will see how to isolate the bottleneck issue through diagnostics. Prerequisite is Web Page Diagnostic feature should be enabled before executing the scenario.

Breaking down transactions helps us to identify the bottleneck issues. Open the Average Transaction Response Time in Analysis, right-click on it and select Web Page Diagnostics for Book Ticket.

Web Page Diagnostics displays breakdown of each page component’s download time of Book Ticket transaction. If the download time is too long, analyze the which measurements is responsible for the lengthy download such as FTP Authentication, DNS Resolution Time, Time to Buffer, Handshaking time etc. If you suspect, there should be any server related problem, check the First Buffer Breakdown option.

It is ideal to check the Network Monitor graphs; it helps to determine what network problem caused the bottlenecks. Also we can identify the bottleneck by using auto-correlating feature. Right-click on Average Transaction Response Time and select Auto Correlate. Mention the time frame for analysis, and the correlate options (which data that you want to correlate) such as SQL Server, Windows Resources, and Web Logic (JMX) etc.

Web Logic (JMX) or JVM Heap Size or Private Bytes might be the cause of slow performance of Book Ticket transaction.

In this way we can pinpoint the issues by analyzing Average Transaction Response Time graph. There are other ways too to determine the cause. In next blog post, we will see about Network and Server Problems.