Visual Studio 2010 Ultimate (VSU) Load Testing SharePoint

This research is from here

Performance is not only the speed of execution; it includes the load and concurrency aspects. Visual Studio is one of the tools used for Performance Test. Visual Studio Test edition or Visual Studio 2010 Ultimate provides the support for test automation.

Performance Test is an answer for the following questions

  • How can we ensure that our new application will support the expected user load?
  • How to avoid the issues which will arise only in real load conditions?
  • How to find the response time?
  • How to plan the capacity of the servers?

Load test simulates multiple users as virtual users and executes the test scripts to simulate the real user load on all Servers in the Farm. Load test tool in VSU can be used with any of the test scripts developed by the Recorder. We can test the system by setting different test mixes, user loads, stress conditions, network mixes, browsers and load patterns. Load test will generate a detailed report, which can be queried from the LoadTest or LoadTest2010 database under SQLExpress. Also, from the result we can create an Excel Trend report or Comparison report. Summary report can be saved as HTML file.

First Make the Test Script for the Load Test 
Recorder

Once you created the test project add a web performance test, which will open the recorder in an internet explorer. Click through you application and start recording the navigation URLs and associated parameters to generate a test script.

Run Test

When you run the test, you will get the following screen, where you can see the status of each URL, how the result appear in Web Browser, what are the parameters passed as part of the request, HTML response, context parameters and the details.

Correlation

Correlation is nothing but linking the response of one web request to the next web request.

For example, when you login to the site it generates a SID for tracking the session. This SID is passed to the client after the login request. Along with the next request, the stored SID will be send to server. When you record this operation using Web test, it records the values and saves as hardcoded value. SID will be different for the next run.

For avoiding such situation, correlate the SID value from the response of the login request to the next request parameters. First we will extract the SID values and save in a context parameter, which will be passed to the next request as a parameter.

Context Parameter

Context Parameters is just like to global variables. If you want to refer one parameter across all URLs, declare the same as Context Parameter.

For example, we need to run the script in multiple environments. Instead of recording the script for each environment separately, define the context parameter as ‘Webserver’ and use the same for URL formation. Context Parameters should be referred inside double curly brackets like {{Webserver}}. All the URLs should be modified using the same context parameter. When you want to run the script in another environment, modify the value of the context parameter, which will automatically take care of the URLs. In test environment the input value will differ depends on the positive testing, negative testing, boundary testing, etc. For passing multiple values to the parameters, we will do the parameterization.

Parameterization

For parameterizing the arguments, add the data source from where we can pick the values. Right click on the Web test -> Select Add Data Source option.

Select the type of data source. Data Source can be a Database like SQl Server, Oracle and Excel, or data can be fetched from CSV or XML files too.

Select the Data file or the database and table contains the input data. Preview of the data will be displayed on the wizard.

Click on the query string and move to the properties of the querystring. Change the value using the Data Source mapping as below.

Extraction Rule

Extraction rules are used for extracting the data from the response of a request. We have the following options for extracting the values; extract the form fields, extract HTTP header, etc. Extracted values can be used as part of the next web request or can be used for making any business decisions.

In following example, I used Extract Text option to extract an id passed from the server. By analyzing the html output of the request, you can form the Starts With and Ends With frames. The HTML response is displayed as part of the result window(will discuss soon).

Validation Rule

Validation rules are to enforce you are in the correct page only. After SignIn or Login In, you may be expecting a page with SignOut text. Following validation rule validates, whether the current response contains the text “SignOut’ or not.

We can form the validation rules using any of the following options. Visual Studio 2010 and 2008 automatically add the Response URL validations to the recorded test.

Transaction

Transactions are a set of operations or round-trips required to perform a particular operation. For example, the process of buying a Book consist of steps for select a Book, Add to cart, Check out and Payment.

Defining transactions will be helpful for analyzing the results. Normally the response time, Response Byte, etc. will be displayed for each URL separately. Once you define the transactions, the response time and all other measures will be displayed for the transaction level.

This is related to single run. If you want to check the parameterization to want to run the test multiple times, click on the Edit run Settings option displayed on top the screen.

Here, you can specify the number of times the test needs to be run. Each run will pick one record from the Parameter data source and run the test. Along with run count, we can specify the browser types too; which simulates how the site appears in different browser.

After setting the run settings, select the Click here to run again option specified on top of the screen.

Generate Code

Generate code option allows you to create the code corresponding to the script.

This will generate the C# code corresponding to the web test and also create a separate Test itself. We can modify the code project without affecting the web test and vice versa.

We can use the C# capabilities to automate or customize the coded web test. Can use loop for iterating one operation or ADO.Net to connect to database and pull some data for request, enforce think times (will discuss in Part II) and process WCF services.
The Load Test

Think Time

Think time is the time taken between two requests. This can be the time taken by the user to fill a form, view a page or reading some text etc. Think times are used for simulating the real user scenario; how the system will work with a real user.

Constant Load

Constant load means same number of users hit the site from the starting of the test till the end. It is like 25 users are using the system for a period of 1hr. All 25 users are hitting the system continuously. This type of testing is mainly used for stress testing.

Step Load

In step load, users will join in a step manner. This is the same as different users hitting the system in various times and the number of users in a system is not constant. Following parameter needs to be specified in step load pattern

Start user count :- How many user should hit the system at the time of test start
Step duration :- After how many seconds the next users should join the system
Step user count :- How many user needs to join the system after step duration
Maximum user count :- what is the maximum user count.

Test Mix

Test Mix specifies how different scenarios are executed or used in the system. Different test mixes are formed by studying the system’s usage.

For example, if we are doing a load testing on online marketing site; around 60% people will search for different products, 30% will buy few products and 10% bookmark the products. From this usage information, we can form a test mix as 60% simulated users will execute the script for search, 30% will execute the script for buying a product and remaining 10% will execute the script for bookmarking a product.

Load test Creation

Right click on the project and select Add -> New Test -> Load test template. It will open the New Load Test Wizard

Next, specify the scenario name like booking the ticket, searching a book, etc. Also specify the Think Time profile. We can either use the think times recorded as part of the web test recording or can use a normal think time distribution. We can also avoid the think times recorded in the web test using the third option.

Next, specify the Load Pattern. Specify whether we are going to use a constant load or a step load pattern. In constant load specify the user count. In case of step load specify the start user count, step duration, step user count and maximum user count.

Next, specify the test mix model. The different options are
[Following definitions are from Visual Studio]

Based on the total number of tests
This model of test mix determines which test is run when a virtual user starts test iteration. At the end of the load test, the number of times that a particular test was run matches the assigned test distribution. Follow this model when you are basing the test mix on transaction percentages in an US log or in production data.

Based on the number of virtual users
This model of test mix determines the percentage of virtual users who will run a particular test. At any point in the load test, the number of users who are running a particular test matches the assigned distribution. Follow this model when you are basing the test mix on the percentage of users running a particular test

Based on user pace
Over the course of the load test, each test is run the specified number of times per user per hour. Follow this model when you want virtual users to run tests at a certain pace throughout the load test.

Based on sequential test order
Each virtual user runs the tests in the order that the tests are defined in the scenario. The virtual user continues cycling through the tests in this order until the load test is complete.

Next, specify the Test Mix. In Test Mix, we are adding different web test scripts to the load test. If we have only one script, all the users are going to perform the same task. If we have 2 or more scripts, we can specify, how many user needs to perform a specific task.

For example, we have two scripts, one for searching a book and another for buying a book. In my load test, I can specify that 70% of users to do search and remaining 30% do the other operation.

Test mix is also for simulating the real user experience. Some of the features are used by many users and others may not be used that much frequently. Depends on the feature usage, we can test the system, which will give a real performance result.


In the following example, we selected two web test scripts and specified the test mix as 65% of the users will execute Webtest1 and 35% will execute Webtest2.

Next, specify the network Mix. Here we can simulate the network like LAN, WAN or internet and compare the performance of the system in different networks.

Next, specify the Browser mix. This will be useful to find how the system is performing in different browsers.

Next, add the systems used as web server, application server and database server for collecting the performance counters. Add the required performance counters for each system.

Next, specify the duration of the test or how many iterations are required. If a warm-up period is set, the load test will automatically increase the load gradually during the warm-up period.

Once, you finish the creation of the load test, it will create a .loadtest file like below. We can edit and add all the settings directly from the below screen itself.

Select the constant Load Pattern and select the corresponding properties. From the property grid, we can change the load pattern as Constant, Step or Goal based.

Goal based load test is used to find the maximum user load which satisfies the condition. Here, we are setting the goal like average response time should be 8min; once the system reaches the goal, it stops execution. Same way we can use goal based testing to find the maximum load which support a processor utilization of 70%.

 

From the Run Settings properties, we can alter the run duration, web test connection pool size, web test connection model, warm-up duration, etc.

We can specify the threshold value for counters. Once it reaches the warning level, it will start displaying warning. When it reaches the Critical threshold value, the requests will start failing.

 

 

You can add custom counters and counter sets to the Counter Sets collection.

 
From the Scenario properties, we can alter the Think time profile and associated think time.


Execute Load Test

Execute the load test by selecting Run test.

Once the test started, you can observe different performance counters and test status in the following screen. You can drag the counters from left side and drop it in the graph area, which will display the graph corresponding to the counter. Requests summary and tests summary can be observed under Overview section.

We can observe the requests, errors, pages, transactions and the details like how many failed, response time, content length from tables option.

You can change the graph display options to only one graph, two vertical graphs, four vertical panels, etc.

Result

Once the load test execution completes, it display the summary report as follows.

We can export the report to Excel using Create Excel Report option. It creates an excel report with multiple sheets representing average response time, runs, page time, etc.
We can create two types of reports.
Trend – create an excel report with the trends in the selected run result.
Comparison – create an excel report by comparing two or more run results.

Detail option will allow us to study the result in granular format. We can select the time period from the bottom area and study the trend in that particular period.

Rig

Sometimes, in performance test, we need to test the system with very high load. Each machine is having limitation in generating the load. From one system we can go a maximum of 500 to 800 user load. If you need more load, we need to add multiple systems to perform the load test. If we are running it from multiple systems, then the analysis of test results and creation of a combined report will be difficult.

Rig is the solution for running the load test from multiple client systems. We need to install the Load controller on one system and Load agent on other systems. VSTS Rig means a group of systems, which consist of one controller and one or more agents. Controller will distribute the work among the agents and will collect the data from all agents and create a single report.

Load Controller and Load Agent installation and configuration, please refer http://blogs.msdn.com/b/edglas/archive/2007/02/17/load-agent-and-load-controller-installation-and-configuration-guide.aspx

 

Performance Counters

Performance counters are predefined or custom counters used to measure the performance of a system. Performance test result analysis fully depends on the performance counters captured as part of the test. So for a better performance result and better analysis, we need to first understand the important counters related to each server.

We can capture the counters either using the Performance Monitor discussed in this articleor using the Visual studio itself. For capturing the corresponding server counters, add the servers to the computers section in Load test.

Add Computers for collecting Performance Counters

Right click on Counter Set Mappings under the Run Settings of Load test and select Manage Counter Set option.

This will open the Manage Counter Sets window, where we can add the computers. Add the systems and select corresponding counter sets.

When the load test executes, we can observe the counters corresponding to each system in Visual Studio itself.

Adding Counters to Performance Monitor

Setting Performance Monitor to capture the counters is discussed in this article. If you use the performance monitor to capture the counters, then we need to ensure to start the custom counter set before the load test start and end the same after the load test completes. Also need to merge the results from various servers and generate the reports.

Important Counters

Depends on the server and application hosted on the same, we have to capture the counters. For example, the counters required for a database server will be different from that of a web server. Also we have generic counters required for every server like the processor utilization or memory utilization, etc. In this section, we will discuss about the Generic counters, important counters required for an application server, counters required for a database server. Here, we will discuss few of the important counters, but not all.

Generic Counters

\Processor(*)\%Processor time – This counter measure the processor utilization. Capture this counter from all servers and measure the average utilization, In an Idle situation, processor utilization should be less than 80%.

\Process(*)\Private Bytes – Indicates the memory allocated to this process in terms of bytes. This indicates the memory usage of the process.

\Network Interface\Bytes Received/sec, \Network Interface\Bytes Sent/sec, \Network Interface\Bytes Total/sec – these three counters determines the network usage of the server.

ASP.Net web Application Server Counters

\.Net CLR Data\ – This counter group contains counters associated with number of pooled connections, number of failed connect attempts and number of connection pools associated with specific process running on the application server.

\.Net CLR LocksAndThreads\ – this counter group contains information on number of physical and logical threads and any contentions occurred.

\.Net CLR Memory\ – this counter collection includes the number of Gen 0, 1 and 2 collections and heap sizes. This determines the garbage collection rates and in turns the memory management in the application.

\.Net Memory Cache4.0\ – this counter group is for understanding the caching implementation in your application.

\ASP.Net Applications\ – this counters determine the overall request processing and application management. How many requests processed, rejected, queued and disconnected, cache hits and misses, errors, authentication failures, Output cache entries and misses, sessions abandoned and active, transactions committed and aborted.

Database Server Counters

\Database\ – General database related counters like database cache size, I/O database reads and writes, Pages converted and records converted. SQL Server specific counters are defined in separate counter groups.

\SQLAgent:Jobs\ – this group contains counters to understand the status of the jobs running in SQL Server. These indicate how many jobs are active, failed, queued and successfully executed.

\SQLServer:Locks\ – this counter determines how many dead locks occurred in the application execution.

Common Terms

Before discussing about the counters and how to interpret them, we need to look into the various terms associated with a performance test.

Response Time

Most of the performance test is done to understand the response time of an application in a given load. If an application is not completed its performance test, then the response time for an expected user load may not be defined.

Response time is the time taken by a web page or a transaction to respond to the user. 8sec is the standard maximum response time for a web page. If the page is having lot of images or video clips, it may take more time to load. For better usage, we can load the page in progress using Asynchronous calls and Ajax. If the response time is very high, the user experience will be low and the usage of the application may affect it. For decreasing the response time at the same time keeping the rich user interface is one of the challenges.

Throughput

Throughput is the number transactions or inputs handled by the server per second. This indicates how much load or requests the server can handle at once. Depends on the throughput and response time requirements we may plan the clustering of servers.

Resource Utilization

Resource utilization includes the servers’ processor, memory and network utilizations. How much the application utilizing the server resources determine, whether we can go with a single server or need to have multiple servers or not.

These are three major terms or measures we use in performance test. Apart from these measures, we have network time, latency time, request time, test mix, load mix, etc. We will discuss later on the different terms associated with a performance test.

Result Analysis

Now, we have the required counter data and the performance data like response time and throughput. Performance result analysis for different scenarios cannot be explained in one or two documents. Depends on the data we receive as a result of performance test, the analysis will vary.

For example, consider you are getting very high response time with good resource utilization. This simple means, the high response time is not due to any resource issue, it may be due to your SQL part or due to front end code. Look into the SQL counters and determine, whether the particular page is making any database calls which yields lot of reads, writes and CPU cycles. If so, then the issue is with database query or stored procedure. Once, we narrow down the issue to database level, and then we can use the database tools like SQL Execution plan or Database engine tuning advisor to understand the issue and fix the same.

If the issue is with application code, then we need to examine, whether the issue is due to any cache issues, images, threading issues, connection pooling, etc.

 

Advertisements