Why is Load Testing Significant?

Load Testing can monitor the response times of the system for every single transaction during a set period of time. This kind of monitoring can bring significant information, precisely for stakeholders and business managers, who look for conclusions based on these outcomes, along with any data for supporting these findings. Such type of performance testing can also raise attention to any complexity and issues in the application software and fix these performance bottlenecks before they become extra problematic. As a developer or QA engineer, one can possibly incorporate several kinds of tests into the code checking: integration tests, unit tests, UI tests, and more. Load testing (kind of performance testing) supports testers or engineers to understand the behavior and accessibility of the application. It critically tackles the following queries:

• Do the codes show the desired performance?
• Does the fresh feature that implemented make a notable effect?
• Does the new modification in the hardware configuration execute well?
• Does the upgrading in the environment work under limits?

You may also like: 5 Core Challenges Associated with Performance Testing

Load Testing Tools

It is also significant to pick out the specific tool that best fits your business needs. Examples of these are Loadrunner, LoadView, and WebLOAD. WebLOAD works on any app using NET, Ajax, HTML5, Oracle Forms, and other technologies. LoadView computes performance in real browsers. LoadRunner an open-source tool can easily manage users at the same time.

We can use SaaS-based BlazeMeter with ease scale and run your load tests. You just need to upload your JMeter file, configure, run, and view real-time results in insightful graphs that can store data over time. BlazeMeter integrates with CI tools like TeamCity, Bamboo, and Jenkins.

Agile, automation continuous and integration are not complete without load testing. Lighter versions, Quick releases, and more automation make a good product and create a perfect working environment for engineer and developers. Load test is an inherent part of any continuous integration process. Thus, when you are setting up your automated continuous integration cycle with tools like open source Jenkins, add load testing to your workflow, and with load testing tools like Apache JMeter™ (open source) .If you require to test in the cloud, try to run your load tests with BlazeMeter.

7 reasons why Load Testing is significant have been listed below:

  • Highly Efficient — As automation testing is in the cloud, it has become more efficient and flexible resource allocation. We can even effortlessly connect several tools via distinct APIs, getting a singular view of these varied pieces means, even more, saving.
  • Detection of bottlenecks before deployment — Evaluating the part of the software or a website before deployment can emphasize bottlenecks; permitting them to be addressed prior to incurring the large real-world costs.
  • Cost Cutting — As it is in the cloud and we can run with varied tools like LoadUI Pro, you can without difficulty save your extra time and costs by creating tests that don’t involve expensive hardware necessities and that are repeatable.
  • Collaborative — We can grab all the benefits of an in-person office, with a singular view of the entire automated testing, however, without the need to be in a similar place. Cloud-based automation test also signifies that you can pick out when you are running it — Continuous testing? Or possibly any time for checking the reports? Or for load testing its best at your peak usage.
  • High productivity in short duration — Speedy results in a short duration of test cycles, quick set up, and deployment. While you are willing to continue to write your own tests, automation testing gets you up and running almost immediately.
  • Load Testing is for anyone — Anybody can potentially test this type of testing whenever he/ she needs to, but the results are crystal clear which specifically allows everybody to participate in this critical business matter.
  • Customer satisfaction — If the response times of the site are short even as it scales up to a higher audience, one-time clients will be more apt to return to.

In short, performance testing or load testing should be a major priority before releasing app or software. It should be implemented early on in software or application development so as to catch extra flaws and bugs and increase client satisfaction while saving precious money and time down the line.

ImpactQA did performance testing for marathon website like New York Road Runner (NYRR), using Blazemeter. Check how we carried out the testing: Performance Testing using Blazemeter for NYRR

Performance Testing using Blazemeter for NYRR

Performance testing generally checks how the system performs and behaves. Performance testing examines reliability, scalability, responsiveness, stability, resource usage and speed of your software and infrastructure. Different kinds of performance tests give you different information. Before performance testing, it’s significant to determine your system’s business objectives, so you can understand if your system behaves perfectly or not according to the user’s needs.

After carrying out performance testing, you can examine different KPIs, such as errors per second, the number of virtual users, hits per second, latency, response time, and bytes per second (throughput), as well as the connection between them. With the help of such reports, you can recognize bugs, flaws, errors, and bottlenecks, and decide what needs to be done.
When should you use Performance Testing? The moment when you want to test your app and website performances plus networks, servers, databases, etc.

Performance Testing for NYRR (New York Road Runners)

New York Road Runners
New York Road Runners

ImpactQA did performance testing for New York Road Runner (NYRR), a marathon website using Blazemeter. Here is how we carried out the testing:

  • PROBLEM REPORT

We wanted to stress test a .NET application and were not sure if we wanted to test using VSTS or JMeter. VSTS has its special advantages while JMeter could easily be configured in BlazeMeter. The .NET application was to be tested for a load of 10K users and the challenge was to test it remotely. The web server was hosted in the US and the tests were being conducted from a remote location in India.

To simulate a real-time load environment we needed the client machine (load generating machine) in the US (the servers were already hosted in the US). We needed 10 machines of 8 GB RAM and 3.0 GHz processor). Acquiring such machines in a short time frame in the US and that too only 2 weeks was not only an expensive affair but also time-consuming and not worth the price.

That’s where BlazeMeter bring into play. With the help of BlazeMeter, we could easily simulate the load using the BlazeMeter as the client machine. The beauty of BlazeMeter is that it allows you to hit a server as if you are based out of that country – the screenshot below.

Blazemeter Features
Blazemeter Features

There is also yet another user-friendly feature of BlazeMeter that allowed the user to set the parameters by dragging the slider. See below.

JMeter Engines
JMeter Engines

We selected 290 threads (number of concurrent users) to load test. The duration of Load Test was 1 hour..

We also want to share our experience of scripting a .NET application using JMeter where we were faced with a unique problem while scripting – the problem with ViewState. Although this had nothing to do with us using BlazeMeter we thought it would be a good idea to put it in this blog.
The problem is if we do not correlate the ViewState variable the JMeter script may run 1st time but in subsequent run the script will create problems. The reason for this is that in the first run, the application may accept your recorded ViewState value but when you run the script next time, it will fail as it will no longer accept the previous view state value.

  • SOLUTION

Correlate the ViewState in the script if they exist in the application. Execute the following steps:

1- Find the ViewState parameter

Blazemeter Performance Testing Step 1
Blazemeter Performance Testing Step 1

2- Now see the HTTP Request Name (in our case its login)

Blazemeter Performance Testing Step 2
Blazemeter Performance Testing Step 2

3- Search for the ViewState (Search in View Result tree) in the response of the login that is just above the login in which you will find the ViewState parameter.

Blazemeter Performance Testing Step 3
Blazemeter Performance Testing Step 3

4- Extract it by using “Regular Expression Extractor” in the recorded “Login” sampler

Blazemeter Performance Testing Step 4
Blazemeter Performance Testing Step 4

5- Now use the “Reference Name” (in our case it is ViewState_Login) as a variable name.

Blazemeter Performance Testing Step 5
Blazemeter Performance Testing Step 5

Conclusion:

BlazeMeter is a great platform and delivers complete shift left testing. It has been trusted by the big giants and SMEs to deliver shift-left continuous testing at scale. It also saves time, improves coverage, accuracy and reduces complexity.

Through BlazeMeter, we have successfully run performance testing for our client NYRR and fixed all loopholes and glitches.

5 Core Challenges Associated with Performance Testing

With the growth and demand of Mobile applications, there is growing competition in the marketplace and performance has a great impact on the app’s reliability and dependability. Therefore, there is a high requirement for performance and load tests with suitable blueprint and execution. In general, performance testing validates the scalability, stability, and speed of the applications. This is significant for providing an ‘Amazing’ and ‘Ultimate’ experience to the users.

Key Purpose of Performance Testing:

To ensure the stability, speed, and accuracy of the software.

The Performance test is critical to authenticate the scalability, reliability, and responsiveness features of the software under test.

In software testing, the rapidity and response time of the specific application is one of the vital attributes. The performance testing uncovers the performance hiccups and bottlenecks to uphold the attention and interest of the customer. Such type of Testing falls into several categories like Stress Testing, Load Testing, and Spike Testing to mention a few.

The most common performance hiccups and glitches observed in any web/ mobile app are as follows:

• Poor response time

• Memory utilization

• Disk usage

• Poor scalability

• CPU utilization

• Loading Bottleneck

• Extended Loading time

• Software configuration problems for the Web server,

• Operating System limitations

• Load balancers, databases, etc.

• Poor network configuration

5 Key Challenges of Performance Testing:

A software application should be thoroughly tested with accurate testing processes before releasing to the customers. Like any other testing procedures, this type of testing also comes with a number of Challenges.

1- Testing Tools and Situation

Performance testing requires you to test the application under reasonable scenarios. However, lots of times the enterprises do not have the budget or environment for executing the tests. One of the most efficient ways to conduct performance testing is to generate a replica of the production environment. Yet, the challenge lies in getting the proper outcome with limited resource and uneven scenarios. For example, creating a scenario where multiple users log into the system at the same time is hard.

There is countless consideration before moving further on with such type of testing. They comprise:

• Technology

• Protocols

• Browser

• Tool

• Platform

• Budget

• Hardware Requirements

2- Test Coverage

It necessitates vast knowledge and talent to cover all the functionalities when doing performance testing. But, the scenarios must be exhibitive of a range of parameters. One can try automating core functionalities by gathering various scenarios. User data must be predicted properly as there would be lots of users using the system in their own context.

3- Performance Test Analysis

Evaluating and examining the performance test outcomes is the most challenging task in Performance and load of testing. It requires you to have comprehensive knowledge and better decision to examine tools and reports. In addition, you need to by and large upgrade the tests based on the definite situation.

4- Non-Functional Requirements

Both functional and non-functional requirements are vital for such kind of test. Functional needs are more precise and include input data types, algorithms, and functionality to be covered. The challenge lies in discovering less specific non-functional requirements. They include capacity, usability, stability, responsiveness, and interoperability.

5- Analyzing the Performance Test Outcomes

This is indeed one of the significant and trickiest of the challenges for Performance testing. A good amount of judgment and thorough knowledge of the system is necessary to examine the several performance result reports besides the tools used for the actual tests. A skilled tester should be able to judge the scenarios and if the test is done or not. The tester should be able to continuously refine the tests and keep adding tests consistent with the situations estimated during the course of time.

JMeter Distributed Testing: Step by Step

Pre-Requisites for Distributed Testing

For distributed testing using JMeter, we need to make some specific configurations both on the Master as well as the Slave machines.

Basic Requirements that both Master and the Slaves machines should have in common:

  • Same Java version
  • Same JMeter version (It’s better to copy the JMeter from one machine to the rest of all)
  • All machines should be on the same subnet

This doc will give you a brief idea about doing Distributed Remote testing using a Windows machine as the master and Linux machines as slaves.

Slave Configuration

After having a similar copy of JMeter in the slaves, follow the below steps for all the Slave machines:

1) Open jmeter-server file present in the bin directory of jmeter and uncomment the below line –

RMI_HOST_DEF=-Djava.rmi.server.hostname=xxx.xxx.xxx.xxx
Also instead of xxx.xxx.xxx.xxx, give the ip of the linux machine you have opened.

2) Give a specific rmi port no. in the below line of the same jmeter-server file –

${DIRNAME}/jmeter ${RMI_HOST_DEF} -Dserver_port=${SERVER_PORT:-2010} -s -j jmeter-server.log “$@”

This port will be further used in the master configuration, hence give different ports for all the different slave machines.

3) After making the above changes, execute JMeter-server on all the slaves.

Master Configuration 

The following changes need to be done for the master setup:

1) Open jmeter.properties present in the bin directory on the Master machine and mention all the slave ips in as remote hosts with the ports assigned in the above slave configuration –
remote_hosts=localhost: 1099, 104.239.234.143:2010
Localhost need not be mentioned if you do not want to use the Master machine to be a part of remote testing to put the load on the application.

2) Set the server port in the same jmeter.properties file 
-server_port=1099

3) After making the above changes, execute jmeter-server.bat on your Windows master machine which will be present in the bin directory.

4) Now execute jmeter.bat on Master machine for your JMeter GUI where the execution will be done and viewed.

Please note that the number of Virtual Users that will be assigned in the thread group in your JMeter GUI, an equal number of users will be generated from each of the machines you are using. For e.g.: let’s say we have one Windows as the master and two Linux machines as the slaves and the VUsers assigned is 10, then a total of 30 users will be making hits on the application under test.

Execution Methods

After doing the slave and master setup, open the scenario to be executed and execute in any of the 2 below methods:

1) Go to Run->Remote Start All. You can also Go to Run->Remote Start and select any one of the machines you wish to generate load.

2) On the top panel, we have a button for the remote start (After the test has completed, c lick on the button beside in red to remote stop all)

Exceptions and Workarounds

“Connection refused” is a common error that we encounter during remote distributed testing. For this, one should try the below checks:

1) Make sure the remote host entries and the ports assigned both on the master and slave are correct. Also, jmeter-server should be executed first followed by the jmeter-server.bat and jmeter.bat on the Master machine.

2) Verify the firewall settings on Windows and Linux machines. Disable if required. On Linux, that can be achieved by the below command:

Increasing Importance of Quality Engineering in Software Testing

How can a company win? One of the key criteria is to ensure good quality of its products and services. But the traditional testing and QC paradigm is not enough in the context of emerging technologies. It has proven to be inefficient: if some shortcomings are revealed, the product may have to be redesigned, requiring additional expenses and extra time. That is why something new is being executed in business — quality engineering solutions. Quality Engineering (QE) is the series of procedures by which software quality is analyzed and improved throughout the application or software development lifecycle. It differs from traditional Quality Assurance in that it prevents defects as well as discovers them.

The QE approach implies that every single stage of the product/ software development cycle is under a scrupulous test of quality engineers. Furthermore, the quality maintenance is offered long after the product is delivered. The execution o such strategy in manufacturing or software development procedures guarantees the sufficiency of the output from the very start reduces imperfections, flaws, and reduces potential losses. In other words, quality engineering is the analysis, development, management, and maintenance of diverse systems compliant with high standards.

What are the rewards of Quality Engineering?

With Quality Engineering, the core benefit for your application development cycle is that you are actually making all the proposed advantages of DevOps and Agile more real. Also the teamwork between developers and testers is more real, more in line with the agile ethos. It is also integrated with Test Management solutions so that the outcomes appear on the dashboard instantly, without a human trigger. With shortened release cycles, time to ensure Quality also reduces considerably. Testers have to be involved at the start of the cycle as they will be setting up the testing environment and framework which will be relied upon for all future sprints. Done right, Quality Engineering offers a great deal more speed in testing. It mainly relies more on Test Automation than manual testing. It is hard to imagine a Quality Engineering function that doesn’t have Test Automation at its center. Yet again, done right, it creates more flexibility and speed for the whole development cycle. It is not considered just functional and non-functional testing, but every single layer and integration that can and should be tested.

In current Digital era, a Quality Engineer should have experience in programming and be supposed to be able to write software as the situations demands. While the Software Development team focuses on constantly upgrading the application, the Quality Engineering team main responsibilities are:

  • Setting up new parameters and standards
  • Optimization of test cases, & improving automation efficiency
  • Identification of drawbacks
  • Generating a plan for improvement
  • Plan execution using different tools and methods
  • Assessing & implementing new technologies and tools
  • Following up to make sure that issues have been solved
  • Creating tailor automation solutions to address application specific use cases
  • Create frameworks & accelerators that help scale QE across manifold channels, Enterprise wide.

Quality engineering is driven by emerging technologies like AI (artificial intelligence), Big Data analytics and IoT. Automation is the driving force behind turning the traditional testing into an effectual quality support model.

Bottom line

Performance of the application/ software is of paramount importance. Every outage, crash, drawbacks, and even slowing down of the app or processing/ working on a client request has the potential to impact revenue directly. It is the responsibility of QE team to not only identify such issues, but also work on identifying/removing the root cause of such problems. This demands a sound understanding of app architecture, monitoring tools, several enterprise sub systems that are catering to the app etc. Overall, Quality Engineering team provides substantial insights about the root cause or issue and solved it in the fastest possible manner.