Performance Testing using Blazemeter for NYRR

Performance testing generally checks how the system performs and behaves. Performance testing examines reliability, scalability, responsiveness, stability, resource usage and speed of your software and infrastructure. Different kinds of performance tests give you different information. Before performance testing, it’s significant to determine your system’s business objectives, so you can understand if your system behaves perfectly or not according to the user’s needs.

After carrying out performance testing, you can examine different KPIs, such as errors per second, the number of virtual users, hits per second, latency, response time, and bytes per second (throughput), as well as the connection between them. With the help of such reports, you can recognize bugs, flaws, errors, and bottlenecks, and decide what needs to be done.
When should you use Performance Testing? The moment when you want to test your app and website performances plus networks, servers, databases, etc.

Performance Testing for NYRR (New York Road Runners)

New York Road Runners
New York Road Runners

ImpactQA did performance testing for New York Road Runner (NYRR), a marathon website using Blazemeter. Here is how we carried out the testing:

  • PROBLEM REPORT

We wanted to stress test a .NET application and were not sure if we wanted to test using VSTS or JMeter. VSTS has its special advantages while JMeter could easily be configured in BlazeMeter. The .NET application was to be tested for a load of 10K users and the challenge was to test it remotely. The web server was hosted in the US and the tests were being conducted from a remote location in India.

To simulate a real-time load environment we needed the client machine (load generating machine) in the US (the servers were already hosted in the US). We needed 10 machines of 8 GB RAM and 3.0 GHz processor). Acquiring such machines in a short time frame in the US and that too only 2 weeks was not only an expensive affair but also time-consuming and not worth the price.

That’s where BlazeMeter bring into play. With the help of BlazeMeter, we could easily simulate the load using the BlazeMeter as the client machine. The beauty of BlazeMeter is that it allows you to hit a server as if you are based out of that country – the screenshot below.

Blazemeter Features
Blazemeter Features

There is also yet another user-friendly feature of BlazeMeter that allowed the user to set the parameters by dragging the slider. See below.

JMeter Engines
JMeter Engines

We selected 290 threads (number of concurrent users) to load test. The duration of Load Test was 1 hour..

We also want to share our experience of scripting a .NET application using JMeter where we were faced with a unique problem while scripting – the problem with ViewState. Although this had nothing to do with us using BlazeMeter we thought it would be a good idea to put it in this blog.
The problem is if we do not correlate the ViewState variable the JMeter script may run 1st time but in subsequent run the script will create problems. The reason for this is that in the first run, the application may accept your recorded ViewState value but when you run the script next time, it will fail as it will no longer accept the previous view state value.

  • SOLUTION

Correlate the ViewState in the script if they exist in the application. Execute the following steps:

1- Find the ViewState parameter

Blazemeter Performance Testing Step 1
Blazemeter Performance Testing Step 1

2- Now see the HTTP Request Name (in our case its login)

Blazemeter Performance Testing Step 2
Blazemeter Performance Testing Step 2

3- Search for the ViewState (Search in View Result tree) in the response of the login that is just above the login in which you will find the ViewState parameter.

Blazemeter Performance Testing Step 3
Blazemeter Performance Testing Step 3

4- Extract it by using “Regular Expression Extractor” in the recorded “Login” sampler

Blazemeter Performance Testing Step 4
Blazemeter Performance Testing Step 4

5- Now use the “Reference Name” (in our case it is ViewState_Login) as a variable name.

Blazemeter Performance Testing Step 5
Blazemeter Performance Testing Step 5

Conclusion:

BlazeMeter is a great platform and delivers complete shift left testing. It has been trusted by the big giants and SMEs to deliver shift-left continuous testing at scale. It also saves time, improves coverage, accuracy and reduces complexity.

Through BlazeMeter, we have successfully run performance testing for our client NYRR and fixed all loopholes and glitches.

5 Core Challenges Associated with Performance Testing

With the growth and demand of Mobile applications, there is growing competition in the marketplace and performance has a great impact on the app’s reliability and dependability. Therefore, there is a high requirement for performance and load tests with suitable blueprint and execution. In general, performance testing validates the scalability, stability, and speed of the applications. This is significant for providing an ‘Amazing’ and ‘Ultimate’ experience to the users.

Key Purpose of Performance Testing:

To ensure the stability, speed, and accuracy of the software.

The Performance test is critical to authenticate the scalability, reliability, and responsiveness features of the software under test.

In software testing, the rapidity and response time of the specific application is one of the vital attributes. The performance testing uncovers the performance hiccups and bottlenecks to uphold the attention and interest of the customer. Such type of Testing falls into several categories like Stress Testing, Load Testing, and Spike Testing to mention a few.

The most common performance hiccups and glitches observed in any web/ mobile app are as follows:

• Poor response time

• Memory utilization

• Disk usage

• Poor scalability

• CPU utilization

• Loading Bottleneck

• Extended Loading time

• Software configuration problems for the Web server,

• Operating System limitations

• Load balancers, databases, etc.

• Poor network configuration

5 Key Challenges of Performance Testing:

A software application should be thoroughly tested with accurate testing processes before releasing to the customers. Like any other testing procedures, this type of testing also comes with a number of Challenges.

1- Testing Tools and Situation

Performance testing requires you to test the application under reasonable scenarios. However, lots of times the enterprises do not have the budget or environment for executing the tests. One of the most efficient ways to conduct performance testing is to generate a replica of the production environment. Yet, the challenge lies in getting the proper outcome with limited resource and uneven scenarios. For example, creating a scenario where multiple users log into the system at the same time is hard.

There is countless consideration before moving further on with such type of testing. They comprise:

• Technology

• Protocols

• Browser

• Tool

• Platform

• Budget

• Hardware Requirements

2- Test Coverage

It necessitates vast knowledge and talent to cover all the functionalities when doing performance testing. But, the scenarios must be exhibitive of a range of parameters. One can try automating core functionalities by gathering various scenarios. User data must be predicted properly as there would be lots of users using the system in their own context.

3- Performance Test Analysis

Evaluating and examining the performance test outcomes is the most challenging task in Performance and load of testing. It requires you to have comprehensive knowledge and better decision to examine tools and reports. In addition, you need to by and large upgrade the tests based on the definite situation.

4- Non-Functional Requirements

Both functional and non-functional requirements are vital for such kind of test. Functional needs are more precise and include input data types, algorithms, and functionality to be covered. The challenge lies in discovering less specific non-functional requirements. They include capacity, usability, stability, responsiveness, and interoperability.

5- Analyzing the Performance Test Outcomes

This is indeed one of the significant and trickiest of the challenges for Performance testing. A good amount of judgment and thorough knowledge of the system is necessary to examine the several performance result reports besides the tools used for the actual tests. A skilled tester should be able to judge the scenarios and if the test is done or not. The tester should be able to continuously refine the tests and keep adding tests consistent with the situations estimated during the course of time.

JMeter Distributed Testing: Step by Step

Pre-Requisites for Distributed Testing

For distributed testing using JMeter, we need to make some specific configurations both on the Master as well as the Slave machines.

Basic Requirements that both Master and the Slaves machines should have in common:

  • Same Java version
  • Same JMeter version (It’s better to copy the JMeter from one machine to the rest of all)
  • All machines should be on the same subnet

This doc will give you a brief idea about doing Distributed Remote testing using a Windows machine as the master and Linux machines as slaves.

Slave Configuration

After having a similar copy of JMeter in the slaves, follow the below steps for all the Slave machines:

1) Open jmeter-server file present in the bin directory of jmeter and uncomment the below line –

RMI_HOST_DEF=-Djava.rmi.server.hostname=xxx.xxx.xxx.xxx
Also instead of xxx.xxx.xxx.xxx, give the ip of the linux machine you have opened.

2) Give a specific rmi port no. in the below line of the same jmeter-server file –

${DIRNAME}/jmeter ${RMI_HOST_DEF} -Dserver_port=${SERVER_PORT:-2010} -s -j jmeter-server.log “$@”

This port will be further used in the master configuration, hence give different ports for all the different slave machines.

3) After making the above changes, execute JMeter-server on all the slaves.

Master Configuration 

The following changes need to be done for the master setup:

1) Open jmeter.properties present in the bin directory on the Master machine and mention all the slave ips in as remote hosts with the ports assigned in the above slave configuration –
remote_hosts=localhost: 1099, 104.239.234.143:2010
Localhost need not be mentioned if you do not want to use the Master machine to be a part of remote testing to put the load on the application.

2) Set the server port in the same jmeter.properties file 
-server_port=1099

3) After making the above changes, execute jmeter-server.bat on your Windows master machine which will be present in the bin directory.

4) Now execute jmeter.bat on Master machine for your JMeter GUI where the execution will be done and viewed.

Please note that the number of Virtual Users that will be assigned in the thread group in your JMeter GUI, an equal number of users will be generated from each of the machines you are using. For e.g.: let’s say we have one Windows as the master and two Linux machines as the slaves and the VUsers assigned is 10, then a total of 30 users will be making hits on the application under test.

Execution Methods

After doing the slave and master setup, open the scenario to be executed and execute in any of the 2 below methods:

1) Go to Run->Remote Start All. You can also Go to Run->Remote Start and select any one of the machines you wish to generate load.

2) On the top panel, we have a button for the remote start (After the test has completed, c lick on the button beside in red to remote stop all)

Exceptions and Workarounds

“Connection refused” is a common error that we encounter during remote distributed testing. For this, one should try the below checks:

1) Make sure the remote host entries and the ports assigned both on the master and slave are correct. Also, jmeter-server should be executed first followed by the jmeter-server.bat and jmeter.bat on the Master machine.

2) Verify the firewall settings on Windows and Linux machines. Disable if required. On Linux, that can be achieved by the below command:

Trends of ERP Testing to Watch for 2019

ERP (Enterprise Resource Planning) systems no longer necessitate any sort of introduction. For businesses, investing in a good system no longer is an alternative. It is a necessity. Enterprise Resource Planning systems have been a part of the business software landscape for a long time. Ever since their foray into the world of business, vendors are incessantly evolving them, so they are more powerful, robust, simpler to use, and affordable.

The 5 Major ERP trends that we should consider in 2019:

1.Competition from Disruptors- The Enterprise resource planning behemoths that have conquered the industry are encountering stiff competition from new, often Software-as-a-Service (SaaS)-only startups & the proliferation of fresh trends threatening to upset how enterprises collect and process data, and also operate. Renowned companies like FinancialForce (already having more than 1,300 Enterprise resource planning customers) and Kenandy are creating solutions based on the Salesforce App Cloud to make them alluring to users of the popular CRM and sales automation tool. On the disruption side, data visualization, big data, and artificial intelligence (AI) top the list of newest technologies that threaten to alter the way Enterprise resource planning systems are built and used. Enterprises looking to update their Enterprise resource planning systems in the year 2019 will need to become aware of to how their new prospects handle such trends. Database performance will be the core performance indicator (KPI) for Enterprise resource planning in 2019.

2.Enterprise resource planning, SaaS, & Hybrid ERP- Enterprise resource planning apps are stored on your servers, which mean you are responsible for long-term hardware maintenance, hardware costs and data backup and recovery. SaaS-based applications are stored on cloud-based servers, which are much less costly, very fast to upgrade and scale, and don’t take up clunky servers. Hybrid ERP systems are becoming famous in some sections as long-time Enterprise resource planning customers enjoy the ability to move certain Enterprise resource planning functions to the cloud while sustaining tight, on-premises control over other facets, particularly those most vulnerable to compliance regulation.

3.Focused on Social Media and Digital Marketing- These days, Enterprise resource planning is specifically focused more and more on functions than marketing, but those modules will need to become social media-savvy by the year 2019. Future Enterprise resource planning systems will need to be able to integrate direct marketing & data gathering links across manifold social media channels to remain on the top list and highly competitive.

4.The Internet of Things (IoT) is going to stay- As more and more products and devices become connected to the internet, more data can be instantly funneled into the Enterprise resource planning system, and that’s an advantage that can’t be ignored. This trend offers better oversight over things like the supply chain and appliance performance, and it also gives overall data pool for good decision making.

5.ERP for the Subsidiary- As more Enterprise resource planning systems are being delivered via the cloud, it is becoming far easier to deploy such SaaS-based tools incrementally. Rather than replacing ERP whole-hog, big giant companies are selecting one slice of the business and plugging in SaaS Enterprise resource planning on a trial basis. This approach lets businesses observe SaaS Enterprise resource planning performance to evaluate how it might fit into the existing on-premises Enterprise resource planning implementation—or whether it should replace on-premises Enterprise resource planning throughout the whole enterprise.

Agile Digital Transformation

Let us understand what exactly AGILE is all about–

AGILE – A term coined in 2011 by a small group of people who were tired of the traditional approach of Software Management of developing projects.

Agile helps teams to provide fast and unpredictable responses to the feedback they receive on their projects.

The twelve principles of agile development include:

1.Customer satisfaction through early & continuous software delivery – Clients are happier if they don’t have to wait for extended periods of time and receive working software at timely intervals between releases.

2.Accommodate changing needs throughout the development process – Whenever there is a change in requirement or feature, it should not cause a delay in the development process and get accommodated easily in the software.

3.Frequent delivery of working software – As Scrum operates in software sprints or iterations, this ensures regular delivery of working software.

4.Collaboration between the developers & business stakeholders during the project – Better decisions are made when the business and technical team collaborate.

5.Support, belief, & motivate the people involved– The unhappy teams cannot deliver their best like the motivated teams. So, support and trust are needed on the team.

6.Enable face-to-face interactions– There is no miscommunication when teams are co-located which not only saves time but also gives better interaction result.

7.Working software is the prime measure of improvement– The ultimate measure of progress is delivering functional software to the customer.

8.Agile processes to support a consistent development pace–Agile process establishes a routine through which teams establish an iterative and maintainable speed through which they can deliver functional software, and they repeat it with each release.

9.Attention to technical aspect & design enhances agility– The apt skills and righteous design ensures the team can maintain the speed, constantly improve the product and back up the change.

10.Simplicity– Develop just enough to get the job done at the moment.

11.Teams which are self-organized encourage great designs, requirements, and architectures– Motivated and Dexterous team members who have the decisiveness, take ownership and interact regularly with other members of the team and share designs that deliver standard products.

12.Regular reflections on how to become more and more effective– Self-evolution, process betterment, advancing expertise, and techniques help the team members to work more coherently.

The aim of Agile is to merge development with the business needs, and the success of Agile is evident. Agile projects are customer-centric and advocate customer guidance and engagement. Because of this, Agile has grown to be an overall view of software development throughout the software development industry and an industry all by itself.

What is a digital transformation?

It is the unification of digital technology into all areas of business, radically changing how you utilize and yield value to customers. It is also an aesthetic change that requires organizations to continuously confront the status quo, investigate, and get comfortable with unfulfillment. 

Digital Transformation is the unprecedented use of digital technology to resolve conventional problems. These digital solutions capacitate inherently new types of innovation and artistry, rather than simply upgrade and support conventional methods.

What is AGILE DIGITAL TRANSFORMATION?

An agile transformation is an act of transfiguring an organization’s form or nature gradually to one that is able to accept and advance in an adaptable, cooperative, self-organizing, fast wavering environment. The Agile Manifesto values and principles can be followed and taught throughout any type of establishment as it does not just apply to development teams.

The whole establishment needs to interpret the clarity of an agile transformation and the value of it in order to gain from the rewards of achieving true, healthy agility. The whole cultural and organizational mindset must change to one that embraces a culture of self-organization and collaboration.

Principles of Agile Digital Transformation:

PRINCIPLE 1:  START WITH A TRANSFORMATIVE VISION

As per a recent Gartner survey, 63% of business leaders stipulated they don’t exactly know what would be the possibilities of next-generation technology. Not astonishingly, just 13% of respondents said they have discovered the next paramount digital business technology investment. The reason for this is that the company lacks a transformation vision that will plan out a digital strategy and, more importantly, give the business the ability to measure progress and make real-time adjustments to improve outcome. Superior management must create, be coherent, and interface the compelling future digital vision.  Transfiguration does not happen bottom-up.

PRINCIPLE 2:  FOCUS ON BUILDING DIGITAL CUSTOMER ENGAGEMENT

Successful digital transformations are always built on front-end customer experience, unlike traditional digital projects. There should be more and more exposure to new growth opportunities by adding digital features to products as well as by changing direction and considering how the products and their services adapt to the digital customer. To create a value we need not use technologies and tools as an integrated package instead there should be sets of applications that can automate the user experience through social, data, cloud and mobile.

PRINCIPLE 3:  SUPPORT THE VISION WITH SECURE DIGITAL PLATFORMS

The organizations must always prioritize risks germane as per their specific operation. As the hottest application in need is for security, there should be smart tools to manage the risks for detecting the intrusions quickly and to respond in real time. The key to liberating the existing IT assets and enabling digital innovation with growth is the interface to digital components. The security of assets and data is paramount, the right strategy will support the speed, safety, and growth required in today’s digital economy.

PRINCIPLE 4: DRIVE INSIGHT WITH DATA-DRIVEN VISUALIZATION

The growing digital organizations not only persistently collect data but identify and envision that data in a context that induces insights that can be acted upon. The solution to unlocking real-time data intelligence for zestful and unified customer engagement is comprehending and address customer personas and micro-segments.

Traditionally companies have a reasonably strong foundation of sales transactional data. However, this data often lacks the necessary dimensionality to create meaningful demographic, attitudinal, and predictive insights. In addition, data rarely is augmented by leveraging publically available and purchased data.

Many organizations are using data visualization to communicate information clearly and efficiently to users through statistical graphics, plots, infographics, and dynamic tables and charts. Effective envisions helps users in examining and reasoning data and documentation.

PRINCIPLE 5: EMBRACE DIGITAL AGILITY TO CREATE ADVANTAGE

Due to the constantly changing customer and market conditions, Business leaders often wrestle to execute extensive projects. The conventional business model features unconnected, unintegrated platforms by business function and projects with 6-18 month lifecycles. Oftentimes, by the time the project is accomplished, market and user requirements have changed with success criteria and ROI seldom realized.

To circumvent these perils, firms must grab adaptable differentiation by developing a “digital agility advantage” that allows a company to embrace market and operational changes as a matter of routine through the use of digital technologies. Digital agility initiatives are rooted in 30-day sprints with new iterations built better and faster. This allows a company to constantly evaluate and modify – the concept of learning, launch, re-learn, re-launch – rectifying the perspective in attainable iterations. Thriving organizations in the digital age must exhibit an awareness of how to be agile; only then, they would be able to execute in an agile way.