Вікторія Горбік Useful Guidelines 26 June 2025, 10:00

AI - threat or helper? How artificial intelligence is changing the tester profession

For a very long time, the specialization of a tester was positioned as the easiest and fastest way to enter IT. And still, some courses and schools periodically speculate on this issue. However, it has already become clear to many that everything is not so simple and unambiguous. In addition, artificial intelligence is developing rapidly, thanks to the tools of which certain transformations are also observed in the work of testers and approaches to the selection of candidates.

Serhiy Romanov, a tester with over 15 years of experience who currently works for Ukrainian company Brightgrove, which collaborates with streaming giant Pluto TV, owned by Paramount Global, shared with dev.ua his observations on changes in QA, as well as tips on AI tools and examples of how AI can be used in testers’ work, which can both help and threaten.

Information about Sergei Romanov

Serhiy Romanov, 36, was born and educated in Kharkiv. There he gained his first significant experience as a PM and Automation QA Engineer at Videal, a company that creates innovative solutions for analyzing, processing, and using big data. Later, he worked as a Software Development Engineer in Test at the software development company SoftServe and the American software development company Red Hat.

For the past five years, Serhiy Romanov has been a manager at Pluto TV, an ad-supported streaming service owned and operated by Paramount Streaming, a division of Paramount Global. He currently manages three teams of testers: a team in Europe, where his colleagues from Kharkiv, Kyiv, and Poland work, a team in the United States, and a team in North America.

About the tester — roles, tasks, tools

It is testers who are responsible for the quality of software. In particular, they check how well the software meets the requirements, and above all, how the program they are creating behaves in different conditions. They also find out what critical errors can be detected when using this program or this product. And although testers actually play the role of the end user, their influence, according to Serhiy Romanov, is currently underestimated.

Serhiy Romanov, tester with over 15 years of experience (Photo from personal archive)

The main tasks of testers performing manual or automated testing are:

requirements analysis;
development of test scenarios;
identification of defects;
checking the correction of previously detected errors.

So, first of all, the tester constantly communicates with the team, in particular with developers, analysts, and product managers, to move together in the same direction to create a stable and high-quality product.

Among the main tools that professionals use for manual testing are:

TestRail is a test management system that helps you structure the process, keep track of test cases, track results, and report on them.
Jira is a commercial system from Atlassian that helps with project management and creating and tracking bugs.
Postman is an API testing and analysis tool that allows you to send requests to the server, receive responses, and analyze their behavior to test the operation of backend applications.
Charles Proxy is a network traffic interception and analysis tool that allows you to view HTTP requests and responses between a client and a server in real time to test, investigate, and debug applications.
Browser DevTools is a set of tools built into modern web browsers (e.g. Chrome, Firefox, Edge) that will help you study the HTML, CSS, and JavaScript code of a page, debug the code and find errors, analyze site performance, and emulate different devices and screen resolutions.

To automate testing, in particular to check the functionality of the system, specialists use programs that allow you to write automated tests and run them in Continuous Integration and Continuous Deployment (CI-CD) processes:

Selenium is an open source framework for web application automation that supports various programming languages and simplifies cross-browser testing.
REST Assured is a Java library that helps create and execute HTTP requests to APIs, as well as verify that the responses match the expected results.
TestNG and JUnit are frameworks for automated testing in Java that allow you to manage the order of test execution, group them, run tests in parallel, and get detailed reports.
Jenkins is an open-source, standalone continuous integration platform written in Java that provides hundreds of plugins to automate the build, testing, deployment, and maintenance processes of any project.

In the early stages of a project, testers typically first write automated tests to verify functionality. The next stage is performance testing, which simulates real-world usage scenarios to assess the load on the system during prime time, the time when the largest number of users are expected. To assess performance, testers run load tests on the backend using tools such as:

JMeter is the simplest tool for measuring acceleration, consisting of plugins that are added to the project and can be configured.
Gatling is a convenient load testing tool that allows you to simulate traffic using code, mostly in Scala. It is easily configured and integrated into automated CI/CD pipelines, and generates detailed reports.

«Personally, for me, a tester is not necessarily someone who is engaged in finding defects, but definitely someone who understands the product and helps the team create high-quality and technological software,» notes Sergey Romanov.

How have tester tasks changed with the advent of AI?

The profession of a tester is not standing still, and artificial intelligence has changed the role of these specialists, transforming it. As Serhiy emphasizes, if earlier the main attention was paid to manual testing or writing automated tests, today artificial intelligence takes over most of the routine tasks. But the specialist is sure that this does not mean that the role of a tester is disappearing.

On the contrary, AI helps testers' tasks transform in favor of technical thinking, when specialists need to think strategically. In particular, decide what the team can do uniquely and analyze what artificial intelligence cannot do to find defects in the product.

«We move from automation to analytics, when we need to think more about how the product should work,» the tester emphasizes.

AI tools for testers

With the advent of artificial intelligence, not only the tasks of testers have changed, but also the tools they use in their work.

In particular, as Serhiy Romanov says, ChatGPT is very helpful — testers can use it to prepare test documentation. The expert notes that previously there were very few testers who prepared high-quality documentation for their projects. Thanks to the chatbot, test documentation began to look much better, and its structure became more logical and easier to understand. However, there are nuances, in particular between the paid and free versions of the program. For example, according to the specialist, a test plan for one of the features, compiled by the paid version of ChatGPT, requires about 20% clarification. When using the free version of the chatbot, the number of necessary corrections increases to 30–40%.

«In general, ChatGPT conveys the essence of the story about 90% correctly, and 10% remains for information that is not confirmed by anyone and which he most likely invented himself,» the expert notes.

Serhiy also emphasizes that for high-quality use of the tool, it is important to prepare a prompt in advance. In the prompt, you need to describe in detail your expectations, emphasize that the chatbot should provide only verified and accurate information, and not invent anything in matters where ChatGPT is incompetent. In addition, in the paid version of the chatbot, there is an opportunity to leave a request for research, for example, regarding some new features.

Also, among other AI tools, Serhiy Romanov recommends using:

MABL is a cloud-based automated testing platform that combines the power of artificial intelligence and machine learning to create, execute, and maintain end-to-end tests.

An example of how Mabl generates a test script in real time (Screenshot by Sergey Romanov)

The platform supports testing of web applications, mobile applications, APIs, and also provides basic capabilities for performance analysis.

Example of how Mabl executes a test script (Screenshot by Sergey Romanov)

Mabl automatically detects changes in the interface, adapts tests to these changes, and reduces false positives. With intelligent analytics, Mabl helps quickly identify issues affecting the user experience and ensures continuous quality throughout all stages of development.

Screenshot from an analytical report in Mabl (Screenshot by Sergey Romanov)

TESTIM is a test automation platform that uses artificial intelligence to accelerate the creation, stabilization, and maintenance of automated tests, and provides TestOps tools to manage the test process.

Testim dashboard with key project indicators (Screenshot by Sergey Romanov)

Machine learning algorithms help identify changes in the interface, reduce false positives, and keep tests relevant even in dynamically changing applications.

Test results in Testim (Screenshot by Sergey Romanov)

Additionally, Testim can be used with Sealights, which allows you to determine which parts of the code were covered during automated tests. This helps focus testing only on changed or risky areas of the application, increasing the efficiency of tests and reducing the feedback hour.

Screenshot from Sealights, showing that due to the analysis of the changed code, the system recommended skipping 99% of the tests (Screenshot by Sergey Romanov)

Functionalize is a cloud-based AI platform for functional test automation that uses generative AI to create, optimize, and maintain automated tests. It is designed to minimize manual work by automatically generating tests based on code changes and user interaction scenarios with the application. The platform automatically identifies the most risky or changed areas of code that need to be reviewed. Functionalize also offers relevant tests, allowing you to reduce scripting time and focus on the most critical modules of the application. The platform also integrates with CI/CD, version control systems Git, Jira, and other development tools.

These tools use machine learning to create so-called self-healing tests. And thanks to AI, the team of testers spends significantly less time on test scenarios that need to be refined with product changes. Thus, according to the expert, thanks to AI tools, adapting a scenario takes a few hours instead of several days.

Among other AI tools that significantly simplify the work of testers, the expert also names GitHub Copilot, which helps specialists write automated tests faster. In addition, this AI tool can do code reviews and find errors in the written code. Also, as Serhiy Romanov emphasizes, with the help of GitHub Copilot you can create not only automated tests, but also test utilities that can be used to perform certain actions on the side of the automated framework almost a hundred times more often.

In addition, the tester has created his own solution, adapted to the unique challenges of the video streaming industry. This project is called HLS AI Analyzer, which uses the ChatGPT API to check the integrity of .m3u8 playlists and media segments. This solution automatically loads the HLS manifest, extracts the .ts segment, extracts metadata segments from the media via ffprobe, and then sends all the data to the ChatGPT API for intelligent analysis for compliance with the RFC-8216 standard. This approach allows you to flexibly perform real-time video stream testing and integrate AI into the test infrastructure. The solution, written in Java, is easily extensible and can be integrated into a CI/CD pipeline.

As part of the HLS AI Analyzer project, Serhiy uses the ChatGPT API and the GPT-4o model to analyze video streams and video files, but if desired, this solution can be adapted for local use. Instead of accessing an external API, you can use a local ML model and perform analysis directly on your machine — without relying on the Internet and cloud services. This approach allows you to maintain control over your data, improve privacy, and customize the model for specific tasks in the streaming pipeline.

Mentioning other tools that help testers in their work, the expert spoke about:

DataDog cloud platform, which is used to monitor and analyze the performance and security of infrastructure and applications. He explains that DataDog testers use it to analyze application logs and metrics. The AI in DataDog allows you to analyze which services were also affected by this failure during a production failure. The platform also allows you to perform Root Cause Analysis, which will generate all the necessary information to eliminate this problem in a more correct way.
Applitools is a platform that uses AI to test user interfaces. Applitools doesn’t just compare pixels, it tries to understand how the interface has changed and whether this change is critical for the end user.

AI — requirements for testers and errors in use

Despite the rather active role that AI already plays in the work of testers, the requirements for specialists during hiring, according to Serhiy Romanov, have not changed much. Although he emphasizes that during interviews, candidates are already asked about their experience using AI tools, in particular GitHub Copilot, namely, how it has helped in the past and how the specialist plans to use it in the future.

«Those testers who are already devoting time to AI systems and tools are safe, because they will always find something to do on the project,» the expert adds, reassuring QA specialists.

He emphasizes that from his own experience he has seen how testers have begun to pay less attention to the skills of writing good code, and have given this work to artificial intelligence. Excessive use of AI in this matter leads to more code reviews.

«From what I see, the code written by a Junior tester using AI requires significant analysis and corrections,» adds Serhiy Romanov. In addition, he adds that it is important to know and be able to use AI tools, but understanding their features and pitfalls. «Well-written and well-structured code can be an invaluable asset in a team,» notes the tester.

Speaking about another common mistake, Serhiy mentions situations when testers do not provide GitHub Copilot with all the information necessary to write automated tests. In particular, they do not immerse the AI in the context of the product or feature for which the automated test is being written. So, if the AI tool does not take into account, for example, certain acceptance criteria specified in Jira Stories, then the test written by it, according to the expert, is more likely to not give complete results.

Examples of AI in testing

Serhiy has already emphasized that AI tools are a good help in preparing documents from testers. However, speaking about practical examples of using artificial intelligence in work, the QA expert emphasizes that no tool will be implemented until a Proof of Concept is developed for it. That is, the team must first familiarize themselves with the tool, and the management must agree on what value it can bring. It is the consideration of the AI tool by managers, engineers, and tech leads that takes place before the implementation stage.

There are several examples of processes that Serhiy Romanov named where AI has proven itself well in testing, including:

Interface testing — here, in most cases, the use of AI will help speed up writing test scripts, issuing releases, and much more.
API testing — in this case, using AI is also appropriate, because there are tools that allow you to develop test scripts based on the information in Swagger and execute these test scripts.

When AI won’t help the tester

But artificial intelligence cannot always be an effective assistant for testers. In particular, as Serhiy Romanov emphasized, when it comes to large projects that use unique protocols and technologies, AI is more likely to not be able to demonstrate its full potential.

As an example, he cited the situation in the video streaming project that Serhiy’s team is currently working on. AI cannot help them write autotests for media files and video protocols. And here’s what it’s all about.

According to the expert, if you give AI the basic task of checking the HLS manifest, having previously loaded it with autotests, then artificial intelligence will most likely cope with this task. It is the documentation that exists for this protocol that will help AI understand what is correct and what is not in this manifest.

It’s a different matter if we’re talking about testing the media files themselves that the player loads when playing a video. Here, AI can no longer do the work itself and will need human help. AI will need a specialist to write the code where this media file is first parsed. After the metadata is received, it will be transmitted to AI, and AI will determine how well this metadata meets the expectations of a particular feature or criterion for the product.

«There are products, in particular, that have a user interface and API, where integration with AI will happen very quickly, and the benefits will be visible already in the early stages. But in products that are unique in themselves, integration with AI is more likely not to show results in the early stages,» the expert concluded. In addition, he added that in order to properly configure AI, it still takes a lot of time to build this entire integration chain and adapt the framework for using AI.

Serhiy also emphasizes that AI will not help the tester in determining the usability of the interface for the end user. In this matter, as before, the competence remains with the person. In addition, AI is more likely not to be creative in finding complex scenarios, because it may not always have the right background. A tester who has good test design skills can find more scenarios that identify bugs in the product, the expert believes. In his opinion, AI also copes with this task, but it has, in essence, patterned thinking, and it does not show such creativity as a person.

But the combination of the right specialist with good test design skills working with AI that will suggest several scenarios, according to Serhiy Romanov, can speed up the tester’s work. In this case, it is the human who will analyze and discover what the artificial intelligence has not yet suggested, and what the specialist himself can suggest.

«We are coming to a point where AI can speed up the tester’s work, but it will be necessary to refine 20% of what artificial intelligence cannot do,» the expert emphasizes.

What future awaits testers with the advent of AI?

The number of news about AI developments and tools, which is multiplying almost daily in geometric progression, gives reason to believe that artificial intelligence has a great future. Looking at how AI can help testers in the future, Sergey Romanov believes that artificial intelligence will take over all the routine work, leaving QA specialists more space for creative ideas.

Currently, there are already tools that are divided by functionality, for example, one focused on the user interface, the other on writing code. In the future, according to Serhiy Romanov, AI tools will also focus on a certain area of human life and activity. For example, there will be AI that will be able to directly work with models for medicine or for real estate agencies.

In addition, Serhiy is confident that specialists will remain safe, and AI will not completely replace them.

«This will be a development area for testers — they will need to reorient themselves from routine tasks towards analyzing and testing the operation of AI systems,» he explains.

In particular, according to the expert, QA specialists will have more responsibilities, where they will need to monitor not only the quality of the product and find defects. In addition, testers need to be prepared to take a direct role in uniting teams and building effective interaction between departments.