Migration of 100 TB under fire and "Nobel laureate in nail sorting": Petro Vavulin on clouds and artificial intelligence at Kyivstar

dev.ua CEO Stanislav Yurasov talked about all this with Petr Vavulin, head of the Kyivstar cloud products and services development department.

— How has customer sentiment in the market changed now after the recent attacks on Kyiv’s IT infrastructure? What is happening now? What are customers saying about their needs?

— In terms of sentiment, there is a slight shift in focus on where some of the IT systems, workloads, and infrastructure are located. So, there is some focus on that. Why not a big one? Most of the clients we help, for whom we implement and support their infrastructure, have had backup sites in the cloud since the beginning of the full-scale invasion.

True, there is now a sharp increase in attention to resource efficiency, i.e., cost optimization. But clients who have had or have had large workloads in Ukraine (infrastructure, computing, etc.) are now considering creating backup sites in other regions and duplicating critical services.

This is not a very big wave, because, I repeat, most customers have done this in advance. Now they are looking for options and scenarios for optimizing costs. In general, most customers have already used Azure, Amazon, Google, Oracle resources in one way or another. In our case, in Azure, because as a Microsoft partner we support and help customers with issues related to the use of this particular cloud platform. To summarize: we do not see any radical changes yet, because most have already been in the cloud in one way or another (to a greater or lesser extent).

— Does Kyivstar have ready-made scenarios in case the company prepares for disaster recovery (make a copy or something similar)? Maybe you have some templates?

— It’s hard to call this a template. Rather, it’s a standardized approach. Our team developed cloud platform usage scenarios 5–7 years ago that are not limited to disaster recovery scenarios or, as they say, BCDR (Business Continuity and Disaster Recovery). They also include scenarios where the client receives part of the computing resources in the cloud. We also update the scenarios from time to time, taking into account current circumstances and the development of technologies and services.

The scenarios take into account different situations and customer challenges. For example, customers work with end users located on other continents or in other countries. In such cases, it makes more sense to use the Azure platform to scale their business abroad.

The scenarios for small companies, medium and small businesses are primarily standardized. For large companies, we offer more complex solutions, which are also present in our portfolio. In addition, we often include other issues that may interest the client: development of reporting solutions, big data processing, web application hosting, artificial intelligence and, of course, cybersecurity issues.

Our team is highly qualified, as confirmed by partner statuses (Microsoft Advanced specializations, Fortinet Select MSSP, etc.). We offer assistance in solving security issues of everything we deploy in the clouds, using different approaches depending on needs and capabilities (Microsoft Cybersecurity Reference Architectures, Microsoft cloud security benchmark v2, Fortinet Security Fabric, etc.).

— What custom options are available for large clients?

— Speaking of large clients, I would like to note that we even have an option that can very rarely be standardized for clients using different services and technologies — the development of a Business Continuity Plan (BCP). We also understand that each client has their own (often unique) recovery requirements.

For example, there is a bank that may lose a certain number of transactions and data in the systems. And there is a client (a small business representative) that ships a small number of products every week, so it has different requirements for quality and execution of this plan. Given the needs and complexity of work processes, a small one may spend more hours in the process of restoring its digital services and for some time carry out the necessary operations in manual mode.

Typically, building and successfully implementing such solutions is a shared responsibility, because we do not physically have access to the client’s data, their credentials, passwords, or services. For example, in terms of data encryption, the client is responsible for encryption (access keys, accounts), but we help implement it and configure everything correctly.

— I understand that cases are different, but is there any timeframe on how long the migration to Azure can take?

— The migration of one of the largest banks in Ukraine with core banking systems and storage comes to mind. At one time, the process took 45 days. Based on this case, I would estimate it to be somewhere around 40–45 days if it is a super-heavy case, i.e. hundreds of terabytes of data, complex applications, and information security systems.

At the beginning of the full-scale invasion, for example, we had a client with databases of about 10–15 terabytes. Then we transferred their data for hours (the data was in several locations, and separate communication channels were used for each). The communication channels held up, we were their provider.

It is necessary to distinguish between cloud migration and backup and replication scenarios. One option is to create backups (long-term data storage), the second is data replication (creating a copy of data to ensure the continuous operation of services). They have different characteristics and complement each other. If we are talking about pre-configured replication and trained IT specialists, then recovery in the cloud can occur from a few minutes after the loss of the site. This is along with the configuration of network connections. But often it is not enough to simply restore a virtual machine or web service — it is necessary to provide users with access to services and IT systems, that is, so that a full switch between sites occurs and the client can continue working.

— You also mentioned artificial intelligence. I would like to know what requests customers have for the use of artificial intelligence? What will they need, what kind of business is it, what are its specifics?

— To paraphrase the former FBI director’s statement regarding cybersecurity (he said that companies are divided into two types: those that have been hacked and those that don’t yet know they’ve been hacked), then about artificial intelligence we can say this: all companies are divided into two types — those that use artificial intelligence and those that don’t know they’re using it.

I can tell you about some typical requests for today.

Internal assistants: allow you to safely work with confidential or sensitive data within the organization (reporting, internal documents), which you don’t want to send to a public chatbot with a subscription, because there is an End-User License Agreement, which allows the model to learn from the sent data and respond to others taking this data into account. This is not suitable for the company, because financial information may end up in public access. We help to safely deploy separate instances, isolated from public subscriptions, regulated by the Microsoft platform (GDPR, ISO, ISO for the safe use of AI has appeared). The advantage of public clouds is the basic certification of services. Instead of spending 3–4 hours looking for a form to return the goods (because there is a shortage of personnel, they hired a new person), an employee asks an assistant — and he answers immediately.

External assistants: also have access to some sensitive information, but work with people outside the company perimeter (clients, contractors). Authorization and validation processes are different there.

Image recognition: for example, price tags on shelves. The promoter comes in, takes a photo, and AI processes and compares prices.

Voice agents: we are working to make this a standard solution. You call the support line, and the AI ​​in live mode processes requests in Ukrainian, English, any other language, switches between languages, and responds to the status of the application or shipment of goods.

— Can we now say that there is a trend in business that the number of requests for artificial intelligence solutions is increasing?

— Yes, the field of artificial intelligence is developing extremely dynamically in our country.

— How do you integrate the Kyivstar cloud and the Azure cloud? When a client comes, what do you offer him?

— First, our team performs three main actions: assesses the client’s tasks, develops an implementation plan, and supports the solution.

Secondly, we approach each client individually. Some have a practically unlimited budget, and the main task is to preserve business processes. And some need to find a middle ground between OpEx and CapEx, infrastructure performance, and the company’s capabilities in the face of shelling, expensive insurance, and uncertainty when solving the issue of fault tolerance and business continuity.

There are many integration options. The simplest option: minimal integration, using storage located in data centers in Europe or the USA — simple but extremely reliable Storage. It starts in 5 minutes: we install a special program, or use a browser. We consult, together with the client we configure access — and then simply drag-and-drop information from your server or the Kyivstar.Cloud server, using our communication channels.

Innovative option: may involve rebuilding the infrastructure using innovative cloud services (for example, using Rearchitect or Rebuild cloud native approaches). In this approach, we evaluate virtual machines, databases, web services, containers (Kubernetes) and look at how to integrate and potentially replace everything using cloud services. A side effect of performing these actions is often the development of a BCP with replication and backup.

"Standard option" — hybrid infrastructure: some of the working nodes are located in the Kyivstar cloud or the client’s data center, and some in Azure. In this option, information security issues and load balancing between sites (automatically or not) are taken into account at the planning and implementation stages.

Network solution: In 2022, during total blackouts, the client had one hardware appliance that routed the entire network. When the power went out, the connection to the virtual device disappeared, and all remote clients stopped working. Together with the client, we developed a solution to move the virtual network device to the Microsoft cloud, which does not turn off. We transferred all links from clients to this virtual appliance in the cloud in Europe, and the traffic passing through the infrastructure in the cloud was routed to the data center where there is power.

— Retail company X says: I want an AI voice assistant, but I don’t understand anything about it. Can Kyivstar create it and deploy it in its cloud?

— These are the clients we mostly work with! When we started in 2023, most companies were trying to run away with Copilot, and we, in turn, in addition to the potential use of Copilot, decided to focus on custom solutions. When a client said they wanted Copilot, we said: you can buy it, but let’s look at your tasks. For internal or external use? Do you need to add voice recognition? Do you need to work with internal databases?

In most cases, we work through this task with the client and arrive at the MVP. The main problem with AI projects is not to make something small that works. The main problem is to integrate the MVP into the workflow within the organization, because at the testing stage everything is fine, but at the full-scale implementation stage, legacy, information security requirements, current systems and services, users, and many other open issues appear.

The next important issue we discuss is the cost of using the solution. At the top level, AI pricing depends on the level of the model and the amount of processed information (tokens). Moreover, the cost takes into account the input (what we ask) and output (the answer, or result we get) information.

I’ve had examples from friends when models started asking off-topic questions (for example, you sell coffee, and instead of the available varieties of beans, a student asks your model how to solve a cubic equation). The model tries to answer, eats up tokens, and the company gets billed.

Therefore, we develop the MVP together with the client’s team, calculate the economics and business case.

We had examples where for some tasks AI turned out to be more expensive than expected, and then the client decided to postpone this issue to the future (when potentially new models could reduce the value of the token).

One client started using AI for regular Q&A within the organization, but instead of the standard model, they chose Claude Opus. It’s like putting a Nobel laureate with half a million dollars in annual income in a nail sorting shop. Can he sort — sort square by square? Maybe, but the cost of this work will be too high.

Another important point during implementation is compliance with the rules of responsible use of AI. Microsoft has 5 principles of responsible use (similar to Isaac Asimov’s three principles of robotics). Some of these principles include the following requirements — the model must be honest, open, and the final decision, if it can harm a person, must be made by a person. Mechanisms must be provided to exclude critical errors. After all, we cannot completely give the autopilot to AI, because then who will be responsible for the consequences?

— If the company has unstructured data, do you help structure it or is that the client’s job?

— So far, we haven’t encountered situations where we couldn’t do anything. Data can be structured and unstructured, but there are tools that allow AI to work with both structured and unstructured data, such as images.

— Do you have ready-made, template AI assistants for an accountant, financier, or sales specialist?

— In general, yes. AI is not limited to chatbots (by the way, the first AI developments appeared back in the 60s of the last century). There are ready-made general-purpose assistants in Azure AI Foundry, for which you simply edit the system prompt. Also, using the Azure cloud, you can tweak the quality of the answers with various parameters, model type, system prompts, weights and penalties.

In addition to assistants, do not forget about other services using artificial intelligence. For example, in Azure Machine Learning Studio, where you can choose a model for sales forecasting, or in Azure AI — a model for optical character recognition (text recognition). For one task with template documents that were not included in standard European or American documents, the task could not be solved in “two clicks”. Therefore, we took the standard model, made the correct markup on three samples for training, and it began to recognize these template documents that are not typical for Europe or the USA with an accuracy of 96%+.

— Could a marketplace appear in the future where a manager goes in, selects the necessary AI functions, puts them in a cart, pays for them, and uses them from the cloud?

— This will appear after a certain number of typical projects are implemented. Right now it looks like the necessary function in the basket is conditionally performed by Kyivstar specialists, adjusting AI to the needs of customers.

Currently, the creation of assistants is conventionally divided into "internal" (those that work with company employees) and "external" (those that work with employees of other companies or clients).

Usually, assistants are created for each department: marketing, IT, legal function, accounting, etc. This is done so that the functions do not see each other’s data (physical separation) and to manage and account for the costs of using AI (model and token consumption). We integrate such bots directly into Microsoft Teams. That is, there is no ready-made marketplace as a site where you can order everything yet, but the process itself is already working.

One good example of an unexpected use of AI: a client had a lack of cybersecurity knowledge, specifically in hardware configuration. We helped them safely deploy the GPT model for IT, and it helped them configure firewalls and write Site-to-Site VPN and Point-to-Site VPN policies for a specific device. This temporarily solved the issue of the lack of the necessary specialist.

— Regarding training: is it better for a manager to study on his own and become a CTO or use the services of companies such as Kyivstar?

— The second way is easier because a person doesn’t waste time becoming a CTO. But who will be responsible if something breaks?

A short story: a friend of mine worked with Google Drive, conditionally buying a 10 TB drive for $5 and providing this space to small sole proprietors for storing accounting documents. At one point, 10 TB of data disappeared. According to him, the support service said: “You didn’t read the EULA, this is an account for personal use, no one gave any guarantees about data storage, here is your 10 TB storage.”

— Are there any legal restrictions on storing data abroad?

— In general, there are no such restrictions for commercial clients. However, some regulation also applies — currently, it is allowed to use clouds for the period of martial law with certain restrictions specified in the Resolution of the Cabinet of Ministers of Ukraine No. 154. There is also the 99th Resolution of the NBU, which regulates the use of cloud services by banks and the financial sector. But returning to small and medium-sized businesses — there are no direct prohibitions.

— Regarding Kyivstar tariff plans: how to choose a plan and what is included?

— We have tariff plans that include Office 365/Microsoft 365 services. Details about commercial offers, costs, and "Superpowers" can always be updated on the Kyivstar website or through B2B managers.

— Due to the rapid evolution of AI, cybersecurity is changing. Does Kyivstar or Azure take on security guarantees?

— The Shared Responsibility Model works. In short, the client is responsible for the data itself, access settings, and enabling multi-factor authentication (MFA). If MFA is disabled and the account is hacked, it is the client’s responsibility. The provider is responsible for the fundamentals, the platform (IaaS, PaaS, SaaS), and infrastructure availability. You can learn more about this model on the website.

— If a company suddenly needs to expand its cloud threefold, how quickly can it be done?

— If it was planned in advance, as quickly as possible. Using Azure follows the Pay-as-you-go model. For example, with backup with replication, the client does not pay for all 15 virtual machines, 10 web applications, and 100 TB of data at once. He pays only for the backup service and support. And he starts paying for the entire infrastructure only when he turns it on during an emergency.

That is, it doesn’t matter whether you planned to increase the number of resources (as in the case of Scale sets) or not (using the platform as a backup site) — you can scale very quickly and, importantly, to geographically different regions (from Europe to the USA or Asia).

The exception is the scarce GPU machines for AI (costing $10,000–20,000 per month). Their expansion due to the currently increased demand must be discussed in advance, but in most cases it is possible to find a solution that meets the needs of the business.

— What about SLA and 24/7 support?

— The first and second lines of support in 99,9% of cases are on the Kyivstar side. The client contacts his manager, engineer or architect. The support team is distributed throughout the country. In some cases, we agree and prescribe severity (incident criticality levels) with the client: for a low level, this is working hours, for critical incidents, this is a 24/7 response. Typical issues can also be resolved through the support line.

— Is there a real Vendor Lock-in when using Azure? Does the customer fall into a trap?

— With the right and balanced approach, there is no vendor lock-in. If you understand at least a little bit what works for you and how, you can always migrate.

There is Cloud Economics for calculating costs. For example, Microsoft has an Azure Hybrid Use Benefit, which allows you to save if you have previously purchased licenses.

Vendor lock occurs when you are locked into a specific SaaS that is available only from one of many providers or when you have a Legacy Line of Business Application. If the developers coded something for MS-DOS in the 90s and hardcoded everything, you will not be able to migrate painlessly even from DOS. But in such cases, you can use different approaches to ensure business continuity and fault tolerance — refactor, perform Re-Architect and replace outdated technologies, or in some cases Rehost or Replatform with the addition of new functionality.

In general, today the most correct approach seems to be to keep some of the expertise to ourselves. That is why we train all our clients (from introductory materials and sessions to certified courses with labs for 3-5 days — from admin to solution architect).

In addition, to improve services, Kyivstar develops partnerships with various vendors (for example, Select partner MSSP status from Fortinet). Combining complementary solutions from different vendors, such as Microsoft or Fortinet, allows solving complex client tasks (from deploying a separate virtual machine to information security services) and optimizing costs, combining services without being tied to a single vendor.