“A useful classifier does not mean a revolution in AI yet.” How Ukrainian IT professionals evaluate the sensational Jev model — use cases in real products

4 Jev use cases from Ukrainian IT professionals

1. Integrating Jev into an AI startup’s pipeline

The integration of the new System One Jev model into the pipeline of AI startup Omev AI solved the problem of frequent re-generations due to small defects, which previously led to a violation of the 90-second SLA and the use of expensive backup models. Jev was installed before the heavy generation stage to instantly check the "skeleton" of the result and classify errors, forming a three-tier system: deterministic rules for mechanical flaws, Jev for filtering false alarms, and a point editor to fix only damaged fragments.

With this new approach, 100% of 723 real-world queries met the SLA, backup model usage was reduced by 70%, and median recovery time dropped from 40–80 seconds to 12 seconds. As a result, generation costs were reduced by double-digit percentages without sacrificing quality, as catching errors at the level of fast typed verdicts is much cheaper than forcing the generative model to waste tokens on self-validation.

According to Omev AI co-founder Dmitry Radionov, the startup team uses Jev as a practical working tool to automate basic information selection — filtering out irrelevant sites, searching for competitors, and evaluating comments on social networks. Although similar classifiers have existed before (such as Google Cloud Natural Language), Jev is distinguished by its high speed and flexibility. At the same time, the company clearly distinguishes between simple initial selection and full-fledged data understanding, noting that complex cases still remain the model’s weak point, and its confidence indicator for many questions is still quite approximate.

The startup plans to use Jev as a flexible, interchangeable component in its pipeline, combining it with the Mercury 2.5 model: Jev will be responsible for fast narrow assessments and initial screening, and Mercury for deeper verification of complex cases and providing explanations. The team does not consider Jev as a universal analyst or a standalone product, but sees it as a great low-cost tool for experiments and an effective part of a larger, more complex system.

2. Testing Jev on a real-world AI product task

The OrchestrAI startup team tested Jev in its e-commerce platform Adminly, which transforms unstructured customer correspondence (with slang, errors, and multiple intents) into prepared business actions. Instead of cumbersome rules with regex and exceptions, Jev was used to quickly classify messages into types (order / support / mixed), determine customer intent, and assess risks. The model proved to be a powerful layer of primary text understanding, capable of significantly replacing some of the costly challenges of traditional LLM and simplifying query routing.

The company views the model as a probabilistic interpreter within a deterministic runtime environment, where Jev performs the classification, the system verifies the result, business rules impose constraints, and a human confirms risky actions. This design allows you to get the most out of the model without giving it control over the business truth.

They also note that Jev is currently in the testing and design phase of integrating with Adminly. Since this is not yet a long-term production implementation, the team is not yet announcing final production metrics, and their current experience is related to initial testing, practical design, and determining safe limits for the model’s application.

3. Jev for handling support requests and opting out of integration

Backend Developer Valentin Nikolaev worked on a task to prioritize requests based on ticket text to help support respond to the most important requests first. He proposed the idea of ​​dynamically recalculating the priority after each new chat message, as the conversation progresses and the initial assessment may become irrelevant. In his opinion, if such reassessment is cheap and helps the team make decisions faster, it has high practical value.

However, during internal testing, the developers discovered a significant limitation that directly affected the integration decision. It turned out that the level of reliability of detecting personal data and sensitive health information in the text of the appeals did not meet the high requirements of the company. Since users can independently specify their names, email addresses or medical details in arbitrary text, a key issue in evaluating this model was the ability to securely transmit such text to an external provider.

4. Experiments with browser control and verification of the claimed capabilities of the model

The founder of Dev AI Consulting, Ph.D Vyacheslav Koldovskyy shared his own experience of experiments with Jev, which focused on browser control and testing the claimed capabilities of the model. In particular, he tested the open project jev-ultrafast from Browser Use. In this architecture, Jev receives a structured description of web page elements and decides which action to perform and which object to interact with, while a separate language model generates text for input.

Vyacheslav noted this example as a successful demonstration of an effective approach to development: instead of trying to put all tasks on one bulky model, the functionality is divided between specialized tools. In his opinion, this concept echoes the structure of the human brain, where separate specialized areas are responsible for different types of tasks.

Vyacheslav Koldovsky plans to use Jev to create “software factories” — systems with varying levels of AI autonomy and human involvement. He sees this model as a way to make such systems faster and cheaper.

In his opinion, it is not necessary to entrust complex AI with absolutely everything. Instead of calling an expensive language model every time for simple steps (for example, choosing the right agent, finding the appropriate context, or checking code against strict criteria), it is better to involve cheaper and specialized tools like Jev. When there are thousands of such small operations, saving on each of them significantly reduces the overall cost and speeds up the entire process.

At the same time, Vyacheslav emphasizes that this is still an expected concept, not final production figures. The main evaluation criterion for him will not be just the low price of one API call, but the final time and cost of obtaining a high-quality result, taking into account possible errors and checks.

Where the model wins and where it needs insurance

The experience of Ukrainian IT specialists who tested Jev in real products shows that the main strength of the model lies in its speed and structure, but it requires a clear understanding of the limits of its capabilities.

For projects where you need to quickly test hypotheses or filter out information noise before feeding data into more expensive models, Jev becomes a great deployment tool. It allows you to avoid complex parsing and work with structured estimates right away.

“The biggest advantage is that you can get started quickly without training your own model and get structured estimates instead of text that still needs to be parsed. This is convenient for testing hypotheses, reducing the input context for large LLMs, and reducing information noise, which often leads to hallucinations in strong models,” notes Dmitry Radionov. “At the same time, the weakness is that a complex question still needs to be correctly decomposed into simple ones. In fact, in many processes, you have to turn one task into a 4- or 5-stage pipeline using Jev. The model will not do this intellectual work for you.”

Dmytro emphasizes that such a tool is primarily a bridge for a quick launch, not a universal solution for years to come: “For stable tasks with large volumes and our own data, we see more sense in retraining our model. That is where we will look for better accuracy and lower cost. Convenience at the start does not equal an advantage forever.”

Another aspect of Jev is its ability to replace cumbersome regex and exception systems. The model allows for natural language processing without creating hundreds of manual rules, but requires caution in ambiguous scenarios.

“In my opinion, the main advantage of Jev is the combination of semantic understanding of the text with a structured result that the program code can use without additional parsing,” explains Denys Kirev. “Instead of a large number of keywords, regular expressions and exceptions, we can describe classes and criteria and get a specific choice, score or probability. This is especially interesting for systems that process many variants of natural language.”

However, Denys warns against over-reliance on structured inference: “At the same time, a structurally correct answer is not necessarily semantically correct. Jev will not return a class outside the specified schema, but may choose the wrong allowed option. The risk increases in ambiguous messages, mixed intentions, slang, long context, or when working with languages ​​and formulations that are not sufficiently represented in the tests. Also, Jev does not replace a full-fledged LLM where you need to create text, explain solutions, or perform complex multi-step reasoning. This is a specialized tool for quick structured decisions, not a universal model.”

Advice from IT professionals based on their own experience

Dmitry Radionov emphasizes the importance of clearly defining the model’s responsibilities. In his opinion, Jev should be used only where probabilistic judgment is required: for classification, routing, risk assessment, or choosing the next scenario.

Dmytro emphasizes that the model should not be a source of business truth. Price, product balances, payment status, delivery, arithmetic, and identification of a specific customer or order should be determined solely by the program and data from the database.

Based on his own experience, he recommends:

• start with one narrow and well-formulated scenario;
• create your own set of real-life depersonalized (or impersonal) examples;
• compare LLM with current logic or LLM-baseline;
• first run the model in shadow mode;
• separately measure errors for different languages ​​and message types;
• do not perceive confidence as a guarantee of correctness;
• provide fallback in case of low confidence, timeout, or API unavailability;
• prevent model responses from bypassing business rules and human approval.

Backend Developer Valentin Nikolaev advises first of all to analyze in detail what data the product will send via the API. In his opinion, special attention should be paid to fields for free text from users, since their actual content often goes beyond the initial purpose. In addition, he emphasizes the need to analyze the legal terms of the provider in advance together with lawyers — even before sending real requests.

As for quality control, Valentin recommends preparing test cases based on your own tasks, the use of which is legitimate. It is important to identify in advance the most critical errors for the business — for example, when an urgent support request mistakenly ends up at the end of the queue. It is also worth testing ambiguous cases separately and clearly agreeing on the criteria under which a decision should be reviewed by a person.

Finally, the developer advises calculating the cost and delay for the entire expected volume of requests, necessarily taking into account the total load if re-evaluation is planned after each new message in the chat.

Vyacheslav Koldovsky emphasizes that the probabilistic assessment of the model is a useful signal, but by no means a guarantee of correctness. In his opinion, such an indicator should be used to determine the limits of automation: when an action can be performed autonomously, and when it needs to be transferred for additional verification to a person or another model. Confidence thresholds must be calculated on your own data, clearly weighing the possible cost of an error.

In the context of building "development factories", Vyacheslav emphasizes that model evaluation should never replace full-fledged testing and independent verification of results. The main goal is not to automate as many steps as possible at any cost, but to build a process that guarantees the necessary quality at lower time and financial costs.

In addition, the expert notes that instead of cloud-based Jev, alternative solutions can be considered. Due to the availability of benchmarks for comparing models, it is worth paying attention to options with local hosting. This allows you to use such tools virtually free of charge, and most importantly, it ensures complete privacy without transferring sensitive data to third parties.

Jev has become the fastest adopted model in the history of Vercel AI Gateway. Why are developers testing it en masse?

On the topic

Jev has become the fastest adopted model in the history of Vercel AI Gateway. Why are developers testing it en masse?

Read the country’s main IT news in our Telegram

On the topic

Read the country’s main IT news in our Telegram