Редакція dev.ua Useful Guidelines 21 September 2026, 16:26

Migrating a high-load service without downtime: how to synchronize data, maintain compatibility, and anticipate reverse migration — SharksCode experience

When a service runs 24/7 and has external and internal consumers, there is virtually no window for maintenance. This is the task that the SharksCode team faced when migrating a high-load backend component along with its data layer to a new infrastructure and a new data storage scheme. The old implementation began to limit scalability and make it difficult to scale the read and write load independently. Meanwhile, any schema changes and releases were becoming increasingly risky. The company was preparing the system for further growth and wanted to do it without a single minute of downtime.

Preparation is half the battle

Before the launch, the team captured a complete picture of the system. This included all producer and consumer services, API and event contracts, database dependencies, critical user scenarios, and the SLO and baseline of the current service. The new system was tested with load tests at rates that exceeded the production baseline, so the engineers saw a safety margin in advance. At the same stage, they outlined the main risks, including data loss or duplication, desynchronization of the state of the two systems, latency degradation, and database overload during double writes.

Both versions worked in parallel for some time. This became the basis for a safe transition. The team followed the principle: first make the consumer compatible, and only then change the manufacturer. Backward-compatible changes to contracts helped in the work. Instead of mandatory fields, the team added optional ones, and adapters worked between the old and new models. Contract versioning was only used
where breaking changes were unavoidable. Thanks to this, switching the service did not require the simultaneous release of all dependent systems.

Data synchronization during transition and protection against duplicates and out-of-order events

For the migration, we chose a double entry with event-based synchronization according to the scheme request → primary system → event/outbox → new system. In the first stage, the old system remained the source of truth, while the new system was filled with historical backfill, current events, and resynchronization of missed records. To keep the process under control
control, engineers set up automatic reconciliation. It constantly compared the number of records between systems, along with their identifiers, statuses, and control units.

Each operation received a unique operation/event ID. On top of that, idempotence with deduplication worked. Repeated requests were handled via retry with backoff and unique constraints. Before updating the state, the system checked the version and timestamp of the events. Without such protection, re-delivery could create an unnecessary operation, and an older event
risked overwriting the newer state. The team separately tested the re-delivery of a single event and the change in the order of its arrival.

This scenario provided one of the most revealing technical lessons. An event with status v2 could be processed before a delayed event with status v1. Without additional protection, the older state overwrote the current one. The problem was noticed through reconciliation and discrepancies between systems. After that, engineers added entity versioning and sequence checking before updating. A separate metric for out-of-order events also appeared.
An abnormal number of rejected updates triggered an alert.

Before working with real traffic, the team tested the entire migration scenario separately. They used integration and contract tests, load tests, schema validation, replay production-like traffic, and end-to-end migration tests. They separately simulated dependency timeouts, repeated and duplicated event delivery, reordering, unavailability of the new database, consumer crashes, and recovery from backlog accumulation. The rollback scenario was tested separately, just like the main migration flow.

Staged switching, metrics monitoring, and rollback, thought out in advance

Big bang was deliberately abandoned. Traffic was transferred in steps shadow → 1% → 5% → 10% → 25% → 50% → 100%. Initially, the new system received shadow traffic without affecting the result. After confirming the correctness through the feature flag, real requests began to be sent. The share was increased only after a stable observation window.

All this time, the team monitored the status of both systems. The speed and error rate, database behavior under load, queue delays, and re-queries were monitored. Separately, engineers monitored data discrepancies between the systems and the success of business operations.

Among the key metrics, the team separately monitored error rate, p95/p99 latency, timeout rate, database CPU/IO, queue lag, retry rate, and the number of data mismatches. The next stage was only reached when the new system's performance remained no worse than the baseline. Error growth, p95/p99 degradation, queue lag accumulation,
data discrepancies or the risk of database overload were a signal to stop the switch.

Routing was kept under control of feature flags and configuration, so it was possible to return traffic to the old system without a new deployment. The old version always received the necessary data. Operations that had already passed through the new system were synchronized back through the event pipeline and reconciliation. Automatic rollback was triggered when the error threshold was violated, data loss or corruption, critical latency degradation, and uncontrolled backlog growth. The rollback scenario was as follows
separate testing itself, as well as the main migration flow.

After the migration was complete, the team saw more stable p95/p99 latency, fewer incidents during peak load, reduced manual operations, and faster recovery. Horizontal scaling became easier, and changes to individual components no longer required a coordinated release of a large number of dependent services.

Main conclusion

Migrating a high-load service without downtime relies on the discipline of preparation and respect for the legacy system, which works until the last proven stage. When all the components of the transition are thought out in advance, the migration ceases to be a leap into the unknown and turns into a managed process with clear criteria at every step. Such
This approach allowed the SharksCode team to update the infrastructure without being noticeable to users and lay the foundation for the smooth growth of the service in the future.