The evolution of backup strategies in DevOps and CI/CD environments

Organizing backups in a DevOps environment

Traditional backup methods were based on the use of a cascaded web application development system and local data storage environments. This assumed the presence of several stages of a stable production process, which allowed data copy operations to be performed without fixing the states of the working environment and their dependencies.

With the advent of virtualized environments and the transition to a pipelined CI/CD process of continuous development and code delivery, the situation has changed. Now it is no longer possible to clearly distinguish any separate stages of the development process, since everything happens simultaneously - making changes to the program body, testing, code delivery, etc. That is, the modern DevOps workflow is a "live" operating environment with many highly dynamic microservices operating in a distributed system of interconnected computing, applications, and services.

Thus, the use of modern development tools in DevOps environments makes it impossible to use traditional approaches to organizing a data backup system without making fundamental changes to them. Thus, in addition to performing active full data backup, it is necessary to introduce a mechanism for dynamic backup of the operating environment states of the system. Backup copies of the operating environment state must "remember" the operating system files, the state of the cluster node, the state of the data pipeline and microservices. This allows you to not stop the development process during the backup and have the ability to return to any initial position of the workflow state.

Integration with CI/CD pipeline

Integrating a backup system into a CI/CD pipeline can be done in different ways, depending on the technological platform it is implemented on. The main thing is that it becomes an integral part of the workflow with the ability to launch the data backup control process in the background on demand.

The best option is to implement it as a microservice with standardized management methods. A similar integration method for the backup system has already been partially implemented for some of the existing virtualized platforms, but it is still incomplete.

Organization of a reservation system based on a "private cloud"

The level of integration of the backup system into the automated CI/CD pipeline of the DevOps environment depends not only on the level of data copying programs, but also on the type of storage. Cloud storages, as a rule, support a technology stack that includes all advanced tools for virtualization, pipelines and ensuring a high level of dynamics when working with data. This significantly brings them closer to the operational DevOps environment and facilitates the conditions for their integration.

And therefore, the most promising option for organizing storage in a DevOps environment may be a storage with "cloud" management principles. However, as we have seen earlier, using "traditional" cloud storage for a backup system has quite significant drawbacks and can prevent us from achieving our goal.

The solution to this situation may be to use a separate server to organize data storage. In this case, we will get an improved version of the “cloud”, the so-called “private cloud”, which will be devoid of many of the shortcomings of its original source, in particular, regarding security. All that remains is to choose the appropriate software for working with a virtualized environment and dynamic data.

A similar option for organizing an automated backup system has been used for several years by the Ukrainian hosting provider FREEhost.UA for cloud VPS hosting clients and when building a private cloud for clients. The provider has backup servers located in Ukraine and EU countries. Automatic full and incremental backups of virtual machines are created daily. In addition, users can independently configure backups of individual files and directories.

Proxmox Backup Server is used as the main software for working with data on the company's servers. Its use allows for a significant level of integration of the backup system with any dynamic virtualized environment, for example, with DevOps CI/CD, and to organize an effective data backup system.

Read the country's main IT news in our Telegram

On the topic

Read the country's main IT news in our Telegram