What is data deduplication in backups and how does it save space?

Data deduplication is an intelligent storage optimization method that eliminates the duplication of repeating information fragments in backups. If a company creates backups from dozens of workstations, they contain many identical system files and office applications, and deduplication records each unique sequence of bytes only once.

The technology operates at the block or file level, analyzing the incoming data stream and creating unique mathematical hash sums for each fragment. When the backup system encounters a block with an already existing hash, it does not create a new record on the disk, but simply places an internal reference to the already saved original.

This approach allows achieving colossal disk space savings, often reducing the required storage volume by five or even ten times compared to the original data size. This makes long-term archive storage financially affordable and reduces the load on communication channels when transferring copies.

However, deduplication has its own features that must be considered when designing the infrastructure. The process of analysis and search for duplicates requires significant CPU and RAM computational resources, so performance of the hardware is critically important for systems with high deduplication.