Rsync friendly compression
Hi AOMEI team! Thanks for your great product.
We wish that AOMEI have the ability to build rsync (and deduplication) friendly backup images. Currently, two images made from the same system leads to two complete unrelated files, so we can't gain any benefits with rsync or storage deduplication.
zlib allows the possibility to use Z_SYNC_FLUSH parameters to allow the compression output to be byte aligned. It does increase the size of the backup file, but the increase is minimal.
See https://github.com/madler/pigz/blob/master/pigz.c for an example.
Thanks
We wish that AOMEI have the ability to build rsync (and deduplication) friendly backup images. Currently, two images made from the same system leads to two complete unrelated files, so we can't gain any benefits with rsync or storage deduplication.
zlib allows the possibility to use Z_SYNC_FLUSH parameters to allow the compression output to be byte aligned. It does increase the size of the backup file, but the increase is minimal.
See https://github.com/madler/pigz/blob/master/pigz.c for an example.
Thanks
Tagged:
Comments
For Backup, our software actually does deduplication when performing backup.
- Since we store multiple system backups within a storage device that supports deduplication, having more byte alignment (for example, once every source input file + at some well chosen interval/pattern for bigger files) will increase the probability of having matching blocks between two different backups of the same system, and even backups of two different systems.
- For all the remote workers, it will make the usage of rsync or remote differential compression way more efficient when an user needs to send his backup to the office through internet. As of 6.3.0, rsync effiency between two full system backups of the same machine (with little changes involved) is around 50%, which is pretty low.
It may be impossible to improve... but it would help us a lot if it happens.
Thanks