Transferring large files
When transferring large files, often users get an impression that Syncrify client is hung and is not doing anything. This is not correct. To ensure the Syncrify process is working, try opening the
Task Manager on Windows or use the
top program on Linux to monitor its activities. You should check its:
- CPU usage
- Disk I/O - both read and write
- Network usage
What takes long
Syncrify uses Rsync algorithm to determine the portion of file that has changed and only transfers those bytes to the remove machine. This is done to save network traffic but at the cost of local disk I/O and CPU usage. At any given time Syncrify client can be doing one of the following:
- Computing and/or comparing blocks - I/O and CPU is used
- Transferring data - Network is used
- Waiting for the server to respond - Idle
When you backup a file that is 20GB in size, it takes a long time even to perform a sequential read on the file.
What is block matching
In order to determine the part of file that has changed (aka delta), Syncrify compares data blocks between client and server. This process takes a long time if the file is large (in Gigabytes). This is a CPU and I/O intensive process with hardly any activity on the network.
Prior to the block matching process, the server splits the files into blocks and sends a list of MD5 signatures to the client. The client then searches for matching blocks on its end. The more blocks are matched the faster the matching process goes. The best case scenario occurs when every block on the client matches with the server and new data is appended at the end of the file. On the other hand, the worst case scenario occurs when none of the blocks match.
A few tips
- Avoid remote disk - Often users try to backup files residing on a remote disk. Tremendous network traffic is generated when Syncrify client tries to backup a file residing on a remote disk. Click here for more information.
- Avoid zipping a file - Zipping file rearranges every block within a file causing very few or none of the blocks to match. This not only increases the time it takes to match blocks, but the resulting delta is usually as big as the original file itself.
- Avoid encryption if possible - When encryption is used, every file is encrypted before block matching starts adding more time to the entire process.
- Avoid versioning - Blocks are matched twice when versioning is used. First on the client machine to determine delta between the file on the client and server and then, on the server to determine the delta between the previous version.
