Remote automation for large 4k Transfers (Multi Terabyte transfers)


Recently got a 4k OLED TV as the prices have dropped. Was getting the 4k content library set up and noticed the RP2 / RP3 slow ethernet and USB was not ideal. Fine for occasional 4k content.

Installed OpenElec on some RP4 and the video support is much improved in recent releases and 4k support works well.

Was backing up some 4k  content with 4 USB3 HDD connected to a single RP4.
Originally ran a CP command (faster than the file manager in Libreelec for bulk file transfers) then realised I needed to disconnect my SSH session so 

nohup cp -a -u -v /var/d1/ /var/d2/ >> copy.txt  2>&1 &

The nohup means "no hang-up" so when I disconnect my connection via SSH the process will not be terminated. I pipe the output (STOUT) to a file and redirect errors (STDERR) to the same file. STOUT is file 1 (on Posix systems including Windows) and file 2 is STDERR). The final lone & makes the command run in the background.

To check the process I can use TOP and see the CPU and memory use - Also looking at the output file provides some insight into progress and errors.


The example above in action shows a real-world use case. The -u flag only copies updated files. You can check the process of the copy with top and grep the output of ps. For large data sets, it may take several minutes to compare files BEFORE the file I/O starts as the cp process first has to read and compare source and target folders to see what needs to be copied.

nohup cp -a -u -v /var/media/10TB/4K/ /var/media/Elements/ >> /var/media/Elements/copy.txt  2>&1 &