Skip to main content

Automatic file transfer between two linux machines [Resolved]

Hi I have two linux machines. Server A (LUbuntu) and Server B (Raspbian). What I want to do is have Server A check in specific a directory(NFS mounted on A) from Server B and IF there are any files transfer them to a location on server A. Ideally after the transfer, any transfered files on B are deleted. Note, that I do not want to sync A and B and I do not want any transfer from A--->B but only from B--->A. Also ideally, i do not want to transfer duplicate files but just delete them.

What is the best way to achieve this without too much srcipting? Since A is not online all the time, resuming file transfer is important.

Are there any existing software to do this or do I have to script everything myself.


Question Credit: vzografos
Question Reference
Asked July 20, 2019
Posted Under: Unix Linux
2 Answers

The basic flaw in this scheme is that there is no way for host B to know when it is safe to copy the files. What if A is writing a file to the NFS-shared dir, and host B picks up on that file before it is completely written? Host B will copy the partial file, and if that (partial) copy is successful, it will delete the entire file, including the part that it didn't copy, because Host A hadn't finished writing it yet.

That said, this seems like a fairly simple application for rsync. Read the man page, particularly the --partial option, perhaps the --checksum option, and especially the caveat in the section about the --remove-source-files option. The suggested workaround of using filenames such as *.new when host A writes files to be copied by host B, then renaming the .new files to whatever its name is supposed to be, may work for you. Renaming within a single filesystem is generally an atomic operation, so host B just has to be patient and ignore any *.new files. Once they get renamed, host B will transfer them the next time the cron job runs.

credit: Jim L.
Answered July 20, 2019

I would suggest rsnapshot.

it is a perl base script, scripting already done for you, just edit its /usr/local/etc/rsnapshot.conf file and specify source and destination for the copy.

Then you run /usr/local/bin/rsnapshot is where I think the executable gets put

To run that automatically: crontab -e and do something like

0 12 * * *   /usr/local/bin/rsnapshot daily

crontab syntax is :

minute   hour   day_of_month   month   day_of_week

per my crontab syntax, would run whatever specified at 12:00pm everyday

0 22 * * 2  /usr/local/bin/rnsapshot daily   # will run at 8:00pm only every Monday

read the rsnapshot instructions regarding the daily input argument to the rsnapshot perl script

rsnapshot can copy between anything, provided you set everything up everything needed [described in instructions]

credit: ron
Answered July 20, 2019
Your Answer