Skip to main content

socat pipe to pipe for multiple open-write-close and open-read-close operations [Resolved]

I'm trying to use socat on two systems in order to send a multivolume tar from one system to the other. Multivolume is an absolute requirement as I'm trying to shift potentially petabytes of data to a mounted tape archive.

Ultimately, I'm want to do something like this:

#  on system1:
socat PIPE:/tmp/pipe1 SYSTEM:"ssh system2 socat - PIPE\:/tmp/pipe2" &
tar -M -L 10G -F /tmp/chvol -cvf /tmp/pipe1  &

#  on system2:
cat /tmp/pipe2 > file001.tar
cat /tmp/pipe2 > file002.tar
...

(The chvol script just echos the path /tmp/pipe1.)

As I've not been able to get this working yet, I'm running the following much-simplified test on one single system:

socat PIPE:/tmp/pipe1 PIPE:/tmp/pipe2 &
sh -c 'while sleep 1; do echo "sending date ..."; date > /tmp/pipe1; done' &
sh -c 'while sleep 1; do echo "slurping ..."; cat < /tmp/pipe2; done' &

which is meant to simulate a writer process repeatedly opening, writing, closing and a reader process repeatedly opening, reading until EOF, closing.

But this doesn't behave as I expected: the writer is seemingly happy, but the reader reads all the accumulated text in one go and thereafter remains blocked in a cat < ... call.

Can anybody explain (1) what is actually happening in the much-simplified test and (2) what I should be doing instead?

I have tried various shut-* options on both pipes (on the assumption that the problem is in the propogation of EOF) and done a lot of googling.


Asked April 21, 2017
Tags: , netcat, socat
Posted Under: Unix Linux
39 views
1 Answers

Your problem is that you have not specified you want unidirectional pipes. The socat man page explains how in this case PIPE: is like an echo.

What probably happened is that when you first wrote the date into the fifo 1, your socat read it, wrote it to fifo 2, then noticed there was input on fifo 2 so read it and wrote it to fifo 1, and so on busy looping until the first cat on fifo 2 managed to empty it before socat read it. Thereafter this first cat will continue reading the fifo, never seeing an end-of-file, and you can continue writing the date into the fifo.

You can see this if you add the -v option to socat to show the i/o.

Adding -u for unidirectional i/o makes it what you probably wanted, but now the socat will exit after the date is written and read, so you will need a loop for it:

while socat -v -u PIPE:/tmp/pipe1 PIPE:/tmp/pipe2; do echo new; done &

With your tar version you could put the socat command inside the /tmp/chvol script instead.


If you have a real tape drive, you might also look at a solution using nbd-server to export a block device over the network.


Answered April 21, 2017
 
I didn't realise that socat's reading of data from fifo1 and writing it to fifo2 made fifo2 readable by socat; I assumed that my own reader process (cat, in my much-simplified example) would be the only process reading. But your explanation fits! Many thanks Meuh! – Alexis Huxley 21 mins ago
 CanDoerz  7 months ago
Your Answer
D:\Adnan\Candoerz\CandoProject\vQA