Sometimes I want to make multiple outputs from multiple processes into one single output. Let's say we have to aggregate K8s pod logs from a deployment. We could do that by redirecting each output to a file and follow the file by
tail -f like below.
tempfile=$(mktemp /tmp/rip-example.XXXXXXXXXX) trap "rm \"$tempfile\"" EXIT pods=$(kubectl get pods | grep Running | grep -oe "app-[a-z0-9\-]\+") for pod in $pods do kubectl logs -f "$pod" >> "$tempfile" & done tail -f "$tempfile"
However, that method consumes disk spaces. Even worse, all the output data remains in the local disk if the process were killed with 9 (SIGKILL). There must be better ways. That's where named pipe comes to play.
Named pipe (a.k.a. FIFO) is a special kind of file and it does not write these kinds of data to disks. Despite an ordinary pipe cannot be shared within the outside of its process, a named pipe can be used from different processes at the same time. A named pipe lives beyond the process lifecycle. Because of its nature, it can be used for IPC (Inter-Process Communication).
A named pipe can be created by passing either
mknod a name for the pipe. Actually the name is a file path. As I described before, it's a kind of file. Like an ordinary file, we can read/write it. Let's revise the last example.
pipe=$(mktemp -u /tmp/rip-example.XXXXXXXXXX) # Just generate a file name by passing "-u"; mkfifo will create the file trap "rm \"$pipe\"" EXIT mkfifo "$pipe" pods=$(ls) for pod in $pods do kubectl logs -f "$pod" > "$pipe" & # Do not need to use ">>" here done cat "$pipe" # `tali -f` does not work as expected here; cat keeps reading until it gets EOF
Since a named pipe is a file, it can be deleted by
rm if any process does not need it anymore. If a process does not know whether a named pipe already exists, it can make sure that by testing the existence of the file path by
test -e. How simple.