Aggregate Multiple Outputs with Named Pipes

Sometimes I want to make multiple outputs from multiple processes into one single output. Let's say we have to aggregate K8s pod logs from a deployment. We could do that by redirecting each output to a file and follow the file by tail -f like below.

tempfile=$(mktemp /tmp/rip-example.XXXXXXXXXX)
trap "rm \"$tempfile\"" EXIT

pods=$(kubectl get pods | grep Running | grep -oe "app-[a-z0-9\-]\+")
for pod in $pods
do
  kubectl logs -f "$pod" >> "$tempfile" &
done

tail -f "$tempfile"

However, that method consumes disk spaces. Even worse, all the output data remains in the local disk if the process were killed with 9 (SIGKILL). There must be better ways. That's where named pipe comes to play.

Named pipe (a.k.a. FIFO) is a special kind of file and it does not write these kinds of data to disks. Despite an ordinary pipe cannot be shared within the outside of its process, a named pipe can be used from different processes at the same time. A named pipe lives beyond the process lifecycle. Because of its nature, it can be used for IPC (Inter-Process Communication).

A named pipe can be created by passing either mkfifo or mknod a name for the pipe. Actually the name is a file path. As I described before, it's a kind of file. Like an ordinary file, we can read/write it. Let's revise the last example.

pipe=$(mktemp -u /tmp/rip-example.XXXXXXXXXX) # Just generate a file name by passing "-u"; mkfifo will create the file
trap "rm \"$pipe\"" EXIT
mkfifo "$pipe"

pods=$(ls)
for pod in $pods
do
  kubectl logs -f "$pod" > "$pipe" & # Do not need to use ">>" here
done

cat "$pipe" # `tali -f` does not work as expected here; cat keeps reading until it gets EOF

Since a named pipe is a file, it can be deleted by rm if any process does not need it anymore. If a process does not know whether a named pipe already exists, it can make sure that by testing the existence of the file path by test -e. How simple.