Sometimes you just can’t get around needing to run an interactive program for an extended period of time; game servers are frequent culprits - remaining as interactive foreground processes which require input from the console to be operated. In this article, we’ll see how to run an interactive program (in this case a Bash shell) as a background daemon using a systemd unit file.

Background

In principle, the procedure we are about to employ would have worked decades ago on even relatively early versions of UNIX - all we really need is POSIX jobs, redirects, and named pipes (FIFOs). As a simple example, consider the following shell command:

bash < input.txt > output.txt 2>&1 &

Readers who are familiar with UNIX shells will immediately see what’s happening

• we’re running a bash shell instance attached to the current shell’s process (as a child process), but backgrounded (it does not prevent continued interaction with the terminal in which the command is run). The input to the bash instance is the file input.txt, and the output is redirected into output.txt. Of course this has a problem - we can’t provide any new input to the bash instance once it has been launched, it will only read the contents of input.txt at the moment the process is started.

The solution to this is to instead use a named pipe, also called a FIFO (First In First Out). This is a special type of file that allows input written after a process has already been started to be streamed into that process. We can create new FIFOs with the mkfifo command. Lets try this again, but with a FIFO this time:

mkfifo input.fifo
bash < input.fifo > output.txt 2>&1 &

Now try sending a command into the FIFO, for example echo 'echo hello' > input.fifo. This will work perfectly, but you will notice that the bash process exits as soon as the command is finished processing (you’ll see a message like [1] + done bash < input.fifo > output.txt 2>&). This still is not quite what we want, as we want our interactive process to continue running no matter how many commands we pipe into it.

The next piece of the puzzle requires understanding the conditions under which FIFOs are closed. When there are no remaining open write handles on a FIFO, and an EOF is sent into the FIFO, it is closed automatically by the kernel. This in turn causes the background process (bash in this example) to receive an EOF character on standard input, causing it to exit. The fix here is to simply force a write handle to remain open on the FIFO; the simplest way to do this is to leave a process running in the background which is guaranteed never to write anything into the FIFO (guaranteeing it will also never send an EOF into the FIFO either). This will prevent the kernel from closing the FIFO even if another process sends an EOF into it.

For this purpose, I like to use while true ; do sleep 1 ; done > some_fifo &. This simply spins in an infinite loop, doing nothing at all at a rate of once per second.

Putting all this together, we arrive at a close approximation of the goal we set out with:

mkfifo input.fifo
while true ; do sleep 1 ; done > input.fifo &
bash < input.fifo > output.txt 2>&1 &

If you try this out on your own system, you’ll see that you can pipe as many commands into input.fifo as you like, and the output will appear in output.txt. The background bash process will not exit unless you issue a command that explicitly causes it to do so (in the case of bash, exit), or you kill it’s parent process (i.e. close the terminal).

Of course, for many use cases where this method makes sense, it is very awkward to interact with the background process by echo-ing the command you want into a FIFO, then running less on a separate log file. We can use the following shell snippet to “attach” to the backgrounded process so you can interact with it (for example to issue a command to a game server):

tail -n 0 -f output.txt &
cat /dev/stdin > input.fifo

This will allow you to enter a command and have the result immediately displayed in the same terminal, while keeping the backgrounded bash session in the background. This snippet works by using tail running in the background to send any output from the backgrounded process to your terminal window. The cat command remains attached to the foreground (i.e. your keyboard is the standard input of the cat command), and shorts its own standard input (/dev/stdin) directly into the FIFO, meaning anything entered on your keyboard is sent to the backgrounded process as if you had been running it interactively.

Astute readers will now be wondering how to handle the parent process (shell session running in your terminal) being closed - with the methodology we have laid out above, this would also cause the background bash session to exit as well. There are several approaches, the least effort would be to simply use the disown builtin to detach the backgrounded processes from the current shell session. Alternatively, FreeBSD users might want to check out my blog post on writing rc.d scripts. However, in the interest of appealing to the most popular init system around today, the rest of this post will discuss utilizing the above method to produce a systemd unit file to run our program as a system daemon.

Creating a Systemd Daemon

Creating services (daemons) with systemd is surprisingly easy. First, locate your system folder. On Debian 9, this is /etc/systemd/system. You’ll know you have found the right folder because it will be filled with .service and .wants files.

You’ll want to create a new unit file, let’s call it exampleservice.service:

[Unit]
Description=Example Service

[Service]
User=someuser
Group=somegroup
ExecStart=/some/path/start_program.sh
ExecStop=echo 'exit' > /some/path/input.fifo

[Install]
WantedBy=multi-user.target

You will usually want to specify User and Group, since if you do not, systemd will run your program as root, which is generally not desirable for networked services for security reasons. The ExecStart program can be a simple shell script that launches your background program using the method described in the previous section (an example follows below). The ExecStop command specifies what systemd needs to do do stop the service - in the case of bash we can simply have it run the exit command, causing the backgrounded process to halt on it’s own. What you need to do here will vary depending on the program you are daemonizing. Finally, the WantedBy simply prevents the service from being run before the system reaches multi-user mode in the event you configure the service to run on boot.

Let’s look at a sample start_program.sh script:

#!/bin/sh

# ATTENTION - this is intended to be run as a systemd service, it should not be
# executed manually.

# make sure the path in /var exists
mkdir -p /var/example

FIFO_PATH=/var/example/input.fifo
BACKGROUND_COMMAND='bash'
OUTPUT_PATH=/var/example/output.txt

# make sure the FIFO does not already exist. If this script is only ever run
# by your unit file, then it should be safe to assume that the background
# process is not alread running, and that the FIFO is stale if it already
# exists
rm -f "$FIFO_PATH"

# create the FIFO
mkfifo "$FIFO_PATH"

# prevent FIFO from closing
while true ; do sleep 1 ; done > "$FIFO_PATH" &
KEEPALIVE_PID=$!

# launch the background program itself, recording it's PID
$BACKGROUND_COMMAND < "$FIFO_PATH" > "$OUTPUT_PATH" 2>&1 &
BACKGROUND_PID=$!

echo "spaned process with PID $BACKGROUND_PID"

# poll every 5 seconds to see if the background process is still running. If it
is no longer running, then we know that it has exited.
while true ; do
	if ! ps aux | grep -v grep | grep "$BACKGROUND_PID" > /dev/null ; then
		break
	fi
	sleep 5
done

echo "process exited at $(date)"

# Kill the keepalive loop so we can delete the FIFO from disk safely
kill -9 $KEEPALIVE_PID

# remove the FIFO itself
rm -f "$FIFO_PATH"

exit 0

You should now be able to have systemd reload all it’s unit files (so it sees the one you just created) by running systemctl daemon-reload. You should now be able to run your service with systemctl start exampleservice, and generally use all systemctl service management commands as you would with any other daemon. You can also attach to the input and output of the backgrounded program at any time by using the method described in the first section.

Motivation

My original use case for this method was to run a Minecraft 1.13.1 (Vanilla) server, which does not have good support for running as a service on systemd based systems (or if it does this was not documented in a way I was able to find). It occurred to me that this same approach would generalize readily to work for any kind of interactive program that needs to be run as a service, but where the interactivity is still required occasionally.