Using systemd to keep your application running even after system reboot

Often times I build applications that needs to be up continuously, I often used a shell script with cron job to keep the program running at all times, but the challenge with that approach is that I have to wait for at least 1minute before my application is restarted. This is because the minimum schedule time (a circle ) a cronjob takes is 1minute before the next trigger.

Now because of this delay, I have resulted to start using a custom systemd to start, stop, or restart my script, as well as configure it to start automatically on boot.

Without wasting much time, let’s start by :

The program

Lets create a simple server using python, but remember your program can be in any language.

Create file pyserver.py in /usr/loca/test/pyserver.py

import socket
serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serv.bind(('0.0.0.0', 8090))
serv.listen(5)
while True:
    conn, addr = serv.accept()
    from_client = ''
    while True:
        data = conn.recv(4096)
        if not data: break
        from_client += data
        print from_client
        conn.send("I am SERVER\n")
    conn.close()
    print 'client disconnected'

We just created a simple server that return “I am server” to the client, now we can test our server to see it work

$ cd /usr/loca/test/
$ python pyserver.py

Open another terminal and run

$ curl http://127.0.0.1:8090
I am SERVER

Now that our application works fine, we will start it as a service.

The aim is for the server  to run at all times, be restarted in case of a failure (unexpected exit), and even survive server restarts. That’s where systemd comes into play

Turning it into a service

Let’s create a custom service file:

$ cd  /etc/systemd/system
$ touch myserver.service

Open with your preferred editor and paste below code.

[Unit]
Description=Myserver demo service
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=loginuser
ExecStart=/usr/bin/python /usr/loca/test/pyserver.py

[Install]
WantedBy=multi-user.target

This defines a simple service. The critical part is the ExecStart directive, which specifies the command that will be run to start the service. You may have to change the user to your login user.

Start the service

Once you have a unit file, you are ready to test the service:

$ sudo systemctl start myserver

Check the status of the service:

$ sudo systemctl status myserver

The service can be stopped or restarted using standard systemd commands:

$ sudo systemctl stop myserver
$ sudo systemctl restart myserver

Finally, use the enable command to ensure that the service starts whenever the system boots:

$ sudo systemctl enable myserver

Some explanations

Starting in the right order

You may have wondered what the After= directive did. It simply means that your service must be started after the network is ready. If your program expects the MySQL server to be up and running, you should add:

After=mysqld.service

Restarting on exit

By default, systemd does not restart your service if the program exits for whatever reason. This is usually not what you want for a service that must be always available, so we’re instructing it to always restart on exit:

Restart=always

You could also use on-failure to only restart if the exit status is not 0.

By default, systemd attempts a restart after 100ms. You can specify the number of seconds to wait before attempting a restart, using:

RestartSec=1

Avoiding the trap: the start limit

I personally fell into this one more than once. By default, when you configure Restart=always as we did, systemd gives up restarting your service if it fails to start more than 5 times within a 10 seconds interval. Forever.

There are two [Unit] configuration options responsible for this:

StartLimitBurst=5
StartLimitIntervalSec=10

The RestartSec directive also has an impact on the outcome: if you set it to restart after 3 seconds, then you can never reach 5 failed retries within 10 seconds.

The simple fix that always works is to set StartLimitIntervalSec=0. This way, systemd will attempt to restart your service forever.

It’s a good idea to set RestartSec to at least 1 second though, to avoid putting too much stress on your server when things start going wrong. As an alternative, you can leave the default settings, and ask systemd to restart your server if the start limit is reached, using StartLimitAction=reboot.