spawnd

Marc Huber

$Id: 4b77e36e5e8ae73d306dce48db920566ca708e74 $

Table of Contents
1. Introduction
1.1. Download
2. Operation
2.1. Command line syntax
3. Configuration file syntax
3.1. Railroad Diagrams
4. Signals
5. Load balancing algorithm
6. Event mechanism selection
7. Sample configuration
8. Startup examples
8.1. Manual startup
8.2. Startup on demand
8.2.1. inetd
8.2.2. xinetd
8.2.3. launchd
8.3. Startup at system boot
8.3.1. Init scripts
8.3.2. launchd
8.3.3. systemd
9. Copyrights and Acknowledgements

1. Introduction

spawnd is a broker with load-balancing functionality that listens for incoming TCP (or SCTP) connections on IP, UNIX or possibly IPv6 sockets, accepts them and finally forwards them (using ancillary messages over UNIX domain sockets) to the spawned server processes.


1.1. Download

You can download the source code from the GitHub repository at https://github.com/MarcJHuber/event-driven-servers/. On-line documentation is available via https://projects.pro-bono-publico.de/event-driven-servers/doc/, too.


2. Operation

spawnd is now actually implemented as a shared library, and the programs that had to be invoked by it are now utilizing that library and are, as such, standalone. This in no way implies that spawnd configuration would be obsolete; only the binary is.


2.1. Command line syntax

Command line syntax is:

spawnd [ -b | -f ] [ -p pidfile-name ] [ -P ] [ -d level ] configuration-file [ id ]

The path to the configuration file is the only command line argument mandatory. If compiled with CURL support, configuration-file may be an URL.

id defaults to spawnd. It may be used to select a non-default section of the configuration file.

The -b switch will tell the daemon to release its controlling terminal on startup and fork itself to the background (just like background = yes in the configuration does, but with higher precedence). Likewise, -f keeps the daemon from forking to the background.

The -p pidfile-name option is equivalent to the pidfile = pidfile-name configuration directive.

The -P option enables config parse mode. Keep this one in mind; it is imperative that the configuration file supplied is syntactically correct, as the daemon won't start if there are any parsing errors at start-up.

The -d switch enables debugging. You most likely don't want to use this. Read the source if you need to.


3. Configuration file syntax

A typical spawnd configuration file consists of multiple id sections: one for spawnd itself, and one for the spawned server process (e.g. tac_plus or ftpd. The actual configuration section used is, by default, the one named after the program evaluating the configuration file. However, a different section may be selected by specifying an id parameter via command line or spawnd directive.

Railroad diagram: Config

For example, have a look at the following configuration snippet:

id = spawnd { exec = /path/to/ftpd }
id = spawnd2 { exec = /path/to/ftpd id = myftpd }
id = myftpd { }
id = ftpd { }

A spawnd started with this will default to evaluating the id = spawnd section, and any ftpd instance started will default to the corresponding ftpd stanza. However, starting spawnd with an additional argument, e.g.

/path/to/spawnd /path/to/configuration_file spawnd2

will choose the spawnd2 section instead, which in turn tells the ftpd to evaluate the myftpd part of the configuration.

Comments within configuration files start with #. At top-level, other configuration files may be included using include = file syntax. Glob pattern matching applies (typical sh(1) wildcards are evaluated).

All the configuration directives given below need to be enclosed in an appropriate id { ... } section.


3.1. Railroad Diagrams

Railroad diagram: SpawndConfig

Railroad diagram: ListenDecl

Railroad diagram: Child

Railroad diagram: AclDecl

Railroad diagram: CIDR

Railroad diagram: MiscDecl

Railroad diagram: SyslogDecl

Railroad diagram: Debug


4. Signals

spawnd will terminate upon receiving a SIGTERM or SIGINT signal. SIGHUP will cause spawnd to restart itself from scratch.

The daemon is only checking for signals every couple of seconds, so actions aren't necessarily immediate.


5. Load balancing algorithm

spawnd allows configuration of upper and lower limits for users and processes. The distribution algorithm will try to assign new connections to one of the running servers with less than users_min connections. If all servers already have at least users_min active connections and the total number of servers doesn't exceed servers_max, an additional server process is started, and the connection is assigned to that process. If no more processes may be started, the connection is assigned to the server process with less than users_max users, which serves the lowest number of connections. Otherwise, the connection will stall until an existing connection terminates.

If the sticky feature is enabled, spawnd will try to assign connections to server processes based on the remote IP address of the peer. Please not that this will not work in combination with HAProxy.


6. Event mechanism selection

Several level-triggered event mechanisms are supported. By default, the one best suited for your operating system will be used. However, you may use the environment variable IO_POLL_MECHANISM to select a specific one.

The following event mechanisms are supported (in order of preference):

Environment variables can be set in the configuration file at top-level:

setenv IO_POLL_MECHANISM = 4

7. Sample configuration

id = spawnd {
    listen { port 21 }
    listen { address = ::0 port = 2121 tls }
    spawn {
        users minimum = 10
        users maximum = 100
        instances minimum = 10
        instances maximum = 100
        exec = /usr/local/libexec/ftpd
        id = ftpd
        config = /usr/local/etc/ftpd.conf
    }
    background = true
}

8. Startup examples

spawnd (either standalone, or utilized via the MAVIS library, which is what ftpd, tac_plus and tcprelay do) is a long-running process. It may be started either manually, on demand, or at system boot time.

The examples in this section focus on tac_plus, but are easily adaptable to the ftpd and tcprelay daemons.


8.1. Manual startup

Starting a daemon manually is fine for testing, but, generally, undesireable for production. A configuration file that specifies at least a port the daemon should listen to is required:

id = spawnd {
    listen {
        port = 49
    }
}
id = tac_plus {
  ...
}

Copy this to, e.g., ./tac_plus.cfg, then start the daemon:

# /usr/local/sbin/tac/plus ./tac_plus.cfg

The daemon will now run in the foreground, blocking your shell until interrupted or being send to the background. If you want to run the daemon in the background, you can either add

    background = yes

to the spawnd section, or use the -b command line option:

# /usr/local/sbin/tac/plus -b /usr/local/etc/tac_plus.cfg

8.2. Startup on demand

The daemon may be started on demand by inetd(8) or compatible applications. The configuration file should not specify a port to bind to, as inetd will pass an already bound socket to the daemon:

id = spawnd {
    listen {
    }
}
id = tac_plus {
  ...
}

8.2.1. inetd

For stock inetd, adding the following line to /etc/inetd.conf and sending a HUP to inetd will activate the daemon:

tacacs  stream  tcp  wait root /usr/local/sbin/tac_plus /usr/local/sbin/tac_plus /usr/local/etc/tac_plus.cfg

8.2.2. xinetd

The equivalent xinetd(8) configuration:

service tacacs
{
    flags       = NAMEINARGS NOLIBWRAP
    socket_type = stream
    protocol    = tcp
    wait        = yes
    user        = root
    server      = /usr/local/sbin/tac_plus
    server_args = /usr/local/sbin/tac_plus /usr/local/etc/tac_plus.cfg
    instances   = 1
}

Depending on your setup this could either be added to /etc/xinetd.conf or be written to /etc/xinetd.d/tacacs.


8.2.3. launchd

Mac OS X comes with launchd(8), and here's a suitable /Library/LaunchDaemons/de.pro-bono-publico.tac_plus.plist:

<xml version="1.0" encoding="UTF-8"?>
<DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label<key>
    <string>de.pro-bono-publico.tac_plus<string>
    <key>ProgramArguments<key>
    <array>
        <string>/usr/local/sbin/tac_plus<string>
        <string>-p<string>
        <string>/var/run/tac_plus.pid<string>
        <string>/usr/local/etc/tac_plus.cfg<string>
    <array>
    <key>KeepAlive<key> <true/>
    <key>Sockets<key>
        <dict>
            <key>Listeners<key>
            <dict> <key>SockServiceName<key> <string>tacacs<string> <dict>
        <dict>
    <key>inetdCompatibility<key>
        <dict> <key>Wait<key> <true/> <dict>
    <key>KeepAlive<key>
        <dict> <key>NetworkState<key> <true/> <dict>
<dict>
<plist>

This needs to be activated using

# sudo launchctl load -w /Library/LaunchDaemons/de.pro-bono-publico.tac_plus.plist

The daemon will write its process id to /var/run/tac_plus.pid, and

# sudo kill `cat /var/run/tac_plus.pid`

will cause it to restart (and, implicitly, to re-read the configuration file).


8.3. Startup at system boot

The daemons may be started at system boot time. Alas, that's very specific to your system. You should definitely know what you're doing, or you may render your system unbootable.


8.3.1. Init scripts

The distribution comes with a couple of System V style init scripts, e.g. tac_plus/doc/etc_init.d_tac_plus. Copy this script to a location appropriate to your system (e.g. /etc/init.d/tac_plus and create the relevant symbolic or hardlinks. See your systems documentation for details.


8.3.2. launchd

On MacOS, create a file /Library/LaunchDaemons/de.pro-bono-publico.tac_plus.plist that consists of:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>de.pro-bono-publico.tac_plus</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/sbin/tac_plus</string>
        <string>-f</string>
        <string>-p</string>
        <string>/var/run/tac_plus.pid</string>
        <string>/usr/local/etc/tac_plus.cfg</string>
    </array>
    <key>KeepAlive</key>
        <dict> <key>NetworkState</key> <true/> </dict>
</dict>
</plist>

Then tell launchd about this configuration:

# sudo launchctl load -w /Library/LaunchDaemons/de.pro-bono-publico.tac_plus.plist

The daemon will write its process id to /var/run/tac_plus.pid, and you may make it re-read its configuration file by issuing

# sudo kill -HUP `cat /var/run/tac_plus.pid`

8.3.3. systemd

For systemd you'll have to create an appropriate configuration unit. Copy

[Unit]
Description=TACACS+ Service
After=syslog.target

[Service]
ExecStart=/usr/local/sbin/tac_plus -f /usr/local/etc/tac_plus.cfg
KillMode=process
Restart=always
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

to /etc/systemd/system/tac_plus.service, then enable and start the service:

# sudo systemctl enable tac_plus.service
# sudo systemctl start tac_plus.service

9. Copyrights and Acknowledgements

Please see the source for copyright and licensing information of individual files.