Set up a Central Log Server with syslog-ng daemon on FreeBSD

In this post I want to show how you can set up a central log server in your network by using the syslog-ng on FreeBSD.

Install syslog-ng daemon

First we need to install the syslog-ng package on FreeBSD.

# pkg install syslog-ng

Next we have to shutdown the default FreeBSD syslogd daemon and disable the start at system boot.

# service syslogd stop
# sysrc syslogd_enable=”NO”

Initial Configuration of syslog-ng

Also we need to enble syslog-ng to start at system boot.

# sysrc syslog_ng_enable=”YES”

To configure syslog-ng we need to edit the main configuration file syslog-ng.conf located in /usr/local/etc.

There is a default base configuration in syslog-ng.conf for logging on the server . These settings are only for the logs of the server itself, as we disabled the default syslog on FreeBSD. You only need to touch this if you want to tune some settings.

To configure the host as a central log server to collect and receive syslog messages from remote hosts in the network, we will create a separate configuration file and import this file into the syslog-ng.conf file.

Detailed informations about the configuration of syslog-ng.conf you can list with the man page command

# man 5 syslog-ng.conf
https://www.freebsd.org/cgi/man.cgi?query=syslog-ng

Now we can start the syslog-ng service with

# sudo service syslog-ng start

And checking if it is running with

# sudo service syslog-ng status

Configure the Host to collect and receive syslog messages from remote servers

We add the following line at the end of the syslog-ng.conf file to import all files in this folder into our syslog-ng.conf file.

@include “/usr/local/etc/syslog-ng/conf.d/”

If the folders above not exists, create them. And now we create a new file in this path, in my case I create a new file named loghost.conf but you can choose any other name as we import all files from the above path into our base configuration file syslog-ng.conf, so it doesn’t matter.

Now adding the following lines:

source s_loghost parameter will tell syslog-ng to listen on the specified IP and UDP Port 514 which is the default Port for Syslog daemon.

source s_loghost
   {
      syslog(ip(192.168.10.200) transport("udp"));
   };

Enable TLS Encryption
Log messages can also be delivered using TLS-encrypted syslog over TCP, as well as over UDP. syslog-ng also supports TCP without TLS.

So you can add a further source in my case I name it source tls_source.
Also use a different Port like 1999. You need to provide a x509 certificate and key to use for authentication and decryption. peer-verify flag will determine which authentication is requested. You have different option like:

peer-verify(optional-untrusted)
peer-verify(optional-trusted)
peer-verify(required-trusted)
peer-verify(required-untrusted)
peer-verify(yes)
peer-verify(no)

source tls_source {
    network(
        ip(192.168.10.200) port(1999)
        transport("tls")
        tls(
            key-file("/usr/local/etc/syslog-ng/ca.d/Server.key")
            cert-file("/usr/local/etc/syslog-ng/ca.d/Server.crt")
	    peer-verify(optional-trusted)
        )
    );
};

filter f_all parameter will tell syslog-ng to log all levels between debug and emerg and not messages from the program devd with levels between debug and info.

About devd daemon
https://www.freebsd.org/cgi/man.cgi?devd(8)

filter f_all
   {
      level(debug..emerg) and not (program("devd") and level(debug..info));
   };

Also we must tell syslog-ng where to save the logs from the remote server. The destination d_host_daily parameter will define the destination /loghost/dailylogs/$HOST.$WEEKDAY.log where $HOST is the remote host and $WEEKDAY is Mon, Tue, Wed, etc. The files will be created automatically with the right owner and permissions. This configuration saves the logs for one week with a separate log file for each host and weekday. Each log file is overwritten the next week automatically.

overwrite-if-older()
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/overwrite-if-older

destination d_host_daily
   {
      file("/loghost/dailylogs/$HOST.$WEEKDAY.log"
         owner(root) group(wheel) perm(0600) dir_perm(0750) create_dirs(yes)
         template("$FULLDATE: $HOST ($FACILITY/$LEVEL) [$PROGRAM] $MSGONLY\n")
         template_escape(no)
         overwrite_if_older(514800) # overwrite if older than 6 days minus 1 hour
      );
   };

Final step is to tell syslog-ng to log it

log
  {
    source(s_loghost);
    filter(f_all);
    destination(d_host_daily);
  };


# If you use TLS encryption with a separate source as above you must also be sure to add a corresponding log entry.

log  
{
    source(tls_source);
    filter(f_all);
    destination(d_host_daily);
  };

To get settings kick in finally restart the syslog-ng daemon with
# sudo service syslog-ng restart

Configure remote servers (clients) to send its syslog messages to the syslog-ng server

From now on the syslog-ng daemon on the central monitoring syslog-ng server can collect and receive syslog messages from remote servers.

In order to use this syslog-ng server we must configure the remote servers (clients) to send their log messages remote to the syslog-ng server.

I use the syslog-ng server primarily to collect the syslog messages from pfSense appliances (Firewall and VPN Gateways) inside the network. So I can access the logs from all pfSense appliances at one location and further do not have to worry about disk space usage on the appliances itself.

The configuration on pfSense to send the syslog messages to a remote syslog server are very easy and briefly. You only have to enable remote logging under Status – System Logs – Settings and need to specify the IP of the remote log server and port, further you can choose what content of logs you want to send, done!

Unfortunately pfSense uses by default the older syslog daemon which is not capable to send log messages encrypted over TLS to a remote log server. It will send the messages in cleartext over UPD. Although it is possible to install the syslog-ng package on pfSense, this will only function as as log server and not as a client, it doesn’t replace the default pfSense syslog server.

To forward syslog messages from a FreeBSD client host you need to add the following line on the clients /etc/syslog.conf file:

*.*              @192.168.10.200:514

On Ubuntu you need to add this line in /etc/rsyslog.conf as it uses by default rsyslog instead of syslog-ng.

As mentioned at the beginning, by default FreeBSD uses Syslog for logging. So the above line only works for the default Syslog system.

If you also want to use syslog-ng on the FreeBSD clients and you still not have it installed, first install also on the client syslog-ng. Refer to the installation from the server.

Then edit the /usr/local/etc/syslog-ng.conf file to tell the syslog-ng client to forward the log messages to our central FreeBSD syslog-ng logging server.

Add the following lines at the end:

destination remote_log_server {
 tcp("192.168.10.200" port(514));
 udp("192.168.10.200" port(514));
 };
 log { source(src); destination(remote_log_server); };

As you see, you can using both TCP and UPD if both enabled on the destination log server.

Configure the clients to use TLS encryption for sending messages to the syslog-ng server

The above configuration forwards the logs from the clients in cleartext to the syslog server. Depending on your network environment you have the demand to send this log messages encrypted to the log server.

Below you will see how to configure TLS encryption on the clients.

On Ubuntu

First you must determine which syslog system your client OS is using.

By default, Rsyslog is installed in Ubuntu 18.04 server and above.

So to send the messages encrypted over TLS to the log server you need to add the following lines into /etc/rsyslog.conf.

$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.d/root-CA.cer
$ActionSendStreamDriver gtls # use gtls netstream driver
#$DefaultNetstreamDriverCertFile /etc/rsyslog.d/ca.d/Cert.pem
#$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/ca.d/Cert.key

$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer <FQDN syslog-ng server>

Also you need to make a change to the forwarding line as follows:

*.*     @@192.168.10.200:1999

Change the @ before the hostname to @@ (2 at-signs), which tells rsyslog to use TCP.
Also set the right Port you use for the TLS source, in my case Port 1999

On FreeBSD

As mentioned at the beginning, by default FreeBSD uses Syslog for logging. Syslog is not capable to use TLS or TCP and can only send the log messages in cleartext over UDP. In this case if you have syslog-ng not still installed, first install also on the client syslog-ng. Refer to the installation from the server.

Then edit the /usr/local/etc/syslog-ng.conf file to tell the syslog-ng client to forward the log messages over TLS to our central FreeBSD syslog-ng logging server.

Add the following lines at the end:

destination remote_log_server {
    network("<FQDN Syslog Server>" port(1999)
        transport("tls")
        tls( ca-dir("/usr/local/etc/syslog-ng/ca.d"))
    );
};


log { source(src); destination(remote_log_server); };

For TLS encryption you need to set the FQDN instead the IP of the remote log server, otherwise you will get an error regarding certificate subject does not match.

Also you need to add the server certificate and the corresponding root certificate which issued the server certificate of the remote log server into the ca-dir directory you set above, in my case /usr/local/etc/syslog-ng/ca.d.

Further you need to create a symbolic link for both certificates with the hash of them to itself.

ln – /usr/local/etc/syslog-ng/ca.d/server.crt <hash.0>
ln – /usr/local/etc/syslog-ng/ca.d/root.crt <hash.0>

On Microsoft Windows, use the following command: mklink .0 cacert.pem

syslog-ng – Administration Guide
https://support.oneidentity.com/de-de/technical-documents/syslog-ng-premium-edition/6.0.14/administration-guide/21

Now restart syslog-ng on the client and from now on the logs should be send encrypted to the remote log server.

Troubleshooting TLS encrypted message transfer

https://support.oneidentity.com/de-de/kb/263658/common-issues-of-tls-encrypted-message-transfer

Testing with the logger command

https://www.poftut.com/linux-logger-command-usage-tutorial-with-examples/

logger can only send cleartext data to either a UDP socket, a TCP socket, or a local UNIX Domain Socket (like /dev/log, which is the default if logger isn’t instructed otherwise).

This will send the message in quote to the syslog-ng server 192.168.1.10 (-n for server name) over TCP and Port 1999 (-T for TCP)

logger -n 192.168.1.10 -T -P 1999 “This is just a simple log line”

This will send the message in quote to the syslog-ng server 192.168.1.10 (-n for server name) over UDP and Port 514 (-d for UDP). As by default the syslog-ng server listens on UDP Port 514 you can also drop them.

logger -n 192.168.1.10 -d -P 514 “This is just a simple log line”

syslog vs rsyslog vs syslog-ng

The Syslog project was the very first project. It started in 1980. It is the root project to Syslog protocol. At this time Syslog is a very simple protocol. At the beginning it only supports UDP for transport, so that it does not guarantee the delivery of the messages.

Next came syslog-ng in 1998. It extends basic syslog protocol with new features like:

• content-based filtering
• Logging directly into a database
• TCP for transport
• TLS encryption

Next came Rsyslog in 2004. It extends syslog protocol with new features like:

• RELP Protocol support
• Buffered operation support

Syslog (daemon also named sysklogd) is the default Log Manger in common Linux distributions.

https://linux.die.net/man/8/syslogd

Sysklogd provides two system utilities which provide support for system logging and kernel message trapping. Support of both internet and unix domain sockets enables this utility package to support both local and remote logging.

System logging is provided by a version of syslogd(8) derived from the stock BSD sources. Support for kernel logging is provided by the klogd(8) utility which allows kernel logging to be conducted in either a standalone fashion or as a client of syslogd.

syslog-ng vs. rsyslog comparison

https://www.syslog-ng.com/products/open-source-log-management/syslog-ng-rsyslog-comparison.aspx

Rsyslog is mainly available for Linux and recently for Solaris. The syslog-ng application is highly portable and available for many more platforms including AIX, HP-UX, Linux, Solaris, Tru64 and most variants of BSD. This makes syslog-ng more suitable for sites with diverse platforms.

Make your work easier!

The syslog-ng application has several useful features not available in Rsyslog, for example the ability to classify, tag, and correlate log messages in real-time, but first off let’s have a look at a snippet picked up from the config file of syslog-ng from a major Linux distribution.

The configuration file of syslog-ng is clean, well-structured.Rsyslog’s configuration format is an extension to the original syslog configuration, which is difficult to read, comprehend, or maintain.

Links

https://www.syslog-ng.com/

syslog-ng Open Source Edition 3.23 – Administration Guide
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.23/administration-guide