Set up a Central Log Server with Rsyslog on Ubuntu

In another post, I described how to set up a central log server with the syslog-ng daemon on FreeBSD, this time I want to do the same but now using the rsyslog daemon on Ubuntu.

The rocket-fast Syslog Server
https://www.rsyslog.com/

RSyslog Documentation
https://www.rsyslog.com/doc/master/index.html

Determine if Rsyslog is installed and enabled already be default

Because I am using here Ubuntu 18.04, which uses rsyslog by default, I will first check if it is really installed with

$ rsyslogd -v

Further I will check if the daemon is still running with

$ sudo systemctl status rsyslog

In case it is not installed, you can install, start an enable rsyslog as follows

$ sudp apt install rsyslog
$ sudo systemctl start rsyslog

# will enable the service at boot
$ sudo systemctl enable rsyslog

Configure RSyslog to receive remote messages

First we need to enable the socket on which rsylog is listening to receive remote messages.

By default rsyslog only logs from local system.

To configure rsylsog to listen and receiving for remote messages we have to edit the following file

/etc/rsyslog.conf

In the MODULES section you will see two sockets, an UDP and TCP socket.

I will enable the default UDP and Port 514 socket plus TCP, so I have to uncomment this module as follows

#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

After editing the rsyslog.conf file, we first need to restart the service to check if the server is listening on UDP Port 514 for remote messages.

systemctl restart rsyslog

# check if the server listening on UDP and TCP Port 514
# ss displays socket statistics and replaces the deprecated netstat command in Linux.
$ ss -a -4
or
$ ss -anu
or
$ netstat -anu
$ netstat -anu | grep 514

In order to restrict access to the log server, we can define senders which will be allowed to send messages.

Therefore we can create AllowedSender directives in the GLOBAL DIRECTIVES section from rsyslog.conf as follows

$AllowedSender <type>, ip[/bits], ip[/bits]

$AllowedSender UDP, 127.0.0.1, 192.0.2.0/24, [::1]/128, *.example.net, somehost.example.com
$AllowedSender TCP, 127.0.0.1, 192.0.2.0/24, [::1]/128, *.example.net, somehost.example.com

Rsyslog also detects some kind of malicious reverse DNS entries. In any case, using DNS names adds an extra layer of vulnerability. We recommend to stick with hard-coded IP addresses wherever possible.

Finally we need to add an rsyslog template in order to store messages from remote server to a specific file and path.

https://www.rsyslog.com/doc/v8-stable/configuration/templates.html

Templates are a key feature of rsyslog. They allow to specify any format a user might want. They are also used for dynamic file name generation. Every output in rsyslog uses templates – this holds true for files, user messages and so on. The database writer expects its template to be a proper SQL statement – so this is highly customizable too. You might ask how does all of this work when no templates at all are specified. Good question ;). The answer is simple, though. Templates compatible with the stock syslogd formats are hardcoded into rsyslogd. So if no template is specified, we use one of those hardcoded templates. Search for “template_” in rsconf.c and you will find the hardcoded ones.

Templates are specified by template() statements. They can also be specified via $template legacy statements.

In order to create a new template, I will add a further section named Templates to the rsyslog.conf file with my custom template RemoteLogs.

This will create a folder named with the IP from the remote host sending the messages, and in this folder a file named sbc.log will be created with the actual remote messages from the remote host.

###########################
####### TEMPLATES #########
###########################

#
#Custom template
$template RemoteLogs, "/var/log/remote/%FROMHOST-IP%/sbc.log"
*.* ?RemoteLogs

‘

A commented quick reference and sample configuration for rsyslog.conf
http://people.redhat.com/pvrabec/rpms/rsyslog/rsyslog-example.conf

Set up the remote Hosts to send messages to the RSyslog Server

To configure remote hosts using also rsyslog as syslog daemon, we have to configure the /etc/rsyslog.conf file on them.

To send all messages over UPD Port 514 add the following line to the end

# Send logs to remote syslog server over UDP Port 514
*.* @@192.0.2.10:514

In case you have critical messages that should not be lost in case the remote rsyslog server is down, you can buffer the messages into the rsyslog disk queue.

The rsyslog queueing subsystem tries to buffer to memory. So even if the remote server goes offline, no disk file is generated

https://www.rsyslog.com/doc/v8-stable/tutorials/reliable_forwarding.html

With the following simple config file, you forward anything you receive to a remote server and have buffering applied automatically when it goes down

$ModLoad imuxsock # local message reception
$WorkDirectory /rsyslog/work # default location for work (spool) files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server:port

Queue Parameters
https://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html

Troubleshooting

logs rsyslogd: error during config processing: omfile: creating parent directories for file ‘Permission denied’ failed: /loghost/192.0.2.146/rsyslogd.log

RSyslog will run into the above error when you create a template to store the messages (logs) to a local path outside /var/log as the RSyslog daemon runs in context of the syslog user and doesn’t have write permissions outside this path.

Solution:
Add an ACL entry for the syslog user to your folder you want to store the logs with:
setfacl -m user:syslog:rwx /loghost/

Links

The rocket-fast Syslog Server
https://www.rsyslog.com/

RSyslog Documentation
https://www.rsyslog.com/doc/master/index.html

Templates
https://www.rsyslog.com/doc/v8-stable/configuration/templates.html

Reliable Forwarding of syslog Messages with Rsyslog
https://www.rsyslog.com/doc/v8-stable/tutorials/reliable_forwarding.html

AllowedSender
https://www.rsyslog.com/doc/v8-stable/configuration/input_directives/rsconf1_allowedsender.html