Syslog fundamentals

[ayomotukoya]

I want to trouble shoot an issue with our syslog servers logs being sent to the last chance index but im realizing I dont understand the syntax for the configuration of the conf files on the syslog servers to do so. Where can I gain fundamental knowledge of syslog ng and how to configure it to send logs into splunk?

[kiran_panchavat]

@ayomotukoya 

Syslog-ng, an open-source Linux utility, is one of the most preferred and easy ways to listen and write logs from a variety of network and security devices and write them to a human-readable format in text files. Once you have setup your syslog-ng system in place, it becomes piece of cake to ingest those logs into Splunk by using a small-weight universal forwarder deployed on the syslog-server. In this post, we will walk you through the steps, best practices, and sample configuration file for syslog-ng and Splunk Universal forwarder inputs.conf.

Note: Please note the steps and suggestions are mentioned for nix* OS only.

Install Syslog-ng

Install the Syslog-ng utility in case it’s not already present on OS.

Check if syslog-ng service is present/running on the system:

systemctl status syslog-ng

If the command gives an error that no such service is present, install the service by using one of the below commands, depending on the type of nix* OS you are using:

yum install syslog-ng
apt-get install syslog-ng

This will download and install the syslog-ng service on the Linux OS. By default, now if you will run the systemctl status command, it should show the status of the service as running.

Configure Custom Syslog Ports on each Device

For collecting data from multiple Syslog devices and subsequently ingesting it into Splunk, it’s easy if you configure each device to send logs a unique port to your Syslog server instead of the default port 514. For example:

Palo Alto Firewall > Syslog_Server:5515/TCP

Fortinet Firewall > Syslong_Server:5516/TCP

CISCO Router > Sysog_Server:5517/TCP

This approach has multiple advantages:

1. Simplicity of configuration required on Syslog-ng: Syslog-ng can be configured to listen to multiple ports. As each port is unique to a device type, further configurations make it simpler to write all the logs listened to from that port to a directory specific to each device.
2. Ease of Troubleshooting: As each device is sending on a unique port, it makes the network troubleshooting also easier in case Syslog is not receiving logs from a particular device. You can easily do a tcpdump on the syslog server for that port and see if there’s any traffic hitting at all or not.

tcpdump -i <interface> | grep <portnumber>

Syslog-ng File Configuration and Troubleshooting

Following is the default location for relevant Syslog-NG configuration files:

/etc/syslog-ng/syslog-ng.conf – This is the master config file which ships along with syslog-ng installation

/etc/syslog-ng/conf.d – Any .conf file created under this directory is processed along with syslog-ng.conf file

As a best practice we create our custom configuration file under the conf.d directory as it’s a more modular approach and allows creating multiple files for different configuration without effecting the overall functionality.

Following is a sample configuration file created under conf.d directory:

syslog_custom_ports.conf

#Listen on port 5515/tcp for logs being sent by Palo Alto Firewall and write it to corresponding folder having date as filename
source palo_alto {

tcp(ip(0.0.0.0) port(5515));

};
destination d_palo_alto {

file(“/opt/syslog-ng/palo_alto/$HOST/$MONTH$DAY.log”);

};

log {

source(palo_alto);

destination(d_palo_alto);

};

#Listen on port 5516/tcp for logs being sent by Fortinet firewall and write it to corresponding folder having date as filename

source fortinet {

tcp(ip(0.0.0.0) port(5516));

};

destination d_fortinet  {

file(“/opt/syslog-ng/fortinet/$HOST/$MONTH$DAY.log”);

};

log {

source(fortinet);

destination(d_fortinet);

};

#Listen on port 5517/tcp for logs being sent by Cisco router and write it to corresponding folder having date as filename

source cisco_router {

tcp(ip(0.0.0.0) port(5517));

};

destination d_cisco_router  {

file(“/opt/syslog-ng/cisco_router/$HOST/$MONTH$DAY.log”);

};

log {

source(cisco_router);

destination(d_cisco_router);

};

Once you save the file you need to reload the syslog-ng service for the new changes to take effect:

# Reloading configuration file without restarting syslog-ng

systemctl reload syslog-ng

#Restarting syslog-ng

systemctl restart syslog-ng

If the configuration file is correct and there are no permissions service should reload correctly post which you can run the following commands to see which all ports are now active on the OS where syslog-ng is listening on

netstat -an

Also, if you have devices sending logs in, navigate to respective directory and verify the logs are being written correcty.

vi /opt/syslog-ng/PaloAlto/<host>/<date in ddmm format>.log

In case there is any issue with syntax of the file run following command to see the errors:

syslog-ng –syntax-only

For other issues further troubleshooting might be required, often analyzing the logs in/var/log/messages is found to be quite helpful. Also, at times SELinux might be causing issues in which case try disabling the SELinux on the OS level.

vi /etc/selinux/config
SELINUX=disabled

# reboot OS for changes to take effect
restart

Read the logs by Splunk

Depending whether you are managing the configurations through Splunk Deployment server or managing them locally on the instance you need to place inputs.conf file on the syslog-server running Universal Forwarder or Heavy Forwarder. In the inputs.conf you need to define the monitor stanzas for each device/log source you configured in the Syslog-ng. For the above syslog-ng following is a sample inputs.conf file:

[monitor:///opt/syslog-ng/palo_alto/*/*.log]
host_segment = 4
index = palo_alto
#sourcetype as per add-on installed or custom sourcetype defined
sourcetype = pan:firewall
no_appending_timestamp = true 

[kiran_panchavat]

@ayomotukoya 

Please check this documentation for syslog-ng 

Using Syslog-ng with Splunk | Splunk 

Sending logs to Splunk using syslog-ng - Blog - syslog-ng Community - syslog-ng Community

syslog-ng Premium Edition 7.0.34 - Administration Guide

Syslog-ng 101, part 4: Configuration and testing - Blog - syslog-ng Community - syslog-ng Community

[marnall]

Which Syslog system are you using? There are likely docs available online for it.

[ayomotukoya]

Syslog ng

[marnall]

You could give https://syslog-ng.github.io/admin-guide/050_The_configuration_file/001_Configuration_syntax.html a shot, for understanding how the configuration file syntax works.