Log Management

Logs are the primary data source for troubleshooting issues, auditing security events, and analyzing system behavior. This article explains how to effectively manage and analyze logs on RHEL-based distributions.

Log System Overview

In CentOS/AlmaLinux/Rocky Linux, the logging system consists of two core components:

• journald (systemd-journald): The systemd journal daemon that collects all systemd unit, kernel, and boot logs, stored in binary format
• rsyslog: The traditional syslog daemon that receives logs forwarded by journald, writes them to text files, and supports remote transmission

The two work together: journald handles collection, while rsyslog handles persistent storage and forwarding.

Applications/Services -> journald (binary log) -> rsyslog -> text logs under /var/log/
                                                     |
                                                Remote log server

journald Configuration and Usage

Basic Queries

# View all logs
journalctl

# View the latest logs (similar to tail -f)
journalctl -f

# View only the last 100 lines
journalctl -n 100

# View logs for a specific service
journalctl -u nginx
journalctl -u sshd -f

# View kernel logs
journalctl -k

# View logs by time range
journalctl --since "2026-03-20 00:00:00" --until "2026-03-20 23:59:59"
journalctl --since "1 hour ago"
journalctl --since today

# Filter by priority (0=emerg, 3=err, 4=warning, 6=info)
journalctl -p err                # Errors and above only
journalctl -p warning -u nginx   # Warning-level nginx logs

# Output in JSON format
journalctl -u nginx -o json-pretty -n 5

# Check disk usage
journalctl --disk-usage

Persistent Storage Configuration

By default, journald logs may only be stored in memory (/run/log/journal/) and are lost after a reboot. To enable persistent storage:

# Create the persistent storage directory
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Edit the configuration
sudo tee /etc/systemd/journald.conf.d/persistent.conf > /dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemMaxFileSize=256M
MaxRetentionSec=3month
Compress=yes
EOF

# Restart journald
sudo systemctl restart systemd-journald

Key parameter descriptions:

Parameter	Description
Storage=persistent	Persist logs to disk
SystemMaxUse=2G	Maximum 2G disk space for logs
SystemMaxFileSize=256M	Maximum 256M per log file
MaxRetentionSec=3month	Retain logs for up to 3 months
Compress=yes	Enable compression

rsyslog Configuration

Default Log Files

# View rsyslog default rules
cat /etc/rsyslog.conf

# Default log layout on RHEL-based systems:
# /var/log/messages    - Most system logs
# /var/log/secure      - Authentication and authorization logs
# /var/log/maillog     - Mail logs
# /var/log/cron        - Cron job logs
# /var/log/boot.log    - Boot logs

Custom Log Rules

# Create a separate log file for a specific program
sudo tee /etc/rsyslog.d/myapp.conf > /dev/null <<'EOF'
# Write logs with the myapp tag to a separate file
if $programname == 'myapp' then /var/log/myapp.log
& stop

# Write all local6 facility logs to a specified file
local6.*    /var/log/custom-app.log
EOF

sudo systemctl restart rsyslog

Log Templates

rsyslog supports custom log formats:

sudo tee /etc/rsyslog.d/custom-template.conf > /dev/null <<'EOF'
# Custom log format template
template(name="CustomFormat" type="string"
    string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%\n"
)

# Apply the custom format to specific logs
local6.*    /var/log/custom-app.log;CustomFormat
EOF

Centralized Logging (Remote Collection)

Configure the receiving end on the log collection server:

sudo tee /etc/rsyslog.d/remote.conf > /dev/null <<'EOF'
# Enable TCP reception (port 514)
module(load="imtcp")
input(type="imtcp" port="514")

# Store remote logs in directories organized by hostname
template(name="RemoteHost" type="string"
    string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
)

# Use custom storage path for remote logs
if $fromhost-ip != '127.0.0.1' then {
    action(type="omfile" dynaFile="RemoteHost")
    stop
}
EOF

sudo systemctl restart rsyslog

# Allow through the firewall
sudo firewall-cmd --permanent --add-port=514/tcp
sudo firewall-cmd --reload

Configure log forwarding on the client:

sudo tee /etc/rsyslog.d/forward.conf > /dev/null <<'EOF'
# Forward all logs to the remote server via TCP
*.* @@log-server.example.com:514

# @  means UDP
# @@ means TCP (recommended for better reliability)

# Enable disk buffer queue if the remote is unavailable
action(
    type="omfwd"
    target="log-server.example.com"
    port="514"
    protocol="tcp"
    queue.type="LinkedList"
    queue.filename="forward_queue"
    queue.maxdiskspace="1g"
    queue.saveonshutdown="on"
    action.resumeRetryCount="-1"
)
EOF

sudo systemctl restart rsyslog

TLS Encrypted Transmission

In production environments, log transmission should be encrypted:

# Install the TLS module
sudo dnf install rsyslog-gnutls -y

# TLS configuration on the log server
sudo tee /etc/rsyslog.d/tls-server.conf > /dev/null <<'EOF'
global(
    defaultNetstreamDriver="gtls"
    defaultNetstreamDriverCAFile="/etc/pki/rsyslog/ca.pem"
    defaultNetstreamDriverCertFile="/etc/pki/rsyslog/server-cert.pem"
    defaultNetstreamDriverKeyFile="/etc/pki/rsyslog/server-key.pem"
)

module(load="imtcp"
    StreamDriver.Name="gtls"
    StreamDriver.Mode="1"
    StreamDriver.AuthMode="x509/name"
)

input(type="imtcp" port="6514")
EOF

# TLS configuration on the client
sudo tee /etc/rsyslog.d/tls-client.conf > /dev/null <<'EOF'
global(
    defaultNetstreamDriver="gtls"
    defaultNetstreamDriverCAFile="/etc/pki/rsyslog/ca.pem"
    defaultNetstreamDriverCertFile="/etc/pki/rsyslog/client-cert.pem"
    defaultNetstreamDriverKeyFile="/etc/pki/rsyslog/client-key.pem"
)

action(
    type="omfwd"
    target="log-server.example.com"
    port="6514"
    protocol="tcp"
    StreamDriver="gtls"
    StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
)
EOF

logrotate Log Rotation

logrotate prevents log files from growing indefinitely by automatically compressing and cleaning up old logs.

Global Configuration

cat /etc/logrotate.conf

# Default settings:
# - Rotate weekly
# - Keep 4 weeks
# - Create new files
# - Include configurations from /etc/logrotate.d/

Creating Rotation Rules for an Application

sudo tee /etc/logrotate.d/myapp > /dev/null <<'EOF'
/var/log/myapp.log
/var/log/myapp-error.log
{
    daily               # Rotate daily
    missingok           # Don't error if the file is missing
    rotate 30           # Keep 30 rotated copies
    compress            # gzip compression
    delaycompress       # Delay compression by one rotation (allows the program to continue writing)
    notifempty          # Don't rotate empty files
    create 0640 myapp myapp  # Permissions and ownership for the new file
    dateext             # Use date as suffix (instead of numbers)
    dateformat -%Y%m%d  # Date format
    sharedscripts       # Run scripts only once for multiple files
    postrotate
        # Notify the application to reopen log files after rotation
        /bin/kill -USR1 $(cat /var/run/myapp.pid 2>/dev/null) 2>/dev/null || true
    endscript
}
EOF

Nginx Log Rotation Example

sudo tee /etc/logrotate.d/nginx > /dev/null <<'EOF'
/var/log/nginx/*.log {
    daily
    missingok
    rotate 60
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    dateext
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}
EOF

Manually Testing Rotation

# Debug mode (don't actually execute, just show what would happen)
sudo logrotate -d /etc/logrotate.d/myapp

# Force rotation
sudo logrotate -f /etc/logrotate.d/myapp

# View the logrotate status file
cat /var/lib/logrotate/logrotate.status

Log Analysis Techniques

Filtering with grep

# Search for error messages
grep -i "error" /var/log/messages
grep -i "failed" /var/log/secure

# Search within a specific time period
grep "Mar 20 1[0-5]:" /var/log/messages   # Logs from Mar 20, 10:00-15:59

# Exclude noise
grep -i "error" /var/log/messages | grep -v "No error"

# Recursive search in a directory
grep -r "out of memory" /var/log/

# Show context around matching lines
grep -B 3 -A 5 "kernel panic" /var/log/messages

# Count occurrences
grep -c "Failed password" /var/log/secure

Analyzing with awk

# Count failed SSH login attempts by IP
awk '/Failed password/ {for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' /var/log/secure \
    | sort | uniq -c | sort -rn | head -20

# Count Nginx requests by hour
awk '{print $4}' /var/log/nginx/access.log \
    | cut -d: -f1-2 | sort | uniq -c

# HTTP status code distribution
awk '{print $9}' /var/log/nginx/access.log \
    | sort | uniq -c | sort -rn

# Find the slowest URLs (assuming response time is in the last column)
awk '{print $NF, $7}' /var/log/nginx/access.log \
    | sort -rn | head -20

# Requests per minute (find traffic peaks)
awk '{print $4}' /var/log/nginx/access.log \
    | cut -d: -f1-3 | sort | uniq -c | sort -rn | head -10

Using sed and Other Tools

# Extract logs for a specific time range (journalctl format)
sed -n '/^Mar 20 14:00/,/^Mar 20 15:00/p' /var/log/messages

# Monitor logs in real time with keyword highlighting
tail -f /var/log/messages | grep --color=auto -i "error\|warning\|fail"

# View login records using last
last -20                    # Last 20 login records
lastb -20                   # Last 20 failed logins
last reboot                 # Reboot history

# Search audit logs using ausearch (requires auditd)
ausearch -m avc             # SELinux denial events
ausearch -k login_events    # Search by audit rule

Practical Log Analysis Script

/usr/local/bin/log_summary.sh

#!/bin/bash
# Daily log summary report

DATE=$(date -d yesterday '+%b %d')
REPORT="/tmp/log_summary_$(date +%Y%m%d).txt"

{
echo "=========================================="
echo "  Log Summary Report - ${DATE}"
echo "=========================================="
echo ""

echo "--- SSH Failed Login TOP 10 ---"
grep "${DATE}" /var/log/secure 2>/dev/null \
    | grep "Failed password" \
    | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' \
    | sort | uniq -c | sort -rn | head -10
echo ""

echo "--- Error Log Statistics ---"
echo "Errors in messages: $(grep -ci "error" /var/log/messages 2>/dev/null || echo 0)"
echo "Failures in secure: $(grep -ci "fail" /var/log/secure 2>/dev/null || echo 0)"
echo ""

echo "--- Disk Space Alerts ---"
df -h | awk 'NR>1 {gsub(/%/,"",$5); if($5+0>80) print $0}'
echo ""

echo "--- Recent Reboot History ---"
last reboot | head -5
echo ""

echo "--- OOM Killer Events ---"
grep -c "Out of memory" /var/log/messages 2>/dev/null || echo "0 events"

} > "$REPORT"

cat "$REPORT"
# Optional: send via email
# mail -s "Log Summary ${DATE}" [email protected] < "$REPORT"

Advanced: Centralized Log Platforms

When the number of servers grows, a dedicated log platform is needed for unified collection, searching, and analysis.

ELK Stack (Elasticsearch + Logstash + Kibana)

ELK is the most popular log analysis platform:

• Elasticsearch: Log storage and search engine
• Logstash: Log collection and processing pipeline
• Kibana: Web visualization interface

Using Filebeat as a lightweight log shipper:

# Install Filebeat (repository pinned to 8.x below; Elastic 9.x is out -- check Elastic's official EOL table before deciding to move)
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

sudo tee /etc/yum.repos.d/elastic.repo > /dev/null <<'EOF'
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

sudo dnf install filebeat -y

# Configure Filebeat to collect system logs
sudo tee /etc/filebeat/filebeat.yml > /dev/null <<'EOF'
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/messages
      - /var/log/secure
    fields:
      type: syslog

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      type: nginx-access

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "elastic"
  password: "your_password"
EOF

sudo systemctl enable --now filebeat

Grafana Loki

Loki is a lightweight log aggregation system from Grafana, positioned as an alternative to ELK with much lower resource consumption:

:::danger[Promtail reached EOL on 2026-03-02]
Grafana's documentation states Promtail's **end of life is 2026-03-02**; commercial support has ended, no further updates ship, and new features go into **Grafana Alloy**. **Use Alloy for new deployments** and follow the official migration notes for existing Promtail setups. The Promtail configuration below is kept only so you can read old deployments; do not install it new.
:::

**Recommended: Grafana Alloy (collector) + Loki (storage/query)**

```bash
# Example versions: Alloy v1.20.1, Loki v3.7.8 (checked 2026-10; see the releases page first)
cd /tmp
curl -LO https://github.com/grafana/alloy/releases/download/v1.20.1/alloy-linux-amd64.zip
unzip alloy-linux-amd64.zip && sudo cp alloy-linux-amd64 /usr/local/bin/alloy && sudo chmod +x /usr/local/bin/alloy

Minimal Alloy collection config (syntax per the official reference):

/etc/alloy/config.alloy

loki.source.file "logs" {
  targets    = [{ __path__ = "/var/log/*.log" }]
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint { url = "http://127.0.0.1:3100/loki/api/v1/push" }
}

Existing Promtail (for reading old configs only; do not install after EOL):

# Example versions: Loki/Promtail 3.0.0 (2024-04, historical example)
cd /tmp
curl -LO https://github.com/grafana/loki/releases/download/v3.0.0/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
sudo cp promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail

# Basic configuration
sudo mkdir -p /etc/promtail
sudo tee /etc/promtail/config.yml > /dev/null <<'EOF'
server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki-server:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          host: __HOSTNAME__
          __path__: /var/log/messages

  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          host: __HOSTNAME__
          __path__: /var/log/nginx/*.log
EOF

# Replace hostname
sudo sed -i "s/__HOSTNAME__/$(hostname)/" /etc/promtail/config.yml

Loki’s advantage is that it does not index the full text of logs. Instead, it organizes log streams using labels, consuming far fewer resources than Elasticsearch, making it ideal for small to medium-scale deployments.

Summary

Task	Tool
View service logs	journalctl -u service-name
View system logs	/var/log/messages or journalctl
Log persistence	journald Storage=persistent
Log rotation	logrotate
Remote centralized collection	rsyslog TCP/TLS forwarding
Log search and analysis	grep / awk / journalctl filtering
Large-scale log platform	ELK Stack or Grafana Loki

Develop a habit of regularly reviewing logs and use automated scripts or log platforms to detect anomalies promptly. This is a key practice for keeping servers running smoothly.