Linux System Logging

Our distribution supports user space and kernel log messages and uses journald as system logger. Its configuration file provides a lot of parameters and can be found under /etc/systemd/journald.conf (for the full detailed information, please check [journald.conf(5)](https://www.man7.org/linux/man-pages/man5/journald.conf.5.html)). This section describes the most important parameters and information regarding logging.

Accessing Log Files

To read out journals the command line tool journalctrl is provided:
journalctl
To follow latest log messages pass argument -f:
journalctl -f
Show filtered by units:
journalctl -u <your-unit.service>

More information in journalctrl man page

Storage of the Logs

There are two modes about how logs can be stored: - volatile = logs are stored to RAM - persistent = logs are stored to flash

NOTE: The flash memory has limited write cycles which needs to be taken into account for the devices lifetime.

IMPORTANT: To not stress unnecessary the flash memory, our distribution logs as default the messages into RAM = volatile.

The logging system can be tailored to the need of the application. Full system logging makes analyzing an issue simpler and purposeful but it may cost valuable disk space and perfomance as well.

The follow subsection explains the most important configuration settings.

Configuration Parameters

In the config file /etc/systemd/journald.conf all parameters can be changed and tailored to the application’s need. Nevertheless, the following parameters are the most important to understand:

  • Storage=

    Can be either “volatile”, “persistent”, “auto” or “none”:

    • volatile (default) - The logs will be stored in memory below /run/log/journal - After a reboot the logs will be gone - The storage parameters with the “Runtime” prefix apply here
    • persistent - The logs will be stored on the disk below /var/log/journal - After a reboot the logs will still be there - The storage parameters with the “System” prefix apply here
    • auto - If the folder /var/log/journal exists, the behavior is the same as with “persistent” otherwise it behaves like “volatile”
    • none - No logs will be saved. Only forwarding will be done (if enabled).
  • SystemMaxUse= / RuntimeMaxUse=

    Control how much disk / memory space the journal may use up at most. Specify values in bytes or use K, M, G, T, P, E as units. Default values: SystemMaxUse=64M, RuntimeMaxUse=16M

  • SystemKeepFree= / RuntimeKeepFree=

    Control how much disk / memory space the journal may keeps free. Specify values in bytes or use K, M, G, T, P, E as units. Systemd-journald will respect both limits (KeepFree/MaxUse) and use the smaller of the two values. Default values: SystemKeepFree=350M, RuntimeKeepFree=not set

  • SyncIntervalSec=

    The timeout before synchronizing journal files to disk. This setting takes time values which may be suffixed with the units “m” for minutes. Default value = 5min

  • ForwardToSyslog=

    Use “yes” or “no”. Enables forwarding to the old syslog (/var/log/messages). If this is enabled, there will still be messages written to the disk regardless of the “Storage=” parameter. Default = no

  • ForwardToConsole=

    Use “yes” or “no”. Useful for debugging purposes Default = no

How to change the settings

If you want to change the default settings like for example persisting the log files, the following steps need to be done…

  1. Change the storage parameter in the config file /etc/systemd/journald.conf to Storage=persistent
  2. Reboot your device (reboot) or restart the logger system (systemctl restart systemd-journald)

NOTE: Don’t forget to clean the logs when you are switching from persistent to volatile storage. More details see section Maintaining the Logs below.

Maintaining the Logs

In embedded devices the disk usage might be essential. The logger system provides several tools to maintain the logs.

Disk Usage

To check the amount of disk space the logs are taking, just run this command:

root@am335x-nmhw21:~# journalctl --disk-usage
Archived and active journals take up 18.0M in the file system.

Cleaning

There are several methods provided to clean the logs.

Cleaning your logs to a specific size run journalctl --vacuum-size=. This removes the oldest archived journal files until the disk space they use falls below the specified size (specified with the usual “K”, “M”, “G” and “T” suffixes):

root@am335x-nmhw21:~# journalctl --vacuum-size=4M
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-00000000000014c8-0005ae5e2a594f10.journal (2.0M).
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-0000000000001af7-0005ae5ecf0899e3.journal (2.0M).
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-00000000000020d0-0005ae5f68b8138f.journal (2.0M).
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-00000000000026c9-0005ae600002f7a6.journal (2.0M).
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-0000000000002cc1-0005ae60973a5468.journal (2.0M).
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-00000000000032b9-0005ae612e720fcf.journal (2.0M).
Vacuuming done, freed 12.0M of archived journals from /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc.

Cleaning your logs using a specific time run journalctl --vacuum-time=. This removes all archived journal files contain no data older than the specified timespan (specified with the usual “s”, “m”, “h”, “days”, “months”, “weeks” and “years” suffixes):

root@am335x-nmhw21:~# journalctl --vacuum-time=1s
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-00000000000038b1-0005ae61c5b11a9d.journal (2.0M).
Deleted archived journal /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc/system@335f2d7f78764e27ba522dc69770346f-0000000000003ef0-0005ae6259e61874.journal (2.0M).
Vacuuming done, freed 4.0M of archived journals from /run/log/journal/c8b8c280f0bc43aba10c21e3574e81fc.

NOTE: When switched from persistent to volatile and after cleaning, it might also be possible to remove the persistent logs by calling rm -rf /var/log/journal/*.