Log Management with LogRotate CentOS 7

If after a long-time work of a server there was no file rotation, compression, or regular deletion, they can occupy all the available memory space.

Logrotate is installed in the default settings on a server. The system is designed to process the history of all the packages and apps installed.

To check the Logrotate version:

logrotate --version 

Command output will be like this:

[root@kvmde54-19861 ~]# logrotate --version

logrotate 3.8.6 

The standard Logrotate configuration is stored by the two roots:

  • The main configuration file - /etc/logrotate.conf.
  • To create certain logs settings, you should use the directory /etc/logrotate.d


Let’s look at the Logrotate configuration file /etc/logrotate.d:

nano /etc/logrotate.conf 

Command output will be like this:

# see "man logrotate" for details

# rotate log files weekly

weekly 

 

# keep 4 weeks worth of backlogs

rotate 4 

 

# create new (empty) log files after rotating old ones

create 

 

# use date as a suffix of the rotated file

dateext 

 

# uncomment this if you want your log files compressed

#compress

 

# RPM packages drop log rotation information into this directory

include /etc/logrotate.d 

 

# no packages own wtmp and btmp -- we'll rotate them here

/var/log/wtmp {

monthly

create 0664 root utmp

     minsize 1M

rotate 1

}

 

/var/log/btmp {

missingok

monthly

create 0600 root utmp

rotate 1

}

 

# system-specific logs may be also be configured here.

include /etc/logrotate.d/web 

The file contains configuration blocks for two different files of a catalog history. The two blocks have the same functions. Any features not set in the configuration blocks accept the default settings and the ones from the file /etc/logrotate.conf.

The main directives for managing and processing logs:

  • monthly – a rotation one time per month. There are other options which include daily, weekly, monthly, size;
  • notifempty – no rotation for an empty log file;
  • rotate – how many old logs you need to store; the number is presented is parameters;
  • create – means that you should create an empty log file after you replace the old one;
  • dateext – the directive adds the rotation date before the old log name;
  • compress – means that you should compress the log;
  • delaycompress – do not compress the last log and the one before it;
  • extension – save the original log file after rotation if it has a set extension;
  • mail – send an email after rotation is finished;
  • maxage – rotate logs if they are older than necessary;
  • missingok – do not show an error if the log file is absent;
  • olddir – replace old logs to a stated folder;
  • postrotate/endscript – random commands after rotation;
  • start – the number from which the old logs start their numeration;
  • size – the log size after it has been replaced;

To manage log files for apps, you can use the two options:

  • Create a new configuration file Logrotate and move it to the catalog /etc/logrotate.d/. The file will work every day as a root user. It will also include other standard LogRotate tasks.
  • Create a new configuration file and launch it using the LogRotate default settings in Ubuntu.


How to create configuration in /etc/logrotate.d/

To make an example, let’s set updates for the server which creates logs to files access.log и error.log stored in the catalog /var/log/example-app/.

To add the configuration to the catalog /etc/logrotate.d/, open a new file:

sudo nano /etc/logrotate.d/example-app 

/var/log/example-app/*.log {

daily

missingok

rotate 14

compress

    notifempty

create 0640 www-data www-data

sharedscripts

postrotate

     systemctl reload example-app

endscript

}

Description of directives:

  • create 0640 www-data www-data – the command creates a new empty file of the log after rotation using the set extensions (0640), owner (www-data), and group (www-data);
  • sharedscripts – this option means that any scripts that you add to the configuration are processed only once during launching after compression. They are not processed for separate updated files. As our configuration will correspond to the two log files (access.log и error.log), the script mentioned it postrotate will be launched only once;
  • postrotate to endscript – the script used in this block will be launched after the log file gets updated. In the example, the app is reset.

After you have set the configuration using the requirements, you can test it with the command:

sudo logrotate /etc/logrotate.conf --debug 

[root@kvmde54-19861 ~]# sudo logrotate /etc/logrotate.conf --debug

reading config file /etc/logrotate.conf 

including /etc/logrotate.d 

Ignoring nginx.rpmnew, because of .rpmnew ending 

reading config file bootlog 

reading config file chrony 

reading config file example-app 

error: example-app:7 unknown user 'www-data' 

error: found error in /var/log/example-app/*.log , skipping 

removing last 1 log configs 

reading config file exim 

reading config file httpd 

reading config file mariadb 

reading config file nginx 

reading config file ppp 

reading config file proftpd 

reading config file roundcubemail 

reading config file syslog 

Ignoring web because it's not a regular file. 

reading config file wpa_supplicant 

reading config file yum 

including /etc/logrotate.d/web 

Allocating hash table for state file, size 15360 B 

 

Handling 13 logs 

 

rotating pattern: /var/log/boot.log 

 after 1 days (7 rotations)

empty log files are not rotated, old logs are removed 

considering log /var/log/boot.log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not day ago yet)

 

rotating pattern: /var/log/chrony/*.log  weekly (4 rotations) 

empty log files are rotated, old logs are removed 

considering log /var/log/chrony/*.log 

  log /var/log/chrony/*.log does not exist -- skipping

not running postrotate script, since no logs were rotated 

 

rotating pattern: /var/log/exim/*log  weekly (4 rotations) 

empty log files are not rotated, old logs are removed 

considering log /var/log/exim/main.log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

considering log /var/log/exim/reject.log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

 

rotating pattern: /var/log/httpd/*log  after 1 days (3 rotations) 

empty log files are not rotated, old logs are removed 

considering log /var/log/httpd/access_log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not day ago yet)

considering log /var/log/httpd/error_log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not day ago yet)

not running postrotate script, since no logs were rotated 

 

rotating pattern: /var/log/nginx/*.log  after 1 days (3 rotations) 

empty log files are not rotated, old logs are removed 

considering log /var/log/nginx/access.log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not day ago yet)

considering log /var/log/nginx/error.log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not day ago yet)

not running postrotate script, since no logs were rotated 

 

rotating pattern: /var/log/ppp/connect-errors  after 1 days (5 rotations) 

empty log files are not rotated, old logs are removed 

considering log /var/log/ppp/connect-errors 

  log /var/log/ppp/connect-errors does not exist -- skipping

 

rotating pattern: /var/log/proftpd/*.log /var/log/xferlog  weekly (4 rotations) 

empty log files are not rotated, old logs are removed 

considering log /var/log/proftpd/controls.log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

considering log /var/log/xferlog 

  log /var/log/xferlog does not exist -- skipping

not running postrotate script, since no logs were rotated 

 

rotating pattern: /var/log/roundcubemail/*.log  30720 bytes (4 rotations) 

empty log files are not rotated, old logs are removed 

switching euid to 0 and egid to 48 

considering log /var/log/roundcubemail/*.log 

  log /var/log/roundcubemail/*.log does not exist -- skipping

switching euid to 0 and egid to 0 

 

rotating pattern: /var/log/cron 

/var/log/maillog

/var/log/messages

/var/log/secure

/var/log/spooler

 weekly (4 rotations)

empty log files are rotated, old logs are removed 

considering log /var/log/cron 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

considering log /var/log/maillog 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

considering log /var/log/messages 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

considering log /var/log/secure 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

considering log /var/log/spooler 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not week ago yet)

not running postrotate script, since no logs were rotated 

 

rotating pattern: /var/log/wpa_supplicant.log  30720 bytes (4 rotations) 

empty log files are not rotated, old logs are removed 

considering log /var/log/wpa_supplicant.log 

  log does not need rotating (log size is below the 'size' threshold)

 

rotating pattern: /var/log/yum.log  yearly (4 rotations) 

empty log files are not rotated, log files >= 30720 are rotated earlier, old logs are removed 

considering log /var/log/yum.log 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not year ago yet)

 

rotating pattern: /var/log/wtmp  monthly (1 rotations) 

empty log files are rotated, only log files >= 1048576 bytes are rotated, old logs are removed 

considering log /var/log/wtmp 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not month ago yet)

  log does not need rotating ('misinze' directive is used and the log size is smaller than the minsize value

rotating pattern: /var/log/btmp  monthly (1 rotations) 

empty log files are rotated, old logs are removed 

considering log /var/log/btmp 

  log does not need rotating (log has been rotated at 2020-5-19 15:23, that is not month ago yet)

 

As a result, you will get the logrotate utility and state the standard configuration file in debug mode.

The information on the logrotate files processed at the moment will get output in the console. The standard Logrotate setting will be processed once a day together with the new configuration.

After this, you can check what you have created:

[root@kvmde54-19861 ~]# logrotate -d /etc/logrotate.d/example-app

reading config file /etc/logrotate.d/example-app 

removing last 1 log configs 

Allocating hash table for state file, size 15360 B 

 

Handling 0 logs 

How to create the LogRotate configuration

In this example, we use an app controlled by user testing, which is the generation of logs stored in the catalog /home/testing/logs/. We need to rotate the logs hourly. That is why we should install it beyond the structure /etc/logrotate.d that you can see in Ubuntu.

Let’s create a configuration file in our catalog using the text editor.

nano /home/testing/logrotate.conf 

Then you should paste the configuration below:

/home/testing/logrotate.conf

/home/testing/logs/*.log {

hourly

missingok

rotate 24

compress

create

}

Then save and close the file.

Such a configuration will rotate files hourly. It will compress them and save twenty-four old logs at the same time saving the new log file to replace the previous one.

Then you should set the configuration based on your app.

Let’s create a log file to check how it works:

cd ~ 

mkdir logs 

touch logs/access.log 

As the logs belong to the testing user, you don’t need to use sudo. Yet, you should state the status file. The file records the facts that logrotate saw or did last time. For this reason, the status file knows what operations it needs to do at the next launch.

Let’s ask Logrotate to place the status file directly to the home catalog for this example. We can state anything available and convenient:

logrotate /home/testing/logrotate.conf --state /home/testing/logrotate-state --verbose --force 

Output

[testing@kvmde54-19861 ~]$ logrotate /home/testing/logrotate.conf --state /home/testing/logrotate-state --verbose --force 

reading config file /home/testing/logrotate.conf 

Allocating hash table for state file, size 15360 B 

 

Handling 1 logs 

 

rotating pattern: /home/testing/logrotate.conf 

/home/testing/logs/*.log  forced from command line (24 rotations)

empty log files are rotated, old logs are removed 

considering log /home/testing/logrotate.conf 

  log needs rotating

considering log /home/testing/logs/access.log 

  log needs rotating

rotating log /home/testing/logrotate.conf, log->rotateCount is 24 

dateext suffix '-2020051916' 

glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' 

renaming /home/testing/logrotate.conf.24.gz to /home/testing/logrotate.conf.25.gz (rotatecount 24, logstart 1, i 24), 

old log /home/testing/logrotate.conf.24.gz does not exist 

renaming /home/testing/logrotate.conf.23.gz to /home/testing/logrotate.conf.24.gz (rotatecount 24, logstart 1, i 23),

 

–verbose prints the detailed information on Logrotate actions. It is the first time when LogRotate has seen this log file. As far as we know, at that moment the file is zero hours. So, we should not apply rotation to it.

If we look at the status file, we can see that Logrotate recorded information on the launch:

cat /home/testing/logrotate-state 

Output

[testing@kvmde54-19861 ~]$ cat /home/testing/logrotate-state

logrotate state -- version 2 

"/home/testing/logs/access.log" 2020-5-19-16:39:6

"/home/testing/logrotate.conf" 2020-5-19-16:39:6

 

Logrotate noticed that it had seen logs and when it had watched their rotation for the last time. If we launch the same command one hour later, the log will be rotated.

If you want to make LogRotate rotate the log file, you need to use the flag –force:

logrotate /home/testing/logrotate.conf --state /home/testing/logrotate-state --verbose --force 

Then you should set the task cron to launce Logrotate hourly. Open crontab of the user:

crontab -e 

You will see a text file opened. The file may have got such comments explaining the supposed main syntax.

Replace the cursor to the new empty line at the end of the file, and add the following:

14 * * * * /usr/sbin/logrotate /home/testing/logrotate.conf --state /home/testing/logrotate-state 

The task will be processed every fourteen minutes of each hour daily.



Blog