Managing log files effectively is an essential task for Linux sysadmin.
In this article, let us discuss how to perform following log file operations using UNIX logrotate utility.
- Rotate the log file when file size reaches a specific size
- Continue to write the log information to the newly created file after rotating the old log file
- Compress the rotated log files
- Specify compression option for the rotated log files
- Rotate the old log files with the date in the filename
- Execute custom shell scripts immediately after log rotation
- Remove older rotated log files
1. Logrotate Configuration files
Following are the key files that you should be aware of for logrotate to work properly.
/usr/sbin/logrotate – The logrotate command itself.
/etc/cron.daily/logrotate – This shell script executes the logrotate command everyday.
$ cat /etc/cron.daily/logrotate
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
/etc/logrotate.conf – Log rotation configuration for all the log files are specified in this file.
$ cat /etc/logrotate.conf
weekly
rotate 4
create
include /etc/logrotate.d
/var/log/wtmp {
monthly
minsize 1M
create 0664 root utmp
rotate 1
}
/etc/logrotate.d – When individual packages are installed on the system, they drop the log rotation configuration information in this directory. For example, yum log rotate configuration information is shown below.
$ cat /etc/logrotate.d/yum
/var/log/yum.log {
missingok
notifempty
size 30k
yearly
create 0600 root root
}
2. Logrotate size option: Rotate the log file when file size reaches a specific limit
If you want to rotate a log file (for example, /tmp/output.log) for every 1KB, create the logrotate.conf as shown below.
$ cat logrotate.conf
/tmp/output.log {
size 1k
create 700 bala bala
rotate 4
}
This logrotate configuration has following three options:
- size 1k – logrotate runs only if the filesize is equal to (or greater than) this size.
- create – rotate the original file and create the new file with specified permission, user and group.
- rotate – limits the number of log file rotation. So, this would keep only the recent 4 rotated log files.
Before the logrotation, following is the size of the output.log:
$ ls -l /tmp/output.log -rw-r--r-- 1 bala bala 25868 2010-06-09 21:19 /tmp/output.log
Now, run the logrotate command as shown below. Option -s specifies the filename to write the logrotate status.
$ logrotate -s /var/log/logstatus logrotate.conf
Note : whenever you need of log rotation for some files, prepare the logrotate configuration and run the logroate command manually.
After the logrotation, following is the size of the output.log:
$ ls -l /tmp/output* -rw-r--r-- 1 bala bala 25868 2010-06-09 21:20 output.log.1 -rwx------ 1 bala bala 0 2010-06-09 21:20 output.log
Eventually this will keep following setup of rotated log files.
- output.log.4.
- output.log.3
- output.log.2
- output.log.1
- output.log
Please remember that after the log rotation, the log file corresponds to the service would still point to rotated file (output.log.1) and keeps on writing in it. You can use the above method, if you want to rotate the apache access_log or error_log every 5 MB.
Ideally, you should modify the /etc/logrotate.conf to specify the logrotate information for a specific log file.
Also, if you are having huge log files, you can use: 10 Awesome Examples for Viewing Huge Log Files in Unix
3. Logrotate copytruncate option: Continue to write the log information in the newly created file after rotating the old log file.
$ cat logrotate.conf
/tmp/output.log {
size 1k
copytruncate
rotate 4
}
copytruncate instruct logrotate to creates the copy of the original file (i.e rotate the original log file) and truncates the original file to zero byte size. This helps the respective service that belongs to that log file can write to the proper file.
While manipulating log files, you might find the sed substitute, sed delete tips helpful.
4. Logrotate compress option: Compress the rotated log files
If you use the compress option as shown below, the rotated files will be compressed with gzip utility.
$ cat logrotate.conf
/tmp/output.log {
size 1k
copytruncate
create 700 bala bala
rotate 4
compress
}
Output of compressed log file:
$ ls /tmp/output* output.log.1.gz output.log
5. Logrotate dateext option: Rotate the old log file with date in the log filename
$ cat logrotate.conf
/tmp/output.log {
size 1k
copytruncate
create 700 bala bala
dateext
rotate 4
compress
}
After the above configuration, you’ll notice the date in the rotated log file as shown below.
$ ls -lrt /tmp/output* -rw-r--r-- 1 bala bala 8980 2010-06-09 22:10 output.log-20100609.gz -rwxrwxrwx 1 bala bala 0 2010-06-09 22:11 output.log
This would work only once in a day. Because when it tries to rotate next time on the same day, earlier rotated file will be having the same filename. So, the logrotate wont be successful after the first run on the same day.
Typically you might use tail -f to view the output of the log file in realtime. You can even combine multiple tail -f output and display it on single terminal.
6. Logrotate monthly, daily, weekly option: Rotate the log file weekly/daily/monthly
For doing the rotation monthly once,
$ cat logrotate.conf
/tmp/output.log {
monthly
copytruncate
rotate 4
compress
}
Add the weekly keyword as shown below for weekly log rotation.
$ cat logrotate.conf
/tmp/output.log {
weekly
copytruncate
rotate 4
compress
}
Add the daily keyword as shown below for every day log rotation. You can also rotate logs hourly.
$ cat logrotate.conf
/tmp/output.log {
daily
copytruncate
rotate 4
compress
}
7. Logrotate postrotate endscript option: Run custom shell scripts immediately after log rotation
Logrotate allows you to run your own custom shell scripts after it completes the log file rotation. The following configuration indicates that it will execute myscript.sh after the logrotation.
$ cat logrotate.conf
/tmp/output.log {
size 1k
copytruncate
rotate 4
compress
postrotate
/home/bala/myscript.sh
endscript
}
8. Logrotate maxage option: Remove older rotated log files
Logrotate automatically removes the rotated files after a specific number of days. The following example indicates that the rotated log files would be removed after 100 days.
$ cat logrotate.conf
/tmp/output.log {
size 1k
copytruncate
rotate 4
compress
maxage 100
}
9. Logrotate missingok option: Dont return error if the log file is missing
You can ignore the error message when the actual file is not available by using this option as shown below.
$ cat logrotate.conf
/tmp/output.log {
size 1k
copytruncate
rotate 4
compress
missingok
}
10. Logrotate compresscmd and compressext option: Sspecify compression command for the log file rotation
$ cat logrotate.conf
/tmp/output.log {
size 1k
copytruncate
create
compress
compresscmd /bin/bzip2
compressext .bz2
rotate 4
}
Following compression options are specified above:
- compress – Indicates that compression should be done.
- compresscmd – Specify what type of compression command should be used. For example: /bin/bzip2
- compressext – Specify the extension on the rotated log file. Without this option, the rotated file would have the default extension as .gz. So, if you use bzip2 compressioncmd, specify the extension as .bz2 as shown in the above example.
My name is Ramesh Natarajan. I will be posting instruction guides, how-to, troubleshooting tips and tricks on Linux, database, hardware, security and web. My focus is to write articles that will either teach you or help you resolve a problem. Read more about
Comments on this entry are closed.
If I’m not mistaken the permissions in your example would be the result of “create 644” not “create 700”. 700 would produce -rwx—— not -rw-r–r– as shown, that is unless I’ve missed something.
Any ideas on how to use logrotate when the file name is dynamic, example:
messages_03956-20101208153330Z-345df1e000.log
Where “20101208153330Z” is the date, but I think the other numbers are related to PIDs.
Thanks.
rf – assuming the numbers are the dynamic part of the name, simply replace that portion with an *. For instance, “messages_*.log”
And a user doesn’t have to be root to use logrotate: this is from the logrotate man page:
One could use “find . -name *.log” in his or her $HOME directory and then add them to their own logrotate scripts in their own $HOME directory.
Thanks. Just what I was looking for 🙂
thanks a lot
This article does cover FreeBSD log rotation
Excellent, it works like a charm.
Good quick start into logrotate for me. Thanks a lot!
As usual the easiest and most comprehensive tutorial on the subject. Thank you very much.
Hi:
should postrotate need to be used for stopping and running the application ? or does logrotate process the new log file after rotation ?
Hello,
Have anyone tried houly log rotation, from what I heard the log rotation is done by a script in cron.daily and can’t be done for lesser frequencies.
Any comments ?
And what will be if I specify both size and daily (or another period) options?
I would like that logrotate rotate files immediatelt after they reach specified size, but, if not, at the specified period of time.
gokulnath: you could probably put the script into /etc/cron.hourly to get your hourly.
I need to run a command every time the logrotate is attempted regardless of whether anything was rotated. Lastaction, firstaction, postrotate and prerotate only appear to work if at least one log file was rotated. Any ideas?
Why do you need the command to run when logrotate runs? You should just be able to crontab the command with crontab -e…
What is the difference between
CREATE 644 & 700
and what is the use of it
hi, thanks for the valuable info posted here about log rotation,
i was just interested to know is their any third party tool that handles all this
i mean my requirement is
archive(compress) the log files and then delete the old log files
and this should be done weekly i.e archiving and deleting of log files
so is their any third party tool for log rotation that makes task lil bit easy other then linux logrotate command ?
thanks
how to setup logrotate for a multi year daily compress of messages /secure/cron ?
/tmp/output.log {
yearly
monthly
daily
copytruncate
create 700 bala bala
rotate 4
compress
dateext
hi, using “create ..” command along with “copytruncate” will not have any effect. This is because copytruncate itself creates a copy of the original log file before truncatnig that original file to zero size (according to MAN page). my ref is this.
hi
quick question
I have a centralized log server, where I need to keep ALL logs, would the following work :
in /etc/logrotate.conf I have :
daily
rotate 4
create
dateext
compress
and in /etc/logrotate.d/syslog:
/var/log/messages
/var/log/secure
/var/log/spooler
{
yearly
monthly
daily
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
Thanks in advance !
Hi,
Many thanks for this. I found it really useful and it had more or less everything. One thing I found out with section 3 involving “copytruncate” is that the original file went back to its original size for me. This is because the process that was writing to the log file was oblivious to any logrotate and just continued at the last offset. This is also described here.
My solution was to turn off logging in this case.
I should add of course that the log file went back to its original size only in certain circumstances and that ‘copytruncate’ usually does the trick very well indeed.
I don’t have logrotate on my FreeBSD box. (Noob alert) How do I install this so that I can set up what you are talking about. Further, what I’m really interested in to use this for FreeRADIUS, and then export to a jumpbox elsewhere, where someone can ftp the particular log files. Is that possible?
Ryan – My friend, you have a tall hill to climb!
assuming terminal as root:
cd /usr/ports/sysutils/logrotate
make install
make clean
cd /usr/local/etc/
nano logrotate.conf
the article basically picks up from here.
Hi Aix,
Thanks. when I tried to to the make install, etc I get:
serv01# make clean
===> Cleaning for logrotate-3.8.7
serv01# make depend
serv01# make install clean
===> Fetching all distfiles required by logrotate-3.8.7 for building
===> Extracting for logrotate-3.8.7
=> SHA256 Checksum OK for logrotate-3.8.7.tar.gz.
===> Patching for logrotate-3.8.7
===> Applying FreeBSD patches for logrotate-3.8.7
===> logrotate-3.8.7 depends on executable: gmake – found
===> logrotate-3.8.7 depends on shared library: libpopt.so – found
===> Configuring for logrotate-3.8.7
===> Building for logrotate-3.8.7
cc -E -Wall -D_GNU_SOURCE -DFreeBSD -DVERSION=\”3.8.7\” -I/usr/local/include -I/usr/local/include -g -M logrotate.c log.c config.c basenames.c > .depend
cc -Wall -D_GNU_SOURCE -DFreeBSD -DVERSION=\”3.8.7\” -I/usr/local/include -I/usr/local/include -g -c -o logrotate.o logrotate.c
cc -Wall -D_GNU_SOURCE -DFreeBSD -DVERSION=\”3.8.7\” -I/usr/local/include -I/usr/local/include -g -c -o log.o log.c
cc -Wall -D_GNU_SOURCE -DFreeBSD -DVERSION=\”3.8.7\” -I/usr/local/include -I/usr/local/include -g -c -o config.o config.c
cc -Wall -D_GNU_SOURCE -DFreeBSD -DVERSION=\”3.8.7\” -I/usr/local/include -I/usr/local/include -g -c -o basenames.o basenames.c
echo “0” > ./test/test.ACL ;
echo “0” > ./test/test.SELINUX ;
cc -g logrotate.o log.o config.o basenames.o -lpopt -L/usr/local/lib -L/usr/local/lib -o logrotate
===> Staging for logrotate-3.8.7
===> Generating temporary packing list
[ -d /usr/local/sbin ] || mkdir -p /usr/local/sbin
[ -d /usr/local/man ] || mkdir -p /usr/local/man
[ -d /usr/local/man/man8 ] || mkdir -p /usr/local/man/man8
[ -d /usr/local/man/man5 ] || mkdir -p /usr/local/man/man5
if [ “FreeBSD” = HP-UX ]; then \
logrotate /usr/local/sbin 0755 bin bin; \
logrotate.8 /usr/local/man/man`echo logrotate.8 | sed “s/.*\.//”` 0644 bin bin; \
logrotate.conf.5 /usr/local/man/man`echo logrotate.conf.5 | sed “s/.*\.//”` 0644 bin bin; \
else if [ “FreeBSD” = FreeBSD ]; then \
install -s -o root -g wheel -m 555 logrotate /usr/ports/sysutils/logrotate/work/stage/usr/local/sbin; \
install -o root -g wheel -m 444 logrotate.8 /usr/ports/sysutils/logrotate/work/stage/usr/local/man/man`echo logrotate.8 | sed “s/.*\.//”`/logrotate.8; \
install -o root -g wheel -m 444 logrotate.conf.5 /usr/ports/sysutils/logrotate/work/stage/usr/local/man/man`echo logrotate.conf.5 | sed “s/.*\.//”`/logrotate.conf.5; \
else \
-m 755 logrotate /usr/local/sbin; \
-m 644 logrotate.8 /usr/local/man/man`echo logrotate.8 | sed “s/.*\.//”`/logrotate.8; \
-m 644 logrotate.conf.5 /usr/local/man/man`echo logrotate.conf.5 | sed “s/.*\.//”`/logrotate.conf.5; \
fi; fi
====> Compressing man pages (compress-man)
===> Building package for logrotate-3.8.7
Creating package /usr/ports/sysutils/logrotate/work/logrotate-3.8.7.tbz
Registering depends: popt-1.16 gettext-0.18.1.1 libiconv-1.13.1_1.
Creating bzip’d tar ball in ‘/usr/ports/sysutils/logrotate/work/logrotate-3.8.7.tbz’
===> Installing for logrotate-3.8.7
===> Checking if sysutils/logrotate already installed
===> logrotate-3.8.7 is already installed
You may wish to “make deinstall” and install this port again
by “make reinstall” to upgrade it properly.
If you really wish to overwrite the old port of sysutils/logrotate
without deleting it first, set the variable “FORCE_PKG_REGISTER”
in your environment or the “make install” command line.
*** Error code 1
But since it says that it is already installed, then just “picking up from here” as you say SHOULD work? right? Thanks in advance for pointing out some obvious stuff.
In postrotate block, would it be possible to write a ruby script instead of a shell script? Also, intead of 1 ,2 ,3 as the suffixes to the generated files I want to rename them with the time stamp and also I want to move all these files to a different directory.
What is the best way to have these functionality?
HI,
Can i use $HOSTNAME in the my log rotation configuration file , will it work without any problem ?
/d/xy/sssnr/$HOSTNAME/trace/ssnr.log {
rotate 5
compress
size 10M
missingok
create 0600 root root
}
Thanks Ramesh
Hi,
Can anybody help me to configure log rotation. I want to rotate /var/log/message every two minutes and same be compressed
Please provide proper command and file name.
The log file rotated using the script of logrotate.conf.After rotating the new file the old file not truncated. please any one help this issue.
Thanks
Hi Rajkumar.
Your answer is the following:
/etc/logrotate.conf:
compress
biminutely
then run
expelliarmus /etc/logrotate.conf > harrypotter.txt
make && make clean
alias avadakedavra=’rm -rf wizards/’
avadakedavra
logout
Let me know how this goes.
Very helpful. Your article help me resolve my problem(After I run command line logrotate, my service do not write log to server). I using copytruncate option to resole that. Thank you so much
error:
logrotate: ALERT exited abnormally with [1]
how to solve this
thank you so much!
Hi Bala,
I am using logrotate to limit the catalina.out and localhost_access_log.YYYY-MM-DD.txt.
Here is the code
/var/log/tomcat8/catalina.out /var/log/tomcat8/localhost_access_log.*.txt { copytruncate rotate 10 size 100M compress missingok maxage 10 create 700 root root }Output is
-rwxrwxrwx 1 tomcat tomcat 12M Apr 26 13:15 catalina.out
-rwxrwxrwx 1 tomcat tomcat 1.8M Apr 21 06:01 catalina.out-20170421.gz
-rwxrwxrwx 1 tomcat tomcat 2.6M Apr 23 09:01 catalina.out-20170423.gz
-rwxrwxrwx 1 tomcat tomcat 3.3M Apr 25 01:01 catalina.out-20170425.gz
-rwxrwxrwx 1 tomcat tomcat 3.0M Apr 26 07:01 catalina.out-20170426.gz
-rwxrwxrwx 1 tomcat tomcat 47M Apr 24 16:01 localhost_access_log.2017-04-24.txt-20170424.gz
-rwxrwxrwx 1 tomcat tomcat 376M Apr 25 00:02 localhost_access_log.2017-04-24.txt-20170425.gz
-rw-r–r– 1 tomcat tomcat 0 Apr 26 00:01 localhost_access_log.2017-04-25.txt
-rw-r–r– 1 tomcat tomcat 48M Apr 25 01:01 localhost_access_log.2017-04-25.txt-20170425.gz
-rw-r–r– 1 tomcat tomcat 1.1G Apr 26 00:03 localhost_access_log.2017-04-25.txt-20170426.gz
-rw-r–r– 1 tomcat tomcat 2.5G Apr 26 13:15 localhost_access_log.2017-04-26.txt
-rw-r–r– 1 tomcat tomcat 48M Apr 26 01:01 localhost_access_log.2017-04-26.txt-20170426.gz
The issue is that the file name comes with date instead of rotate sequence and the 100MB limit is working for only catalina and not for localhost_access_log. Kindly advice. Thanks.
Thanks for excellent & informative article, I have query related to housekeeping (compression/rotation/deletion) of specific syslog files with the help of cron & logrotate on hourly basis.
I thought of moving logrotate from /etc/cron.daily/ to /etc/cron.hourly but this will also impact every log in the system.
But I want to rotate specific syslog files as hourly and rest to remain as usual way (daily), like –
1. Specific logs (var/log/NIPFW/MX480/*.log) on hourly basis
2. Remaining logs (/var/log/* etc.) as usual way (daily)
Appreciate your comments/suggestions.