Interpreting Log Files

Log files have great importance in AR System since they help you to identify the root-cause of an incident. Once you have a production release, logs might be full of errors but you have to order these errors according to their impact and then solve one by one till you have a clean log file. The most important error is that there some sort of loop on the system. You can see this in the log file as "Too many filter operations...". Once you see this, your system is in trouble and this error must be solved as soon as possible. In this case, you can turn on filter and escalation log and find the reason for this error. Another fatal error is the one related with licenses. If you do not have enough license, your system might behave abnormal, e.g. you would not create a licensed user.

Another point in log analysis is that AR System might throw the same error multiple times which cause a big log file. In order to read it, you have to go through many pages which is sometimes impossible. For this case, I recommend you to write a simple log parser, e.g. in Java, which goes through the whole log file and brings you distinct errors. In this way, your logs will be more readable.

It is also important to rotate the logs because they get larger everyday. It is a good idea to rotate logs, esp. arerror.log on daily basis. In this way, you can find errors easier when you have an incident. BMC always requests log files when you open a ticket to them. Having daily logs will also be an advantage because you can submit smaller logs to BMC. We use the following dummy shell script for rotation and zipping. You can adapt it to your environment:

#!/bin/sh
#------------------Rotating and Zipping 'XXX_error.log'--------------#
DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%Y`
#------------------Rotation of 'XXX_error.log'----------------#
LOGFILEJ=api_error.log
for dir in <directory>
do
        if [ -x $dir ]; then
                cd $dir
                mv $LOGFILEJ $LOGFILEJ.$YEAR$MONTH$DAY
                echo "-------------------------------------------------"        
              echo "$YEAR-$MONTH-$DAY : '$LOGFILEJ' file is rotated";
        else
              echo "-------------------------------------------------"
                echo "$YEAR-$MONTH-$DAY : ERROR:'$LOGFILEJ' file cannot be rotated!";
        fi
done
#------------------Zipping of 'api_error.log'----------------#
for dir3 in  <directory>
do
        if [ -x $dir3 ]; then
                 cd $dir3
               gzip -c $LOGFILEJ.$YEAR$MONTH$DAY  > $LOGFILEJ.$YEAR$MONTH$DAY.gz
               rm $LOGFILEJ.$YEAR$MONTH$DAY
               mv $LOGFILEJ.$YEAR$MONTH$DAY.gz api_error/$LOGFILEJ.$YEAR$MONTH$DAY.gz
                 echo "-------------------------------------------------"       
               echo "$YEAR-$MONTH-$DAY : '$LOGFILEJ' file is zipped.";
        else
               echo "-------------------------------------------------"
                 echo "$YEAR-$MONTH-$DAY : ERROR:'$LOGFILEJ' file cannot be zipped!";
        fi
done
exit