Logback configuration with log rotation and retention based on active spring boot profile

Logging is an important aspect of a production application. They are a single source of truth when you require an issue to be debugged or an event to be traced once your application has gone into production. Logback is one of the logging utilities supported in spring based applications. It’s even better in spring boot as it supports different profiles and configurations based on the profile.

What is the objective?

We will see how we can use a single logback configuration file to effectively and efficiently manage the log files generated by your service.

For production and staging, we would like to have the logs stored in a file so that they can be referred back to investigate an issue or bug. Following would be our requirements for logging.

1. The name of the log file should be <name of application>.log. Eg: (order-service.log )
2. When the log file is more than 80 MB, we need to rotate the file and the older content should be available in a file with the following format:
   <app-name>.<date>.<fileindex>.log
   Eg: order-service.2018-08-20.1.log,order-service.2018-08-20.1.log
   Keeping the file at 80MB would allow us to open and read through them in a normal text application.
3. When the date changes, the files for the previous day should be moved to a directory with the name as the previous day date.
4. Delete the older logs when they are older than 30 days.

For our development purposes, we don’t need to have the logs sent to a file. They can just be sent to console of the IDE.

In the following sections, we will see how we can accomplish the above requiments using the logback configuration.

Adding support for logback in our spring boot

When you create a spring boot application and add a starter dependency like the web ( spring-boot-starter-web), the logback dependencies (spring-boot-starter-logging) are already pulled into the mix. There are no other additional dependencies required for supporting logback.

Configuring logback

If you are using spring-boot, you can configure logback using a single file in the classpath. The file needs to be named as logback-spring.xml.

We will see how we can configure a logback for different profile using a sample file. Following would be the sample file and each sections are defined in detail below.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <!-- Get the property for app name from the properties file -->
    <springProperty scope="context" name="appName" source="spring.application.name"/>

    <!-- Configuration when the profile is staging or prod -->
    <springProfile name="staging,prod">
        <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
            <file>logs/${appName}.log</file>
            <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                <!-- daily rollover -->
                <fileNamePattern>logs/%d{yyyy-MM-dd,aux}/${appName}.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
                <!-- keep 30 days' worth of history -->
                <maxHistory>30</maxHistory>
                <timeBasedFileNamingAndTriggeringPolicy
                        class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                    <!-- or whenever the file size reaches 80MB -->
                    <maxFileSize>80MB</maxFileSize>
                </timeBasedFileNamingAndTriggeringPolicy>
            </rollingPolicy>
            <encoder>
                <pattern>%d %5p | %t | %-54logger{55} | %m %n</pattern>
            </encoder>
        </appender>

        <logger name="${appName}-logger">
            <level value="INFO"/>
        </logger>

        <root>
            <level value="INFO"/>
            <appender-ref ref="FILE"/>
        </root>
    </springProfile>

    <!-- Configuration when the profile is dev -->
    <springProfile name="dev">
        <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
            <encoder>
                <pattern>%d %5p | %t | %-55logger{55} | %m %n</pattern>
            </encoder>
        </appender><springProperty scope="context" name="appName" source="spring.application.name"/>
        <logger name="${appName}-logger">
            <level value="INFO"/>
        </logger>
        <root>
            <level value="INFO"/>
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>

Note that the above file content need to put in logback-spring.xml file in the classpath ( Preferably in the resources folder )

Sections in the configuration file

The configuration file is basically an XML with the root element as  <configuration>. Under this, we can specify different configurations per spring profile.

Defining a property in the logback context

The first line indicates that we need to refer appName as the property value of spring.application.name from the currently active profile’s property. We will be using this in the subsequent configuration. You can use this method to refer to any other property you have defined in the application.yml or application.properties file.

<springProperty scope=”context” name=”appName” source=”spring.application.name”/>

name is the name of the property in the configuration context and source is property key from which the value needs to be derived.

Configuring the logback property per profile

Configuration specific to profile are put in the <SpringProfile> tag with the applicable name of the profiles specified as comma separated list.

<springProfile name=”staging,prod”>

….

</springProfile>

If you want more details on how to configure multiple profiles and its advantages, refer to the following blog post.

Appenders

An appender is basically a location to which the logs will be appended to. This could a file or a console ( IDE console, output stream etc.). In our example, we would be using a File-based appender for our staging and production profile and a console based appender for our development profile.

Define a rolling file appender with a retention period and file name format

Keeping in mind the requirements we specified initially, we are going to define our file-based appender for staging and prod profile as below

<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>logs/${appName}.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <!-- daily rollover -->
        <fileNamePattern>logs/%d{yyyy-MM-dd,aux}/${appName}.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
        <!-- keep 30 days' worth of history -->
        <maxHistory>30</maxHistory>
        <timeBasedFileNamingAndTriggeringPolicy
                class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
            <!-- or whenever the file size reaches 80MB -->
            <maxFileSize>80MB</maxFileSize>
        </timeBasedFileNamingAndTriggeringPolicy>
    </rollingPolicy>
    <encoder>
        <pattern>%d %5p | %t | %-54logger{55} | %m %n</pattern>
    </encoder>
</appender>
<logger name="${appName}-logger">
    <level value="INFO"/>
</logger>
<root>
    <level value="INFO"/>
    <appender-ref ref="FILE"/>
</root>

1. Start by defining the name for the appender and then refer to the logback class that would provide the functionality. For our purpose, we are using ch.qos.logback.core.rolling.RollingFileAppender
2. Specify the name and location of the current log file (This will contain the most recent logs of the application ).
   We will keep it as <app-name>.log under logs directory which is specified using in <file>

    

   Please note that the logs directory is relative to the application location.

3. Create a file rolling policy based on time ( date ). We use the following logback class for the same ch.qos.logback.core.rolling.TimeBasedRollingPolicy
4. Define the location of files and the filename pattern for rolled files.
   In the above sample, we are asking logback to put the files under logs directory of the application with the directory name as the date of the day. The name of the rotated file would be <app-name>.<date>.<fileindex>.log

    

   Please note that this is only for the rotated file. The most recent file would be created as <appname>.log in the logs directory  as specified in step 2

5. Next, we need to rotate the file whenever the recent file reaches 80MB.
   This is configured under the timeBasedFileNamingAndTriggeringPolicy tag
6. The maxHistory tag specifies the maximum days worth of files to be retained.
7.  Use the encoder to define the pattern of the logs in the file. You can refer here for more details on the logback encoder.
8. Define the logger name as appName-logger
9. Define the level or logging under the root tag. The available values are INFO, TRACE,ERROR, DEBUG.
   The root element is referred to the file appender using the appender name.

That’s it for the file based rolling and retention configuration. Be sure to define the spring.application.name in the property file and also use the names of the profiles according to your setup.

Creating a console based appender for development

During the development phase, we are not bothered about the logging and most probably you would be logging to the IDE console. So let’s define an appender for our dev profile.

<!-- Configuration when the profile is dev -->
<springProfile name="dev">
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d %5p | %t | %-55logger{55} | %m %n</pattern>
        </encoder>
    </appender>


    <logger name="${appName}-logger">
        <level value="INFO"/>
    </logger>

    <root>
        <level value="INFO"/>
        <appender-ref ref="CONSOLE"/>
    </root>
</springProfile>

Here the definition is pretty straightforward.

1. Put the appender configuration under respective SpringProfile
2. Create the appender as console appender using the logback class as ch.qos.logback.core.ConsoleAppender
3. Use the encoder pattern same as the file appender
4. Define the log lever and refer to the appender in the root element

Conclusion

Logs are an important source of truth in production. It is important to log critical events and its even more important to make sure that we have logs organized in a manner so that they are manageable as well as under control.

It is very easy to overlook the nuances of logging and end up with a 10 GB log file which renders the entire logging exercise useless.

Modern systems using containers use tools like Filebeat to ship the logs to a central system like an ELK stack.  Even in those cases, it is advisable to have the log files organized and backed up for secondary reference.

If you are interested more in filebeat and ELK stack, you could refer to  the following links

• Filebeat for shipping logs
• ELK stack for central logging

Please let us know your queries and comments in the discussion section below.

regards
Microideation