Skip to end of metadata
Go to start of metadata

Contents

Purpose

Out of the box, Repose only provides application logging which is different than access logging. For access logging, the Simple Logging Facade for Java (SLF4J) HTTP Logging filter uses Log4j 2.x to allow logging the information in HTTP requests that are sent to and the responses sent from Repose. It is based on the Apache HTTPD Logging Standard.

If a user-supplied logging configuration file is not found, Repose programmatically sets default log4j properties.  The default properties add a ConsoleAppender to the ROOT logger. The output will be formatted using a PatternLayout set to the pattern "%d %-4r [%t] %-5p %c %x - %m%n".  The default log level is set to DEBUG.

For information regarding the migration from log4j 1.x to log4j 2.x, see the Apache documentation here:

General filter information

Filter name: slf4j-http-logging

Filter configuration: slf4j-http-logging.cfg.xml

Released: version 4.1.5

This filter replaced the HTTP Logging filter which was deprecated in version 4.1.5.

Prerequisites

Required headers: The SLF4J HTTP Logging filter has no required request headers.
Required preceding filters: The SLF4J HTTP Logging filter has no required preceding filters.
Standard filter order:  When you set the order of Repose filters, place the SLF4J HTTP Logging filter first in the sequence.

Basic SLF4J HTTP Logging configuration

Configure the SLF4J HTTP Logging filter by editing the slf4j-http-logging.cfg.xml file.

1. Set Up Repose  

Updated the logging-configuration in the container.cfg.xml.

container.cfg.xml
...
<logging-configuration href="file:///etc/repose/log4j2.xml"/>
...

2. Add the SLF4J HTTP Logging Filter  
Add the  slf4j-http-logging filter to your system-model.cfg.xml within the filters element. Place this filter at the top of the filter sequence.

system-model.cfg.xml
<filters>
	<filter name="slf4j-http-logging" />
	<!-- Place other filters below the slf4j-http-logging filter ->	
</filters>

3. Configure the SLF4J HTTP Logging Filter 

Within the slf4j-http-log element:

  • Configure the id to the desired logger name.
  • Configure either the format attribute or element for what will be logged.
slf4j-http-logging.cfg.xml
...
    <slf4j-http-log id="my-special-log">
        <format>
            <![CDATA[
            { "timestamp": "%t" ... }
            ]]>
        </format>
    </slf4j-http-log>
...

4. Configure Log4J logging level

In the file named in the logging-configuration in the container.cfg.xml (log4j2.xml by default), configure the logger named by the slf4j-http-log element's id attribute in the slf4j-http-logging.cfg.xml file to INFO or below.

log4j2.xml
...
    <Loggers>
        <Root level="warn">
            <AppenderRef ref="RollingFile"/>
        </Root>
        ...
        <Logger name="my-special-log" level="info"/>
    </Loggers>
...

5. Test SLF4J HTTP Logging

  • Run requests.
  • Confirm your configuration by checking the logs in the target file.

Configurable parameters

XML schema definition

Example configuration

The slf4j-http-logging.cfg.xml file contains the following elements and attributes. Add the filter to your Repose deployment through the System Model Configuration.

ElementsAttributes

Required/

Optional

Description
<slf4j-http-log>-RequiredDefines specified logs.
idRequired

Names the SLF4J logger appender name. It can be used in the backend logger configuration (Current log4j 1.2) to direct it's output to an appender. See log4j to learn more.

If the id doesn't match a logger named in the logging configuration, then the default Root logger is used.

formatRequired if the format element is not used.Specifies what format will be used for the logs. See "What is Logged" below for formatting instructions.
<format>-Required if the format attribute is not used.

Specifies what format will be used for the logs. See "What is Logged" below for formatting instructions.

The format attribute and element are mutually exclusive and one is required. If neither or both are present, then the configuration file will fail to validate.


Configuration example

slf4j-http-logging.cfg.xml
<?xml version="1.0" encoding="UTF-8"?>

<slf4j-http-logging xmlns="http://docs.openrepose.org/repose/slf4j-http-logging/v1.0">
        <!-- The id attribute is the named target of the log output,
         it can then be used in log4j backend to determine which appender to go to -->
        <!-- The format includes what will be logged.  The arguments with % are a subset of the apache mod_log_config
             found at http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats -->
        <slf4j-http-log
                id="my-special-log"
                format="Response Time=%T seconds\tResponse Time=%D microseconds\tResponse Code Modifiers=%200,201U\tModifier Negation=%!401a\tRemote IP=%a\tLocal IP=%A\tResponse Size(bytes)=%b\tRemote Host=%hRequest Line=&quot;%r&quot;\tRequest Method=%m\tRequest Protocol=%H\tServer Port=%p\tQuery String=%q\tTime Request Received=%t\tStatus=%s\tRemote User=%u\tURL Path Requested=%U\n"/>
        <slf4j-http-log id="my-other-log" format="Remote IP=%a Local IP=%A" />
</slf4j-http-logging>


In the log4j2.xml file, add settings similar to the following:


log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <Console name="STDOUT">
            <PatternLayout pattern="%d %-4r [%t] %-5p %c - %m%n"/>
        </Console>
        <RollingFile name="RollingFile" fileName="/var/log/repose/current.log"
                     filePattern="/var/log/repose/current-%d{yyyy-MM-dd_HHmmss}.log">
            <PatternLayout pattern="%d %-4r [%t] %-5p %c - %m%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="200 MB"/>
            </Policies>
            <DefaultRolloverStrategy max="2"/>
        </RollingFile>
    </Appenders>
    <Loggers>
        <Root level="warn">
             <AppenderRef ref="STDOUT"/>
             <AppenderRef ref="RollingFile"/>
        </Root>
        <Logger name="com.sun.jersey" level="off"/>
        <Logger name="net.sf.ehcache" level="error"/>
        <Logger name="org.apache" level="warn"/>
        <Logger name="org.eclipse.jetty" level="off"/>
        <Logger name="org.openrepose" level="info"/>
        <Logger name="org.springframework" level="warn"/>
        <Logger name="my-special-log" level="info"/>
    </Loggers>
</Configuration>


What is logged

Currently, Repose can be configured to log information from the request URI and headers but not the request message content.The SLF4J HTTP Logging filter can log static text specified in the slf4j-http-log format attribute and specific information about the request/response.  The table below lists the request/response information the user may choose to log:

* This table is a subset of the Apache logging functionality listed at Apache HTTPD Logging Standard. Repose only supports this subset of the Apache logging standard plus the listed non-standard extras.

Format string

Description

\t

Tab

\n

Newline

%%

The percent sign

%a

Remote IP-address

%A

Local IP-address

%b

Size of response in bytes, excluding HTTP headers.  In CLF format, i.e. a '-' rather than a 0 when no bytes are sent.

%B

Size of response in bytes, excluding HTTP headers.  0 is logged when no bytes are sent.

%D

The time taken to serve the request, in microseconds. [As of release 2.3.6]

%gThe internal GUID assigned to the request. [As of release 7.1.2.0]
%HThe request Protocol. [As of release 2.5.0]

%h

Remote host

%{header}i%{header type out-format}i%{header type out-format in-format}i

Request header. Parameters typeout-format and in-format are optional. If not specified, then the header value is output as is.Currently, the only type supported is DATE. Out and in date formats supported are RFC_1123 (HTTP-Date) and ISO_8601.If type is specified, then out-format is required. in-format is optional and defaults to RFC_1123 for DATE types if not specified.Note: Headers are treated in a case insensitive manner.  For example, %{X-Auth-Token}i is equivalent to %{x-auth-token}i.

%MThe Status message if set. [As of release 2.3.6]

%m

The request method.

%{header}o%{header type out-format}o%{header type out-format in-format}o

Response header. Parameters typeout-format and in-format are optional. If not specified, then the header value is output as is.Currently, the only type supported is DATE. Out and in date formats supported are RFC_1123 (HTTP-Date) and ISO_8601.If type is specified, then out-format is required. in-format is optional and defaults to RFC_1123 for DATE types if not specified.An example to output the retry-after header using the ISO_8601 format would be %{retry-after DATE ISO_8601}oNote: Headers are treated in a case insensitive manner.  For example, %{LOCATION}o is equivalent to %{location}o.

%p

The canonical port of the server serving the request.

%q

The query string (prepended with a ? if a query string exists, otherwise an empty string).

%rFirst line of request. [As of release 2.5.0]

%s

Status.  For requests that got internally redirected, this is the status of the \*original\* request \--\- %>s for the last.

%t

Time the request was received. The format for Repose versions 4.1.5 - 7.3.2.0 is: "dd-MM-yyyy-HH:mm:ss.SSS", for versions 8.0.0.0 - current: "yyyy-MM-dd HH:mm:ss".

%{format}tTime the request was received using the specified date format (SimpleDateFormat patterns; format string limited to letters, numbers, dashes, spaces, periods, and colons, e.g. "%{yyyy-MM-dd HH:mm:ss}t"). [As of release 7.1.5.0]

%T

The time taken to serve the request, in seconds. [As of release 2.3.6]

%u

Remote user (from auth; may be bogus if return status (%s) is 401)

%U

The URL path requested, not including any query string.


The user can restrict what gets printed for responses with specific HTTP status codes.  The table below shows examples of this functionality:

Format string

Description

%403,401U

Logs the URL path requested on 403 and 401 responses only.

%\!200,304,302U

Logs the URL path requested for all responses except 200, 304, and 302.

Return codes and conditions

This filter does not return specific response codes. The request will simply pass through to the next filter or to the origin service.


Request headers created

The SLF4J HTTP Logging filter does not create any request headers.

Tips and tricks

Logging To A Separate File

You can configure your log4j2.xml to log the SLF4J HTTP output to a different file to keep your application logging and access logging separate.  Here's a sample log4j2.xml that accomplishes this (which assumes the default slf4j filter configuration):

log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration monitorInterval="15" packages="org.apache.logging.log4j.flume.appender">
    <Appenders>
        <Console name="STDOUT">
            <PatternLayout pattern="%d %-4r [%t] %-5p %c - %m%n"/>
        </Console>

        <RollingFile name="RollingFile" fileName="/var/log/repose/current.log"
                     filePattern="/var/log/repose/current-%d{yyyy-MM-dd_HHmmss}.log">
            <PatternLayout pattern="Trans-Id:%X{traceGuid} - %d %-4r [%t] %-5p %c - %m%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="200 MB"/>
            </Policies>
            <DefaultRolloverStrategy max="2"/>
        </RollingFile>

        <!-- Create a new RollingFile configuration element for the separate access log file. -->
        <!-- Optional: The archive file pattern ends with ".gz" to compress these files. Use tools like 'zless' and 'zgrep' to work with them. -->
        <RollingFile name="AccessRollingFile" fileName="/var/log/repose/access.log"
                     filePattern="/var/log/repose/access-%i.log.gz">
            <!-- You only need to configure the message (%m) here. -->
            <PatternLayout pattern="%m"/>
            <Policies>
                <!-- For testing purposes, rollover the log file at 20 KB. -->
                <SizeBasedTriggeringPolicy size="20 KB"/>
            </Policies>
            <!-- Keep the last 10 archive logs. -->
            <DefaultRolloverStrategy max="10"/>
        </RollingFile>

        <File name="PhoneHomeMessages" fileName="/var/log/repose/phone-home.log" append="false">
            <PatternLayout>
                <Pattern>%m</Pattern>
            </PatternLayout>
        </File>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="STDOUT"/>
            <AppenderRef ref="RollingFile"/>
        </Root>
        <Logger name="com.sun.jersey" level="off"/>
        <Logger name="net.sf.ehcache" level="error"/>
        <Logger name="org.apache" level="warn"/>
        <Logger name="org.eclipse.jetty" level="off"/>
        <Logger name="org.springframework" level="warn"/>
        <Logger name="intrafilter-logging" level="info"/>
        <Logger name="phone-home-message" level="info" additivity="false">
            <AppenderRef ref="PhoneHomeMessages"/>
        </Logger>

        <!-- Include all of your slf4j-http-log IDs here (e.g. my-special-log). -->
        <!-- Set the "additivity" to false to have your access logs only go to your access log file. Do not set the additivity if you want the logs going to current.log as well. -->
        <Logger name="my-special-log" level="info" additivity="false">
            <appenderRef ref="AccessRollingFile"/>
        </Logger>
        <Logger name="my-other-log" level="info" additivity="false">
            <appenderRef ref="AccessRollingFile"/>
        </Logger>
    </Loggers>
</Configuration>




  • No labels

16 Comments

  1. See log4j to learn more.

    Is there a specific page on the log4j website that would offer more insight?

    1. Unfortunately, we're still on a very old version of log4j, version 1.2, and so finding the documentation is difficult. There isn't really a specific page on how to configure it.

      We have a backlog item to upgrade to log4j 2.x which has substantially better documentation, but also requires you to change your configuration: REP-810 - Getting issue details... STATUS

      http://logging.apache.org/log4j/1.2/manual.html Might help

      http://stackoverflow.com/questions/4570072/how-to-configure-log4j-properties-for-springjunit4classrunner Also might provide good examples.

  2. slf4j-http-logging.cfg.xml
    ...
    <slf4j-http-log
                    id="my-special-log"

    Where in the log4j.properties file do I define the log name (e.g. 'my-special-log')? Can you please provide an example configuration for the log4.properties file that would work with the 'my-special-log' example?

    1. Here's an example that uses a syslog appender:

      log4j.logger.my-special-log=INFO, httpSyslog, httpLocal
      
      log4j.appender.httpSyslog=org.apache.log4j.net.SyslogAppender
      log4j.appender.httpSyslog.Facility=local1
      log4j.appender.httpSyslog.FacilityPrinting=false
      log4j.appender.httpSyslog.Header=true
      log4j.appender.httpSyslog.syslogHost=some.Syslog.host
      log4j.appender.httpSyslog.port=514
      log4j.appender.httpSyslog.protocol=tcp
      log4j.appender.httpSyslog.layout=org.apache.log4j.PatternLayout
      log4j.appender.httpSyslog.layout.ConversionPattern=%m%n
      
      
      1. Thanks David, I added this example to the documentation.

        Are these settings typical with a Repose installation  (e.g. port, Facility)? Would localhost suffice for the syslogHost value?

      2. Also, would the FileAppender work here? It may be easier to document/configure.

  3. 2. Add the SLF4J HTTP Logging Filter  
    Add <filter name="slf4j-http-logging"> to your system model configuration, in the <filters> category.

    I am getting a syntax error when I add this to my <filters> section. Should it look like the following instead?

    system-model.cfg.xml
    <filters>
            <filter name="slf4j-http-logging" /> <!-- Self-closing? -->
    </filters>
  4. is it possible to add something like a socket server through which the log can be access remotely? i was told that log4j can do this. if a yes applies to repose, how can i do that? i tried to add a socket appender in log4j2.xml, it didn't work.thanks!

    1. Hi Yegui Cai,

      As far as I know, log4j does not actually store logs. It just writes out log content using provided log appenders. The socket appender, for example, writes log contents out over a socket. So just using Repose, I do not believe that there is a way to remotely access logs.

      However, it should not be too hard to get you remotely accessible logs. The most obvious solution would just be to use ssh/scp/ftp to access the logs on your Repose box. Of course, if desired, you could also use something like Logstash to manage your logs. If you went that route, you would want to add a socket appender to your Repose log4j2 configuration to write the logs to Logstash. For further information on that approach, see the following links:

      https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html

      https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html

      http://stackoverflow.com/a/26227002

      Hopefully that's enough to get you heading down the right path. Let me know if you have any other questions!

      Thanks,

      Damien

  5. Apologize that my previous question is not clear. My question is how i can add something to the log4j2.xml file so that the we can read the info as what is shown in the log locally? I know there is socket appender. I tried to configure repose using socket appender, but it is not successful. could you mind giving me an example on how to achieve this? Thank you very much!

    1. Yegui Cai,

      I think I may still be missing something. Where do you want the log data to end up?

      The socket appender will write a serialized Log Event to a socket. You need something to received that data. Log4J2 provides a SocketServer which can do just that. For more information and an example, take a look at:

      https://logging.apache.org/log4j/2.x/manual/appenders.html#SocketAppender

      Once the LogEvent is received, it can be handled in a number of ways. Logstash, for instance, may do some transformations on it before adding it to elasticsearch. A log4j2 SocketServer will use another log4j2 configuration file to determine where to write the LogEvent and in what format.

      As far as formatting goes, many log4j2 appenders have a format configuration where you can specify how data should be written. By default, Repose writes out to a log file using a RollingFile appender with the PatternLayout set to <PatternLayout pattern="Trans-Id:%X{traceGuid} - %d %-4r [%t] %-5p %c - %m%n"/>. If you have another appender with output that you want to look the same, you'll have to set the format/pattern for that appender to the same format/pattern as the file appender.

      If you need additional help, I would be happy to provide it, but I will need more details. In particular, I need to know what problem you are trying to solve.

      Thanks,

      Damien

      P.S. A more detailed example of setting up a SocketAppender with a SocketServer: http://howtodoinjava.com/2013/04/08/log4j-socketappender-and-socket-server-example/

  6. Hi Damien
    Our use case is the following
    We have our app deployed as a repose filter. We'd like to put together a test infrastructure that would test the logs.
    When we add a socket appender to our app log configuration (the same way as it was shown in those URLs you refer. In fact we have seen them and used as references)  we do not see anything received on the test client.
    Do you know whether repose would anyhow block the traffic from that log server.
    Have you guys ever tried anything like that? If you did an example would’ve been really helpful
    Thanks

  7. Here is out configuration file:

    <?xml version="1.0" encoding="UTF-8"?>
    <Configuration monitorInterval="15" packages="org.apache.logging.log4j.flume.appender">
    <Appenders>
    <Console name="STDOUT">
    <PatternLayout pattern="%d %-4r [%t] %-5p %c - %m%n"/>
    </Console>
    <RollingFile name="RollingFile" fileName="/var/log/repose/current.log"
     filePattern="/var/log/repose/current-%d{yyyy-MM-dd_HHmmss}.log">
    <PatternLayout pattern="GUID:%X{traceGuid} - %d %-4r [%t] %-5p %c - %m%n"/>
    <Policies>
    <SizeBasedTriggeringPolicy size="200 MB"/>
    </Policies>
    <DefaultRolloverStrategy max="2"/>
    </RollingFile>

    <Socket name="Socket" host="dev-deskl2368" port="8082">
    <SerializedLayout />
    </Socket>

    </Appenders>
    <Loggers>
    <Root level="info">
    <AppenderRef ref="STDOUT"/>
    <AppenderRef ref="RollingFile"/>
    <AppenderRef ref="Socket"/>

    </Root>
    <Logger name="com.sun.jersey" level="off"/>
    <Logger name="net.sf.ehcache" level="error"/>
    <Logger name="org.apache" level="warn"/>
    <Logger name="org.eclipse.jetty" level="off"/>
    <Logger name="org.springframework" level="warn"/>
    <Logger name="intrafilter-logging" level="info"/>
    <Logger name="com.irdeto" level="trace"/>
    </Loggers>
    </Configuration>
  8. Yegui Cai,

    Thanks, all of the context is a huge help. Unfortunately, to the best of my knowledge, we have not used the Socket appender. We have, however, used the Flume appender to send log data from Repose to a Flume instance. You can read more about that here: CF Flume Sink
    Repose itself should not be blocking this traffic -- Repose essentially hands that log data to log4j2 which then decides what to do with it. So the log4j2 system is what we're looking at here.

    I suppose the first thing I would do is verify that the host (i.e., dev-desk12368) is reachable from your Repose node. Try to ping the host from the Repose node. If that goes through, perhaps curl random data to the SocketServer (or whatever you happen to be using) listener port. I would imagine that you will see logs in the SocketServer (or whatever you happen to be using) stating that a message was received but invalid. You could also use something like Wireshark if you really want to get down to it. Rule out the possibility that something is wrong with the receiving host.

    Once that is done, make sure that the log4j2 is actually sending the data. Again, you may want to use something like Wireshark. For every Repose log line, I would expect to see data being sent to dev-desk12368:8082. If that is not happening, I'm not sure how you would move forward. You may have to attach a remote debugger to the Repose process and start walking through log4j2 to find out what the Socket appender is actually doing.

    i suspect that, if the host is working as expected, either a firewall is blocking traffic, or the log4j2 configuration you've provided is not being loaded, and as a result, the Socket appender is not actually being used. For the latter case, if you can make any change to your log4j2 config and have it reflected in the logs, you will know that the correct config is being used by log4j2 inside of Repose.

    I know most of this is vague advice, but unfortunately, it's very difficult to know or find out exactly what is causing your issue without either being in your environment, or having an exact replica to test with. On top of that, I only really have a moderate knowledge of log4j2. Regardless, I hope this helps!

    Thanks,

    Damien

  9. Hi Damien.

    Thank you very much for your detailed suggestions.

    Maybe i am asking a dumb question. However my understanding of the socker appender configuration is

    1. the repose node is serving as a server in a socket connection for providing the information (ie, the text of any logging events). so if the configuration works, we should be able to see the port 8082 open for incoming requests. What is see from my repose node is that there is no port 8082 open yet.
    2. the hostname should be the some hostname of the repose node.

    Does my understanding correct?

     

     

    1. Yegui Cai,

      Although Repose is a server, the SocketAppender that is instantiated based on the Repose log4j2.xml configuration is a socket client. In other words, it expects there to be a server socket listener running on the the specified host and port that it can connect to. Take a look at this section of the Log4J 2.x documentation for more information on their SocketAppender:

      Kindest Regards,

      Bill