Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Implement Logging Service

Description

A new logging service is necessary to provide flexible, reliable logging (especially access logging). The service will track HTTP interactions along with relevant data for those interactions. It will use this data to render user-provided templates which will then be logged.

The service should provide additional support certain data formats (i.e., JSON). Additional support should include proper escaping and validation.

The JTwig templating engine can be used to render templates.

See the linked spike for more details.

We should be able to specify the logger (like the SLF4J and Herp filters currently do).

Acceptance Criteria

  • This will be a new service.

  • Users should be able to write their own templates on the formatting of the access logs.

    • Template format should not collide with the other template formats already in use in Repose.

  • There should be a mechanism for users to specify that the template is JSON. This will do two things. Template values should be JSON escaped, and the template should be validated on Repose startup service loading to ensure it's valid JSON.

    • If the template doesn't validate with sample/dummy data applied, a WARNING log should be emitted on service loading. Otherwise, no other action should be taken (i.e. use the config anyway).

  • It must be possible to log every request.

  • It should be able to log the following attributes of a request and/or response:

    • HTTP Header values (%i, %o)

      • The values should be the last values we can reasonably get

        • Request values

          • Right before going to the origin service or

          • Right before a filter rejects a request or throws an error. We do not need the values added by the filter that rejected the request or threw an error.

        • Response values

          • Right before the filter chain finishes process (i.e. right before returning to the client)

    • Remote/Local IP address (%a, %A)

    • Size of request and response (%b, %B)

      • This will simply be the value in the Content-Length header

    • Remote host (%h)

    • Request method (%m)

    • Request protocol (%H)

    • Response protocol (if possible/feasible)

      • I'd like to know if the client used HTTP 1.0 or 1.1 and if the origin service responded with HTTP 1.0 or 1.1.

    • Response status message (%M)

    • Query string of the URL (%q)

    • Response status code (%s)

    • Time the request was received (%t)

      • ISO 8601

    • Time taken to serve the request (%D, %T)

    • Time taken to serve the request by the origin service

      • Default to some value when the origin service isn't hit

      • This value might also end up being some large value if we timed out waiting for the origin service

    • URL requested (%U)

    • Username (%u)

    • Tracing ID (%g)

  • When writing headers (which can be a list of values), we should enable the user to specify how it should be outputted.

    • They should have the option of generating a comma-separated list at the very least.

  • A reasonable default template should be supplied.

    • That template is: (TBD)

Environment

None
100% Done
Loading...

is duplicated by

Activity

Damien Johnson May 7, 2019 at 3:01 PM
Edited

I am considering not implementing the “crush” feature (to condense whitespace) at this time for the following reasons:

  • Unanswered questions:

    • When should whitespace be “crushed”?

      • When the configuration is read (i.e., on the templates)? What if the spacing is necessary for the template logic to function? What if the template logic adds whitespace when it runs that isn’t obviously present in the templated message?

      • After rendering the message? This is probably the right time, but adds additional processing for every request. Also, we may be removing characters from templated values which may alter their meaning. The template author may not know all valid values of all templated values when writing the template.

    • How should whitespace be crushed? Removing leading and trailing whitespace seems like a good start. How should whitespace within the message be handled? HERP only replaces repeating whitespace around newlines.

    • What is the intent behind “crushing” whitespace? If it is to reduce the size of messages over the network, should we not “crush” all whitespace (rather than just around newlines)? Is “crushing” around newlines a sufficient compromise?

  • JTwig may give us whitespace control features.

  • XML/XSD may give us the tools we need to condense whitespace as part of XML processing. That couples our implementation to XML configuration and only allows “crushing” when the configuration is read. See xml:space and xsd:whiteSpace. Additionally, if CDATA is not used, XML already ignores leading whitespace on every line (like what HERP is doing). Of course, CDATA must be used for XML templates, but those are not yet formally supported as a format.

  • Looking over the acceptance criteria, “crushing” is not listed. It seems to me like it is not necessary for the MVP, and would be gold-plating.

Damien Johnson May 1, 2019 at 5:12 PM

This was a helpful issue opened on Jetty for someone who was also looking to leverage Jetty listeners to achieve reliable auditing:
https://github.com/eclipse/jetty.project/issues/3314

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Capitalizable

True

Story Points

Time remaining

0h

Sprint

Fix versions

Priority

Parent

Created April 23, 2019 at 4:57 PM
Updated June 27, 2019 at 6:38 PM
Resolved June 19, 2019 at 12:57 PM