Highly Efficient Record Processor (HERP) filter

Purpose

The Highly Efficient Record Processor (HERP) filter logs an event for each API request. These logs provide information regarding the processing of each request. The logs can be inspected to view the usage of a service and to perform auditing.  Events are recorded even if the API request is rejected by Repose before reaching the origin service. 

General filter information

Filter name: HERP

Filter configuration: highly-efficient-record-processor.cfg.xml

Released: version 7.0.0.1

Prerequisites

Required headers: The HERP filter has no required headers.

Required preceding filters: The HERP filter has no required preceding filters.  

Recommended follow-on (succeeding) filters: The DeRP filter is optional but commonly follows the HERP filter.  

Basic HERP configuration

To set up the HERP filter, edit the highly-efficient-record-processor.cfg.xml file. 

  1. Add the HERP filter to your system model. 
  2. Configure the logger-name attribute to note the SLF4j logger
  3. Configure the service-code attribute to note the service being accessed.
  4. Configure the region attribute to note the region of the service.
  5. Configure the data-center attribute to note the data center of the service.
  6. Configure the template element to define the format of the messages to be logged by the HERP filter. 

Optional user-configurable filtering

The HERP filter allows user-configurable filtering along with optional pre and post log identifiers. 

Each filterOut block contains all of the criteria required for a single filter. If any of the individual criterion in the match elements contained within a filterOut element fail, then that filter fails and it goes on to evaluate the next one. If any of the filterOut blocks fully match, then the event is filtered and not logged to the post-filter log. In other words, the match elements are logically ANDed and the filterOut blocks are logically ORed.

1. Within the match element:

  • Set the field attribute to name the event field to be matched. If the field is a map (e.g. parameters), then a dot separates the field from the map key and both are placed in the field attribute.
  • Set the regex attribute to determine if the regular expressions match the fields' values. The regex is currently considered to match if any part of a field's value matches. For example, a regex of "racker" would match a value of "rackerTwo". To strictly match, use a more explicit regex such as "^racker$".

2. Optionally, you can: 

  • Specify a pre-filter-logger-name that logs all events to it prior to performing any filter logic.
  • Specify a post-filter-logger-name that logs all events to it that are not filtered out.

User Access Event Recording With Flume

For information about using the Flume with the HERP filter for user access event recording, see CF Flume Sink.

Configurable parameters

XML Schema definition

Example configuration

ElementsAttributes

Required/

Optional

Description
<highly-efficient-record-processor>-RequiredSpecifies the sub-elements and attributes to define your HERP configuration.
pre-filter-logger-nameOptional

Used to note the SLF4j logger target which can be used in the backend logger configuration to direct it's output to an appender.

Defaults to org.openrepose.herp.pre.filter

See Log4j documentation for more information.

post-filter-logger-nameOptional

Contains all of the criteria required for a single filter to direct it's output to an appender.

See Log4j documentation for more information.

service-codeRequiredUsed in logging to note the service being accessed.
regionRequiredUsed in logging to note the region of the service.
data-centerRequiredUsed in logging to note the data center of the service.
<template>-Required

Mustache-based template used to output events. See the supported fields in the table below.

crushOptionalIndicates whether or not to replace a newline sequence (i.e. \r, \n, \r\n) plus any following whitespace with a single space. Default value is false.
<filterOut>-Optional

Contains the set of criteria applied to the events to determine if an event is sent through to the named post filter logger. Multiple filterOut elements are logically OR'd together.

<match>-Required if using the filterOut element

Contains individual criterion used to determine if an event passes through. Multiple match elements are logically AND'd together.

fieldRequiredEvent field to compare the regular expression against.
regexRequiredRegular expression to apply to the event field.

In the template element, certain template keys may be used to add dynamic content to the text being logged. Any valid key enclosed in brackets (i.e., {{<key>}}) will be replaced, brackets included, by the value associated with that key at runtime. Note that key names are case sensitive. A list of supported keys follows:

Supported fields

DescriptionPrerequisite Filter(s)Version release

clusterId

The cluster ID of the Repose node on which the filter is running, configured in the Repose system model.-7.0.0.1
dataCenter

The data center, configured in the highly-efficient-record-processor.cfg.xml file.

This string will be used in logging to note the data center of the service.

-7.0.0.1
guidA globally unique identifier generated for a particular request.-7.0.0.1
impersonatorNameThe impersonator making the request on behalf of a user.

client-auth

or

openstack-identity-v3

7.0.0.1
methodLabelA label which describes the API call. This is pulled from the WADL used by the api-validator filter.api-validator7.0.1.0
nodeIdThe node ID of the Repose node on which the filter is running, configured in the Repose system model.-7.0.0.1
parameters

The query parameters for a request.

Note that this field is a map, and specific parameters can be inserted using the following form: "{{parameters.<parameter-key>}}".

-7.0.0.1
projectID

The tenant ID or project ID of the user making the request authenticated through an OpenStack Identity service.

Note that, if the client-auth filter is used, the tenantId from that filter will be inserted into this field.

client-auth

or

openstack-identity-v3

7.0.0.1
region

The region, configured in the highly-efficient-record-processor.cfg.xml file.

This string will be used in logging to note the region of the service.

 -7.0.0.1
requestMethodThe HTTP method of the request.-7.0.0.1
requestQueryStringThe full query string of the request. For example, a request to /path?a=1&b=1 would have a query string of a=1&b=2.-7.0.0.1
requestURLThe URL of the request.-7.0.0.1
requestorIpThe IP of the user making the request.-7.0.0.1
responseCodeThe response code to be returned to the user.-7.0.0.1
responseMessageThe response message to be returned to the user. This is not the body of the response.-7.0.0.1
rolesThe roles of the user making the request authenticated through an OpenStack Identity service.

client-auth

or

openstack-identity-v3

7.0.0.1
serviceCode

The service code, configured in the highly-efficient-record-processor.cfg.xml file.

This string will be used in logging to describe the service being accessed.

-7.0.0.1
targetHostThe host portion of the request.-7.0.0.1
timestampA timestamp indicating when the HERP filter processed the request.-7.0.0.1
userAgentThe user agent of the request.-7.0.0.1
userNameThe username of the user making the request.

client-auth

or

openstack-identity-v3

7.0.0.1

The HERP filter also provides helper functions to the templating engine. These helpers work like the keys above, but take some input. For example, assuming the HTTP request method is a GET, {{cadfMethod requestMethod}} would be replace by "read/get". The following helpers are available:

Supported helpers
DescriptionVersion release
cadfOutcome <responseCode>Converts a HTTP response code into either "success" or "failure".7.0.1.0
cadfTimestamp <timestamp>Converts a timestamp into the CADF format.7.0.1.0
cadfMethod <requestMethod>Converts a HTTP method into the CADF format.7.0.1.0

Recommendation for managing disk space

If you expect heavy usage, you should use a logging tool, such as Logstash, for managing events and logs and should not write events to files. Even if you rely on auditing, it is not recommended that you use the file system with the audit logs. Repose has developed CF Flume Sink to eliminate disk space and log file issues. 

Return codes and conditions

This filter does not return specific response codes.

Request headers created

This filter does not create any request headers.