Repose uses one of two datastore implementations to store various types of data. The two datastore implementations are:
The local datastore is enabled by default. The distributed datastore is enabled by including the appropriate service into your system model configuration.
If no other datastores are configured, then Repose will use the local datastore. The local datastore will store data using the cache on each node. Data will not be shared among the nodes, so each Repose node will have its own copy of the data. For example, if using rate limiting with the local datastore, then each node will track its own limit information and limit updates will not be shared other nodes.
The Distributed Datastore Service was implemented in version 2.7.0.
A Repose cluster may, at times, need to share information between cluster nodes. The Distributed Datastore component allows Repose to host a simple hash-ring object store that shares consistency between all of the participating cluster nodes. This, in turn, allows other hosted components as well as external programs to use the Repose cluster as a whole to store information.
Instead of cache operations communicating through the Repose Service port (which is the port which Repose service requests to pass to the Origin Service), the Distributed Datastore Service will communicate through configured port(s) within the distributed datastore configuration. If the Distributed Datastore Service is unable to communicate with the service on other nodes, it will fall back on the local datastore temporarily. Once other nodes become reachable, the Distributed Datastore Service will return to being distributed.
The Distributed Datastore service can be added to a Repose deployment by adding it as a service (dist-datastore) within the services list of the system-model.cfg.xml file. Adding the Distributed Datastore Service to a Repose deployment requires that listening ports be configured within the dist-datastore.cfg.xml file. The configuration schema can be found at dist-datastore-configuration.xsd.The <port> element is the port configuration for the Distributed Datastore. When you configure Repose to start with the Distributed Datastore, the running Repose instance will try to find the <port> configuration that matches it's own cluster and node. If only the 'cluster' attribute is defined, the running Repose instance will assume that is the port in which to open a listener for the Distributed Datastore.
The following is a sample configuration.
The distributed datastore can use either HTTP or HTTPS as its transport protocol. HTTPS will be used if a keystore is specified in the configuration, otherwise HTTP will be used.
The distributed datastore provides the option to encrypt communication between nodes using HTTP over SSL/TLS. As mentioned above, this is achieved by configuration a keystore in the dist-datastore.cfg.xml file. When this feature is used (at time of writing), the distributed datastore will also enforce client authentication during the SSL handshake. By both validating the client and encrypting communication with the client, data in the datastore is made more secure and reliable.
Assuming all Repose nodes are configured identically, the most straight-forward way to make use of this security would be to use a single unique keystore as both the keystore and the truststore. This can be achieved by not explicitly configuring a separate truststore. Since each datastore node will have a copy of the keystore, each node will trust every other node.
Client authentication in SSL/TLS can act as as alternate form of client validation, performing a task similar to that of an access control list. As such, the usage of client authentication may replace the need to configure the allowed-hosts section of the dist-datastore.cfg.xml file.
Note that that distributed datastore will use a connection pool to communicate across nodes. If a connection pool is not configured, the default will be used. In nearly all cases, the connection pool being used should not be the default, but rather, a connection pool should be configured to use a keystore/truststore matching the keystore/trustore configured in the distributed datastore. That is, the distributed datastore may be thought of as a server, and clients in the connection pool as clients. Both the client and server need to be aware of how to communicate, and so they both must be configured with the appropriate secrets.
For managing keystores and truststores, the aptly named keytool can be used.
To configure the distributed datastore to use a keystore/truststore, the following would need to be added to the dist-datastore.cfg.xml file:
For more details, see:
The distributed datastore shares key-space with all of the enabled cluster nodes.
Key-space is determined by the maximum value of the distributed datastore's hashing algorithm. At current the only supported hashing algorithm is MD5.
Addressing a key is done by first normalizing all of the participating cluster nodes. This is done by an ascending sort. After the participating nodes have had their order normalized, the key-space is sliced up by dividing the maximum possible number of addresses by the total number of participating nodes. The given key is then reduced to its numeric representation and a cluster node is looked up by performing a modulus such that (<key-value> % <number-of-cluster-members>).
By default, the internal Repose client implementation for the distributed datastore will obscure key-space by storing only the MD5 hash value of a given key and not the key's actual value. This is important to note since external gets against the distributed datastore must be aware of this functionality.
The MD5 hash is represented as a 128bit UUID.
Example Key Addressing
String Key: object-key MD5 Hash: cd26615a30a3cdce02e3a834fed5711a UUID: cecda330-5a61-26cd-1a71-d5fe34a8e302
If an external application makes a request for data stored by Repose components, it must first hash the key using MD5 before sending the request such that...
Obscuring key-space is not a function of the distributed datastore service. This functionality is only present in the internally consumed java cache client. If an external application puts an object into the distributed datastore, the object will be stored under the value of the key given.
The repose distributed datastore component is a filter component that hosts a simple RESTful API that can be contacted to perform remote object store operations. These operations are defined below.
GET /powerapi/dist-datastore/objects/<object-key> HTTP/1.1
Gets a stored object from the datastore by its key.
PUT /powerapi/dist-datastore/objects/<object-key> HTTP/1.1
Puts an object into the datastore by its key.
DELETE /powerapi/dist-datastore/objects/<object-key> HTTP/1.1
Deletes an object from the datastore by its key.
PATCH /powerapi/dist-datastore/objects/<object-key> HTTP/1.1
Patches an object in the datastore by its key. If the object does not exist create a new one. Return the modified/new object. The object must be Patchable.
In the event that a node with in a datastore cluster falls off line or is unable to respond to requests, it is removed from the node's cluster membership for a period of time. During this time, the online node will then re-address its key-space in order to continue operation. After certain periods of rest, the node may attempt to introduce the damaged cluster member into its cluster membership. A damaged cluster member must go through several validation passes where the member is introduced back into the addressing algorithm before it can be considered online. In order to keep healthy nodes from attempting to route requests to the damaged node, a participating node may tell it's destination that the destination may not route the request and must store, delete or get the value locally.
The Repose node will open sockets each time it has to communicate with other Repose nodes to share information. During times of load this can affect performance and data integrity as when one node cannot communicate with another it will mark that node damaged and store/create information locally. One way this can happen is if the user running repose hits their open file limit. Luckily this can be mitigated by increasing the open file limit for the user running Repose.
Connecting to a Repose instance running the Distributed Datastore via JMX gives access to cache statistics implemented by the Ehcache Metrics Library.
MBean: net.sf.ehcache -> Cache > PAPI_LOCALrepose-cluster-node:repose-node-id:LocalDatastoreCacheManager
The authentication components will use the local datastore to cache token, role, and group information.
Rate limiting can be configured to use any of the three to store rate limiting information for users. See the documentation for Rate Limiting to find out how to specify which datastore to use. By default, rate limiting will use the distributed datastore if available. If no distributed datastores are available, then rate limiting will use the local datastore.
In the following example, the Repose instance with the node id of node1 will launch a Distributed Datastore service which will listen on port 9999. The Repose instance with the node id of node2 will launch it's own Distributed Datastore service which will listen on port 7777. With allow-all set to false, a host that is not in the cluster cannot communicate with the Distributed Datastore.