Dan The Engineer

Keycloak on Distributed SQL (CockroachDB) - Part 2/2

Adam Spindler -

In part one of this post, we configured a single Keycloak node to store it's data in a resilient and available CockroachDB. In this post we cover how to add more nodes to the cluster and ensure that the keycloak nodes are able to register themselves on CockroachDB which, then allows Keycloak to share cache information.

Target Architecture

For this implementation, the target architecture will be as shown below:


In this post we will be adding regions two (us-east-1) and three (eu-west-2)

Adding a node for Resilience and Availability

From the previous post, we now have a single keycloak node connecting to a load balanced distributed CockroachDB. However, in this configuration it is not resilient, to resolve this we will now add nodes in two more regions, each one connecting to local CockroachDB nodes which are part of the existing Cockroach cluster.

As AWS does not allow multicast between EC2 nodes, we will need to use a different method to allow Keycloak to discover other nodes and share data between them. We will do this first with TCPPING and then use the Cockroach DB.

Stop Keycloak and copy the conf/cache-ispn.xml file to a new file which for this example we called conf/cache-ispn-tcpping.xml. We then edit the conf/cache-ispn-tcpping.xml file to include the following configuration(note, this will require different IP addresses in other installations):

<!-- custom stack goes into the jgroups element -->
<jgroups>
	<stack name="tcpping" extends="tcp">
	<TCP bind_port="7800"/>
	<TCPPING
initial_hosts="10.15.2.27[7800],10.15.0.25[7800],10.15.1.27[7800]"
	port_range="0"
stack.combine="REPLACE"
stack.position="MPING"
	/>
	</stack>
</jgroups>
<cache-container name="keycloak">
	<transport lock-timeout="60000" stack="tcpping"/>

Note that the <transport> tag now has an additional value stack=”tcpping”.

We now edit the cache settings in the conf/keycloak.conf file by including the following lines:

# cache
cache=ispn
cache-config-file=cache-ispn-tcpping.xml

Again we start the Keycloak server and we can see from the output that the JGroups channel is being started with the “tcpping” stack:

We now need to start adding more nodes to confirm they are communicating and sharing cache information.

In another region (AWS us-east-1 in this example) create a new Keycloak node:

  • Install openssl
  • Install java
  • Install keycloak
  • Set UDP buffers
  • Generate the SSL certificates for the new host
  • Copy the conf directory from the 1st node to the second node
  • Edit the conf/keycloak.conf file to change the db-url value to the local load balancer for the database.
  • Edit the conf/keycloak.conf file to change the hostname value to the new node.

We can then start the second node:

From the startup messages we can see that both nodes, keycloak01 and keycloak02, are referenced in the logs.

We can now log in once on host keycloak01 and again on host keycloak02 as one of the users we created earlier. When logged in as Admin on either node we can see both of the sessions:

Making the Keycloak nodes self-register and add a new region

We now have the 2 keycloak nodes using the Infinispan cluster, but the config has the IPs of the hosts hard-coded in the configuration file. This is not particularly flexible in the long run for operations so the final step is to use the CockroachDB for the keycloak nodes to register themselves to avoid this hard-coding.

Stop all Keycloak servers and on the first node with postgres installed, copy the conf/cache-ispn-tpcping.xml to conf/cache-ispn-jdbcping.xml and edit to replace the jgroups stack and <transport> tag as shown:.

<!-- custom stack goes into the jgroups element -->
<jgroups>
	<stack name="jdbcping" extends="tcp">
	<JDBC_PING connection_driver="org.postgresql.Driver"
connection_username="keycloak"
connection_password="keycloak"
connection_url="jdbc:postgresql://localhost:5432/jgroups"
	initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, bind_addr VARCHAR(200) NOT NULL, created timestamp NOT NULL, cluster_name varchar(200) NOT NULL, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
	insert_single_sql="INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?,'${jboss.bind.address:127.0.0.1}',NOW(), ?, ?);"
	delete_single_sql="DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?;"
select_all_pingdata_sql="SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?;"
info_writer_sleep_time="500"
remove_all_data_on_view_change="true"
stack.combine="REPLACE"
	stack.position="MPING" />
	</stack>
</jgroups>
<cache-container name="keycloak">
	<transport lock-timeout="60000" stack="jdbcping"/>

We now change the conf/keycloak.conf to point to the new configuration

cache-config-file=cache-ispn-jdbcping.xml

Build and start the Keycloak server on node 1:

Again, from the startup messages we can see that Keycloak is now using the “jdbcping” stack. Now to check data is in the jgroups db in postgres

Again, we are using the postgres database to create the data structure as this cannot be done directly in CockroachDB. Once we have the data in postgres we need to stop keycloak and extract the data from the postgres database and import it into CockroachDB.

pg_dump --host=localhost -U keycloak -d jgroups > jgroups.dmp

We created the CockroachDB user and database for jgroups earlier in the process, so we can proceed direxctly to populate database using our sql client:

cockroach sql --host=ajs-db01:26257 --certs-dir=certs -u keycloak -d jgroups -f ../new_kc_dump/jgroups.dmp

We confirm the database has been created in the Cockroach cluster:

Now we have the data in the CockroachDB cluster we can re-point to cockroachdb by editing the conf/cache-ispn-jcdbping.xml file and replacing the postgres connection string with the appropriate CockroachDB connection string:

connection_url="jdbc:postgresql://ajs-db-demo-int-1b85bcdea0b73e74.elb.us-west-1.amazonaws.com:26257/jgroups?sslmode=require&amp;sslrootcert=${kc.home.dir}/crdb_certs/ca.crt"

Note the database url string has now has the database value changed to “jgroups” which is different to the “keycloak” database in the conf/keycloak.conf file.

Build and start keycloak sever again and confirm the data is being put into Cockroach jgroups database:

We now copy the conf/cache-ispn-jdbcping.xml to the second node and edit it to include the load balancer for the local CockroachDB servers in the connection_url value.

connection_url="jdbc:postgresql://ajs-lb-int-688c72c587b28502.elb.us-east-1.amazonaws.com:26257/keycloak?sslmode=require&amp;sslrootcert=${kc.home.dir}/crdb_certs/ca.crt"

Now change the conf/keycloak.conf on node 2 to point to the new cache-ispn-jdbcping.xml file.

cache-config-file=cache-ispn-jdbcping.xml

Again we build and start keycloak on the second node to check both nodes are registered in cockroach.

Now we have both nodes using CockroachDB to register themselves there is no need for hard-coding of host IPs in configuration files, so we can add new nodes simply.

  • Install java
  • Set buffers
  • Install keycloak
  • Create open ssl certificates
  • Copy cockroachdb ca.crt across
  • Copy config files across
  • Change hostname and load balancer in conf/keycloak.conf
  • Change load balancer in conf/cache-ispn-jdbcping.xml
  • Build keycloak
  • Start keycloak

Confirm all nodes registered in cockroach and users created in one node are visible to the other nodes.

We now have a 3-node keycloak cluster in 3 regions, connecting to a CockroachDB distributed cluster also across 3 regions, providing resilience against failures in any AWS region.