CockroachDB - Fun with Encryption

Problem we are trying to Solve

Migrating from a single-tenant architecture to a multi-tenant architecture where each tenant can use “Bring Your Own Key” (BYOK) to securely encrypt their data at rest in the shared platform.

The Result

The success criteria for the specified use case

Tables can be stored in a specified store
Multiple stores on a node, each with independent encryption

Documents

Cockroach start - https://www.cockroachlabs.com/docs/v20.2/cockroach-start.html
Encryption - https://www.cockroachlabs.com/docs/v20.2/encryption.html
Enterprise features: https://www.cockroachlabs.com/docs/v20.2/enterprise-licensing.html

Process Overview:

Create Database and apply enterprise licence
Rolling restart with encrypted stores
Assign tables to an encrypted store
Verify data and stores is encrypted

Step by Step Process

Create DB
Create the cockroach DB, initialise and apply the enterprise licence as Documentation
Rolling restart including the definition of the encrypted store and the associated keys
Working through each node in the cluster in turn, perform a graceful shutdown and restart the node with the following options for each encrypted store required:

--store=path=<path to store encrypted data>,attrs=<label>

--enterprise-encryption=path=<path to store encrypted data>,key=<path to key file>,old-key=plain

In this example, the database is being started with 3 stores in the /home/cockroach/DB directory:

Store	Label	Encryption key
.../repl1	None	None
.../crypt_repl1	encrypt1	.../keys/aes-128-rep1.key
.../crypt_repl2	encrypt2	.../keys/aes-128-rep2.key

For the purpose of this example, the keys were generated via the cli using:

cockroach gen encryption-key -s 128 /home/cockroach/DB/keys/<keyname>.key

Once the node has been successfully added to the cluster, this process is then repeated on each of the remaining nodes in turn until all nodes have been restarted with the encrypted stores.

Create tables and assign to an encrypted store
Once all nodes have restarted, the tables which will hold data can be assigned to the respective stores:

alter table <schema>.<tablename> configure zone using constraints’[+<label>]’;

For example the following command would assign the table ‘crypt1’ to the store that was labelled as “encrypt1”, which for this example is the store held at “/home/cockroach/DB/crypt_rep1” and encrypted with the “home/cockroach/DB/keys/aes-128-rep1.key” key.

alter table enc_demo.crypt1 configure zone using constraints=’[+encrypt1]’;

Populate data in tables
Data can now be added to the tables as any other table in the database.
Verify data is encrypted at rest
Checking the contents of the .sst files within the encrypted store now shows all data being encrypted:

The encryption status of the various stores can been seen through the web console at (http://:/#/reports/stores/local) as shown below:

Be aware that cockroach requires an old-key value for the --encryption-enterprise option to allow for key rotation. This case used ‘plain’ as this was the first time encryption was applied. When rotating keys this needs to be set to the old key so Cockroach can decrypt the data and re-encrypt with the new key. See the documentation on “Encryption” in the Documents section.