SoftHSM Setup (Encrypted Key Storage)¶

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Instead of storing the private key as a raw PEM file on disk, you can use SoftHSM2 to store the key encrypted in a PKCS#11 token database. This provides stronger key protection without requiring hardware.

With SoftHSM:

The private key is stored encrypted in a token database, never as a raw PEM file.
Key access requires a PIN.
Keys are marked non-extractable even with the PIN, the key can be used for signing but cannot be exported.
Token database persists across container restarts via the existing bind mount.

Important

Before following the steps in this topic, complete Steps 1 through 4 on the Integrate Agent with Secrets Manager Using IAM Roles Anywhere.

SoftHSM support requires Auth Service version 5.14.0.2469 or later.

Comparison: Filesystem-Based vs SoftHSM ¶

Aspect	Filesystem-Based	SoftHSM
Private key storage	Raw PEM file on host	Encrypted in SoftHSM token DB
What attacker needs	Read the file	Token DB + PIN + PKCS#11 tooling
Key extractable	Always (it’s a file)	No (`CKA_EXTRACTABLE=false`)
`credential_process` references	File paths (`--certificate /path/to/client.crt`)	PKCS#11 URIs (`--certificate "pkcs11:token=..."`)

Step 1: Set Up the Alation Agent Machine ¶

Use SSH to connect to the Alation Agent machine.
Check if the .aws directory exists under /etc/hydra/auth/. If not, create it:
```
sudo mkdir -p /etc/hydra/auth/.aws
```

Prepare the private key. SoftHSM requires the private key in PKCS#8 format. Check the current format:

head -1 client_private.key

First line	Format	Action
`-----BEGIN PRIVATE KEY-----`	PKCS#8	Ready to use, skip conversion
`-----BEGIN RSA PRIVATE KEY-----`	PKCS#1	Needs conversion
`-----BEGIN EC PRIVATE KEY-----`	EC key	Needs conversion
`-----BEGIN ENCRYPTED PRIVATE KEY-----`	Encrypted PKCS#8	Decrypt first

If conversion is needed:

openssl pkcs8 -topk8 -nocrypt -in client_private.key -out client_private_pkcs8.key

# Verify:
head -1 client_private_pkcs8.key
# Should show: -----BEGIN PRIVATE KEY-----

Use client_private_pkcs8.key (or the original if already PKCS#8) for the import step below.

Exec into the running Auth Service container:
```
sudo docker exec -it auth bash
```

Create the token directory on the persistent mount:

mkdir -p /opt/alation/site/config/authserver/softhsm/tokens
chmod 700 /opt/alation/site/config/authserver/softhsm/tokens

Initialize a new PKCS#11 token:
```
softhsm2-util --init-token --free \
    --label "roles-anywhere" \
    --so-pin <SO_PIN> \
    --pin <USER_PIN>
```
Replace <SO_PIN> (Security Officer PIN) and <USER_PIN> with your chosen PINs.
Note
- The SO PIN is for administrative operations (reinitialize token, reset User PIN). Only needed during setup and recovery.
- The User PIN is used at runtime by aws_signing_helper to access the key for signing. This PIN will appear in the credentials file.

Verify the token was created:

softhsm2-util --show-slots
# Should show a slot with token "roles-anywhere"

Import the private key. Pipe from stdin so the key never touches the container filesystem.

Use the content of client_private_pkcs8.key (or client_private.key if already PKCS#8):

cat <<'EOF' | softhsm2-util --import /dev/stdin \
    --token "roles-anywhere" \
    --pin <USER_PIN> \
    --label "ra-key" \
    --id 01 --force
-----BEGIN PRIVATE KEY-----
<paste full content of client_private_pkcs8.key here>
-----END PRIVATE KEY-----
EOF

Import the client certificate. Write it to a temp file first (SoftHSM does not support certificate import from stdin).

Use the content of client.crt (the certificate signed by the CA, not ca.crt):
```
cat > /tmp/ra-cert.pem <<'EOF'
-----BEGIN CERTIFICATE-----
<paste full content of client.crt here>
-----END CERTIFICATE-----
EOF
```
Import using pkcs11-tool. The --id 01 must match the key’s ID so SoftHSM pairs them together:
```
pkcs11-tool \
    --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --write-object /tmp/ra-cert.pem \
    --type cert --label "ra-cert" --id 01
```
Clean up the temp file:
```
rm /tmp/ra-cert.pem
```
Important

The --id 01 must match for both the key and the certificate so SoftHSM pairs them together.

Verify the objects are in the token:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --list-objects
# Should show both Private Key and Certificate objects

Exit the container:
```
exit
```

Create the AWS credentials file at /etc/hydra/auth/.aws/credentials with PKCS#11 URIs instead of file paths:

sudo vi /etc/hydra/auth/.aws/credentials

Add the following content. The entire credential_process value must be on a single line (no line breaks):

[<profile-name>]
credential_process = /usr/local/bin/aws_signing_helper credential-process --certificate "pkcs11:token=roles-anywhere;object=ra-cert;id=%01?pin-value=<USER_PIN>" --private-key "pkcs11:token=roles-anywhere;object=ra-key;id=%01?pin-value=<USER_PIN>" --pkcs11-lib /usr/lib/softhsm/libsofthsm2.so --trust-anchor-arn <trust-anchor-arn> --profile-arn <profile-arn> --role-arn <role-arn>

Replace the following placeholders:

Placeholder	Description	Example
`<profile-name>`	The profile name configured in the Alation UI authentication profile	`iam_roles_anywhere_profile`
`<USER_PIN>`	The User PIN chosen during token initialization	(customer-chosen)
`<trust-anchor-arn>`	The Trust Anchor ARN from Step 3	`arn:aws:rolesanywhere:us-east-1:248135293344:trust-anchor/bdd70097-...`
`<profile-arn>`	The Roles Anywhere Profile ARN from Step 4	`arn:aws:rolesanywhere:us-east-1:248135293344:profile/71b7f5ac-...`
`<role-arn>`	The IAM Role ARN from Step 2	`arn:aws:iam::248135293344:role/AlationSecretsManagerAccess`

Important

The credential_process value must be on a single line with no line breaks. Multi-line values will cause a parsing error.
The aws_signing_helper binary path is /usr/local/bin/aws_signing_helper (pre-installed in the container), not a host path.

Set file permissions:

sudo chmod 640 /etc/hydra/auth/.aws/credentials
sudo chmod 750 /etc/hydra/auth/.aws
sudo chown root:alationdocker /etc/hydra/auth/.aws
sudo chown root:alationdocker /etc/hydra/auth/.aws/credentials

Step 2: Create an Authentication Profile ¶

This step is performed in Alation

To create an authentication profile for the Secrets Manager integration:

Log in to your Alation instance as a Server Admin.
Click the Admin Settings gear icon on top right to open the Admin Settings page.
Click Authentication to open the Authentication tab. Locate the section Authentication Configuration Methods for External Systems.

Next to See configurations for, click the drop-down menu and select the relevant Alation Agent.
Click Add Configuration, and then select AWS Secrets Manager as the method type. The Authentication Configuration Method page will open in a new browser tab.
In Config Name, enter a unique name for the configuration. Save it for future reference when configuring the data source.
Under Region, select the appropriate AWS region for the Secrets Manager service (the region under which your secrets are stored).
Under Authentication Type, select IAM Roles Anywhere.
Click Save. Alation attempts to create a connection, and if the connection is successful, the configuration is saved.

Now, you can use your integration with an OCF connector. See next: Configure Authentication with AWS Secrets Manager for a Data Source.

Troubleshooting ¶

Error: `CKR_TOKEN_NOT_PRESENT`¶

Cause: SoftHSM cannot find the token database. This typically happens when:

The token directory does not exist or was not created.
The softhsm2.conf file points to a wrong path.

Fix:

Exec into the container and verify the token exists:

sudo docker exec -it auth bash
softhsm2-util --show-slots

If no token is listed, reinitialize the token (see Step 1).

Error: `CKR_PIN_INCORRECT`¶

Cause: The User PIN in the credentials file does not match the PIN set during token initialization.

Fix: Verify the pin-value in the PKCS#11 URIs in the credentials file matches the User PIN you set when initializing the token.

Error: `PKCS11 token not found`¶

Cause: The token label in the PKCS#11 URI does not match the label used during initialization, or the token was not initialized.

Fix:

Exec into the container and list the available tokens:

sudo docker exec -it auth bash
softhsm2-util --show-slots

Verify the token label matches what is in the credentials file (token=roles-anywhere).

Error: `Certificate not found in PKCS#11 token`¶

Cause: The certificate was not imported, or the object label or ID does not match the PKCS#11 URI.

Fix:

Exec into the container and list objects:

sudo docker exec -it auth bash
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --list-objects

Verify the certificate object exists with label ra-cert and ID 01.
If missing, re-import the certificate (see Step 1).

Error: `AccessDeniedException` when accessing Secrets Manager¶

Cause: The IAM role does not have sufficient permissions for Secrets Manager, or the Roles Anywhere session policy is too restrictive.

Fix:

Verify the IAM role has the required Secrets Manager permissions.
If using session policies on the Roles Anywhere profile, ensure they allow the required Secrets Manager actions.
Verify the secret’s resource policy (if any) allows access from the assumed role.

What happens when the Auth Service container is upgraded?¶

The SoftHSM packages and aws_signing_helper binary are baked into the Docker image. On upgrade (new image pulled and container recreated):

Component	Stored where	Survives upgrade?
`softhsm`, `opensc` packages	Container image	Reinstalled fresh in new image
`aws_signing_helper` binary	Container image (`/usr/local/bin/`)	Reinstalled fresh in new image
SoftHSM token database	Persistent mount (`/etc/hydra/auth/softhsm/tokens/`)	Yes
AWS credentials file	Persistent mount (`/etc/hydra/auth/.aws/credentials`)	Yes

Nothing is lost on upgrade. The token database and credentials file are on the bind mount, not in the container filesystem. The SoftHSM2 token database format is stable across minor versions.

What if the PIN in the credentials file is compromised?¶

Even with the PIN and access to the container, the private key cannot be extracted as a raw file. By default, SoftHSM sets CKA_EXTRACTABLE=false on imported keys.

With the PIN, an attacker can:

List objects in the token (labels, IDs, key types).
Use the key for signing operations (which is what aws_signing_helper does).
Read the certificate (certificates are not marked sensitive).

With the PIN, an attacker cannot:

Extract the raw private key bytes (blocked by CKA_EXTRACTABLE=false).
Copy the key to another token or file.

So even if the PIN leaks, the attacker can only use the key from inside that container — they cannot steal it and use it elsewhere. This is the fundamental difference from a raw PEM file, which once read can be copied anywhere.

What does the SoftHSM token database look like on disk?¶

SoftHSM creates a UUID-named subdirectory per token:

softhsm/tokens/
+-- a3b2c1d4-5678-9abc-def0-123456789abc/
    +-- token.object        # token metadata
    +-- 00000001.object     # private key (encrypted)
    +-- 00000002.object     # certificate (encrypted)

Running cat on any .object file shows binary data — the content is encrypted with a key derived from the User PIN. It is not PEM, not DER, and not parseable by openssl.

How do I rotate the client certificate and key?¶

There are two scenarios: rotating only the certificate (keeping the same key) or replacing both the certificate and the private key.

Rotate the certificate only¶

Use this when the existing private key is still valid and you only need a new certificate (for example, the certificate is expiring).

Generate a new CSR using the existing key and sign it with your CA.
Exec into the Auth Service container:
```
sudo docker exec -it auth bash
```

Delete the old certificate from the token:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --delete-object --type cert --label "ra-cert"

Write the new certificate to a temp file:

cat > /tmp/ra-cert.pem <<'EOF'
-----BEGIN CERTIFICATE-----
<paste full content of new client.crt here>
-----END CERTIFICATE-----
EOF

Import the new certificate with the same label and ID:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --write-object /tmp/ra-cert.pem \
    --type cert --label "ra-cert" --id 01

Clean up and exit:
```
rm /tmp/ra-cert.pem
exit
```
Update the AWS Trust Anchor only if the CA certificate changed. No changes are needed in the credentials file.

Replace both the certificate and the private key¶

Use this when you need to replace both the private key and certificate (for example, the key was compromised or you are rotating to a new key pair).

Prepare the new private key in PKCS#8 format and the new certificate signed by your CA.
Exec into the Auth Service container:
```
sudo docker exec -it auth bash
```

Delete the old private key:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --delete-object --type privkey --label "ra-key"

Delete the old public key:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --delete-object --type pubkey --label "ra-key"

Delete the old certificate:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --delete-object --type cert --label "ra-cert"

Verify all old objects are removed:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --list-objects

Import the new private key:

cat <<'EOF' | softhsm2-util --import /dev/stdin \
    --token "roles-anywhere" \
    --pin <USER_PIN> \
    --label "ra-key" \
    --id 01
-----BEGIN PRIVATE KEY-----
<paste full content of new PKCS#8 private key here>
-----END PRIVATE KEY-----
EOF

Write the new certificate to a temp file:

cat > /tmp/ra-cert.pem <<'EOF'
-----BEGIN CERTIFICATE-----
<paste full content of new client.crt here>
-----END CERTIFICATE-----
EOF

Import the new certificate:

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --write-object /tmp/ra-cert.pem \
    --type cert --label "ra-cert" --id 01

Clean up the temp file:
```
rm /tmp/ra-cert.pem
```

Verify all three new objects are present (Private Key, Public Key, and Certificate):

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --login --pin <USER_PIN> \
    --list-objects

Exit the container:
```
exit
```
Update the AWS Trust Anchor only if the CA certificate changed. No changes are needed in the credentials file.

What if the Auth Service is not using SoftHSM?¶

If SoftHSM is never used, everything remains dormant. The softhsm and opensc packages are installed but no processes are started, no ports are opened, and no CPU or memory is used. The aws_signing_helper binary sits at /usr/local/bin/ and is never executed unless referenced in a credential_process. Existing authentication flows (static keys, IAM instance profile, IAM user credentials) are completely unaffected.

Why SoftHSM instead of a hardware TPM?¶

SoftHSM requires no hardware dependency — it works on any Linux machine without kernel modules or device passthrough. No changes to the Alation Agent configuration are needed. It uses the same PKCS#11 interface as hardware HSMs, providing an easy migration path to TPM later if needed (same aws_signing_helper flags, just change the PKCS#11 library path).

SoftHSM Setup (Encrypted Key Storage)¶

Comparison: Filesystem-Based vs SoftHSM¶

Step 1: Set Up the Alation Agent Machine¶

Step 2: Create an Authentication Profile¶

Troubleshooting¶

Error: CKR_TOKEN_NOT_PRESENT¶

Error: CKR_PIN_INCORRECT¶

Error: PKCS11 token not found¶

Error: Certificate not found in PKCS#11 token¶

Error: AccessDeniedException when accessing Secrets Manager¶

What happens when the Auth Service container is upgraded?¶

What if the PIN in the credentials file is compromised?¶

What does the SoftHSM token database look like on disk?¶

How do I rotate the client certificate and key?¶

Rotate the certificate only¶

Replace both the certificate and the private key¶

What if the Auth Service is not using SoftHSM?¶

Why SoftHSM instead of a hardware TPM?¶

Alation User Documentation PDFs

Comparison: Filesystem-Based vs SoftHSM ¶

Step 1: Set Up the Alation Agent Machine ¶

Step 2: Create an Authentication Profile ¶

Troubleshooting ¶

Error: `CKR_TOKEN_NOT_PRESENT`¶

Error: `CKR_PIN_INCORRECT`¶

Error: `PKCS11 token not found`¶

Error: `Certificate not found in PKCS#11 token`¶

Error: `AccessDeniedException` when accessing Secrets Manager¶