bella.network Blog

Internal ACME server

Issue certificates with your own certificate authority and the automation benefits of Let's Encrypt
Most websites use an encrypted connection to secure transferred data
Most websites use an encrypted connection to secure transferred data (Photo by Azamat Esenaliev from Pexels)

Let’s Encrypt was instrumental in driving the uptake of the encrypted web. Not only by providing free certificates, but also by providing a simple way to get validated and trusted certificates automatically. In addition, renewal of these certificates can be automated without any human interaction.
For these automations, the Automatic Certificate Management Environment - short ACME - protocol was created. Tools and programs like certbot, acme.sh, Caddy and Traefik use it to issue certificates.

For internal use, it may be preferred to use an internal certificate authority (CA) because of several reasons (rate limit, internal domain/tld, non-validateable externally, extremely short-lived, …). In such cases, we can use the ACME protocol to issue certificates automatically with an internal CA. This way we can avoid externally issued and manually issued certificates which allows us to simplify and automate tasks.

The software Step-CA by Smallstep offers us all functinality needed to operate an ACME server for automated certificate issuance.

For clients to validate certificates successfully and display them as trusted, we need to distribute the root certificate. Within enterprise environments, this is mostly done using Group Policies in Active Directory and MDM on mobile phones.
When using Active Directory, a infrastructure for certificates is already set up using Active Directory Certificate Services. A instance of AD CS issued the root certificate and another instance issued a intermediate certificate used to sign end/leaf certificates.

To operate the ACME server, we need to create a new intermediate certificate which will be signed by the root certificate authority. This way no new root certificate needs to be distributed and the distribution of the newly created intermediate certificate is only required optionally.
To not cause any validation problems, I strongly advertise to also distribute intermediate certificates to clients.

Requirements and things to prepare:

  • Existing Windows Active Directory Certificate Services instance / alternatively any other CA (not covered by this article)
    • Permission to sign a intermediate certificate (SubCA template)
    • Copy of root CA public key
  • Linux VM for step-ca ACME Server

Installation

For this setup you should create a new VM whose only task is to issue certificates by providing an ACME server. I am using Ubuntu 22.04 with 2 vCPU, 512 MB RAM and 8 GB disk size.
We need to install the step-ca package first, which can be found on GitHub smallstep/certificates > Releases.

1
2

wget -O step-ca_0.26.1_amd64.deb https://dl.smallstep.com/gh-release/certificates/gh-release-header/v0.26.1/step-ca_0.26.1_amd64.deb
dpkg -i step-ca_0.26.1_amd64.deb

We will initialisize Step-CA into /var/lib/step-ca/ and prepare the intermediate certificate to be signed by the root CA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

mkdir /var/lib/step-ca/
cd /var/lib/step-ca/

# Initialisize Root-CA and configuration files
step ca init
# Delete the root CA
rm .step/secrets/root_ca_key
# Copy your root CA public key to step-ca
mv <path-to-your-public-root-ca.crt> .step/certs/root_ca.crt
# Create a intermediate CA and generate a CSR for it
# save and secure the password, it is very important and decrypts the private key
step certificate create "Intermediate CA Name" intermediate.csr intermediate_ca_key --csr

Copy the intermediate.csr file from your Linux system to your Windows CA. We need to sign the certificate with the SubCA template. This allows the issued certificate to issue leaf certificates, allowing basic intermediate CA operations (pathLenConstraint not set to 0).

1

certreq -submit -attrib "CertificateTemplate:SubCA" intermediate.csr intermediate.crt

Copy back the newly created intermediate.crt file to /var/lib/step-ca/intermediate.crt and proceed with the following commands:

1
2

mv intermediate.crt .step/certs/intermediate_ca.crt
mv intermediate_ca_key .step/secrets/intermediate_ca_key

Configuration

Using the following command, you can add an ACME provisioner to the step-ca instance. A provisioner with the name acme is added which acts as ACME server.

1

step ca provisioner add acme --type ACME

The file /var/lib/step-ca/.step/config/ca.json contains the basic configuration of our intermediate CA.
The service will be bond to the priviledged port 443 (HTTPS) and all used hostnames will be defined within the configuration file. In addition we override the x509 issuer template and make some addional adjustments, allowing us to modify every issued certificate.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

"address": ":443",
"dnsNames": [
	"ca.bella.pm",
	"10.20.23.222"
],
[...]
"authority": {
	"provisioners": [
		{
			"type": "ACME",
			"name": "acme",
			"claims": {
				"maxTLSCertDuration": "4320h0m0s",
				"defaultTLSCertDuration": "2160h0m0s"
			},
			"options": {
				"x509": {
					"templateFile": "/var/lib/step-ca/.step/templates/issuer.tpl"
				}
			}
		}, {
			[...]
		}
	]
},
"tls": {
	"cipherSuites": [
		"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
		"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
	],
	"minVersion": 1.2,
	"maxVersion": 1.3,
	"renegotiation": false
}

In the file /var/lib/step-ca/.step/templates/issuer.tpl the default certificate template will be extended to add some additional fields.
Especially the issuingCertificateURL field allows us to provide a destination URL, where the intermediate certificate can be obtained by a client. This is needed if the webserver wasn’t configured to serve the end certificate with the intermediate certificate and only the root certificate is placed in the trust store. This allows the client to complete the chain as far as possible to build a path to a stored root certificate.
Please note that the issuingCertificateURL should use a HTTP URL to avoid problems validating the given domain itself (chicken-and-egg problem).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
	"subject": {{ toJson .Subject }},
	"sans": {{ toJson .SANs }},
{{- if typeIs "*rsa.PublicKey" .Insecure.CR.PublicKey }}
	"keyUsage": ["keyEncipherment", "digitalSignature"],
{{- else }}
	"keyUsage": ["digitalSignature"],
{{- end }}
	"extKeyUsage": ["serverAuth", "clientAuth"],

	"issuingCertificateURL": ["http://ca.bella.pm/intermediate_ca.crt"]
}

If the certificate chain is not complete, the client is now able to download the intermediate certificate from the given URL.
Access logs show on my server that the URL is requested by some Go applications, Microsoft CryptoAPI and Apple trustd:

1
2
3

10.20.15.78 - - [09/May/2024:00:02:21 +0000] "GET /intermediate_ca.crt HTTP/1.1" 200 572 "-" "Go-http-client/1.1"
10.20.15.79 - - [11/May/2024:08:25:59 +0000] "GET /intermediate_ca.crt HTTP/1.1" 200 572 "-" "Microsoft-CryptoAPI/10.0"
10.20.15.80 - - [20/May/2024:06:45:48 +0000] "GET /intermediate_ca.crt HTTP/1.1" 200 572 "-" "com.apple.trustd/3.0"

To automatically start step-ca on system start, the file /etc/systemd/system/step-ca.service is added with the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

[Unit]
Description=A private certificate authority (X.509 & SSH) & ACME server
Documentation=https://github.com/smallstep/certificates
After=network.target
Before=nss-lookup.target
Wants=nss-lookup.target

[Service]
NonBlocking=true
WorkingDirectory=/var/lib/step-ca
ExecStart=/usr/bin/step-ca .step/config/ca.json --password-file=/etc/stepca-password
ProtectHome=yes
ProtectControlGroups=yes
ProtectKernelModules=yes
User=_step-ca
Group=_step-ca
CacheDirectory=step-ca
LogsDirectory=step-ca
RuntimeDirectory=step-ca
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

The file /etc/stepca-password contains the password used do decrypt our previously generated intermediate certificate.
Using AmbientCapabilities=CAP_NET_BIND_SERVICE we allow the program to bind to port 443 even when running as the non-root user _step-ca.

The service is finally enabled and started using the following command:

1

systemctl enable --now step-ca.service

Issuing a certificate

As ACME client I prefer to use acme.sh from https://github.com/acmesh-official/acme.sh. Using the following command, I am trying to issue a certificate for internal.bella.pm against the previously created ACME server.
The following command uses /var/lespace as http root folder for the ACME validation process and reloads the nginx configuration after the certificate was issued.

1
2
3
4
5

# ECDSA with 256bit
/root/.acme.sh/acme.sh --issue -d internal.bella.pm --server https://ca.bella.pm/acme/acme/directory -w /var/lespace --keylength ec-256 --ecc --reloadcmd "nginx -s reload"

# RSA with 3072bit
/root/.acme.sh/acme.sh --issue -d internal.bella.pm --server https://ca.bella.pm/acme/acme/directory -w /var/lespace --keylength 3072 --reloadcmd "nginx -s reload"

Manually issuing a certificate using JWK

Besides ACME we are able to manually issue certificates using the integrated JWK backend. This allows us to locally issue certificates for domains which can’t be validated automatically.
First, we can create a certificate with csr if we do not have any available yet:

1

step-cli certificate create blog-dev.bella.network dev.csr dev.key --csr --no-password --insecure

Next we issue a certificate locally using our stored account:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

root@CA.bella.pm ~ $ step ca sign request.csr dev.crt --ca-config /var/lib/step-ca/.step/ --not-after 8760h
✔ Provisioner: thomas@bella.network (JWK) [kid: ......]
✔ Please enter the password to decrypt the provisioner key:
✔ CA: https://ca.bella.pm
✔ Certificate: dev.crt
root@CA.bella.pm ~ $ cat dev.crt
-----BEGIN CERTIFICATE-----
MIICfzCCAiSgAwIBAgIRAPW16X4Um+R6uxsM+9XFWVswCgYIKoZIzj0EAwIwKDEm
MCQGA1UEAxMdYmVsbGEubmV0d29yayBJbnRlcm1lZGlhdGUgQ0EwHhcNMjExMjA1
MTY1NDQzWhcNMjIxMjA1MTY1NTQzWjAhMR8wHQYDVQQDExZibG9nLWRldi5iZWxs
YS5uZXR3b3JrMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEmo8DzDf3PwY23nNa
Lnoo9Z+ytv64rph4okkaqV4yd1nn/3V4ekg/FgX40MeLzJjqZ8WmaE1Axqhn2JYY
DY4sx6OCATQwggEwMA4GA1UdDwEB/wQEAwIHgDAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwHQYDVR0OBBYEFLfNyhbe3N+j415Tf7rxppABCnyiMB8GA1Ud
IwQYMBaAFPeB9bSUs9F22sIiC0TsJVaYxIr0MEIGCCsGAQUFBwEBBDYwNDAyBggr
BgEFBQcwAoYmaHR0cDovL2NhLmJlbGxhLnBtL2ludGVybWVkaWF0ZV9jYS5jcnQw
IQYDVR0RBBowGIIWYmxvZy1kZXYuYmVsbGEubmV0d29yazBYBgwrBgEEAYKkZMYo
QAEESDBGAgEBBBR0aG9tYXNAYmVsbGEubmV0d29yawQrNER5OHBON1hXeEZsdWRW
NGtpc2Nnak8xVFphb2hjOUxYMUtCRTI4TEYtWTAKBggqhkjOPQQDAgNJADBGAiEA
/D5qgtmh6rv3/GqRMf8Mn4GA6n1GKl1VQd/p9tsOULgCIQC7wdAEn34M9vzc53ax
kvG2wbq8HcIKExah3nciQklxLQ==
-----END CERTIFICATE-----

Nice to know and what’s missing?

All in all, step-ca provides all functionality needed to operate an internal ACME server. There are some features missing which can be extended manually or are currently in the pipeline to be included in a future release.

One of this features is Support for CRL which allows certificate revocation in the future. In addition, after a merge is completed there shouldn’t be much work needed for an OCSP integration making the CA respond to revocation checking requests by servers and browsers. - Support for CRL is now available since v0.23.1. See addition below on how to configure it.

If you are planning to use step-ca on a bigger scale, consider switching from the currently used database Badger to MySQL/MariaDB or PostgreSQL. This allows you to scale to a bigger environment and to access the database by third-party tools (e.g. to audit all issued certificates). A guide on how to switch to a different database is available at Configuring step-ca > Databases.

If 3 months validity are no suitable for you, you can change the validity to up to 13 months. Going over 13 months may cause problems with some clients due to the requirements to official certificates not exceeding 397 days. See Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates at 6.3.2 Certificate operational periods and key pair usage periods.
Keep also in mind that most ACME clients like certbot and acme.sh will renew only if the certificate is valid less than 30 days. Issuing certificates with a validity of 30 days may require some config changes of your ACME clients.

Automated renewal of certificates saves energy and makes all services more efficient, allowing us to use our time for much more important and interesting tasks.
Happy certificate issuing!

New since step-ca v0.23.1 - 2023-01-21

Since step-ca v0.23, support for CRL was added which allows us to use step-ca within a Windows environment with strict revocation checking enabled.
Browsers usually don’t check for CRLs, but Windows itself with its built-in certificate store does. This primarily affects PowerShell, .NET and other applications.\

To enable CRL support, we need to add the following part within the ca.json configuration file:

1
2
3
4
5
6
7

[...]
"crl": {
	"enabled" : true,
	"generateOnRevoke": true,
	"idpURL": "http://ca.bella.pm/acme/crl"
},
[...]

step-ca serves the generated CRL at https://ca.bella.pm/1.0/crl by default which has one big disadvantage:
The CRL is served using HTTPS from step-ca itself, which also generates a certificate which references the CRL. This prevents the CRL from being served to clients successfully failing the validation.
To solve this problem, we need to add a reverse proxy in front of step-ca which serves the CRL using HTTP.
We can use the following configuration for nginx:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

server {
	listen 80;
	listen [::]:80;
	server_name ca.bella.pm;

	location /acme/crl {
		rewrite /acme/crl /1.0/crl break;
		proxy_pass https://ca.bella.pm/;
		proxy_redirect     off;
		proxy_set_header   Host $host;
	}
}

This allows us to use the CRL URL http://ca.bella.pm/acme/crl which is configured within the ca.json configuration file.
Windows is now able to successfully check the revocation status of certificates issued by step-ca.

A look into the nginx access log shows that the CRL is requested by Windows CryptoAPI:

1

10.20.15.77 - - [15/Jan/2023:20:55:26 +0000] "GET /acme/crl HTTP/1.1" 200 167 "-" "Microsoft-CryptoAPI/10.0"