No More Expiry Surprises: Avoid Disruptions with Automated AWS Certificate Inspection

At VirtueCloud , we’ve observed that managing certificate rotations is a common challenge many DevOps teams face. During our consultations with clients, it became clear that automating certificate expiration alerts is a crucial yet often overlooked aspect of day-to-day operations.

To address this, we propose an efficient way to streamline the process using AWS services, helping clients avoid potential disruptions and downtime. Although AWS Certificate Manager (ACM) with Private Certificate Authority (PCA) and KMS is great for signing certificates, a more straightforward approach using AWS Secrets Manager is sometimes a better fit.

Here’s a breakdown of how we did it:

Prerequisites

- AWS account

- Basic knowledge of AWS services

- AWS CLI configured (optional)

The Problem

We decided to store our certificates in AWS Secrets Manager. The challenge that followed was how to track the expiration dates and rotate certificates before they expired.

Manual tracking of expiration dates is:

- Error-prone

- Easy to forget

- Time-consuming

- Stressful (nobody wants an expired certificate in production!)

The Solution

We set up a simple, automated system using three key AWS services:

- EventBridge for scheduling

- Lambda for checking expiration dates

- SNS for sending alerts

How It Works

- Lambda periodically inspects secrets stored in Secrets Manager.

- EventBridge triggers the inspection based on a schedule.

- SNS sends alerts to the team when a certificate is nearing expiration.

The Code

Here’s a basic Lambda function to inspect certificates and send alerts:

import json

import boto3

from datetime import datetime

import OpenSSL.crypto

def lambda_handler(event, context):

    secretsmanager = boto3.client('secretsmanager')

    sns = boto3.client('sns')

    try:

        # Get list of secrets

        secrets = secretsmanager.list_secrets()

        for secret in secrets['SecretList']:

            # Get the secret value

            secret_value = secretsmanager.get_secret_value(SecretId=secret['Name'])

            secret_dict = json.loads(secret_value['SecretString'])

            # Look for certificate entries (ending in .cert)

            for key, value in secret_dict.items():

                if key.endswith('.cert'):

                    # Parse the certificate

                    cert_data = value.replace('\\n', '\n')

                    cert = OpenSSL.crypto.load_certificate(

                        OpenSSL.crypto.FILETYPE_PEM, 

                        cert_data

                    )

                    # Get expiration date

                    expiry = datetime.strptime(

                        cert.get_notAfter().decode('ascii'),

                        '%Y%m%d%H%M%SZ'

                    )

                    # Calculate days until expiration

                    days_until_expiry = (expiry - datetime.utcnow()).days

                    # Alert if expiring within 60 days

                    if days_until_expiry <= 60:

                        sns.publish(

                            TopicArn='arn:aws:sns:REGION:ACCOUNT_ID:secret-rotation-alerts',

                            Subject=f'Certificate Expiration Alert - {key}',

                            Message=f'''

                            Certificate {key} in secret {secret["Name"]} is expiring soon!

                            Expiration Date: {expiry.date()}

                            Days until expiration: {days_until_expiry}

                            Please rotate this certificate soon.

                            '''

                        )

        return {

            'statusCode': 200,

            'body': 'Certificate Inspection completed successfully'

        }

    except Exception as e:

        print(f"Error: {str(e)}")

        return {

            'statusCode': 500,

            'body': f'Error checking certificates: {str(e)}'

        }
        

Setting It Up

Step 1: Create an IAM Role for Lambda

• Go to the IAM Console.
• Click Roles > Create role.
• Select AWS Service and choose Lambda.
• Add the following permissions:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": [

                "secretsmanager:ListSecrets",

                "secretsmanager:DescribeSecret"

            ],

            "Resource": "*"

        },

        {

            "Effect": "Allow",

            "Action": [

                "sns:Publish"

            ],

            "Resource": "arn:aws:sns:REGION:ACCOUNT_ID:secret-rotation-alerts"

        },

        {

            "Effect": "Allow",

            "Action": [

                "logs:CreateLogGroup",

                "logs:CreateLogStream",

                "logs:PutLogEvents"

            ],

            "Resource": "arn:aws:logs:*:*:*"

        }

    ]

}    

• Name the role secret-rotation-inspector-role.

Step 2: Create an SNS Topic

aws sns create-topic --name secret-rotation-alerts

aws sns subscribe --topic-arn <your-topic-arn> --protocol email --notification-endpoint your@email.com

Then confirm the subscription via email.

Step 3: Create Lambda Function

• Go to the Lambda Console.
• Click Create function and choose Author from scratch.
• Name the function secret-rotation-inspector.
• Use Python 3.9 runtime and assign the IAM role secret-rotation-inspector-role.
• Paste the Lambda code and deploy.

Step 4: Create EventBridge Rule

1. Go to the EventBridge Console and click Rules > Create rule.

Expand Image

2. Configure the rule to run quarterly:

- Name: inspect-secret-quarterly

- Description: “Inspect secrets every quarter for rotation”

- Rule type: Schedule

- Cron expression: 0 9 1 1,4,7,10 ? *

Expand Image

3. Set the Lambda function as the target for this rule

Expand Image

Why Every 3 Months?

Most certificates are valid for one year, so quarterly inspections offer ample time for renewal and avoid unnecessary Lambda executions. This frequency helps catch any missed rotations.

Step 5: Test the Setup

1. Go to the Lambda Console.

2. Select your function and click the Test tab.

3. Create a test event and execute the function.

4. Check CloudWatch logs for execution results and your email for any alerts.

Customization Options

You can easily modify the Lambda function to inspect specific secrets using tags or adjust the alert threshold from 60 days to another time frame. For more flexible alerts, consider integrating with Slack or Teams via SNS.

Cost Considerations

This setup is very cost-effective, with minimal Lambda executions, only a few SNS notifications, and low CloudWatch logging costs. In most cases, the entire solution costs under a dollar per month.

Solution Comparison

While this blog post demonstrates a simple solution using Secrets Manager, AWS offers other approaches for certificate management, such as ACM Private Certificate Authority (PCA) with KMS. However, ACM PCA comes with different cost implications:

• Higher costs for standard CA: Approximately $350/month per CA, which can add up for organizations with multiple certificates or environments.
• Cheaper short-lived certificate mode: Available at $50/month, but it’s designed specifically for certificates with short validity periods (typically hours or days), making it less suitable for long-term certificate management.

Given these factors, the Secrets Manager solution is ideal when you need:

• Simple certificate storage and rotation
• Direct key access for signing operations
• A cost-effective solution, usually costing just a few dollars per month
• Quick implementation with minimal setup

In contrast, ACM PCA is better suited for enterprise environments that require:

• Comprehensive PKI management
• More frequent certificate rotations
• Environments with higher security and compliance needs

Ultimately, the right solution depends on your specific requirements and the scale of your operations. Select the method that aligns with your goals, whether it’s Secrets Manager for simplicity and cost-efficiency or a more advanced setup like ACM PCA for larger, more complex environments.

Expanding the Solution

If you're interested in automating the entire rotation process, AWS Secrets Manager offers a built-in rotation feature for certain types of secrets. For custom secrets, you could extend this setup by creating an additional Lambda function that:

• Generates new secrets
• Updates the necessary applications
• Validates the new secret is working properly
• Archives the old secrets

However, implementing full secret automation is a more advanced topic — one we’ll cover in a future post!

Key Takeaways

Automating secret rotation tracking reduces human error and saves significant time. By leveraging AWS services like Lambda, EventBridge, and SNS, you can create a robust and low-cost solution to ensure certificate rotation happens on time without manual intervention.

We hope this guide helps others streamline their secret management process! Feel free to reach out to us at connect@virtuecloud.io you have any questions or suggestions for enhancing this solution.