How To Cache Secrets Using AWS Lambda Extensions

What is the AWS Lambda extension?

• Internal extensions → Run as part of the runtime process, in-process with your code. Internal extensions enable use cases such as automatically instrumenting code.
• External extensions → Allow you to run separate processes from the runtime but still within the same execution environment as the Lambda function. External extensions can start before the runtime process and can continue after the runtime shuts down. These extensions run as companion processes to Lambda functions.

Usecase

• Prefetch secrets from AWS Secret manager and save it into an in-memory cache.
• The extension will read “config.yaml” file in the lambda function to get the list of secrets that needs to be prefetched and cached. This way, the Lambda function controls what needs to be cached, and the extension code can be generic and reused across multiple Lambdas for a similar use case.
• Cache validity is specified through an environment variable in Lambda, instead of extension defining it.

Architecture

• Hosts a local HTTP server, which will be invoked by lambda function to retrieve the secret instead of directly going to AWS Secret manager
• The extension using config.yaml file inside the Lambda to get the list of secrets that needs to get cached in memory
• Secrets cache is refreshed based on the value (in minutes) set in the environment variable “CACHE_TIMEOUT”. If no value is specified its defaulted to 10

Note: This extension requires the Node.js 12 runtime to be present in the Lambda execution environment of your function.

Initializing extension and reading secrets from the cache

Code

Deploy

Pre-requisites

• Zip utility needs to be installed in the local systemAWS CLI needs to be installed in the local system, for more information click
• Create new secret in AWS Secret ManagerCreate a new secret using the following command

Note: This command assumes you’ve placed your secret, such as this example JSON text structure {“username”:”anika”,”password”:”aDM4N3*!8TT”}, in a file named mycreds.json.

aws secretsmanager create-secret --name secret_now --secret-string file://mycreds.json

Lambda function

• After successfully creation of Lambda, attach “SecretsManagerReadWrite” policy to the IAM role associated with the Lambda function
• Replace the code inside “index.js” with the following

Note: Sample of the below code can be found in “example-function/index.js” under the code repo

Note: The code invokes the local HTTP server hosted inside lambda extension to read the secret’s value instead of directly going to AWS Secret’s manager

• Create a new file named “config.yaml” under the root directory with the following contents:

Note: Sample of the below yaml file can be found in “example-function/config.yaml” under the code repo. “secret_now” is the name of the secret we have created in the previous step, and we want to be cached by the extension. If you want more secrets to be cached you can keep adding them here

SecretManagers:

1. secrets:

• secret_now

• Click on “Deploy” button to deploy the latest changes
• Create a new environment variable “CACHE_TIMEOUT” and set the value in minutes based on which the cache will be refreshed

Note: If the environment value not found, then cache gets refreshed every 10 minutes

• Increase the memory of Lambda to 1200 MB & Timeout to 30 seconds

Deploy extension

• Run the following command to deploy the extension and associate the layer to the Lambda function

> chmod +x deploy.sh
> ./deploy.sh

Invoke Lambda function

aws lambda invoke \
 --function-name "Secrets-Extension-Lambda-Test" \
 --payload '{"payload": "hello"}' /tmp/invoke-result \
 --cli-binary-format raw-in-base64-out \
 --log-type Tail

View logs

Resources