Efficiently send log data from AWS S3 to Axiom via Lambda function
This page explains how to set up an AWS Lambda function to send logs from an S3 bucket to Axiom. The Lambda function triggers when a new log file is uploaded to an S3 bucket, processes the log data, and sends it to Axiom.
To determine the best method to send data from different AWS services, see Send data from AWS to Axiom.
Before creating the Lambda function, package the requests module so it can be used in the function:
Create a Lambda function with Python runtime and upload the packaged zip file containing the requests module and your function code below:
In the environment variables section of the Lambda function configuration, add the following environment variables:
DATASET_NAME
is the name of the Axiom dataset where you want to send data.API_TOKEN
is the Axiom API token you have generated. For added security, store the API token in an environment variableIn the Amazon S3 console, select the bucket where your log files are stored. Go to the properties tab, find the event notifications section, and create an event notification. Select All object create events as the event type and choose the Lambda function you created earlier as the destination. For more information, see the AWS documentation.
Ensure the log file you upload to the S3 bucket is in the correct format, such as JSON or newline-delimited JSON (NDJSON) or CSV. Here’s an example:
After uploading a test log file to your S3 bucket, the Lambda function automatically processes the log data and sends it to Axiom. In Axiom, go to the Datasets tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.
Efficiently send log data from AWS S3 to Axiom via Lambda function
This page explains how to set up an AWS Lambda function to send logs from an S3 bucket to Axiom. The Lambda function triggers when a new log file is uploaded to an S3 bucket, processes the log data, and sends it to Axiom.
To determine the best method to send data from different AWS services, see Send data from AWS to Axiom.
Before creating the Lambda function, package the requests module so it can be used in the function:
Create a Lambda function with Python runtime and upload the packaged zip file containing the requests module and your function code below:
In the environment variables section of the Lambda function configuration, add the following environment variables:
DATASET_NAME
is the name of the Axiom dataset where you want to send data.API_TOKEN
is the Axiom API token you have generated. For added security, store the API token in an environment variableIn the Amazon S3 console, select the bucket where your log files are stored. Go to the properties tab, find the event notifications section, and create an event notification. Select All object create events as the event type and choose the Lambda function you created earlier as the destination. For more information, see the AWS documentation.
Ensure the log file you upload to the S3 bucket is in the correct format, such as JSON or newline-delimited JSON (NDJSON) or CSV. Here’s an example:
After uploading a test log file to your S3 bucket, the Lambda function automatically processes the log data and sends it to Axiom. In Axiom, go to the Datasets tab and select the dataset you specified in the Lambda function. You now see your logs from your IoT devices in Axiom.