How to Use AWS S3 Event Notifications to Trigger Lambda, SNS, and SQS

Amazon S3, a scalable and reliable object storage service, is a cornerstone of many cloud-based architectures. But did you know that it also enables seamless automation of workflows through Amazon S3 Event Notifications?

Whether you are triggering image processing upon file uploads or keeping track of file deletions, S3 Event Notifications are a powerful tool to integrate with other AWS services and streamline operations.

This blog attempts to cover how they work, their configuration, and best practices to get the most out of them.

---

What Are S3 Event Notifications?

S3 Event Notifications allow you to automatically respond to specific events in your S3 bucket. These events include:

• Object creation: E.g., PutObject, PostObject, or multipart uploads.
• Object removal: E.g., DeleteObject.
• Object restoration: E.g., restoring from Glacier.

When such events occur, S3 can send notifications to a destination of your choice, enabling you to trigger downstream processes.

---

How S3 Event Notifications Work

The workflow is straightforward:

1. An event occurs in an S3 bucket (e.g., a file upload).
2. S3 generates an event message.
3. The event message is sent to a specified destination.
4. The destination (e.g., an SQS queue or a Lambda function) processes the event.

This mechanism enables real-time or near-real-time processing of S3 bucket activity. Here’s a simplified flow:

S3 Bucket -> Event Notification -> Destination -> Action Triggered

---

Supported Destinations

S3 Event Notifications can send event messages to the following destinations:

Amazon Simple Queue Service (SQS):

• Ideal for decoupling applications.
• Allows for message persistence and processing retries.

Amazon Simple Notification Service (SNS):

• Suitable for broadcasting messages to multiple subscribers.

AWS Lambda:

• Perfect for real-time processing (e.g., resizing images or analyzing text).

---

Configuration Methods

You can configure S3 Event Notifications using:

1. AWS Management Console

• Navigate to the bucket.
• Select the Properties tab.
• Scroll to Event notifications and create a new configuration.

2. AWS CLI

Example command to set up a notification:

aws s3api put-bucket-notification-configuration \
    --bucket my-bucket \
    --notification-configuration file://notification.json

3. Infrastructure as Code (IaC)

Example using CloudFormation:

Resources:
  S3Bucket:
    Type: "AWS::S3::Bucket"
    Properties:
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: "s3:ObjectCreated:Put"
            Function: "arn:aws:lambda:region:account-id:function:my-function"

---

Key Use Cases

1. Real-Time Data Processing: Use Lambda to process uploaded images (e.g., resizing or tagging).
2. Application Decoupling: Route notifications to an SQS queue for asynchronous processing.
3. Event-Driven Architectures: Notify multiple systems of changes using SNS.
4. Monitoring and Auditing: Log changes in bucket contents for compliance.

---

Best Practices

• Use Least Privilege: Ensure IAM roles grant only the necessary permissions.
• Test Configurations: Simulate events to validate configurations.
• Handle Failures: Implement retries for failed notifications.
• Monitor Metrics: Use CloudWatch to track and troubleshoot event delivery.

---

Limitations and Considerations

• Single Configuration Per Event Type: You can only define one destination per event type per bucket.
• Event Delivery Consistency: Event delivery is eventually consistent.
• Costs: Destinations like SQS, SNS, or Lambda may incur additional costs.

---

Sample Implementation

Here’s a quick tutorial to set up an S3 Event Notification that triggers a Lambda function:

Step 1: Lambda Function Code

Create a Lambda function and add the following sample code:

Example Python script (using Boto3):

import json
def lambda_handler(event, context):
    print("Event Received:", json.dumps(event))
    for record in event['Records']:
        bucket_name = record['s3']['bucket']['name']
        object_key = record['s3']['object']['key']
        print(f"File uploaded: {bucket_name}/{object_key}")

Step 2: Configure S3 Bucket Event Notification

• I have already created a bucket named test-bucket-with-event-notification.
• Go to Properties of S3 bucket and navigate to Event notifications.

Once the Event type is selected, choose the destination. Here I am selecting the Lambda function created in the first step.

Step 3: Test the Setup

Once you have created the resources, it’s time to test the setup.

For this, upload a file to your S3 bucket and verify that the Lambda function processes the event.

Once the file is uploaded to the S3 bucket, it will trigger the lambda function, and print the received event in CloudWatch logs.

---

Key Points to Consider:

• The above setup is basic. 🙂
• The lambda function is not inside a VPC.
• S3 bucket encryption is not enabled.
• When encryption is enabled, the respective permissions must be provided. (Example, IAM Role of lambda)🤔

References:

---

Conclusion

Amazon S3 Event Notifications are a powerful way to automate workflows and build event-driven architectures. Understanding their configuration and integrating them with other AWS services can unlock immense productivity and scalability in your cloud solutions. Whether you’re building serverless applications or decoupling complex systems, S3 Event Notifications have you covered.

Ready to take the next step? Try configuring your first S3 Event Notification today and integrate it with AWS Lambda or SQS. For more information, check out the AWS S3 Documentation. Also, explore my other blogs for more AWS insights!

In the next blog, we will configure multiple destinations like AWS SNS, AWS SQS, etc. 😊