AWS S3 Event Fan-Out to Lambda, SQS, and SNS

A Practical Guide to Delivering One S3 Event to Many AWS Services

Introduction

Amazon S3 Event Notifications allow you to configure event-driven workflows by triggering various AWS services whenever specific actions occur in an S3 bucket. That said, a common challenge arises when you need to configure multiple event notifications for the same event type but with different destinations.

In this blog, I will share my experience handling this issue, discuss possible scenarios where it can be encountered, and outline how to work around it using AWS features.

---

Personal Experience

I encountered this challenge when setting up an S3 Event Notification for the s3:ObjectCreated:Put event. Initially, an event notification was already configured with an SNS topic as the destination. Then, I realized I also needed to add a new notification targeting SQS.

Surprisingly, AWS does not allow multiple event notifications for the same event type on a single bucket.

This limitation forced me to rethink my approach and look for alternative solutions.

---

Understanding the Limitation

AWS S3 enforces a restriction where you cannot create multiple event notifications for the same event type. If an event notification already exists for s3:ObjectCreated:Put with one destination, you cannot create another event notification for the same event with a different destination.

Possible Scenarios Where This Can Be an Issue

1. Multiple Consumers of the Same Event: Different services might need to process the same event differently, e.g.,

• An SNS topic to notify users.
• An SQS queue for a background processing service.

2. Event Processing with Different Latency Requirements: One service might need real-time processing, while another might require batch processing.

3. Cross-Account Event Handling: One event destination might be within the same AWS account, while another could be in a different account.

4. Monitoring and Logging Requirements: Sending the same event to a monitoring tool (like a Lambda function) and another service for further data transformation.

---

Workaround: Using SNS as a Fan-Out Mechanism

A practical solution to this limitation is to use an SNS topic as the event notification destination and then configure multiple subscribers to the SNS topic.

Steps to Implement the Workaround

1. Create an SNS Topic
2. Subscribe Multiple Destinations to the SNS Topic

• SQS Queue
• Lambda Function
• Another SNS Topic (if required for cross-account handling)

3. Configure S3 Event Notification to Publish to the SNS Topic

Example Implementation

Step 1: Create an SNS Topic

aws sns create-topic --name S3EventFanoutTopic

Step 2 :

A] Create an SQS Queue and Subscribe it to SNS.

aws sqs create-queue --queue-name MyS3EventQueue
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:S3EventFanoutTopic \
    --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:123456789012:MyS3EventQueue

Add following SQS Queue Policy to receive events from SNS:

{
      "Sid": "Allow-SNS-SendMessage",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:MyS3EventQueue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:S3EventFanoutTopic"
        }
      }
    }

B] Create a Lambda function and Subscribe it to SNS

1. Create an IAM role with Lambda execution permissions and SNS subscribe access.

2. Create a deployment package (zip file) for your Lambda function

Sample Node.js Lambda function code (index.js):

exports.handler = async (event) => {
    console.log("Received SNS event:", JSON.stringify(event, null, 2));
    for (const record of event.Records) {
        const snsMessage = record.Sns.Message;
        console.log("Message from SNS:", snsMessage);
    }
    return {
        statusCode: 200,
        body: JSON.stringify('Event processed successfully'),
    };
};

3. Create a zip package.

zip function.zip index.js

4. Create the Lambda Function.

aws lambda create-function \
  --function-name MyS3EventProcessor \
  --runtime nodejs18.x \
  --role arn:aws:iam::123456789012:role/LambdaExecutionRole \
  --handler index.handler \
  --zip-file fileb://function.zip

5. Add permission for SNS to invoke Lambda.

aws lambda add-permission \
  --function-name MyS3EventProcessor \
  --statement-id sns-invoke \
  --action "lambda:InvokeFunction" \
  --principal sns.amazonaws.com \
  --source-arn arn:aws:sns:us-east-1:123456789012:S3EventFanoutTopic

6. Subscribe the Lambda function to the SNS topic.

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:S3EventFanoutTopic \
  --protocol lambda \
  --notification-endpoint arn:aws:lambda:us-east-1:123456789012:function:MyS3EventProcessor

Step 3: Create an S3 Event Notification

Go to the S3 bucket console and configure the event notification with the SNS topic (S3EventFanoutTopic) as the destination.

Step 4: Verify Event Propagation

Upload a file to the S3 bucket and check if the SQS queue receives the event notification.

aws s3 cp myfile.txt s3://my-bucket/

Check messages in the SQS queue:

aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/MyS3EventQueue

Also check CloudWatch log of Lambda Function:

Conclusion

AWS S3’s event notification limitation can be a blocker when you need multiple destinations for the same event. However, by using SNS as a fan-out mechanism, you can efficiently distribute event notifications to multiple AWS services without hitting this restriction.

This approach ensures flexibility while keeping the architecture scalable and event-driven.

Additionally, it’s important to ensure proper permissions are set up for each subscriber:

• SNS → SQS: SQS must allow sns.amazonaws.com to send messages.
• SNS → Lambda: Lambda must have a resource-based policy allowing SNS to invoke it.

Always verify that subscriptions are in the Confirmed state, check CloudWatch logs for Lambda executions, and consider enabling SNS delivery status logging if you’re troubleshooting issues.

This solution has helped me overcome a real-world challenge with minimal complexity and high reliability. I hope this post helps you architect your S3 workflows more effectively!