EventBridge¶

Introduction¶

Amazon EventBridge is a serverless event bus service (Formerly CloudWatch Event), which is a router to send messages to Lambda functions, SQS, SNS, or some SaaS providers, using event routing rules. This is not a queueing, topic, or streaming product.

EventBridge is a service for building scalable event-driven applications, enabling highly agile software development via fully managed and secure integrations, cutting costs and reducing time to production.

The following figure illustrates the typical source of events and how EventBridge can process them and sends JSON documents to different sinks. Filtering logic may be applied.

In each AWS account there is an EventBridge default event bus already created, but we can add our own, via CLI, CloudFormation, CDK,... An event bus is a pipeline that receives events. To receive events from SaaS partners, we need to create a partner event source connected to a partner event bus.

EventBridge supports 28+ AWS service as targets. We can also integrate to Partners SaaS services like Datadog, Zendeck, git, Spotify, another EventBridge event bus...

Target can be an API destination which is made up of an HTTP endpoint, HTTP method (such as GET, POST, PUT), and a connection, which defines the type of authorization and credentials used to call the endpoint.

EventBridge event bus as target can be use for cross region replication, for events aggregation in a same region. This could be mono or cross AWS accounts. Routing events between event buses in different accounts allows to support workloads grow in size and complexity (the receiver account must grant permission on the target event bus to allow the sending account to send events).

Value Propositions¶

• No infrastructure to manage.
• Fully managed event backbone to pass events from sources to sinks using routing logic.
• A lot of native, easy, integration with AWS Services.
• Combined with DynamoDB streams, API Gateway, SQS, SNS and Lambda, it is relatively easy to develop quick applications with no server and complex environment to configure.
• No need to write custom integration code, as could be done in classical ESB, but still support filtering and transformations.
• Enabled archive to support replayability.
• End point to end-point integration, with pipe to do advanced filtering and enrichments.
• Support scheduled event creation.
• EventBridge supports three methods for authorization: basic auth, OAuth, and API key

Read features presentation.

Concepts¶

• Event represents a change that happened in the past. Event, in EventBridge has the following structure with metadata and payload in the detail element. The combination of the "source" and "detail-type" fields serves to identify the fields and values found in the "detail" field.

{
  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "detail-type": "EC2 Instance State-change Notification",
  "source": "aws.ec2",
  "account": "111111111111",
  "time": "2017-12-22T18:43:48Z",
  "region": "us-west-1",
  "resources": [
    "arn:aws:ec2:us-west-1:123456789012:instance/ i-1234567890abcdef0"
  ],
  "detail": {
    "instance-id": " i-1234567890abcdef0",
    "state": "terminated"
  }
}

• Event buses - An event bus receives events. In term of design, it should include all the events related to the same business entity. It can be seen as a queue or a topic, or a channel.

• Event Pattern are JSON objects with the same structure of the event that it is matching. The field names must also appear in the event with the same nesting structure. Any additional fields in the event that aren't included in the event pattern are ignored. Event patterns are defined in the context of event rule.

  {
    "source": [ "aws.ec2" ],
    "detail": {
      "state": [ "terminated" ]
    }
  }
  

• Routing rule matches incoming events and routes them to targets for processing. When we create a rule, we associate it with a specific event bus, and the rule is matched only to events received by that event bus. Source field is a meta-data attribute that is set by the producer application to uniquely identify itself, and can be used to filter and route to different consumer. It is useful to avoid looping on events coming in the event bus from different sources.

  • Rule matching operates in constant time. The rule engine is in this event-rules repository.
  • There are limits on the number of rules per bus (2000).
  • We can only use Scheduled rules on the default event bus.
  • EventBridge rules help to filter the events before they are sent to the targets, making sure that each target receives only the events they are interested in. A routing rule can customize the JSON sent to the target, by passing only certain parts of the message or by overwriting message attributes with constant values.
  • For each target of a rule, we can define a retry policy that defines how many times and how long EventBridge will try to send the event again. SQS DLQ can be used for undelivered events.

• Target is a resource, EventBridge sends events when the rule pattern matches. We can create multiple targets for one rule.

• EventBridge Pipes: is used for point to point connection between producer and consumer with filtering, enrichment and transformation capabilities. It may keep message order from source to one destination. The sources can be DynamoDB stream, Kinesis stream, MQ broker, MSK stream, Kafka topic, SQS queue.

• Global endpoints is used to facilitate failing over a secondary region, in case of primary disruption. It is linked to CloudWatch Alarm and Route 51 health checks.

• With Schema Discovery enabled, we can automatically get a schema definition from the event sent to the bus, saved in a schema registry. Schema Discovery also detects changes to event schema and automatically generates new schema version. From the schema in the registry, we can get code binding for Java, TypeScript, and Python, to jump start any consumer apps.

• With the archive and replay feature, EventBridge can record any events processed by any type of event bus. Replay stores these recorded events in archives. We can choose to record all events, or filter events to be archived by using the same event pattern matching logic used in rules.

Pattern examples¶

• A or statement in a rule:

  {
    "detail": {
      "location": ["eu-west", "eu-east"]
    }
  }
  

  Or between attribute

  {
    "detail": {
      "$or": [
        {"location": ["eu-west"]},
        {"nb_passengers": [ {"numeric": [ ">", 10 ]}]}
      ]
    }
  }
  

• A prefix based statement:

Security¶

EventBridge uses AWS Identity and Access Management, to manage both identity-based and resource-based policies. To make API calls against the resources we own, EventBridge needs the appropriate permissions. We can attach resource-based policies to the target resource with permission for the EventBridge service to call the required API.

To invoke an API Gateway endpoint using an EventBridge rule, you could add an allow statement on the "execute-api:Invoke" API action.

Identity-based policies are attached to an IAM user, group, or role that is assumed by the EventBridge service when the rule is invoked. See fine-grained access control documentation.. Amazon EventBridge has two AWS managed policies you can attach to users or groups: AmazonEventBridgeFullAccess or AmazonEventBridgeReadOnlyAccess.

We can authorize other account to PutEvents, PutRule or PutTargets API calls to an existing Event Bus via resource-based policy. It can be done for cross-region too.

See Example policy: Send events to a custom bus in a different account.

It provides encryption at rest and in transit by default.

CLI - SAM¶

Develop can use AWS CLI events to create an event bus:

aws events create-event-bus --name account_events

SAM can be used with the following CLI options to select an existing schema: Infrastructure event management > ... > EventBridge App from scratch (100+ Event Schemas) >... Schema registry > EventSchema...

The same can be done with AWS CLI:

aws schemas put-code-binding --registry-name discovered-schemas --schema-name com.aws.orders@OrderNotification --language Python36

When code bindings are published, we can download them using get-code-binding-source command.

aws schemas get-code-binding-source --registry-name discovered-schemas --schema-name com.aws.orders@OrderNotificatio

Deployment patterns¶

A a managed service, it is running under AWS VPC, scale horizontally and is highly available. Custom buses are defined per region. We can enable event replication to send from primary event bus to a secondary region. Events are processed in the secondary Region asynchronously.

EventBridge provides at-least-once event delivery to targets, and is designed to retry sending events that are not delivered. Therefore we need to implement idempotency. The default retry policy is 24 hours up to 185 times. It uses exponential back-off and jitter. If a retry policy is unsuccessful in delivering the event, the event is dropped. To avoid losing events after they failed to be delivered to a target, configure a DLQ.

AWS services send event to EventBridge using best effort delivery (in some rare cases an event might not be delivered) or with durable delivery (at least once).

It is possible for application within a VPC to produce events to event bus, using Interface VPC endpoints, and to be an API destination (ECS task) also using Interface VPC endpoints.

See EventBridge product documentation and access an AWS service using an interface VPC endpoint.

EventBridge with API Destination as target can also use private HTTP endpoint within a VPC. (See this blog) and the git repository aws-sample eventbridge-events-to-vpc.

When using different regions for a disaster recovery architecture, RTO includes time period for triggering CloudWatch alarms and updating statuses for Route 53 health checks. For RPO, the time includes events that are not replicated to the secondary Region and are stuck in the primary Region until the service or Region recovers. (RTO and RPO can be max of 420s).

Solution patterns¶

Below is a standard end to end solution using common patterns:

• EventBridge can be used to Fan-out to different sinks or delivery targets but with limited way, see next bullet.
• With high transaction per second throughput, developers need to consider the number of delivery targets. For 1 target and 1 rule, it is common to be over 500k tps. Growing the number of delivery targets adds latency.
• So when we need to fan-out more, we can add SNS as delivery target after EventBridge. EB is then used for filtering events.
• We can define a central event hub to aggregate all the events from AWS Organizations into a single AWS account or region. Apps in different accounts can be authorized to send events to this central hub via resource-based policy.
• As events in archive aren't necessarily replayed in the same order that they were added to the archive, it is important to add a timestamp on the event message.
• Add a sequence_id or idempotency_id in the produced event for idempotency supports by consumers.

Solution development approach with EventBridge¶

• Assess who own the rules, and try to limit the number of rules, and focus on trying to get one rule to one delivery target.
• Define the schema of the event with JSONSchema, including the definition of the AWSEvent envelop.
• Upload the schema to the schema registry
• Create EventBridge bus per business entity (DDD aggregate). For example an OrderEventBus will support multiple event types: OrderCreated, OrderUpdated...
• Create routing rule with the event patterns that match certain event attribute and route to a target.
• Generate the Java, Nodejs or Python bindings

• EventBridge is defined at the region level. So producer and consumer apps need to get region information in environment variables.

• The AWS CLI support EventBridge via aws events.

• Cloud Formation EventBridge template

• EventBridge CDK Construct library in Python

• Classical steps to develop a Lambda function from an Event definition in the schema registry:

• Create a sam project

• By getting the code binding from the event model
• In the Lambda function, use boto3 to access the backend and return response back to EventBridge event bus.
• Implement business logic in the Lambda handler function.

How tos and best practices¶

ServerlessLand has a lot of CloudFormation examples to deploy different EDA patterns.

CarRideEventsLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 1
      LogGroupName: "/aws/events/acr/carrides"
CarRideEventsToLogRule:
  Type: AWS::Events::Rule
  Properties:
    Description: Catch all rule for development purposes- go to cloudwatch
    EventBusName: !Ref EventBusName
    EventPattern:
      source:
        - acr.com.car-ride-simulator
    Name: CarRideEventsDevRule
    State: ENABLED
    Targets:
      - Id: Ide60ca354-ebb9-47d2-ad81-6a52867eb9e6
        Arn: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CarRideEventsLogGroup}"

• Create a rule for the event bus that will act as a "catch-all" for every event passed to the bus, irrespective of source.

    "PublishOrderProcessedEvent": {
        "Type": "Task",
        "Resource": "arn:aws:states:::events:putEvents",
        "Parameters": {
        "Entries": [
            {
            "Detail": {
                "OrderId": "new_id",
                "OrderDetails.$": "$.detail"
            },
            "DetailType": "Order Processed",
            "EventBusName": "Orders",
            "Source": "com.aws.orders"
            }
        ]
    },

aws events put-events --entries file://OrderNotification_v1.json

See CDK doc.

FAQs¶

• Product FAQs

Deeper dive¶

• Product documentation Tutorials
• Building event-driven architectures on AWS
• Event Driven Architecture with Amazon API Gateway, Amazon EventBridge and AWS Lambda
• Amazon EventBridge CDK Construct Library Event
• Webinars and videos
• Integration with SaaS Partners
• Working with events and the Amazon EventBridge schema registry
• Amazon EventBridge Scheduler User Guide