When to use a Lambda function, and when not?

Since its release in 2014, AWS Lambda has seen an incredible growth. Today, billions of functions are invoked every day and they are now the way to go for many different workloads and operations tasks. AWS Lambda supports tens of event sources: from Amazon SQS and Amazon EventBridge for event-driven architectures to AWS Config for resource configuration, passing by Amazon API Gateway to handle HTTP requests, and many others.

But just because you can use Lambda functions almost everywhere doesn’t mean you should. To improve performance, reduce latency and optimise costs, it is sometimes recommended to use direct integration between services. This blog post will go through different use cases and analyse wether a Lambda function is relevant or not. In the latter case, it will describe how to replace it and what are the advantages.

Use case #1: High-criticality API

In this use case, users are calling a REST API to place orders. These orders are sent to a SQS Queue, so that a backend can handle them asynchronously. The main point here is the criticality of the request as we don’t want to loose any order.

Do we need a Lambda function? No

Explanation:

• The objective here is to avoid loosing orders, so we want to store them as quickly as possible.
• Having a Lambda function between the API Gateway and the SQS Queue introduces some code, which increases the risk of failure and therefore the risk of loosing orders.
• This pattern is called ‘Storage-First Pattern’. It consists in the ability for API Gateway to directly integrate with SQS to reliably store the incoming request before any process occurs. You can get more details here and

Implementation:

• If you use AWS CDK, there is a solution construct on the shelves for this pattern: aws-apigateway-sqs.
• If you use SAM or other frameworks, you will find different implementations on serverlessland.com.
• Under the hood, we leverage direct integration between API Gateway and SQS, and the following mapping template:

Action=SendMessage&MessageBody=$util.urlEncode("$input.body")

Going further:

• This pattern also works with Amazon Kinesis Data Stream (CDK solution construct: aws-apigateway-kinesisstreams) or any other messaging service that provides persistence of the messages.
• Implementing filters and data transformation is possible within the mapping template (using Velocity Template Library) but can be difficult to test and troubleshoot. You can choose to replace it with a Lambda function but you will loose the storage-first capability. I would recommend to do it in several steps:

1. Send the raw message on the SQS Queue,
2. Have a Lambda function that does the filtering/data transformation,
3. Push the filtered message to the backend (through another SQS queue).

Use case #2: CQRS

In this use case, we want to implement the CQRS pattern (Command Query Responsibility Segregation) as we have different usage patterns for our data:

• Write-intensive: we need an important throughput to write our data, so we select Amazon DynamoDB, which is a NoSQL database that provides single-digit millisecond performance at any scale.
• Low and structured read: on the other side, we don’t perform heavy read but we need to do complex querying. A relational database is best suited so we select Amazon Aurora, the AWS relational database compatible with PostgreSQL and MySQL.

The CQRS patterns consists in separating commands (write) from queries (read) so that we can leverage the best tool for the job when usage patterns are very different (heavy write/low read or low write/heavy read).

To achieve this, we need a way to synchronise the Aurora database with the data inserted in DynamoDB : we leverage DynamoDB streams which will stream all the changes to a downstream service.

Do we need a Lambda function? Yes

Explanation:

• DynamoDB offers the two following options to capture changes in the data and to stream them.

– DynamoDB Streams, that we use here and that can only stream to Lambda or an application that uses the Kinesis Client Library (KCL).

– Using Kinesis Data Streams. With this option, we also need some compute (a Lambda function or an application with the KCL) to insert the data in Aurora.

• A relational database (here Aurora) cannot scale as much as DynamoDB. Using a Lambda function with a limited reserved concurrency will avoid overloading the database with too many connections.

Implementation:

• With CDK, you can leverage the solution construct aws-dynamodbstreams-lambda. Note however that it’s still in development and subject to breaking changes.
• When using SAM, you can find different implementations in serverlessland.com (example).

Going further:

• It is possible to filter changes coming from DynamoDB to Lambda, for example to select only modifications done to a subset of the data, or to capture only data creation. This will reduce the number of executions of the function and the amount of code in the function, driving costs down.

Use case #3: Event-driven architecture

In this third use case, we have an event-driven architecture with two (micro)services. This architecture leverages 2 common patterns:

• Choreography: each service is independent and unaware of other services and the overall flow. However they know the choreography and what to do when receiving a specific piece of information. EventBridge is used to send these pieces of information (events). This pattern ensures a very low coupling between services and provides a better resilience.
• Orchestration: Just like the conductor directs an orchestra, this pattern leverages a central service to orchestrate multiple operations/services. In our use case, service B makes use of AWS Step Functions to orchestrate several tasks.

In event driven architectures, we generally leverage both:

• Choreography between different domains (ex: order / products / delivery / …) so that each domain is independent from the others.
• Orchestration within a domain so that you can perform multiple related tasks with proper error handling, retry mechanism and more globally orchestration capabilities outside of your business code.

Do we need a Lambda function? No

Explanation:

• EventBridge provides direct integration with multiple services, including Step Functions, so that you can start the execution of a state machine based on events. Lambda is not required.

Implementation:

• When using CDK, you can leverage the aws-eventbridge-stepfunctions solution construct.
• If you use SAM or Terraform, you can find examples in the patterns section of serverlessland.com.

Going further:

• EventBridge also permits to filter and transform events sent to targets. This reduces event more the need to write a Lambda function to perform these tasks.
• This integration can also be used to start a state machine execution in response to Amazon S3 events, for example when objects are created in a S3 bucket. That way, you don’t need a Lambda just to trigger the execution of the state machine.

Use case #4: Services orchestration

In this use case, we have a Step Functions state machine in charge of analysing documents uploaded by customers and performing verifications and operations on the extracted data. The creation of an object in the S3 bucket triggers the execution of the state machine (see use case #3) and we leverage Amazon Textract to analyse the document and extract meaningful information. Step Functions is able to integrate with more than 200 AWS services and 10 000 APIs directly without any Lambda (see the full list of compatible services and APIs) and Textract is part this list.

Do we need a Lambda function? Yes

Explanation:

• The AnalyzeDocument API returns a very verbose response, with all the elements contained in the document, their position, the confidence, relationships between elements and so on. The bigger the document is, for example a multi-page PDF, the bigger the response payload will be. This requires a robust parser, like amazon-textract-response-parser (Python, JS and C#) to make it easier to browse and find information in the response.
• Therefore, a Lambda function is needed to use this library and process the result of Textract.

Implementation:

• Use the amazon-textract-response-parser with your preferred language in your Lambda function.

Going further:

• If we used the Amazon Translate TranslateText API instead of Textract, we could use the direct integration. Indeed, the response payload of this API is much lighter and you can extract the TranslatedText, directly from your state machine (using “$.TranslatedText“). It’s important to understand the API that you are trying to call and the data you want from this API, so that you can select the appropriate approach:
• If you can easily extract part of the API response, use direct integration. Also have a look at Step Functions intrinsic functions which can be used to perform common tasks without a function.
• If you need more advanced capabilities (ex: a parser), use a Lambda function.

Use case #5: GraphQL API triggering a workflow

AWS AppSync can be seen as an alternative to API Gateway to expose data and APIs to a client. It is particularly useful for frontends (web and mobile) as it exposes GraphQL instead of REST, thus reducing the number of queries and also the amount of data that transit between the backend and the frontend. AppSync offers a tight integration with DynamoDB, Aurora and Amazon Opensearch Service, where it can retrieve the data to expose. AppSync also provides an integration with Lambda when you need to implement more complex logic. Finally AppSync also provides pipeline resolvers, when a single step is not enough and you need multiple steps to perform an operation.

In this use case, we don’t want to use the pipeline resolver but leverage a Step Functions workflow which provides additional orchestration and error handling mechanisms to perform multiple tasks. When the client performs the request, we should start the execution of the workflow, wait for the response and return the result back to the client.

Do we need a Lambda function? No

Explanation:

• While there is no built-in integration between AppSync and Step Functions, Appsync provides HTTP Resolvers. This enables you to perform any operations on any HTTP endpoint. It can be a backend deployed on premises, it can be a partner API, or … an AWS API.
• AWS provides different options to access its services: the AWS Console, the AWS CLI, AWS SDKs but they all end up with an API call (see all service endpoints). Looking at Step Functions, we have 2 endpoints:
• states.<region>.amazonaws.com for asynchronous workflows (either standard or express)
• sync-states.<region>.amazonaws.com for synchronous workflows (express)
• In our use case, we want to get an immediate response from the workflow, so we leverage the synchronous express workflow and use the StartSynchronousExecution API with the sync-states.<region>.amazonaws.com endpoint. AppSync will perform a simple HTTP call the Step Function endpoint.

Implementation:

• This blog post explains in lot more details how to implement this integration, especially it gives the payload you have to use to properly call the Step Functions endpoint.
• Looking for some code, you can have a look at this pattern (using CDK) on serverlessland.com.

Going further:

• AppSync will timeout if the workflow takes more than 30 seconds to respond. If you need more time in your workflow, you can actually start an asynchronous execution of it, and leverage a Subscription to get notified when the workflow has finished the processing. This blog post also explains how to implement that.

Conclusion

In this blog post, we have described different use cases and wether a Lambda function was needed or not between different services. We saw that Lambda is not always required. Services like API Gateway, EventBridge or Step Functions provide native integration with multiple other AWS services, which make the architecture more resilient and cost effective. When this kind of native integration is not available, a Lambda function is needed. As a general recommendation, use Lambda function to transform data, not to transport data between services.