When to use a Lambda function, and when not?

Use case #1: High-criticality API

Explanation:

  • The objective here is to avoid loosing orders, so we want to store them as quickly as possible.
  • Having a Lambda function between the API Gateway and the SQS Queue introduces some code, which increases the risk of failure and therefore the risk of loosing orders.
  • This pattern is called ‘Storage-First Pattern’. It consists in the ability for API Gateway to directly integrate with SQS to reliably store the incoming request before any process occurs. You can get more details here and

Implementation:

  • If you use AWS CDK, there is a solution construct on the shelves for this pattern: aws-apigateway-sqs.
  • If you use SAM or other frameworks, you will find different implementations on serverlessland.com.
  • Under the hood, we leverage direct integration between API Gateway and SQS, and the following mapping template:
Action=SendMessage&MessageBody=$util.urlEncode("$input.body")

Going further:

  • This pattern also works with Amazon Kinesis Data Stream (CDK solution construct: aws-apigateway-kinesisstreams) or any other messaging service that provides persistence of the messages.
  • Implementing filters and data transformation is possible within the mapping template (using Velocity Template Library) but can be difficult to test and troubleshoot. You can choose to replace it with a Lambda function but you will loose the storage-first capability. I would recommend to do it in several steps:
  1. Have a Lambda function that does the filtering/data transformation,
  2. Push the filtered message to the backend (through another SQS queue).

Use case #2: CQRS

In this use case, we want to implement the CQRS pattern (Command Query Responsibility Segregation) as we have different usage patterns for our data:

  • Low and structured read: on the other side, we don’t perform heavy read but we need to do complex querying. A relational database is best suited so we select Amazon Aurora, the AWS relational database compatible with PostgreSQL and MySQL.

Explanation:

  • DynamoDB offers the two following options to capture changes in the data and to stream them.

Implementation:

  • With CDK, you can leverage the solution construct aws-dynamodbstreams-lambda. Note however that it’s still in development and subject to breaking changes.
  • When using SAM, you can find different implementations in serverlessland.com (example).

Going further:

  • It is possible to filter changes coming from DynamoDB to Lambda, for example to select only modifications done to a subset of the data, or to capture only data creation. This will reduce the number of executions of the function and the amount of code in the function, driving costs down.

Use case #3: Event-driven architecture

  • Orchestration: Just like the conductor directs an orchestra, this pattern leverages a central service to orchestrate multiple operations/services. In our use case, service B makes use of AWS Step Functions to orchestrate several tasks.
  • Orchestration within a domain so that you can perform multiple related tasks with proper error handling, retry mechanism and more globally orchestration capabilities outside of your business code.

Explanation:

Implementation:

Going further:

  • EventBridge also permits to filter and transform events sent to targets. This reduces event more the need to write a Lambda function to perform these tasks.
  • This integration can also be used to start a state machine execution in response to Amazon S3 events, for example when objects are created in a S3 bucket. That way, you don’t need a Lambda just to trigger the execution of the state machine.

Use case #4: Services orchestration

Explanation:

  • The AnalyzeDocument API returns a very verbose response, with all the elements contained in the document, their position, the confidence, relationships between elements and so on. The bigger the document is, for example a multi-page PDF, the bigger the response payload will be. This requires a robust parser, like amazon-textract-response-parser (Python, JS and C#) to make it easier to browse and find information in the response.
  • Therefore, a Lambda function is needed to use this library and process the result of Textract.

Implementation:

Going further:

  • If we used the Amazon Translate TranslateText API instead of Textract, we could use the direct integration. Indeed, the response payload of this API is much lighter and you can extract the TranslatedText, directly from your state machine (using “$.TranslatedText“). It’s important to understand the API that you are trying to call and the data you want from this API, so that you can select the appropriate approach:
  • If you can easily extract part of the API response, use direct integration. Also have a look at Step Functions intrinsic functions which can be used to perform common tasks without a function.
  • If you need more advanced capabilities (ex: a parser), use a Lambda function.

Use case #5: GraphQL API triggering a workflow

Explanation:

  • While there is no built-in integration between AppSync and Step Functions, Appsync provides HTTP Resolvers. This enables you to perform any operations on any HTTP endpoint. It can be a backend deployed on premises, it can be a partner API, or … an AWS API.
  • AWS provides different options to access its services: the AWS Console, the AWS CLI, AWS SDKs but they all end up with an API call (see all service endpoints). Looking at Step Functions, we have 2 endpoints:
  • states.<region>.amazonaws.com for asynchronous workflows (either standard or express)
  • sync-states.<region>.amazonaws.com for synchronous workflows (express)
  • In our use case, we want to get an immediate response from the workflow, so we leverage the synchronous express workflow and use the StartSynchronousExecution API with the sync-states.<region>.amazonaws.com endpoint. AppSync will perform a simple HTTP call the Step Function endpoint.

Implementation:

  • This blog post explains in lot more details how to implement this integration, especially it gives the payload you have to use to properly call the Step Functions endpoint.
  • Looking for some code, you can have a look at this pattern (using CDK) on serverlessland.com.

Going further:

  • AppSync will timeout if the workflow takes more than 30 seconds to respond. If you need more time in your workflow, you can actually start an asynchronous execution of it, and leverage a Subscription to get notified when the workflow has finished the processing. This blog post also explains how to implement that.

Conclusion

In this blog post, we have described different use cases and wether a Lambda function was needed or not between different services. We saw that Lambda is not always required. Services like API Gateway, EventBridge or Step Functions provide native integration with multiple other AWS services, which make the architecture more resilient and cost effective. When this kind of native integration is not available, a Lambda function is needed. As a general recommendation, use Lambda function to transform data, not to transport data between services.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jérôme Van Der Linden

Jérôme Van Der Linden

103 Followers

Senior Solution Architect @AWS - software craftsman, agile and devops enthusiastic, cloud advocate. Opinions are my own.