Step Functions with API Gateway integration, good idea?

Use case

In a registration workflow, I need to validate the address provided by the user with a 3rd party API. In my case, I’m using an open API from the french government: https://adresse.data.gouv.fr/api-doc/adresse. The process is quite simple:

  • Assert there is a valid address corresponding to that input: at least one value, with a confidence score > 0,82.
  • Retrieve the “official” label for this

Implementation

Using a Lambda function

The naive approach is to have a Lambda function that will request this API and perform the validation logic:

Step Functions with intermediary Lambda function that performs the API call

Using API Gateway

You can actually invoke an API Gateway endpoint directly from Step Functions. To do so, you first need to create an HTTP proxy to the 3rd party API in API Gateway, and then have the state machine invoking it:

Step Functions with API Gateway HTTP Proxy to the 3rd party API
Step Functions With API Gateway HTTP Proxy to the 3rd party API and business logic

Configuration

I’m using HTTP API because it offers up to 60% reduction in latency over REST API (source) and it is also cheaper: $1.11/million requests vs $3.50/million requests (source). And I don’t need the additional features provided by the REST API (throttling, caching, …). Have a look here for the differences.

Tuning of the Lambda function

Comparison

Performance

Using a dataset of thousands of addresses, I wrote a load test (in python) that starts the execution of each state machine (in a random order, to avoid a potential cache effect from the 3rd party API). Running the test for about 20 minutes, I have the following execution times for my state machines (with Lambda in blue and API Gateway in orange):

Workflow execution time comparison

Costs

Indicated prices are for the Ireland region.

CloudWatch Logs Insights request
Price comparison of a Lambda function and API Gateway HTTP API
Number of state transitions per workflow
Total price comparison (Step Functions + Lambda / API GW)

Conclusion

In this article, I’ve shown that the direct integration with API Gateway in Step Functions is quite competitive versus the use of an intermediary Lambda function, both in terms of performance and potentially in terms of pricing. So if you need to retrieve some data from an API, that makes sense to use this direct integration. Be sure to use the HTTP API which provides lower latency and better prices than the REST API.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jérôme Van Der Linden

Jérôme Van Der Linden

103 Followers

Senior Solution Architect @AWS - software craftsman, agile and devops enthusiastic, cloud advocate. Opinions are my own.