Skip to main content

Converting an AWS Lambda application to Fargate

Overview

In this tutorial, we'll show you how simple it is to take a Klotho-annotated application, and migrate it from Lambda to Fargate ECS. You make one config change, and Klotho handles the rest. image showcasing topology transition from lambda to fargate In this example, we can see that Klotho understood a Network Load Balancer (NLB) was required for your existing API Gateway to function with the new ECS containers. Klotho has generated infrastructure-as-code (IAC) for the intermediary NLBs and configured them on your behalf.

Why?

Service requirements may change, and that may mean the type of compute you initially deployed with is no longer appropriate. Klotho enables developers to quickly and easily migrate from one compute service to another.

Getting Started

We're going to start by cloning the ts-serverless-gateway app which has already been Klotho-annotated. This sample app showcases multiple execution units which we'll deploy as Lambdas initially. We will then update the Klotho Configuration to swap from Lambdas to Fargate ECS and redeploy.

Prerequisites

Repository

Clone our sample apps git repo and install the npm packages for the ts-serverless-gateway app.

git clone https://github.com/klothoplatform/sample-apps.git
cd sample-apps/ts-serverless-gateway
npm install

App Overview

The ts-serverless-gateway application utilizes the following annotations

/**
* @klotho::execution_unit {
* id = "srvless-quotes"
* }
*/

export const router = require('express').Router();
...

/**
* @klotho::persist {
* id = "quoteKV"
* }
*/
let quoteStore = new Map<string, string>();
...

Klotho compile the application

Let's start by compiling our TypeScript application into JavaScript.

npx tsc

Next, log in to Klotho. This will allow us to support you if you run into any issues, and give you the opportunity to shape the product in this early development stage.

klotho --login # if you haven't already

Then, compile the the app with Klotho.

klotho . --app ts-sg --provider aws

The output will be similar to:

██╗  ██╗██╗      ██████╗ ████████╗██╗  ██╗ ██████╗
██║ ██╔╝██║ ██╔═══██╗╚══██╔══╝██║ ██║██╔═══██╗
█████╔╝ ██║ ██║ ██║ ██║ ███████║██║ ██║
██╔═██╗ ██║ ██║ ██║ ██║ ██╔══██║██║ ██║
██║ ██╗███████╗╚██████╔╝ ██║ ██║ ██║╚██████╔╝
╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚═════╝

Adding resource input_file_dependencies:
Adding resource exec_unit:srvless-userpost
Adding resource exec_unit:srvless-quotes
Adding resource exec_unit:srvless-userget
index.js:27:0: Found 1 route(s) for middleware 'router' var: app
index.js:30:0: Found 1 route(s) for middleware 'userPost' var: app
Adding resource gateway:app
index.js:27:0: Found 1 route(s) for middleware 'router' var: app
index.js:28:0: Found 2 route(s) for middleware 'quotes' var: app
Not adding duplicate route / for srvless-quotes. Exists in srvless-userpost unit: srvless-quotes
index.js:27:0: Found 1 route(s) for middleware 'router' var: app
index.js:29:0: Found 1 route(s) for middleware 'userGet' var: app
Not adding duplicate route / for srvless-userget. Exists in srvless-userpost unit: srvless-userget
Adding resource persist_kv:quoteKV
Adding resource topology:ewu-ts-serverless
Adding resource infra_as_code:Pulumi (AWS)
Pulumi.user123-ts-sg.yaml: Make sure to run `pulumi config set aws:region YOUR_REGION --cwd 'compiled/' -s 'user123-ts-sg'` to configure the target AWS region.

In the ./compiled folder, you'll find the infrastructure-as-code for your cloud-native application ready for deployment. You'll also find a file called user123-ts-sg.png, visualizing the topology output of your compiled application:

lambda deployment topology image

Deploying the application

Deployment Dependencies

Now that the last-section has produced infrastructure-as-code, it is possible to deploy the cloud-native version of the application. The AWS account set up on your computer will be used.

But first, we need to install and set up a few more dependencies:

  • Docker
  • an AWS account, set up with either:
    • The AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY environment variables for a user
    • OR, $HOME/.aws/credentials (eg. via AWS CLI: aws configure) setup
  • Pulumi CLI
    • Follow the Pulumi installation instructions and set up for local usage:
      pulumi login --local
      export PULUMI_CONFIG_PASSPHRASE=""

Deploying with Pulumi

# region should be an aws region, for example us-west-1

pulumi config set aws:region <region> --cwd 'compiled/' -s ts-sg

If this is your first time deploying this application, you will be prompted to create the user123-ts-sg stack. Just press ENTER and the stack will be created for you.

The stack 'user123-ts-sg' does not exist.
If you would like to create this stack now, please press <ENTER>, otherwise press ^C:

Now that the Pulumi stack is created, simply install the project dependencies and deploy.

cd compiled
npm install
pulumi up

You'll see a preview of the changes to be applied to your AWS account: (you'll be able to delete these later on)

Previewing update (user123-ts-sg):
Type Name Plan
+ pulumi:pulumi:Stack user123-ts-sg-user123-ts-sg create
+ ├─ awsx:ecr:Repository user123-ts-sg create
+ │ └─ aws:ecr:LifecyclePolicy user123-ts-sg create
+ ├─ aws:apigateway:RestApi app create
+ │ ├─ aws:apigateway:Method -GET create
+ │ │ └─ aws:apigateway:Integration lambda--GET create
+ │ ├─ aws:apigateway:Resource v1 create
+ │ ├─ aws:apigateway:Resource users create
+ │ │ ├─ aws:apigateway:Method users-POST create
+ │ │ │ └─ aws:apigateway:Integration lambda-users-POST create
+ │ │ └─ aws:apigateway:Method users-GET create
+ │ │ └─ aws:apigateway:Integration lambda-users-GET create
+ │ ├─ aws:apigateway:Resource quote create
+ │ │ └─ aws:apigateway:Method quote-POST create
+ │ │ └─ aws:apigateway:Integration lambda-quote-POST create
+ │ ├─ aws:apigateway:Resource quote-list create
+ │ │ └─ aws:apigateway:Method quotelist-GET create
+ │ │ └─ aws:apigateway:Integration lambda-quotelist-GET create
+ │ └─ aws:apigateway:Deployment app-deployment create
+ │ └─ aws:apigateway:Stage app-stage create
+ ├─ aws:cloudwatch:LogGroup srvless-userget-function-api-lg create
+ ├─ aws:cloudwatch:LogGroup srvless-userpost-function-api-lg create
+ ├─ aws:s3:Bucket user123-ts-sg-payloads create
+ ├─ aws:cloudwatch:LogGroup srvless-quotes-function-api-lg create
+ ├─ aws:ecr:Repository user123-ts-sg create
+ ├─ aws:dynamodb:Table KV_user123-ts-sg create
+ │ └─ aws:sqs:Queue user123-ts-sg-kv-queue create
+ ├─ aws:iam:Role user123-ts-sg_1231f_LambdaExec create
+ │ ├─ aws:iam:Policy user123-ts-sg-srvless-quotes-exec create
+ │ └─ aws:iam:RolePolicyAttachment user123-ts-sg-srvless-quotes-exec create
+ ├─ aws:iam:Role user123-ts-sg_86e4e_LambdaExec create
+ │ ├─ aws:iam:Policy user123-ts-sg-srvless-userget-exec create
+ │ └─ aws:iam:RolePolicyAttachment user123-ts-sg-srvless-userget-exec create
+ ├─ aws:iam:Role user123-ts-sg_bc115_LambdaExec create
+ │ ├─ aws:iam:Policy user123-ts-sg-srvless-userpost-exec create
+ │ └─ aws:iam:RolePolicyAttachment user123-ts-sg-srvless-userpost-exec create
+ ├─ aws:iam:RolePolicyAttachment user123-ts-sg-srvless-userpost-lambdabasic create
+ ├─ aws:iam:RolePolicyAttachment user123-ts-sg-srvless-userget-lambdabasic create
+ ├─ aws:iam:RolePolicyAttachment user123-ts-sg-srvless-quotes-lambdabasic create
+ ├─ aws:lambda:Function srvless-userget create
+ ├─ aws:lambda:Function srvless-userpost create
+ ├─ aws:lambda:Function srvless-quotes create
+ ├─ aws:lambda:Permission post-v1users-permission create
+ ├─ aws:lambda:Permission get-v1users-permission create
+ ├─ aws:lambda:Permission get--permission create
+ ├─ aws:lambda:Permission get-v1quotelist-permission create
+ └─ aws:lambda:Permission post-v1quote-permission create

Resources:
+ 47 to create

Do you want to perform this update? [Use arrows to move, enter to select, type to filter]
yes
> no
details

Respond with yes when you're ready and wait for your resources to be created. This usually takes 3 minutes. When it's done deploying, you'll see the completion status and the AWS provided API Gateway URL for your API:

Outputs:
apiUrls : [
[0]: "https://qxy99avspl.execute-api.us-east-1.amazonaws.com/stage/"
]

To test your endpoints, copy the base URL you see on your screen and use these curl commands:

# Make sure to change apiUrl
$ curl -X POST --data "quote=HelloWorld" "https://apiUrl/v1/quote"
Added HelloWorld

# Make sure to change apiUrl
$ curl "https://apiUrl/v1/quote-list"
["HelloWorld"]

Update the Klotho configuration

Now that we have our sample app deployed with Lambdas, let's review and update the generated Klotho Configuration.

First, make a copy of the configuration so we can pass the file in on future Klotho compiles.

cp ./compiled/klotho.yaml .

Open the copied ./klotho.yaml file in an editor of your choice and update the type field of each entry in execution_units from lambda to fargate.

---
execution_units:
srvless-quotes:
type: fargate #lambda -> fargate
infra_params:
memorySize: 512
timeout: 180
srvless-userget:
type: fargate #lambda -> fargate
infra_params:
memorySize: 512
timeout: 180
srvless-userpost:
type: fargate #lambda -> fargate
infra_params:
memorySize: 512
timeout: 180

Recompile the application

Recompile the application and provide the klotho.yaml config file, which we copied and edited previously. This will generate new infrastructure as code to deploy ECS containers and associated NLBs.

## We specify the klotho.yaml config file which you copied and edited to use fargate

klotho . --config klotho.yaml

In the ./compiled folder, you'll find the updated user123-ts-sg.png visualization:

fargate deployment topology image

Redeploy the application

Now redeploy the application. Refer to the Deploying with the CLI section of the guide if you need a refresher. Note the Pulumi preview process and subsequent deployment step may take several minutes longer than the previous deploy.

The Pulumi preview will look a little different this time around. All of the resources assocaited with the previous Lambda deployment, such as the Lambdas and Lambda to Gateway integrations, are replaced with those necessary for a Fargate ECS deployment, such as ECS Services and NLBs.

Previewing update (user123-ts-sg):
Type Name Plan Info
pulumi:pulumi:Stack user123-ts-sg-user123-ts-sg running.
+ ├─ awsx:lb:NetworkLoadBalancer srvless-quotes-nlb create
+ │ └─ awsx:lb:NetworkTargetGroup srvless-quotes-tg create
+ │ ├─ awsx:lb:NetworkListener srvless-quotes-listener create
+ │ └─ aws:lb:TargetGroup srvless-quotes-tg create
+ ├─ awsx:x:ecs:FargateTaskDefinition srvless-quotes-task create
+ ├─ awsx:x:ec2:Vpc user123-ts-sg create
+ │ ├─ awsx:x:ec2:InternetGateway user123-ts-sg create
+ │ │ └─ aws:ec2:InternetGateway user123-ts-sg create
+ │ ├─ awsx:x:ec2:Subnet user123-ts-sg-private-0 create
+ │ │ ├─ aws:ec2:RouteTable user123-ts-sg-private-0 create
+ │ │ └─ aws:ec2:Subnet user123-ts-sg-private-0 create
+ │ ├─ awsx:x:ec2:Subnet user123-ts-sg-private-1 create
+ │ │ ├─ aws:ec2:RouteTable user123-ts-sg-private-1 create
+ │ │ └─ aws:ec2:Subnet user123-ts-sg-private-1 create
+ │ ├─ awsx:x:ec2:Subnet user123-ts-sg-public-1 create
+ │ │ ├─ aws:ec2:RouteTable user123-ts-sg-public-1 create
+ │ │ ├─ aws:ec2:Subnet user123-ts-sg-public-1 create
+ │ │ ├─ aws:ec2:Subnet user123-ts-sg-public-1 create
+ │ │ └─ aws:ec2:RouteTableAssociation user123-ts-sg-private-1 create
+ │ │ └─ aws:ec2:RouteTableAssociation user123-ts-sg-public-1 create
+ │ ├─ awsx:x:ec2:NatGateway user123-ts-sg-0 create
pulumi:pulumi:Stack user123-ts-sg-user123-ts-sg running. read aws:ec2:SecurityGroup srvless-userpost-service-0
+ │ │ └─ aws:ec2:Eip user123-ts-sg-0 create
+ │ │ └─ aws:ec2:RouteTableAssociation user123-ts-sg-private-0 create
+ │ ├─ awsx:x:ec2:Subnet user123-ts-sg-public-0 create
+ │ ├─ awsx:x:ec2:Subnet user123-ts-sg-public-0 create
+ │ │ └─ aws:ec2:NatGateway user123-ts-sg-0 create
+ │ │ ├─ aws:ec2:RouteTable user123-ts-sg-public-0 create
+ │ │ ├─ aws:ec2:Route user123-ts-sg-public-0-ig create
+ │ │ ├─ aws:ec2:Route user123-ts-sg-public-0-ig create
+ │ │ └─ aws:ec2:RouteTableAssociation user123-ts-sg-public-0 create
+ │ └─ awsx:x:ec2:NatGateway user123-ts-sg-1 create
+ │ └─ aws:lb:LoadBalancer srvless-quotes-nlb create
+ │ ├─ aws:ec2:Eip user123-ts-sg-1 create
+ │ ├─ aws:ec2:Eip user123-ts-sg-1 create
+ │ │ └─ aws:ec2:Route user123-ts-sg-private-0-nat-0 create
+ ├─ awsx:lb:NetworkLoadBalancer srvless-userpost-nlb create
+ ├─ awsx:lb:NetworkLoadBalancer srvless-userpost-nlb create
+ │ ├─ awsx:lb:NetworkTargetGroup srvless-userpost-tg create
+ │ │ │ └─ aws:lb:Listener srvless-quotes-listener create
+ │ │ │ └─ aws:lb:Listener srvless-userpost-listener create
+ │ │ └─ aws:lb:TargetGroup srvless-userpost-tg create
+ │ └─ aws:lb:LoadBalancer srvless-userpost-nlb create
+ ├─ awsx:x:ecs:Cluster user123-ts-sg-cluster create
+ │ ├─ awsx:x:ec2:SecurityGroup user123-ts-sg-cluster create
+ │ │ ├─ awsx:x:ec2:IngressSecurityGroupRule user123-ts-sg-cluster-containers create
+ │ │ │ └─ aws:ec2:SecurityGroupRule user123-ts-sg-cluster-containers create
+ │ │ ├─ awsx:x:ec2:EgressSecurityGroupRule user123-ts-sg-cluster-egress create
+ │ │ │ └─ aws:ec2:SecurityGroupRule user123-ts-sg-cluster-egress create
+ │ │ ├─ awsx:x:ec2:IngressSecurityGroupRule user123-ts-sg-cluster-ssh create
+ │ │ │ └─ aws:ec2:SecurityGroupRule user123-ts-sg-cluster-ssh create
+ │ │ └─ aws:ec2:SecurityGroup user123-ts-sg-cluster create
+ │ └─ aws:ecs:Cluster user123-ts-sg-cluster create
+ ├─ awsx:x:ecs:FargateTaskDefinition srvless-userget-task create
+ ├─ awsx:x:ecs:FargateTaskDefinition srvless-userpost-task create
+ │ └─ aws:ecs:TaskDefinition srvless-userpost-task create
+ │ └─ aws:ecs:TaskDefinition srvless-userpost-task create
+ │ └─ aws:ecs:TaskDefinition srvless-userpost-task create
+ │ └─ aws:ecs:TaskDefinition srvless-userpost-task create
+ │ ├─ awsx:lb:NetworkTargetGroup srvless-userget-tg create
+ │ │ ├─ awsx:lb:NetworkListener srvless-userget-listener create
+ │ │ │ └─ aws:lb:Listener srvless-userget-listener create
+ │ │ └─ aws:lb:TargetGroup srvless-userget-tg create
+ │ └─ aws:lb:LoadBalancer srvless-userget-nlb create
+ ├─ aws:cloudwatch:LogGroup user123-ts-sg-srvless-userget-lg create
+ ├─ aws:cloudwatch:LogGroup user123-ts-sg-srvless-quotes-lg create
+ ├─ aws:cloudwatch:LogGroup user123-ts-sg-srvless-userpost-lg create
├─ aws:iam:Role user123-ts-sg_86e4e_LambdaExec
~ │ └─ aws:iam:Policy user123-ts-sg-srvless-userget-exec update [diff: ~policy]
+ ├─ aws:iam:RolePolicyAttachment srvless-userget-fargateAttach create
+ ├─ aws:iam:RolePolicyAttachment srvless-quotes-fargateAttach create
+ ├─ aws:iam:RolePolicyAttachment srvless-userpost-fargateAttach create
├─ aws:iam:Role user123-ts-sg_bc115_LambdaExec
~ │ └─ aws:iam:Policy user123-ts-sg-srvless-userpost-exec update [diff: ~policy]
+ ├─ awsx:x:ec2:SecurityGroup srvless-userget-service-0 create
+ ├─ awsx:x:ecs:FargateService srvless-userget-service create
+ │ └─ aws:ecs:Service srvless-userget-service create
├─ aws:iam:Role user123-ts-sg_1231f_LambdaExec
~ │ └─ aws:iam:Policy user123-ts-sg-srvless-quotes-exec update [diff: ~policy]
+ ├─ awsx:x:ec2:SecurityGroup srvless-quotes-service-0 create
+ ├─ awsx:x:ecs:FargateService srvless-quotes-service create
+ │ └─ aws:ecs:Service srvless-quotes-service create
+ ├─ awsx:x:ec2:SecurityGroup srvless-userpost-service-0 create
+ ├─ awsx:x:ecs:FargateService srvless-userpost-service create
+ │ └─ aws:ecs:Service srvless-userpost-service create
├─ awsx:ecr:Repository user123-ts-sg
+ ├─ aws:servicediscovery:PrivateDnsNamespace user123-ts-sg-privateDns create
+ ├─ aws:ec2:SecurityGroup user123-ts-sg create
+ ├─ aws:servicediscovery:Service srvless-userget create
+ ├─ aws:servicediscovery:Service srvless-userpost create
+ ├─ aws:servicediscovery:Service srvless-quotes create
+ ├─ aws:ec2:VpcEndpoint dynamodbVpcEndpoint create
+ ├─ aws:ec2:VpcEndpoint s3VpcEndpoint create
+ ├─ aws:ec2:SecurityGroupRule user123-ts-sg-ingress create
+ ├─ aws:ec2:VpcEndpoint sqsVpcEndpoint create
+ ├─ aws:ec2:VpcEndpoint secretsmanagerVpcEndpoint create
+ ├─ aws:ec2:VpcEndpoint lambdaVpcEndpoint create
+ ├─ aws:ec2:VpcEndpoint snsVpcEndpoint create
+ ├─ aws:apigateway:VpcLink srvless-userpost-vpc-link create
+ ├─ aws:apigateway:VpcLink srvless-quotes-vpc-link create
+ ├─ aws:apigateway:VpcLink srvless-userget-vpc-link create
├─ aws:apigateway:RestApi app
│ ├─ aws:apigateway:Resource users
│ │ ├─ aws:apigateway:Method users-POST
+ │ │ │ ├─ aws:apigateway:Integration fargate-users-POST create
- │ │ │ └─ aws:apigateway:Integration lambda-users-POST delete
│ │ └─ aws:apigateway:Method users-GET
+ │ │ ├─ aws:apigateway:Integration fargate-users-GET create
- │ │ └─ aws:apigateway:Integration lambda-users-GET delete
│ ├─ aws:apigateway:Resource quote
│ │ └─ aws:apigateway:Method quote-POST
+ │ │ ├─ aws:apigateway:Integration fargate-quote-POST create
- │ │ └─ aws:apigateway:Integration lambda-quote-POST delete
│ ├─ aws:apigateway:Method -GET
+ │ │ ├─ aws:apigateway:Integration fargate--GET create
- │ │ └─ aws:apigateway:Integration lambda--GET delete
│ └─ aws:apigateway:Resource quote-list
│ └─ aws:apigateway:Method quotelist-GET
+ │ ├─ aws:apigateway:Integration fargate-quotelist-GET create
- │ └─ aws:apigateway:Integration lambda-quotelist-GET delete
- ├─ aws:lambda:Permission get-v1quotelist-permission delete
- ├─ aws:lambda:Permission get--permission delete
- ├─ aws:lambda:Permission post-v1users-permission delete
- ├─ aws:lambda:Permission post-v1quote-permission delete
- ├─ aws:lambda:Permission get-v1users-permission delete
- ├─ aws:lambda:Function srvless-userget delete
- ├─ aws:lambda:Function srvless-userpost delete
- ├─ aws:lambda:Function srvless-quotes delete
- ├─ aws:cloudwatch:LogGroup srvless-userpost-function-api-lg delete
- ├─ aws:cloudwatch:LogGroup srvless-quotes-function-api-lg delete
- └─ aws:cloudwatch:LogGroup srvless-userget-function-api-lg delete

Resources:
+ 99 to create
~ 3 to update
- 16 to delete
118 changes. 28 unchanged

Do you want to perform this update? [Use arrows to move, enter to select, type to filter]
yes
> no
details

If you wish to move forward with the deploy, respond with yes. Note this deploy will take several minutes longer than the previous one as there are more resources to create and destroy this time around.

Cleanup

To clean up your AWS deployment simply run:

cd compiled
pulumi destroy

You will see a Pulumi preview of all the resources pending deletion. Response with yes to begin deleting.

That's it! By simply changing the execution_unit: type in the Klotho configuration from lambda to fargate, we were able to migrate our application from one compute resource to another.