HOW TO HANDLE MILLIONS OF SHOPIFY WEBHOOKS WITH API GATEWAY AND TERRAFORM

Here at Orange Collar, we build custom apps for the Shopify platform on a regular basis. Some of these apps require the consumption of webhooks being sent from the Shopify platform. Whenever we have to consume webhooks as part of an app requirement we take into account the following realities.

1. Shopify will stop sending webhooks if they consistently take over 400ms to respond.
2. Shopify will stop sending webhooks if a large number of them do not respond with a 200 success response.
3. Shopify can, and sometimes will send a webhook more than 1 time.

Because the volume and cadence of the webhooks being sent from Shopify are pretty much unpredictable we need a solution that scales quickly and handles the above-mentioned realities.

By using an Amazon API Gateway to SQS integration we can solve for all of three of these realities easily. The APIG to SQS integration responds in just over 100ms, scales up to handle thousands of requests and can deduplicate repeat messages.

If you are not building your infrastructure with Terraform yet you are missing out. All of our infrastructures are being built with Terraform.

Here is the main.tf file broken down by area. Don’t worry, at the end of the post I will provide the entire main.tf file so you can easily copy and paste.

SETUP YOUR TERRAFORM ENVIRONMENT.

I personally hate it when I am looking for a tutorial online and the post creator omits the setup part of the tutorial. This setup will store the Terraform state in an S3 bucket in the us-east-1 datacenter. Just add your bucket name and the key (filename) that you want to use to store the Terraform state.

This also defines the provider to be AWS and creates two variables. The AWS region and the name of the app that you are building. Finally, it creates the tags to be used later when the Amazon resources are created.

CREATE THE AMAZON SQS QUEUE

This creates an SQS FIFO queue. Why use a FIFO queue? This type of queue allows for the deduplication of messages based either on a key-value or the hash of the message body. We want to use the content-based deduplication for our purposes so if Shopify sends over a duplicate webhook it is handled here and we do not have to write any additional logic for it.

By default, a FIFO queue supports 300 messages per second. If you need to allow more a request to Amazon can increase this limit significantly.

CREATE API GATEWAY

This creates an API Gateway with a path of /webhook/shopify. Since webhooks are sent as a POST we have created a POST method here. This part of the script creates the API Gateway, the “webhook/shopify” resources. It then creates the POST method and attaches it to the “webhook/shopify” resource.

CREATE THE METHOD SETTINGS AND INTEGRATION

This part is where we set the settings for the POST method so we can log out error events and collect metrics. This is also where the integration between the APIG and the SQS queue occurs.

It is important to note here that the integration header must be set to this value set up in this file or you will receive 500 errors back from the integration.

ADD THE METHOD RESPONSE TO THE INTEGRATION

For our purposes, we want to return a 200 success response for every single POST request. We will handle the authentication and validation of the POSTS later on and want to ensure that all responses are returned as successful. This ensures that Shopify never runs into errors and stops sending this endpoint webhooks.

CREATE THE IAM ROLE AND POLICY

We need to create an IAM role and policy to allow the APIG to send messages to the SQS queue.

ATTACH THE IAM ROLE TO THE POLICY

One the policy and role have been created we need to attach them.

CREATE THE CLOUDWATCH LOGS

CREATE THE STAGE AND DEPLOYMENT

There is a bug in terraform where the stage name that has been defined up in aws_api_gateway_method_settings has an issue with the deployment. If you want your Terraform script to auto-deploy you will run into this bug. In this case, we have worked around it by creating a second stage that gets deployed named “dev-temp”.

The bug report is tracked here; if anyone has a better solution for this I would love to hear it.

That is all there is to set up this up in Terraform. I hope this helps someone out that is trying to tackle integrating Amazon’s API Gateway with SQS.

Here is the full text-based script for main.tf

#Remote state
terraform {
  backend "s3" {
    region = "us-east-1"
    bucket = "PUT BUCKET NAME HERE"
    key    = "PUT KEY HERE"
  }
}

variable "region" {
  default = "us-east-1"
}

variable "app_name" {
  default = "PUT YOUR APP NAME HERE"
}

provider "aws" {
  region = "${var.region}"
}


locals {
  common_tags = {
    Environment = "Development"
    Application = "${var.app_name}"
  }
}

data "aws_caller_identity" "current" {}


// ******************** SQS SETUP ******************** //
resource "aws_sqs_queue" "myapp_sqs_queue" {
  name                        = "${var.app_name}-inbound-queue.fifo"
  fifo_queue                  = true
  content_based_deduplication = true
  tags                        = "${local.common_tags}"
}



// ******************** API GATEWAY SETUP ******************** //
resource "aws_api_gateway_rest_api" "myapp_apig" {
  name = "${var.app_name}-apig"
}

resource "aws_api_gateway_resource" "webhook_resource" {
  path_part   = "webhook"
  parent_id   = "${aws_api_gateway_rest_api.myapp_apig.root_resource_id}"
  rest_api_id = "${aws_api_gateway_rest_api.myapp_apig.id}"
}

resource "aws_api_gateway_resource" "webhook_shopify_resource" {
  path_part   = "shopify"
  parent_id   = "${aws_api_gateway_resource.webhook_resource.id}"
  rest_api_id = "${aws_api_gateway_rest_api.myapp_apig.id}"
}

resource "aws_api_gateway_method" "webhook_shopify_post_method" {
  rest_api_id   = "${aws_api_gateway_rest_api.myapp_apig.id}"
  resource_id   = "${aws_api_gateway_resource.webhook_shopify_resource.id}"
  http_method   = "POST"
  authorization = "NONE"
}


resource "aws_api_gateway_method_settings" "webhook_shopify_post_method_settings" {
  rest_api_id = "${aws_api_gateway_rest_api.myapp_apig.id}"
  stage_name  = "${aws_api_gateway_stage.myapp_deployment_stage.stage_name}"
  method_path = "${aws_api_gateway_resource.webhook_shopify_resource.path_part}/${aws_api_gateway_method.webhook_shopify_post_method.http_method}"

  settings {
    metrics_enabled = true
    logging_level   = "INFO"
  }
}

resource "aws_api_gateway_integration" "webhook_shopify_post_integration" {
  rest_api_id             = "${aws_api_gateway_rest_api.myapp_apig.id}"
  resource_id             = "${aws_api_gateway_resource.webhook_shopify_resource.id}"
  http_method             = "${aws_api_gateway_method.webhook_shopify_post_method.http_method}"
  integration_http_method = "POST"
  type                    = "AWS"
  credentials             = "${aws_iam_role.apig-sqs-send-msg-role.arn}"
  uri                     = "arn:aws:apigateway:${var.region}:sqs:path/${data.aws_caller_identity.current.account_id}/${aws_sqs_queue.myapp_sqs_queue.name}"

  request_parameters = {
    "integration.request.header.Content-Type" = "'application/x-www-form-urlencoded'"
  }

  request_templates = {
    "application/json" = <<EOF
Action=SendMessage&MessageGroupId=1&MessageBody=
{
  "body" : $input.json('$'),
  "rawbody" : "$util.base64Encode($input.body)",
  "headers": {
    #foreach($header in $input.params().header.keySet())
    "$header": "$util.escapeJavaScript($input.params().header.get($header))" #if($foreach.hasNext),#end

    #end
  },
  "method": "$context.httpMethod",
  "params": {
    #foreach($param in $input.params().path.keySet())
    "$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end

    #end
  },
  "query": {
    #foreach($queryParam in $input.params().querystring.keySet())
    "$queryParam": "$util.escapeJavaScript($input.params().querystring.get($queryParam))" #if($foreach.hasNext),#end

    #end
  }
}
EOF
  }
  passthrough_behavior    = "WHEN_NO_TEMPLATES"
}


resource "aws_api_gateway_method_response" "webhook_shopify_post_method_response_200" {
  rest_api_id = "${aws_api_gateway_rest_api.myapp_apig.id}"
  resource_id = "${aws_api_gateway_resource.webhook_shopify_resource.id}"
  http_method = "${aws_api_gateway_method.webhook_shopify_post_method.http_method}"
  status_code = "200"
}

resource "aws_api_gateway_integration_response" "webhook_shopify_post_integration_response_200" {
  rest_api_id = "${aws_api_gateway_rest_api.myapp_apig.id}"
  resource_id = "${aws_api_gateway_resource.webhook_shopify_resource.id}"
  http_method = "${aws_api_gateway_method.webhook_shopify_post_method.http_method}"
  status_code = "${aws_api_gateway_method_response.webhook_shopify_post_method_response_200.status_code}"
}



resource "aws_iam_role" "apig-sqs-send-msg-role" {
  name = "${var.app_name}-apig-sqs-send-msg-role"
  tags = "${local.common_tags}"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "apig-sqs-send-msg-policy" {
  name        = "${var.app_name}-apig-sqs-send-msg-policy"
  description = "Policy allowing APIG to write to SQS for ${var.app_name}"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
           "Effect": "Allow",
           "Resource": [
               "*"
           ],
           "Action": [
               "logs:CreateLogGroup",
               "logs:CreateLogStream",
               "logs:PutLogEvents"
           ]
       },
       {
          "Effect": "Allow",
          "Action": "sqs:SendMessage",
          "Resource": "${aws_sqs_queue.myapp_apig.arn}"
       }
    ]
}
EOF
}

## IAM Role Policies
resource "aws_iam_role_policy_attachment" "terraform_apig_sqs_policy_attach" {
  role       = "${aws_iam_role.apig-sqs-send-msg-role.id}"
  policy_arn = "${aws_iam_policy.apig-sqs-send-msg-policy.arn}"
}


resource "aws_cloudwatch_log_group" "webhook_shopify_log_group" {
  name              = "APIG-Execution-Logs_${aws_api_gateway_rest_api.myapp_apig.name}"
  retention_in_days = 30
}

## Setup the stages and deploy to the stage when terraform is run.
resource "aws_api_gateway_stage" "myapp_deployment_stage" {
  stage_name    = "dev-temp" // This a hack to fix the API being auto deployed.
  rest_api_id   = "${aws_api_gateway_rest_api.myapp_apig.id}"
  deployment_id = "${aws_api_gateway_deployment.myapp_deployment.id}"

}

resource "aws_api_gateway_deployment" "myapp_deployment" {
  rest_api_id     = "${aws_api_gateway_rest_api.myapp_apig.id}"
  stage_name      = "dev"
}