Background Processing

AWS Lambda is a service that executes code in response to triggers and data received from other AWS services. It is our preferred solution for any kind of background processing in a project. Key benefits over past approaches include:

• Automated Administration - All of the infrastructure for running your code is handled so you don't need to worry about EC2 instances, operating systems, or maintenance. Also, logging and monitoring is provided by CloudWatch.
• Fault Tolerance - Capacity is distributed across multiple Availability Zones per region, protecting against machine and datacenter failures.
• Automatic Scaling - Code is executed only when needed and scales to handle the rate of incoming requests without configuration.
• Cost - Billing happens in increments of 100 milliseconds for only the requests that are served. Applications will no longer incur monthly EC2 charges for idle time in a worker process.

Triggers

Historically background processing on projects have been triggered in two ways: by schedule (cron) or by message publication (SQS). AWS Lambda has a suitable replacement for both of these scenarios.

• Scheduled Processing
  Any time you think you need a cron job, you can use a Lambda that is invoked by a CloudWatch Events Schedule.

• Message Publication
  Instead of publishing messages to an SQS queue, send them to SNS. Lambdas that subscribe to SNS Topics will receive the payload of messages that are published, from there the world is yours.

Deploying Lambdas

Once a Lambda has been initially configured using the AWS Console, you can push updates during the CI process using Fabric. The following fabfile.py snippet demonstrates creating a ZIP file for Lambda code in a local path and updating an existing function by name using boto3:

import io
import os
from zipfile import ZipFile
from boto3.session import Session
from fabric.api import env, task


AWS_ACCESS_KEY_ID = env.AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY = env.AWS_SECRET_ACCESS_KEY
AWS_REGION = env.get('AWS_REGION', 'us-east-1')


session = Session(
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    region_name=AWS_REGION)
aws_lambda = session.client('lambda')


def files_to_zip(path):
    for root, dirs, files in os.walk(path):
        for f in files:
            full_path = os.path.join(root, f)
            archive_name = full_path[len(path) + len(os.sep):]
            yield full_path, archive_name


def make_zip_file_bytes(path):
    buf = io.BytesIO()
    with ZipFile(buf, 'w') as z:
        for full_path, archive_name in files_to_zip(path=path):
            z.write(full_path, archive_name)
    return buf.getvalue()


@task
def deploy_my_cool_lambda():
    lambda_name = 'my-cool-lambda'
    lambda_dir = os.path.join('./path/to', lambda_name)
    aws_lambda.update_function_code(
        FunctionName=lambda_name,
        ZipFile=make_zip_file_bytes(path=lambda_dir))

Additional Resources

• Lambda Function Handler (Python) - Info about the signature of a function handler and the parameters it receives when invoked.
• The Context Object (Python) - Info about how to get runtime info from the Lambda service in your function handler.
• Logging (Python) - Info about logging in your function handler.
• Exceptions (Python) - Info about how Lambda serializes any exceptions raised by the function handler.
• Creating a Deployment Package (Python) - See the Advanced scenario section for info on including resources in your Lambda that you would normally pip install.