Getting HTTP Requests Working in AWS Lambda with the Requests Library

When building AWS Lambda functions in Python, the Requests library is a popular choice for making HTTP requests to external services. However, developers often run into issues getting Requests to function properly in the Lambda environment.

In this guide, we'll cover the most common problems and solutions for using Requests in Lambda:

Understanding the Lambda Environment

The key to fixing issues with Requests in Lambda is understanding how the Lambda environment differs from a typical Python runtime.

Lambda functions run in an ephemeral container which is initialized from scratch on each invocation. This means you don't have access to system libraries beyond the AWS SDK and what's included in your deployment package.

Additionally, Lambda functions have limited disk space, memory, and CPU available. This can cause problems for libraries trying to cache data or do CPU/memory intensive operations.

Common Issue #1: Missing Dependencies

When you deploy a Lambda function, you must bundle all function code and dependencies into a deployment package. If you rely on native system libraries not included in Lambda, or fail to vendor dependencies, you'll run into errors like:

Unable to import module 'lambda_function': No module named 'requests'

The solution is to use a virtual environment to install Requests and all other dependencies, then bundle the site-packages folder in your deployment package.

This ensures Lambda has access to all the necessary libraries.

Common Issue #2: SSL Certificate Verification Failures

You might see errors like SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed when trying to make HTTPS requests. This occurs because Lambda doesn't include common CA certificates for validating SSL connections.

The solution is to set the verify argument to False when making requests:

import requests

url = "https://api.example.com/data"
response = requests.get(url, verify=False) 

While disabling SSL verification works, it exposes you to potential MITM attacks. A more secure solution is to bundle CA certificates and pass them to Requests.

Common Issue #3: Connection Timeouts

Sometimes requests seem to hang or time out unexpectedly. This can occur if the remote server is taking a long time to respond, and Lambda kills the function before getting a response.

By default, the Lambda environment times out after 3 seconds. For longer running requests, you need to configure longer timeouts:

import requests

url = "https://api.example.com/data"

# Set connection timeout to 20 seconds
response = requests.get(url, timeout=20)

Increasing timeouts gives Requests more time to handle network I/O, avoiding unexpected timeouts.

Optimizing Performance

Even when you get Requests working, performance can suffer in Lambda compared to EC2 due to environment constraints.

Some best practices:

Following these troubleshooting tips and optimization best practices will help you be successful using the powerful Requests library within AWS Lambda.