Scrape Like a Pro! Get Your Free API Key

The Complete HTTPBin CheatSheet in Python

Mohan Ganesan
Dec 6, 2023 · 11 min read

What is Httpbin?

Httpbin is a popular online service that provides a simple HTTP request & response service for testing and debugging HTTP libraries and clients. Some key things you can do with httpbin:

  • Make different types of HTTP requests to see responses
  • Test authorization mechanisms and headers
  • GET and POST data to endpoints that return the data in the response
  • Test connectivity to a public HTTP service
  • Explore HTTP response formats like JSON and XML
  • Generally build and test HTTP client implementations

    • It is used by developers testing HTTP client code, those experimenting with API requests, and anyone wanting a quick public HTTP service to interact with.

    Here are some of the key scenarios where httpbin is useful:

    Testing HTTP Client Code

    Since httpbin provides a simple HTTP service, it's extremely useful for testing code that makes HTTP requests like:

  • REST and API clients
  • web scrapers
  • crawlers
  • bots
  • wrappers around requests libraries

    • You can test making GET, POST requests, custom headers, status codes etc.

    Experimenting with APIs

    httpbin allows you to prototype and experiment with APIs quickly without needing to set up a server. You can shape any JSON response, get common status codes, test auth easily. Helpful when exploring integrations or trying things out.

    Learning HTTP

    httpbin exposes all major features of HTTP protocol like methods, headers, redirection, caching etc. So its a great tool when learning core HTTP concepts or tools like cURL and requests modules in various languages. Lets you try things easily.

    Debugging Issues

    If you face an issues with HTTP requests in your application like headers not reaching, redirects looping, auth failures etc, httpbin can help debug and identify the root cause quickly.

    Documentation Samples

    When writing docs for an API client/library, httpbin provides great sample requests and responses to demonstrate usage. Saves having to mock things up.

    CI/Testing

    httpbin endpoints can be leveraged in CI pipelines and test suits to validate HTTP connectivity and perform protocol level integration tests.

    So basically anytime you need a freely hosted HTTP test server, httpbin delivers that and helps test and debug all components related to HTTP clients and servers.

    How is it useful in web scraping/ web crawling?

    httpbin is extremely helpful when working on web scraping projects. Here are some specific use cases:

    Testing Crawlers & Parsers

    Since httpbin provides endpoints that return HTML, XML, JSON responses - you can leverage it to unit test scrapers, parsers, DOM traversal code without needing a live site. Allows testing core scraping logic in isolation.

    response = requests.get('<http://httpbin.org/html>')
html = response.text

# parse and extract data from html

    Experimenting with Target Sites

    You can shape mock responses similar to the target site you want to scrape using custom headers, status codes, response formats. Helps build robust scrapers.

    headers = {'Content-Type': 'text/html; charset=utf-8'}
response = requests.get('<http://httpbin.org/response-headers>',
                        headers=headers)
html = response.text

    for example, my Chrome produces this…

    Testing Proxy Configurations

    httpbin provides an endpoint /ip that returns Origin IP address. Useful for testing scraping through different proxies to validate they are configured correctly:

    response = requests.get('<http://httpbin.org/ip>', proxies=proxy)
origin_ip = response.json()['origin']
print(origin_ip)

    Debugging Intermittent Issues

    Web scrapers often face intermittent issues like blocked IPs, connectivity issues etc. httpbin provides endpoints to help replicate and tests these scenarios in a controlled way during development.

    Httpbin Functions CheatSheet

    Making Requests

    To make requests to httpbin endpoints, you need to use the requests module:

    import requests

response = requests.get('<https://httpbin.org/get>')
print(response.text)

    /get

    The /get endpoint accepts GET requests and returns the request data in JSON format:

    import requests

data = {'key1': 'value1', 'key2': 'value2'}
response = requests.get('<https://httpbin.org/get>', params=data)
print(response.json())

# Response contains the data dictionary

    You can test adding headers, authentication, proxies and more.

    /post

    The /post endpoint allows you to make POST requests and returns the payload and headers:

    import json
import requests

data = {'name': 'John'}
headers = {'Content-type': 'application/json'}

response = requests.post('<https://httpbin.org/post>',
                         data=json.dumps(data), headers=headers)

print(response.json())

    /status

    The /status endpoint lets you test statuses other than 200 OK:

    import requests

response = requests.get('<https://httpbin.org/status/201>')
print(response.status_code) # 201
print(response.json)

    /response-headers

    The /response-headers endpoint allows you to specify custom response headers:

    import requests

headers = {'custom': 'header'}
response = requests.get('<https://httpbin.org/response-headers>', headers=headers)
print(response.json()['headers']['Custom']) # 'header'

    /basic-auth

    The /basic-auth endpoint allows testing Basic HTTP authentication:

    import requests

response = requests.get('<https://httpbin.org/basic-auth/user/passwd>',
                        auth=('user', 'passwd'))
print(response.json())

    /digest-auth

    The /digest-auth endpoint allows testing Digest HTTP authentication:

    import requests
from requests.auth import HTTPDigestAuth

response = requests.get('<https://httpbin.org/digest-auth/auth/user/passwd>',
                        auth=HTTPDigestAuth('user', 'passwd'))
print(response.status_code)

    /delay

    The /delay endpoint adds a specified delay before responding:

    import requests

response = requests.get('<https://httpbin.org/delay/3>') # delays for 3 seconds
print(response.json())

    /drip

    The /drip endpoint sends partial responses every specified duration:

    import requests

response = requests.get('<https://httpbin.org/drip?duration=5&numbytes=100&code=200>')
for data in response.iter_content(100):
    print(data) # prints 100 bytes every 5 seconds

    This is fun to try in your browser. Notice how the download moves ahead every few seconds

    /range

    The /range endpoint streams Response data allowing testing Range headers:

    import requests

response = requests.get('<https://httpbin.org/range/10>', headers={'Range': 'bytes=0-9'})
print(response.text) # prints first 10 bytes

    /deny

    The /deny endpoint denies you access by sending various status codes for testing :

    import requests

response = requests.get('<https://httpbin.org/deny>')
print(response.status_code) # 401 Unauthorized

    /redirect

    The /redirect endpoint redirects you to another url:

    import requests

response = requests.get('<https://httpbin.org/redirect/1>')
print(response.history[0].status_code) # 302 Found
print(response.url) # Redirected URL

    /relative-redirect

    The /relative-redirect endpoint issues a relative redirect:

    import requests

response = requests.get('<https://httpbin.org/relative-redirect/6>')
print(response.history[0].status_code) # 302 Found
print(response.url) # Current URL after redirect

    /cookies

    The /cookies endpoint sets, deletes and shows cookies:

    import requests

cookies = {'test': 'cookie'}
response = requests.get('<https://httpbin.org/cookies>', cookies=cookies)
print(response.json()) # Contains cookie data

    /set-cookie

    The /set-cookie endpoint allows setting cookies:

    import requests

response = requests.get('<https://httpbin.org/cookies/set?name=value>')
print(response.headers['Set-Cookie'])

    /delete-cookies

    The /delete-cookies endpoint deletes cookies:

    import requests

cookies = {'test': 'cookie'}
response = requests.get('<https://httpbin.org/cookies>', cookies=cookies)
response = requests.get('<https://httpbin.org/cookies/delete?test>')
print(response.json()) # test cookie deleted

    /headers

    The /headers endpoint returns the request headers in a JSON:

    import requests

headers = {'user-agent': 'my-test-app'}
response = requests.get('<https://httpbin.org/headers>', headers=headers)
print(response.json()["headers"])

    /user-agent

    The /user-agent endpoint returns browsers and OS from the User-Agent header:

    import requests

headers = {'user-agent': 'Mozilla/5.0 (Linux; Android 6)'}
response = requests.get('<https://httpbin.org/user-agent>', headers=headers)
print(response.json()["user-agent"]) # Details parsed from header

    /gzip

    The /gzip endpoint returns gzip encoded content:

    import requests

response = requests.get('<https://httpbin.org/gzip>')
print(response.headers["Content-Encoding"]) # gzip
print(response.text) # Decompressed text

    /deflate

    The /deflate endpoint returns deflate encoded content:

    import requests

response = requests.get('<https://httpbin.org/deflate>')
print(response.headers["Content-Encoding"]) # deflate
print(response.text) # Decompressed text

    /brotli

    The /brotli endpoint returns brotli encoded content:

    import requests

response = requests.get('<https://httpbin.org/brotli>')
print(response.headers["Content-Encoding"]) # br
print(response.text) # Decompressed text

    /stream

    The /stream endpoint streams newline delimited JSON data:

    import requests

response = requests.get('<https://httpbin.org/stream/10>', stream=True)
for line in response.iter_lines():
    print(line.decode('utf-8')) # Parses and prints json

    /html

    The /html endpoint returns some HTML content:

    import requests

response = requests.get('<https://httpbin.org/html>')
print(response.text[:15]) # '<!DOCTYPE html>'
print(response.headers["Content-Type"]) # 'text/html'

    /robots.txt

    The /robots.txt endpoint returns example robots.txt restricting all:

    import requests

response = requests.get('<https://httpbin.org/robots.txt>')
print(response.text)

    /deny

    You can test various http response status codes:

    import requests

response = requests.get('<https://httpbin.org/status/404>')
print(response.status_code) # 404

response = requests.get('<https://httpbin.org/status/403>')
print(response.status_code) # 403

response = requests.get('<https://httpbin.org/status/500>')
print(response.status_code) # 500

response = requests.get('<https://httpbin.org/status/504>')
print(response.status_code) # 504

    Response Inspection

    You can inspect the raw response including:

  • Content bytes
  • Encoding
  • Stream state
  • Apparent encoding
  • History
    • import requests

response = requests.get('<https://httpbin.org/get>')
print(response.content) # Raw bytes

print(response.encoding) # Encoding

print(response.is_redirect)  # Is redirect response
print(response.is_permanent_redirect)

print(response.apparent_encoding) # Apparent encoding

print(response.history) # Redirect history

    Advanced Usage

    Authentication

    Different auth protocols require dedicated code flows:

    # Basic Auth
response = requests.get(url, auth=('user', 'pass'))

# Digest Auth
auth = HTTPDigestAuth('user', 'pass')
response = requests.get(url, auth=auth)

# OAuth 1.0
import oauthlib
from requests_oauthlib import OAuth1

oauth = OAuth1(client_key, client_secret, resource_owner_key, resource_owner_secret)
response = requests.get(url, auth=oauth)

    Cookies, Sessions

    Use a session to persist cookies across requests:

    session = requests.Session()
session.get(url1) # cookies saved for url2
resp = session.get(url2)

    Proxies

    Specify HTTP, SOCKS proxies:

    proxies = {'http': '<http://1.2.3.4:5678>'}
requests.get(url, proxies=proxies)

    Use proxy auth:

    proxies = {'http': 'user:pass@1.2.3.4'}
requests.get(url, proxies=proxies)

    Error Handling

    Catch and recover from specific errors:

    try:
    response = requests.get(url)
    response.raise_for_status()
except requests.exceptions.HTTPError as err:
    # Handle HTTP error
except requests.exceptions.Timeout:
    # Retry after timeout

    Set timeouts to avoid hanging:

    requests.get(url, timeout=3)

    Debugging

    Inspect request details:

    import logging
logging.basicConfig(level=logging.DEBUG)

requests.get(url)
# -> logs request method, path, headers, body

    Use a proxy like Fiddler to capture traffic.

    Example Use Cases

    Test API Client Module

    Httpbin provides an excellent way to test API client code without relying on actual production backends. For example:

    import appapi
import requests

# Test authentication
response = requests.get("<https://httpbin.org/basic-auth/user/passwd>",
                        auth=("user", "passwd"))
assert response.status_code == 200

# Mock client module method
appapi.get_user = lambda x: requests.get("<https://httpbin.org/anything>")

response = appapi.get_user(123)
assert b'"httpbin"' in response.content

    Here httpbin acts as a dummy API that can return realistic response codes and payloads to validate the client module works as expected, without needing real implementations on the server side.

    We can take advantage of endpoints like:

  • /basic-auth
  • /bearer
  • /status/200
  • /json
  • /bytes/1024

    • To stand-in for actual APIs the client code will interact with.

    Identify Security Issues

    The httpbin service provides ways to detect security anti-patterns:

    import requests

# Test for SQL injection
payload = "' OR 1=1--"
response = requests.post("<https://httpbin.org/post>",
                         data={"username": payload})

# If API is vulnerable, might see database errors
if b'error' not in response.content:
    print("SQL Injection vulnerability detected")

# Test overflow with long string
long_str = "A"*5000
requests.post("<https://httpbin.org/post>", data=long_str)

    Here we can send intentional malicious payloads designed to trigger exceptions and unstable behavior that indicate problems like SQLi, buffer overflows etc.

    Since httpbin simply echoes back the request, it allows testing the actual API backend in an isolated way.

    Tooling and Libraries

    Requests

    Popular HTTP client with options like:

  • params, json, headers
  • Cookies, auth, timeout
  • Response content, status
  • SSL verify flag
  • Streaming, callbacks

    • Httpx - AsyncRequests analogue

    Features like Requests plus:

  • Async/await support
  • HTTP/2 support
  • Build on ASGI
    • import httpx

async with httpx.AsyncClient() as client:
    resp = await client.get(url)

    PyRESTTest - Declarative REST API Testing

    Test cases specified in YAML:

    - test:
    name: Get user
    url: /api/users
    method: GET
    headers:
        Auth: abc123
    validators:
        - compare: {header: Content-Type, expected: application/json}
        - compare: {jsonpath_mini: .id, expected: 123}

    Similar Services

    Here are some useful links for some services similar to httpbin:

    JSONPlaceholder

    JSONPlaceholder provides a fake REST API for testing without needing real backends.

  • Website: https://jsonplaceholder.typicode.com/
  • Supports standard GET/POST/PUT/DELETE methods
  • Returns placeholder fake data in JSON format
  • Can be used to mock responses during development
  • No setup required, just call API endpoints

    • Example usage:

    <https://jsonplaceholder.typicode.com/posts/1>
# Returns fake post data

    WireMock

    WireMock allows stubbing and mocking web services for testing how code handles different scenarios and responses.

  • Website: http://wiremock.org/
  • Java library that runs as a stand-alone process
  • API for defining request matching and configurable responses
  • Verifies requests received against expectations
  • Records interactions for later inspection
  • Useful for testing edge cases and failures

    • Example usage:

    stubFor(get(urlEqualTo("/api/user"))
    .willReturn(aResponse()
        .withStatus(500))
);

    Mocky

    Mocky provides a simple way to mock APIs by returning static or dynamic JSON data.

  • Website: https://designer.mocky.io/
  • Graphical editor to build responses
  • Support for JSON schema
  • Customize status codes
  • Dynamic values via {{guuid}} syntax
  • Share mock APIs with others
  • Disposable endpoints, no setup

    • Example usage:

    <https://run.mocky.io/v3/{{guuid}>}
# Returns dynamic guid value

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you


    Try ProxiesAPI for free

    curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

    <!doctype html>
    <html>
    <head>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
    ...


    Tired of getting blocked while scraping the web?

    Get access to 1,000 free API credits, no credit card required!

    Try for free


    X

    Don't leave just yet!

    Enter your email below to claim your free API key: