Accessing Array Data in URLs with Python's urllib

Feb 6, 2024 ยท 2 min read

When building web applications in Python, you may need to pass array data in a URL to send it between pages or access it on the server. Python provides the urllib module to encode and decode these URLs containing array data.

To create a URL with an array, you can use the urllib.parse.urlencode() method:

import urllib.parse

my_data = {'an_array': [1, 2, 3]} 
url_str = urllib.parse.urlencode(my_data)
print(url_str)
# an_array%5B%5D=1&an_array%5B%5D=2&an_array%5B%5D=3

This encodes the array into a format that can be passed in a URL with separate key/value pairs for each array element.

On the receiving end, you can use urllib.parse.parse_qs() to decode this back into a Python dictionary with an array:

import urllib.parse

url_str = "an_array%5B%5D=1&an_array%5B%5D=2&an_array%5B%5D=3"
data = urllib.parse.parse_qs(url_str) 
print(data)
# {'an_array': ['1', '2', '3']}

A few things to note:

  • The array indices are omitted in the encoding, so you may lose ordering information
  • Array values are converted to strings, so you may need to cast them back to integers or other types
  • Beware of properly escaping special characters in the array data
  • To preserve more array structure, you could encode the data as JSON instead using the json module. The main advantage of encoding as URL params is that it integrates seamlessly with existing APIs that use this style.

    In summary, Python's urllib provides simple utilities to encode array data into URLs and restore it on the other end. With some care taken in handling the data format changes, this allows easy transmission of array data through web applications.

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you


    Try ProxiesAPI for free

    curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

    <!doctype html>
    <html>
    <head>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
    ...

    X

    Don't leave just yet!

    Enter your email below to claim your free API key: