HTTP Requests with Python: GET, POST, Headers, Parameters, and JSON

ByJu Yeon Eum

Data engineers and analysts frequently need to retrieve data from websites, APIs, and web services. Python’s requests library provides a convenient interface for communicating with these services through HTTP.

This article introduces:

  • The basic structure of HTTP communication
  • Sending GET and POST requests
  • Adding query parameters
  • Inspecting request and response headers
  • Processing text, binary, and JSON responses
  • Sending form data and JSON payloads
  • Handling timeouts and HTTP errors
  • Reusing connections with sessions

What Is HTTP?

HTTP, or Hypertext Transfer Protocol, defines how clients and servers exchange information over the web.

An HTTP interaction normally consists of two parts:

  1. A client sends an HTTP request.
  2. A server returns an HTTP response.

For example, a Python program might request weather data from an API. The API server processes the request and returns a JSON response containing the requested information.

An HTTP request can contain:

  • A request method, such as GET or POST
  • A URL
  • Headers
  • Query parameters
  • An optional request body

The response can contain:

  • A status code
  • Response headers
  • Text, JSON, binary data, or another response body

The Python Requests Library

Requests is a third-party Python library that provides a straightforward interface for sending HTTP requests.

It is not part of Python’s standard library. Install it with:

python -m pip install requests

Then import it into Python:

import requests

Python also includes standard-library HTTP tools such as urllib.request and http.client. Requests is often preferred because it provides a more concise and readable interface.

Sending a Basic GET Request

A GET request is normally used to retrieve a resource from a server.

import requests

url = "https://www.ibm.com/"

response = requests.get(url, timeout=10)
response.raise_for_status()

print(response.status_code)

The requests.get() function sends the request and returns a Response object.

The timeout argument prevents the program from waiting indefinitely. Requests does not apply a timeout unless one is explicitly supplied.

The raise_for_status() method raises an exception when the server returns an unsuccessful HTTP status such as 404 Not Found or 500 Internal Server Error.

Understanding HTTP Status Codes

The response status code indicates the outcome of the request.

print(response.status_code)

Common status-code groups include:

RangeMeaning
200–299Successful response
300–399Redirection
400–499Client-side error
500–599Server-side error

Some common individual codes are:

Status codeMeaning
200OK
201Created
204No Content
301Moved Permanently
400Bad Request
401Unauthorized
403Forbidden
404Not Found
429Too Many Requests
500Internal Server Error
503Service Unavailable

A 200 response is common, but it is not the only successful status code.

Inspecting the Final URL

The response object records the final URL associated with the request:

print(response.url)

This can be useful when the server redirects the request or when Requests adds encoded query parameters to the URL.

Request Headers and Response Headers

Request headers contain information sent from the client to the server.

print(response.request.headers)

Response headers contain information returned by the server.

print(response.headers)

The two should not be confused:

# Headers sent to the server
request_headers = response.request.headers

# Headers returned by the server
response_headers = response.headers

Requests stores response headers in a case-insensitive dictionary-like object. For example:

content_type = response.headers.get("Content-Type")
print(content_type)

A content type might look like:

text/html; charset=UTF-8

or:

application/json

The Date response header, when present, generally represents the server’s response origination time. It does not represent the exact time at which the client sent the request.

Examining the Request Body

The outgoing request body is available through:

print(response.request.body)

A conventional GET request generally does not include a body, so this will normally return:

None

Technically, HTTP does not make it impossible for a GET request to contain content. However, GET request content has no generally defined semantics and may be rejected by some systems. Data used to refine a GET request should normally be supplied through query parameters. This behavior is described in the current HTTP Semantics specification.

Reading a Text Response

When the response contains text, use the text attribute:

html = response.text

print(html[:200])

Requests determines an encoding and decodes the response into a Python string.

The selected encoding can be inspected with:

print(response.encoding)

If the server declares the wrong encoding, it can be changed before accessing response.text:

response.encoding = "utf-8"
html = response.text

Reading Binary Content

Use response.content when the response contains raw bytes:

binary_data = response.content

print(type(binary_data))

This is appropriate for content such as:

  • Images
  • PDF documents
  • ZIP archives
  • Audio files
  • Other non-text resources

The difference is:

response.text       # Decoded Python string
response.content    # Raw bytes

Sending Query Parameters with GET

APIs frequently use query parameters to filter or customize a response.

A URL containing query parameters might look like:

https://httpbin.org/get?name=Joseph&ID=123

The query string begins with ?. Each parameter is represented as a name-value pair, and multiple pairs are separated with &.

Instead of constructing this string manually, pass a dictionary to the params argument:

import requests

url = "https://httpbin.org/get"

parameters = {
    "name": "Joseph",
    "ID": "123"
}

response = requests.get(
    url,
    params=parameters,
    timeout=10
)

response.raise_for_status()

print(response.url)

The resulting URL will resemble:

https://httpbin.org/get?name=Joseph&ID=123

Requests automatically performs URL encoding. This is safer and more reliable than joining strings manually.

For example:

parameters = {
    "search": "data engineering",
    "page": 2
}

response = requests.get(
    "https://httpbin.org/get",
    params=parameters,
    timeout=10
)

print(response.url)

The space in data engineering is encoded appropriately.

The Requests documentation provides additional examples of passing parameters in URLs.

Processing a JSON Response

Many APIs return data in JSON format.

Use response.json() to parse the response into a Python object:

data = response.json()

print(type(data))
print(data)

For the httpbin.org/get endpoint, query parameters are available under the args key:

print(data["args"])

Expected result:

{
    "ID": "123",
    "name": "Joseph"
}

Before assuming that a response contains JSON, it can be useful to inspect its content type:

content_type = response.headers.get("Content-Type", "")

if "application/json" in content_type:
    data = response.json()

However, the content type alone is not a guarantee that the body contains valid JSON. JSON parsing can still fail if the server returns malformed content.

Also note that successfully parsing JSON does not mean the HTTP request succeeded. Check the status with raise_for_status() first.

Sending a POST Request

POST requests are commonly used to submit data to a server.

Unlike a conventional GET request, POST data is usually placed in the request body.

Sending Form Data

Pass a dictionary to the data argument to send form-encoded data:

import requests

url = "https://httpbin.org/post"

payload = {
    "name": "Joseph",
    "ID": "123"
}

response = requests.post(
    url,
    data=payload,
    timeout=10
)

response.raise_for_status()

print(response.url)
print(response.request.body)

The URL normally does not contain the form values because they were placed in the request body.

The test service returns the submitted form data under the form key:

result = response.json()

print(result["form"])

Expected result:

{
    "ID": "123",
    "name": "Joseph"
}

Sending JSON Data with POST

Modern APIs frequently expect JSON rather than form-encoded data.

Use the json argument:

payload = {
    "name": "Joseph",
    "ID": 123
}

response = requests.post(
    "https://httpbin.org/post",
    json=payload,
    timeout=10
)

response.raise_for_status()

result = response.json()

print(result["json"])

Using json=payload tells Requests to:

  • Serialize the Python object as JSON
  • Place it in the request body
  • Set an appropriate JSON content type

This is usually preferable to manually calling json.dumps() and setting the header yourself.

Form Data and JSON Are Different

The following requests do not send the same representation:

requests.post(url, data=payload, timeout=10)
requests.post(url, json=payload, timeout=10)

The first normally sends:

application/x-www-form-urlencoded

The second normally sends:

application/json

Use the format required by the API documentation.

POST Requests Can Also Have Query Parameters

POST data is usually sent in the body, but a POST URL can still contain query parameters.

response = requests.post(
    "https://httpbin.org/post",
    params={"source": "tutorial"},
    json={"name": "Joseph"},
    timeout=10
)

response.raise_for_status()

print(response.url)

This request has:

  • source=tutorial in the URL
  • {"name": "Joseph"} in the request body

Therefore, GET versus POST should not be understood simply as “URL data versus body data.” Their primary difference is their HTTP semantics and intended use.

GET and POST Compared

FeatureGETPOST
Typical purposeRetrieve a resourceSubmit or process data
ParametersCommonly in the URLCan be in the URL or body
Request bodyUsually omittedCommonly used
Safe operationIntended to be safeMay change server state
Idempotent semanticsYesNot necessarily
Common API useSearch and retrievalCreation, submission, actions

A GET request should not normally change server data. Repeating a POST request, however, may create duplicate records or repeat an operation unless the API provides idempotency protection.

Adding Custom Headers

HTTP headers can specify the expected response format, identify the client, or provide authentication credentials.

headers = {
    "Accept": "application/json",
    "User-Agent": "DataSphere-Tutorial/1.0"
}

response = requests.get(
    "https://httpbin.org/headers",
    headers=headers,
    timeout=10
)

response.raise_for_status()

print(response.json())

An API token might be provided using an authorization header:

headers = {
    "Authorization": f"Bearer {access_token}",
    "Accept": "application/json"
}

Do not hardcode production credentials directly into source code. Use environment variables or an appropriate secrets-management system.

Sensitive credentials should generally not be placed in URL query parameters because URLs can appear in logs, browser histories, analytics systems, and monitoring tools.

Handling Errors and Timeouts

Production code should account for network errors, timeouts, invalid responses, and unsuccessful HTTP status codes.

import requests

def fetch_json(url, params=None):
    try:
        response = requests.get(
            url,
            params=params,
            timeout=(3.05, 15)
        )

        response.raise_for_status()
        return response.json()

    except requests.exceptions.Timeout:
        print("The request timed out.")

    except requests.exceptions.HTTPError as error:
        print(f"HTTP error: {error}")

    except requests.exceptions.JSONDecodeError:
        print("The response did not contain valid JSON.")

    except requests.exceptions.RequestException as error:
        print(f"Request failed: {error}")

    return None

The timeout tuple represents:

timeout=(connection_timeout, read_timeout)

In this example:

  • Requests has approximately 3.05 seconds to establish a connection.
  • It has 15 seconds to wait between response data operations.

A timeout is not necessarily a limit on the total duration of the complete request.

Reusing Connections with a Session

When making several requests to the same service, use a Session.

import requests

with requests.Session() as session:
    session.headers.update({
        "Accept": "application/json",
        "User-Agent": "DataSphere-Tutorial/1.0"
    })

    first_response = session.get(
        "https://httpbin.org/get",
        params={"page": 1},
        timeout=10
    )

    second_response = session.get(
        "https://httpbin.org/get",
        params={"page": 2},
        timeout=10
    )

    first_response.raise_for_status()
    second_response.raise_for_status()

Sessions can provide:

  • Connection pooling
  • Shared headers
  • Cookie persistence
  • Shared authentication configuration
  • Better performance for repeated requests

Additional configuration options are documented in the Requests guide to session objects.

A Reusable API Client Example

The following function combines several recommended practices:

import requests

def get_api_data(base_url, parameters=None):
    headers = {
        "Accept": "application/json",
        "User-Agent": "DataSphere-Client/1.0"
    }

    try:
        response = requests.get(
            base_url,
            params=parameters,
            headers=headers,
            timeout=(3.05, 15)
        )

        response.raise_for_status()
        return response.json()

    except requests.exceptions.Timeout:
        raise RuntimeError("The API request timed out.")

    except requests.exceptions.HTTPError as error:
        raise RuntimeError(
            f"The API returned an HTTP error: {error}"
        ) from error

    except requests.exceptions.JSONDecodeError as error:
        raise RuntimeError(
            "The API returned invalid JSON."
        ) from error

    except requests.exceptions.RequestException as error:
        raise RuntimeError(
            f"The API request failed: {error}"
        ) from error


parameters = {
    "name": "Joseph",
    "ID": "123"
}

result = get_api_data(
    "https://httpbin.org/get",
    parameters
)

print(result["args"])

The httpbin.org service is useful for demonstrations and testing. Sensitive or confidential information should not be sent to a public testing service.

Practical Reliability and Security Guidelines

When using HTTP requests in a data pipeline:

  • Use HTTPS whenever possible.
  • Always specify a timeout.
  • Call raise_for_status() before processing the response.
  • Validate the expected content type.
  • Handle JSON parsing errors.
  • Do not place secrets in query parameters.
  • Do not hardcode API keys in source code.
  • Respect API rate limits.
  • Use pagination when retrieving large datasets.
  • Retry only appropriate operations.
  • Apply backoff when handling temporary failures.
  • Log request metadata without exposing credentials.
  • Use sessions for repeated requests.
  • Validate downloaded or externally supplied data before processing it.

Be especially careful when retrying POST requests. Repeating a non-idempotent POST operation can create duplicate records or execute the same action multiple times.

Key Takeaways

  • Requests is a third-party Python library for working with HTTP services.
  • GET is generally used to retrieve resources.
  • POST is commonly used to submit or process data.
  • Query parameters should be passed through the params argument.
  • Form data is sent with data.
  • JSON request bodies are sent with json.
  • Request headers and response headers are separate.
  • response.text returns decoded text.
  • response.content returns raw bytes.
  • response.json() parses JSON into a Python object.
  • raise_for_status() detects unsuccessful HTTP responses.
  • Every request should have a timeout.
  • Sessions improve repeated communication with the same server.

Conclusion

Python’s Requests library provides an accessible way to communicate with APIs and other HTTP services. A basic request may require only one line of code, but reliable data engineering applications must also handle parameters, headers, response formats, timeouts, unsuccessful status codes, authentication, and security.

Understanding these elements makes it possible to build safer and more dependable API clients, ingestion scripts, and data pipelines.

One-sentence summary: Python’s Requests library enables applications to retrieve and submit web data through HTTP while providing convenient support for query parameters, headers, JSON, error handling, and connection management.

Similar Posts

Questions, corrections, or additional insights?