Working with JSON and APIs

Modern applications often need to exchange data with web services and APIs. JSON (JavaScript Object Notation) has become the standard format for data exchange, and Python makes working with JSON and APIs straightforward.

In this lesson, you'll learn how to parse and create JSON data, make HTTP requests using the requests library, interact with REST APIs, and handle errors gracefully. These skills are essential for building applications that integrate with web services, fetch data from APIs, and communicate with other systems.

What You'll Learn

JSON format and structure
Reading and writing JSON files
Parsing JSON data
Making HTTP requests (requests library)
Working with REST APIs
Error handling for API calls

Understanding JSON

JSON is a lightweight data format that's easy for humans to read and machines to parse:

import json

# JSON string
json_string = '{"name": "Alice", "age": 25, "city": "London"}'

# Parse JSON string to Python dict
data = json.loads(json_string)
print(data)  # {'name': 'Alice', 'age': 25, 'city': 'London'}
print(data["name"])  # Alice

# Convert Python object to JSON string
person = {"name": "Bob", "age": 30, "city": "Manchester"}
json_output = json.dumps(person)
print(json_output)  # {"name": "Bob", "age": 30, "city": "Manchester"}

Working with JSON Files

Read and write JSON files:

import json

# Write to JSON file
data = {
    "users": [
        {"name": "Alice", "age": 25},
        {"name": "Bob", "age": 30}
    ]
}

with open("data.json", "w") as file:
    json.dump(data, file, indent=2)  # indent for pretty printing

# Read from JSON file
with open("data.json", "r") as file:
    loaded_data = json.load(file)
    print(loaded_data)

Making HTTP Requests

The requests library makes HTTP requests simple:

import requests

# Install with: pip install requests

# GET request
response = requests.get("https://api.github.com")
print(response.status_code)  # 200
print(response.json())       # Response as JSON

# GET with parameters
params = {"q": "python", "page": 1}
response = requests.get("https://api.github.com/search/repositories", params=params)
data = response.json()
print(data["total_count"])  # Number of results

Basic API Interaction

Here's a complete example of API interaction:

import requests
import json

# Example: GitHub API
def get_user_info(username):
    """Get GitHub user information."""
    url = f"https://api.github.com/users/{username}"
    response = requests.get(url)

    if response.status_code == 200:
        return response.json()
    else:
        return None

user = get_user_info("octocat")
if user:
    print(f"Name: {user.get('name')}")
    print(f"Followers: {user.get('followers')}")
    print(f"Public repos: {user.get('public_repos')}")

POST Requests

Send data to APIs:

import requests
import json

# POST request with JSON data
url = "https://httpbin.org/post"
data = {
    "name": "Alice",
    "age": 25
}

response = requests.post(url, json=data)
print(response.status_code)  # 200
print(response.json())

# POST with form data
form_data = {"username": "alice", "password": "secret"}
response = requests.post(url, data=form_data)
print(response.json())

Error Handling

Handle API errors gracefully:

import requests

def safe_api_call(url):
    """Make API call with error handling."""
    try:
        response = requests.get(url, timeout=5)
        response.raise_for_status()  # Raises exception for bad status codes
        return response.json()
    except requests.exceptions.Timeout:
        print("Request timed out")
        return None
    except requests.exceptions.ConnectionError:
        print("Connection error")
        return None
    except requests.exceptions.HTTPError as e:
        print(f"HTTP error: {e}")
        return None
    except requests.exceptions.RequestException as e:
        print(f"Request error: {e}")
        return None

data = safe_api_call("https://api.github.com/users/octocat")

Headers and Authentication

Many APIs require headers or authentication:

import requests

# Custom headers
headers = {
    "User-Agent": "MyApp/1.0",
    "Accept": "application/json"
}
response = requests.get("https://api.github.com", headers=headers)

# API key authentication
api_key = "your-api-key-here"
headers = {"Authorization": f"Bearer {api_key}"}
response = requests.get("https://api.example.com/data", headers=headers)

# Basic authentication
from requests.auth import HTTPBasicAuth
response = requests.get(
    "https://api.example.com/data",
    auth=HTTPBasicAuth("username", "password")
)

Practical Examples

Here are real-world API examples:

# Example 1: Weather API
import requests

def get_weather(city, api_key):
    """Get weather data for a city."""
    url = f"http://api.openweathermap.org/data/2.5/weather"
    params = {
        "q": city,
        "appid": api_key,
        "units": "metric"
    }

    try:
        response = requests.get(url, params=params)
        response.raise_for_status()
        data = response.json()
        return {
            "city": data["name"],
            "temp": data["main"]["temp"],
            "description": data["weather"][0]["description"]
        }
    except requests.exceptions.RequestException as e:
        print(f"Error fetching weather: {e}")
        return None

# Example 2: REST API client class
class APIClient:
    def __init__(self, base_url, api_key=None):
        self.base_url = base_url
        self.api_key = api_key
        self.session = requests.Session()
        if api_key:
            self.session.headers.update({"Authorization": f"Bearer {api_key}"})

    def get(self, endpoint, params=None):
        """Make GET request."""
        url = f"{self.base_url}/{endpoint}"
        response = self.session.get(url, params=params)
        response.raise_for_status()
        return response.json()

    def post(self, endpoint, data=None):
        """Make POST request."""
        url = f"{self.base_url}/{endpoint}"
        response = self.session.post(url, json=data)
        response.raise_for_status()
        return response.json()

# Using the client
client = APIClient("https://api.example.com", api_key="secret")
users = client.get("users")
new_user = client.post("users", data={"name": "Alice"})

# Example 3: Fetching and processing JSON data
import requests
import json

def fetch_and_save_data(url, filename):
    """Fetch data from API and save to file."""
    try:
        response = requests.get(url)
        response.raise_for_status()
        data = response.json()

        with open(filename, "w") as file:
            json.dump(data, file, indent=2)

        return data
    except requests.exceptions.RequestException as e:
        print(f"Error: {e}")
        return None

# Example 4: Paginated API requests
def fetch_all_pages(base_url, max_pages=10):
    """Fetch all pages from a paginated API."""
    all_data = []
    page = 1

    while page <= max_pages:
        url = f"{base_url}?page={page}"
        try:
            response = requests.get(url)
            response.raise_for_status()
            data = response.json()

            if not data:  # No more data
                break

            all_data.extend(data)
            page += 1
        except requests.exceptions.RequestException as e:
            print(f"Error on page {page}: {e}")
            break

    return all_data

Try It Yourself

Practice working with JSON and APIs:

JSON File Manager: Create a class that reads, writes, and updates JSON configuration files.
API Wrapper: Create a wrapper class for a public API (like GitHub, Reddit, or a weather API) with methods for common operations.
Data Fetcher: Write a script that fetches data from multiple API endpoints, combines the results, and saves to a JSON file.
Error Handler: Create a robust API client with retry logic, timeout handling, and comprehensive error messages.
JSON Validator: Write a function that validates JSON structure against expected schema and provides helpful error messages.

Summary

JSON is the standard format for data exchange, and Python's json module makes working with it straightforward. The requests library simplifies HTTP requests and API interactions. Understanding how to parse JSON, make API calls, handle errors, and work with authentication is essential for modern Python applications.

These skills enable you to build applications that integrate with web services, fetch real-time data, and communicate with other systems. As you continue learning, you'll find APIs are everywhere in modern software development.

What's Next?

In the next lesson, we'll explore testing in Python. You'll learn how to write unit tests using the unittest framework, test functions and classes, and understand test-driven development basics. Testing is crucial for building reliable, maintainable code.