Build a Zero-Trust URL Pipeline for AI Agents

AI agents that fetch external URLs represent a critical attack surface. Without validation at every stage, agents become vectors for data exfiltration, SSRF attacks, and prompt injection. This article outlines a defense-in-depth approach that assumes every input is hostile.

The Core Problem: Implicit Trust

Most agent implementations validate URLs at a single point—often just before the HTTP request. This creates gaps where attackers manipulate URLs through encoding tricks, redirect chains, or DNS rebinding. The 2023 incidents involving major LLM systems demonstrated how a single missed validation step enables zero-click data exfiltration.

The fundamental issue is treating URL validation as a gate rather than a continuous process. Once a URL passes initial checks, subsequent redirects and final destinations often receive minimal scrutiny. Attackers exploit this by starting with benign-looking URLs that redirect to internal endpoints after validation completes.

Layer 1: Parse-Time Validation

Validation must begin the moment your agent receives a URL string, before normalization occurs. This layer catches obvious attacks and establishes baseline checks.

from urllib.parse import urlparse, unquote
from ipaddress import ip_address

def parse_validate(raw_url: str) -> dict:
    DANGEROUS_SCHEMES = {'data', 'javascript', 'file', 'ftp'}

    # Decode to catch encoding tricks
    decoded = unquote(raw_url)
    parsed = urlparse(decoded)

    if parsed.scheme in DANGEROUS_SCHEMES:
        raise ValidationError(f"Scheme '{parsed.scheme}' blocked")

    hostname = parsed.hostname
    if not hostname:
        raise ValidationError("No hostname")

    # Block IP literals
    try:
        addr = ip_address(hostname)
        if addr.is_private or addr.is_loopback:
            raise ValidationError("Internal IPs blocked")
    except ValueError:
        pass

    return {'scheme': parsed.scheme, 'hostname': hostname, 'raw': raw_url}

Key checks: scheme allowlisting, encoding normalization, IP literal detection, and hostname extraction. Eliminate malformed URLs before they enter your pipeline.

Layer 2: DNS-Time Validation

DNS resolution is where SSRF protections often fail. Attackers use short TTLs, CNAME chains, or timing attacks to return different IPs on different lookups.

import dns.resolver
from ipaddress import ip_address, ip_network

def dns_validate(hostname: str, context: dict) -> dict:
    answers = dns.resolver.resolve(hostname, 'A')
    ips = [str(rdata) for rdata in answers]

    BLOCKED = [
        ip_network('10.0.0.0/8'), ip_network('172.16.0.0/12'),
        ip_network('192.168.0.0/16'), ip_network('127.0.0.0/8'),
        ip_network('169.254.0.0/16')
    ]

    for ip_str in ips:
        addr = ip_address(ip_str)
        for network in BLOCKED:
            if addr in network:
                raise ValidationError(f"DNS resolved to internal IP: {ip_str}")

    context['resolved_ips'] = ips
    return context

DNS can change between validation and connection. Re-resolve at connection time and compare against original results—terminate if they differ.

Layer 3: Connection-Time Controls

Even with parse and DNS validation, the HTTP connection requires additional safeguards. Redirects, content-type changes, and response handling each represent escape points.

import requests

def fetch_validated(context: dict) -> bytes:
    session = requests.Session()
    session.max_redirects = 0  # Handle redirects manually

    response = session.get(
        context['raw'],
        stream=True,
        verify=True,
        timeout=(5, 30)
    )

    content_type = response.headers.get('content-type', '').lower()
    ALLOWED = {'text/plain', 'application/json', 'text/html'}

    if not any(ct in content_type for ct in ALLOWED):
        raise ValidationError(f"Content-type '{content_type}' blocked")

    content = response.content[:1024*1024]  # 1MB limit
    return content

Connection controls enforce: redirect handling with re-validation, content-type filtering, size limits, and certificate validation.

Implementation Checklist

Verify each control in your pipeline:

1. Parse-time: Scheme allowlist, encoding normalization, IP literal detection
2. DNS-time: Re-resolve before connection, block internal ranges
3. Connection-time: IP pinning, redirect validation, content-type filtering
4. Response-time: Content sanitization, metadata extraction
5. Audit-logging: Log all validation decisions and resolved IPs

Test against SSRF payloads: IPv6 localhost representations, DNS rebinding domains, URL encoding variations, and redirect chains to internal endpoints.

Conclusion

A zero-trust URL pipeline treats every external request as potentially hostile. Validating at parse, DNS, and connection stages eliminates the single-point-of-failure enabling data exfiltration attacks. The architecture adds complexity but provides defense-in-depth against SSRF and injection vectors. Document your validation stages clearly—if you cannot explain each layer, an attacker will find the gap.