How to Scrape Amazon or eBay Using Free Proxies

Choosing Your Arsenal: Free Proxies in the Wild

In the digital agora, proxies stand as ephemeral sentinels—gateways to anonymity, freedom, and, alas, fragility. The free proxy, that elusive creature, offers passage but at a price: instability, throttling, or, in the worst scenario, betrayal. Let us examine, with a Cartesian clarity, the landscape:

Warning: Free proxies are public and may be compromised. Never send credentials or sensitive data through them.

Harvesting Proxies: The Ritual

A dance with the ephemeral demands automation. Let us summon Python and its acolytes, requests and BeautifulSoup, to fetch proxies:

import requests
from bs4 import BeautifulSoup

def fetch_proxies():
    url = 'https://free-proxy-list.net/'
    soup = BeautifulSoup(requests.get(url).content, 'html.parser')
    proxies = []
    for row in soup.find('table', id='proxylisttable').tbody.find_all('tr'):
        tds = row.find_all('td')
        if tds[6].text == 'yes':  # HTTPS only
            proxy = f"{tds[0].text}:{tds[1].text}"
            proxies.append(proxy)
    return proxies

Proxies in Rotation: The Art of Disguise

Amazon and eBay, those digital fortresses, wield banhammers with mechanical precision. The solution? Rotate proxies, change user-agents, and inject delays—a choreography of misdirection.

import random
import time

proxies = fetch_proxies()
user_agents = [
    # A bouquet of user-agents
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64)...',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...',
    # Add more
]

def get_random_headers():
    return {'User-Agent': random.choice(user_agents)}

def get_random_proxy():
    return {'http': f"http://{random.choice(proxies)}", 'https': f"http://{random.choice(proxies)}"}

def request_with_proxy(url):
    for attempt in range(5):
        proxy = get_random_proxy()
        headers = get_random_headers()
        try:
            response = requests.get(url, headers=headers, proxies=proxy, timeout=5)
            if response.status_code == 200:
                return response.text
        except Exception:
            continue
        time.sleep(random.uniform(1, 3))
    return None

Scraping Amazon: Navigating the Labyrinth

Amazon weaves anti-bot spells: CAPTCHAs, dynamic content, IP bans. For small-scale scraping, focus on product listings; for anything more, consider ethical limits and legal boundaries.

Example: Extracting Product Titles

from bs4 import BeautifulSoup

def scrape_amazon_product_title(asin):
    url = f"https://www.amazon.com/dp/{asin}"
    html = request_with_proxy(url)
    if not html:
        print("Failed to retrieve page.")
        return None
    soup = BeautifulSoup(html, 'html.parser')
    title = soup.find('span', id='productTitle')
    return title.text.strip() if title else None

asin = 'B08N5WRWNW'  # Example ASIN
print(scrape_amazon_product_title(asin))

Scraping eBay: Through the Bazaar

eBay, a less vigilant sentinel, still employs rate-limiting and bot-detection—less severe, but present. Focus on the item page (e.g., https://www.ebay.com/itm/ITEMID).

Example: Extracting Item Price

def scrape_ebay_price(item_id):
    url = f"https://www.ebay.com/itm/{item_id}"
    html = request_with_proxy(url)
    if not html:
        print("Failed to retrieve page.")
        return None
    soup = BeautifulSoup(html, 'html.parser')
    price = soup.find('span', id='prcIsum')
    return price.text.strip() if price else None

item_id = '234567890123'  # Example Item ID
print(scrape_ebay_price(item_id))

Obfuscation: The Poetry of Evasion

• Randomize request intervals:
  python
time.sleep(random.uniform(2, 6))
• Shuffle proxies and user-agents with each request.
• Pause or switch proxies on HTTP 503, 403, or CAPTCHA detections.

Limits and Legalities:

Best Practices:

• Test proxies for liveness before use (many die within hours).
• Respect robots.txt—do not trespass where forbidden.
• Limit concurrency (avoid thread storms with free proxies).
• Parse gracefully—site layouts mutate like spring undergrowth.

Tools & Libraries:

Sample Proxy Validation Routine:

def validate_proxy(proxy):
    try:
        r = requests.get('https://httpbin.org/ip', proxies={'http': proxy, 'https': proxy}, timeout=3)
        return r.status_code == 200
    except:
        return False

proxies = [p for p in proxies if validate_proxy(p)]

A Final Note on Persistence:

To scrape with free proxies is to chase the horizon—ever-changing, always just out of reach. Rotate, adapt, and never forget that each request is a drop in the ocean of digital commerce. The web is a living thing; treat it as such, and it may yet yield its secrets.