DEV Community

elliott maqy
elliott maqy

Posted on

Price comparison: Amazon, AliExpress, eBay, Walmart, Etsy with Python

What does a ceramic mug cost online? It depends where you look: Amazon, AliExpress, eBay, Walmart and Etsy each show different products, different sellers, different shipping terms and very different prices for the same words. Sellers who source or price a product, and anyone building a price comparison page, need those five answers side by side, in the same columns. This tutorial does a price comparison across Amazon, AliExpress, eBay, Walmart and Etsy with Python: one keyword, one run, one table, then the cheapest product on each marketplace and a price summary per marketplace with pandas. The data comes from the Marketplace Price Comparison Actor on Apify, called through the official Python client.

What you need

  • An Apify account (the free plan is enough to try this) and its API token, from Apify Console → Settings → API & Integrations, exported as an environment variable: export APIFY_TOKEN=...
  • Python 3 and two packages:
pip install "apify-client>=3,<4" pandas
Enter fullscreen mode Exit fullscreen mode

The common table

The Actor searches every marketplace for the same keywords in one run and returns one flat row per product, with the same columns whatever the marketplace:

  • marketplace (amazon, aliexpress, ebay, walmart or etsy), searchQuery and rankInMarketplace, the product's position in that marketplace's best-match order;
  • title, url, imageUrl, platformProductId (the marketplace's own id) and, when shown, brand and category;
  • price and currency, originalPrice and discountPercent for a deal, inStock;
  • shippingFree, shippingCost, deliveryEstimate, shipsFrom;
  • rating on 5, reviewCount, soldCount (units sold, where the marketplace publishes it), storeType (platform, business, individual or unknown) and isSponsored.

A value a marketplace does not show is null, so a CSV export keeps the same columns from run to run. Private sellers are never named; review texts and their authors are never collected.

Step 1: the input

run_input = {
    "searchQueries": ["ceramic mug"],
    "marketplaces": ["amazon", "aliexpress", "ebay", "walmart", "etsy"],
    "maxResultsPerMarketplace": 20,
    "country": "US",
}
Enter fullscreen mode Exit fullscreen mode

This is the public task "Compare ceramic mug prices on Etsy, eBay and AliExpress", with Amazon and Walmart added. country picks each marketplace's site (amazon.com, ebay.com, walmart.com, etsy.com, AliExpress shipping to the US) and the currency, USD here. maxResultsPerMarketplace caps the products per keyword and marketplace, so this run returns at most 100 rows.

Choose the keyword with Etsy in mind: Etsy only has results for common product words such as "ceramic mug" or "phone case". For "wireless earbuds" or "air fryer", keep the default marketplaces (Amazon, AliExpress, eBay and Walmart). Other useful fields: condition (new or used), and priceMin / priceMax for a price range, applied by each marketplace's filter and checked again on every product.

Step 2: run it and save the table

Save this as compare_prices.py:

import os

import pandas as pd
from apify_client import ApifyClient

run_input = {
    "searchQueries": ["ceramic mug"],
    "marketplaces": ["amazon", "aliexpress", "ebay", "walmart", "etsy"],
    "maxResultsPerMarketplace": 20,
    "country": "US",
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("myth84/marketplace-price-compare-scraper").call(run_input=run_input)
rows = list(client.dataset(run.default_dataset_id).iterate_items())

for item in rows[:8]:
    print(item["marketplace"], item["rankInMarketplace"], item["price"], item["currency"], item["title"][:60])

df = pd.json_normalize(rows)
columns = [
    "marketplace", "rankInMarketplace", "title", "price", "currency", "shippingFree", "shippingCost",
    "deliveryEstimate", "rating", "reviewCount", "soldCount", "storeType", "isSponsored", "url",
]
df[columns].to_csv("ceramic-mug-5-marketplaces.csv", index=False)
print(df["marketplace"].value_counts())
Enter fullscreen mode Exit fullscreen mode

pd.json_normalize turns each row into one line and flattens attributes, the small object of facts that only some marketplaces show, into columns such as attributes.condition (eBay and Walmart) or attributes.boughtPastMonth (Amazon's "bought in past month", a monthly lower bound). The CSV is the common table: one line per product, whatever the marketplace.

Step 3: the cheapest on each marketplace

Three things to know before calling a product "the cheapest":

  • Welcome deals. AliExpress shows part of its products at a price for first-time buyers only, flagged by attributes.newShopperDeal. That price is not one anybody can pay, so leave those rows out.
  • No featured offer. Amazon shows some products without a price, when no seller has the featured offer: price is empty, and the filter below leaves those rows out.
  • Shipping. price is the displayed price, shipping not included. shippingCost gives the shipping price when the marketplace shows it; shippingFree is true when shipping is free and empty when the marketplace says nothing.

Add this to the script:

welcome = df.get("attributes.newShopperDeal", pd.Series("false", index=df.index)).eq("true")
anyone = df[~welcome & df["price"].notna() & (df["currency"] == "USD")].copy()
anyone["withShipping"] = anyone["price"] + anyone["shippingCost"].fillna(0)

cheapest = anyone.sort_values("withShipping").groupby("marketplace").head(1)
print(cheapest[["marketplace", "title", "price", "shippingCost", "withShipping", "url"]].to_string(index=False))

summary = anyone.groupby("marketplace").agg(
    products=("title", "count"),
    lowest=("price", "min"),
    median=("price", "median"),
    highest=("price", "max"),
    medianRating=("rating", "median"),
    medianSold=("soldCount", "median"),
)
print(summary.sort_values("median"))
Enter fullscreen mode Exit fullscreen mode

cheapest holds one line per marketplace: the lowest price shipping included, as far as each marketplace shows it, with the link. summary gives, per marketplace, the number of products compared, the lowest, median and highest price, the median rating and the median units sold (empty where the marketplace publishes no sales figure, such as Amazon and Walmart). The median is the number to look at: one very cheap accessory or one premium set can make the lowest and highest misleading, and the median says what the typical product of the first results costs.

The comparison stays within one currency: no exchange rate is applied, which is why the filter keeps the US dollar rows. For the full rows of a marketplace, filter the DataFrame on marketplace and sort it by soldCount or rating.

Several keywords, every day

searchQueries takes several lines; each keyword is compared separately, and searchQuery tells the rows apart. Group by ["searchQuery", "marketplace"] instead of "marketplace" and the summary covers your whole catalogue in one run.

To follow a market over time, run the script every morning with cron and append the summary to a CSV with today's date: the median price of each marketplace, day after day, shows who moves first. Without code, open the public task in Apify Console, change the keyword, save it as your own task and add a daily schedule; the Output tab has a Comparison view of the products and a summary record per keyword and marketplace with the lowest, median and highest price.

Good to know

  • Each marketplace's own best-match order, sponsored products included and flagged in isSponsored; rankInMarketplace is the position among the rows of the run.
  • One marketplace failing does not stop the others: its status and the reason are in the run log and the summary record. The run fails only when no marketplace returned anything.
  • Amazon is searched on amazon.com, .co.uk, .ca, .de, .fr, .it and .es, Walmart in the US and Canada only; for another country, the marketplace is skipped and the log says so.
  • eBay is searched for fixed-price ("Buy It Now") listings: an auction's current bid is not a price to compare.
  • A product found by two of your keywords on the same marketplace appears once, under the first keyword.
  • No personal data: products, prices, average ratings and review counts; sellers are named only when they are the platform itself or a registered business.

Pricing is pay-per-event: a fixed price per product row, $2.00 per 1,000 on Apify's Free plan, down to $1.29 on the higher plans, whatever the marketplace (this run of up to 100 rows costs at most $0.20). A marketplace that returns nothing costs nothing. Use the data for analysis, respect each marketplace's terms and local law, and do not republish listings.

Disclosure: I build these Actors and may earn a commission if you sign up to Apify through these links.

Top comments (0)