What does a ceramic mug cost online? It depends where you look: Amazon, AliExpress, eBay, Walmart and Etsy each show different products, different sellers, different shipping terms and very different prices for the same words. Sellers who source or price a product, and anyone building a price comparison page, need those five answers side by side, in the same columns. This tutorial does a price comparison across Amazon, AliExpress, eBay, Walmart and Etsy with Python: one keyword, one run, one table, then the cheapest product on each marketplace and a price summary per marketplace with pandas. The data comes from the Marketplace Price Comparison Actor on Apify, called through the official Python client.
What you need
- An Apify account (the free plan is enough to try this) and its API token, from Apify Console → Settings → API & Integrations, exported as an environment variable:
export APIFY_TOKEN=... - Python 3 and two packages:
pip install "apify-client>=3,<4" pandas
The common table
The Actor searches every marketplace for the same keywords in one run and returns one flat row per product, with the same columns whatever the marketplace:
-
marketplace(amazon,aliexpress,ebay,walmartoretsy),searchQueryandrankInMarketplace, the product's position in that marketplace's best-match order; -
title,url,imageUrl,platformProductId(the marketplace's own id) and, when shown,brandandcategory; -
priceandcurrency,originalPriceanddiscountPercentfor a deal,inStock; -
shippingFree,shippingCost,deliveryEstimate,shipsFrom; -
ratingon 5,reviewCount,soldCount(units sold, where the marketplace publishes it),storeType(platform, business, individual or unknown) andisSponsored.
A value a marketplace does not show is null, so a CSV export keeps the same columns from run to run. Private sellers are never named; review texts and their authors are never collected.
Step 1: the input
run_input = {
"searchQueries": ["ceramic mug"],
"marketplaces": ["amazon", "aliexpress", "ebay", "walmart", "etsy"],
"maxResultsPerMarketplace": 20,
"country": "US",
}
This is the public task "Compare ceramic mug prices on Etsy, eBay and AliExpress", with Amazon and Walmart added. country picks each marketplace's site (amazon.com, ebay.com, walmart.com, etsy.com, AliExpress shipping to the US) and the currency, USD here. maxResultsPerMarketplace caps the products per keyword and marketplace, so this run returns at most 100 rows.
Choose the keyword with Etsy in mind: Etsy only has results for common product words such as "ceramic mug" or "phone case". For "wireless earbuds" or "air fryer", keep the default marketplaces (Amazon, AliExpress, eBay and Walmart). Other useful fields: condition (new or used), and priceMin / priceMax for a price range, applied by each marketplace's filter and checked again on every product.
Step 2: run it and save the table
Save this as compare_prices.py:
import os
import pandas as pd
from apify_client import ApifyClient
run_input = {
"searchQueries": ["ceramic mug"],
"marketplaces": ["amazon", "aliexpress", "ebay", "walmart", "etsy"],
"maxResultsPerMarketplace": 20,
"country": "US",
}
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("myth84/marketplace-price-compare-scraper").call(run_input=run_input)
rows = list(client.dataset(run.default_dataset_id).iterate_items())
for item in rows[:8]:
print(item["marketplace"], item["rankInMarketplace"], item["price"], item["currency"], item["title"][:60])
df = pd.json_normalize(rows)
columns = [
"marketplace", "rankInMarketplace", "title", "price", "currency", "shippingFree", "shippingCost",
"deliveryEstimate", "rating", "reviewCount", "soldCount", "storeType", "isSponsored", "url",
]
df[columns].to_csv("ceramic-mug-5-marketplaces.csv", index=False)
print(df["marketplace"].value_counts())
pd.json_normalize turns each row into one line and flattens attributes, the small object of facts that only some marketplaces show, into columns such as attributes.condition (eBay and Walmart) or attributes.boughtPastMonth (Amazon's "bought in past month", a monthly lower bound). The CSV is the common table: one line per product, whatever the marketplace.
Step 3: the cheapest on each marketplace
Three things to know before calling a product "the cheapest":
-
Welcome deals. AliExpress shows part of its products at a price for first-time buyers only, flagged by
attributes.newShopperDeal. That price is not one anybody can pay, so leave those rows out. -
No featured offer. Amazon shows some products without a price, when no seller has the featured offer:
priceis empty, and the filter below leaves those rows out. -
Shipping.
priceis the displayed price, shipping not included.shippingCostgives the shipping price when the marketplace shows it;shippingFreeis true when shipping is free and empty when the marketplace says nothing.
Add this to the script:
welcome = df.get("attributes.newShopperDeal", pd.Series("false", index=df.index)).eq("true")
anyone = df[~welcome & df["price"].notna() & (df["currency"] == "USD")].copy()
anyone["withShipping"] = anyone["price"] + anyone["shippingCost"].fillna(0)
cheapest = anyone.sort_values("withShipping").groupby("marketplace").head(1)
print(cheapest[["marketplace", "title", "price", "shippingCost", "withShipping", "url"]].to_string(index=False))
summary = anyone.groupby("marketplace").agg(
products=("title", "count"),
lowest=("price", "min"),
median=("price", "median"),
highest=("price", "max"),
medianRating=("rating", "median"),
medianSold=("soldCount", "median"),
)
print(summary.sort_values("median"))
cheapest holds one line per marketplace: the lowest price shipping included, as far as each marketplace shows it, with the link. summary gives, per marketplace, the number of products compared, the lowest, median and highest price, the median rating and the median units sold (empty where the marketplace publishes no sales figure, such as Amazon and Walmart). The median is the number to look at: one very cheap accessory or one premium set can make the lowest and highest misleading, and the median says what the typical product of the first results costs.
The comparison stays within one currency: no exchange rate is applied, which is why the filter keeps the US dollar rows. For the full rows of a marketplace, filter the DataFrame on marketplace and sort it by soldCount or rating.
Several keywords, every day
searchQueries takes several lines; each keyword is compared separately, and searchQuery tells the rows apart. Group by ["searchQuery", "marketplace"] instead of "marketplace" and the summary covers your whole catalogue in one run.
To follow a market over time, run the script every morning with cron and append the summary to a CSV with today's date: the median price of each marketplace, day after day, shows who moves first. Without code, open the public task in Apify Console, change the keyword, save it as your own task and add a daily schedule; the Output tab has a Comparison view of the products and a summary record per keyword and marketplace with the lowest, median and highest price.
Good to know
-
Each marketplace's own best-match order, sponsored products included and flagged in
isSponsored;rankInMarketplaceis the position among the rows of the run. - One marketplace failing does not stop the others: its status and the reason are in the run log and the summary record. The run fails only when no marketplace returned anything.
- Amazon is searched on amazon.com, .co.uk, .ca, .de, .fr, .it and .es, Walmart in the US and Canada only; for another country, the marketplace is skipped and the log says so.
- eBay is searched for fixed-price ("Buy It Now") listings: an auction's current bid is not a price to compare.
- A product found by two of your keywords on the same marketplace appears once, under the first keyword.
- No personal data: products, prices, average ratings and review counts; sellers are named only when they are the platform itself or a registered business.
Pricing is pay-per-event: a fixed price per product row, $2.00 per 1,000 on Apify's Free plan, down to $1.29 on the higher plans, whatever the marketplace (this run of up to 100 rows costs at most $0.20). A marketplace that returns nothing costs nothing. Use the data for analysis, respect each marketplace's terms and local law, and do not republish listings.
Disclosure: I build these Actors and may earn a commission if you sign up to Apify through these links.
Top comments (0)