Amazon Scraping: What the Rules Actually Allow

Amazon scraping sits between two documents that say different things. The robots.txt file leaves product pages open to general crawlers and closes the offer lists, wishlists, and availability pages. The Conditions of Use, in the edition dated August 14, 2026, grant a license that excludes data mining and robots outright. Nothing in between is secret: the boundary is written down, and Amazon enforces it technically, with captcha, IP blocks, and account blocks. I build scrapers that work inside this boundary, the Amazon scraping to order service, and this article is the map of the rules themselves, checked against the live robots.txt and the current terms.

Two documents with different force

People asking whether Amazon scraping is allowed are usually holding three documents in one question: the law, the terms of use, and robots.txt. They are not the same thing.

• The law sets the outer frame, and it varies by country. This article does not issue legal verdicts. The method-and-data-type frame that decides the question is mapped in Is web scraping legal in 2026?.
• The Conditions of Use are a contract between Amazon and the visitor. Break them and the site’s response is technical and immediate: captcha, then IP and account blocks.
• robots.txt is a technical instruction to crawlers. It says which paths the site asks general crawlers not to request. It is a courtesy convention, but a scraper that respects it is a scraper that lives longer, and it is the document Amazon actually keeps precise about paths.

A workable project reads all three and fixes its own red lines before the first request: public pages only, no bypassing of logins or technical protections, no personal data from reviews, no load that outweighs the result.

What Amazon’s robots.txt actually allows

The live file at amazon.com, checked again for this article, addresses the wildcard section to all general crawlers. The paths that matter for product data:

Path	Rule for general crawlers	What it means in practice
/dp/<ASIN> product pages	allowed	prices, stock signals, ratings, review counts sit on open pages
/gp/offer-listing/	disallowed	the full list of seller offers per product is closed, with two narrow exceptions
/gp/product/product-availability	disallowed	the dedicated availability page is closed
/wishlist/, /gp/registry/wishlist/	disallowed	wishlists are personal, not market data
/gp/cdp/member-reviews/, review RSS feeds	disallowed	profile-level review feeds are closed

So the product page itself is the open surface. Price, availability signal, rating, review text visible on the page, the Buy Box offer with its price and fulfillment channel, all of it renders for a plain visitor without an account. The offer list behind the page, every seller competing on one ASIN, is the part robots.txt closes, outside the two legacy Allow exceptions for addresses starting with B000 and 9000.

This split is not an accident. Amazon keeps the surface a buyer needs open, and closes the bulk-collection surfaces that exist purely for automation. A scraper built around open product pages by ASIN list works inside that intent.

What the Conditions of Use say

The terms, edition of August 14, 2026, put the restriction inside the license language. The original wording:

“This license does not include any collection and use of any product listings, descriptions, or prices; any use of data mining, robots, or similar data gathering and extraction tools.”

Read literally, that sentence covers every automated read of a product page, including the ones robots.txt leaves open. This is the actual tension in Amazon scraping: the two documents disagree, one permits what the other prohibits. What resolves it in practice is not a court ruling but Amazon’s own behavior. The enforcement Amazon describes and applies is technical: captcha challenges, IP blocks, account blocks. A project that treats the terms as the strict document and robots.txt as the minimum floor, that stays on public pages, keeps a human-like pace, and skips anything behind a login, is running the line most working scrapers actually hold.

Two things I do not do regardless of documents: bypass technical protections, captcha solving farms and header spoofing races, and collect personal data from reviews. Both convert a data project into a different kind of project.

Is there an official API?

Yes, and it is worth knowing what it does not do. The official channel for Amazon product data is the Creators API, aimed at publishers, authors, and partners of the affiliate program. The Product Advertising API 5.0 that preceded it is deprecated: the documentation now points to the Creators API REST interface, and calls to the old endpoint return HTTP 403 with a pointer to the migration guide. Its rate limits sit behind a login, so there are no public numbers to quote here.

The boundary of the official channel is the affiliate catalog. If your question is “what does this product cost and what do I earn linking to it”, the API answers. If your question is a market dataset, every ASIN in a category with competitor prices, offer counts, and who holds the Buy Box on each round, the API does not answer it. That dataset is what a scraper over public pages produces: a fixed field list per ASIN, collected on a schedule, each round stamped and comparable with the previous one.

The Buy Box is the field that decides money for sellers. When several sellers offer one ASIN, Amazon places one of them in the main buy block, and the placement can flip. A dataset that records who held the box, at what price, through which fulfillment channel, on every round, is how a seller learns about the flip on the day it happens. I run exactly this kind of collection, on my list of ASINs or yours, as part of web scraping to order, with the general scraping line collected under web scraping services.

How to stay inside the line

The rules translate into a working checklist, fixed before the first request:

1. Fix the field list and the ASIN list first. Price, stock signal, rating, review count, Buy Box owner, fulfillment channel. A scraper without a fixed field list collects noise.
2. Stay on public pages. Anything behind a login is out of scope, no exceptions.
3. Respect the robots.txt floor. Open /dp/ pages yes, offer lists and the closed paths no.
4. Pace like a person. Delays between requests, a restrained total load, and a schedule instead of one giant run. Amazon’s anti-bot answers volume, and a scraper that behaves like a catalog browser gets treated like one.
5. Skip personal data. Review text tied to a product is market data. Reviewer profiles and identities are not collected.
6. Stamp every round. A dated row per ASIN per round is what turns a scrape into a dataset, and it is what makes price-drop and Buy Box-flip alerts possible.

Run this way, the output is a table that grows more valuable every round: CSV, Google Sheets, or PostgreSQL, with alerts when a price moves past a threshold or the Buy Box flips to a competitor. The same shape works for competitor price monitoring across any set of sites.

Common mistakes

• Scraping everything reachable instead of the fixed field list. The export becomes landfill no analyst opens twice.
• One heroic run instead of a schedule. The data is stale between runs, which cancels the point of collecting it.
• No pacing. Request volume is the fastest way to meet Amazon’s captcha and IP blocks.
• Ignoring the offer list boundary in robots.txt, then wondering why the scraper’s IP pool burns through.
• Reading reviews for identities. Profiles are personal data, the review text on the product page is the market signal.
• Checking the table by hand. A threshold alert exists precisely so nobody spends an evening diffing columns.

The short version

Amazon’s robots.txt keeps product pages open and closes offer lists, wishlists, and availability pages. The Conditions of Use, in the August 14, 2026 edition, exclude data mining and robots from the license they grant. The response to crossing the line is technical, captcha and blocks, and the working practice is to hold both documents: treat the terms as the strict rule, robots.txt as the floor, keep to public pages, pace the requests, and collect a fixed field list on a schedule. The official Creators API serves the affiliate catalog, and a market dataset with competitor prices and Buy Box owners comes from public pages, inside that boundary.