The question
If you write anything that reads a product page — a price tracker, a wishlist, a comparison tool, a personal script — you eventually discover that some shops will not let you. What is rarely written down is which ones, how they refuse, and whether the refusal is consistent.
So on 25 October 2026 I measured twenty of India’s largest online shops and wrote down exactly what came back. The numbers below are the whole result, including the parts that complicate the story.
Method
One HTTP GET per shop, to the homepage, following redirects, with a 25-second timeout. A current Chrome User-Agent and an ordinary HTML Accept header. Node’s fetch, so Node’s TLS fingerprint — not a real browser’s. From residential broadband in India, not a data centre. Requests were paced roughly a second apart; this was twenty requests, not a crawl.
Responses were bucketed by status and body size. Anything under ten kilobytes with a 2xx status counts as a stub rather than a page, which is the same threshold the tool this came from uses in production.
Two things that follow from this, and both cut against over-reading the result. The homepage is not a product page, and a shop may well guard the two differently. And a residential IP is the easy case — the interesting failures happen from hosted infrastructure, which is where anything you deploy will actually run.
A third thing emerged from re-running it, and it is the most useful caveat of the three: the measurement perturbs what it measures. Bewakoof answered the first request with a 250-kilobyte page. After four requests across twenty minutes it began returning 83 bytes of JSON reading “Access denied”. Nothing about the client changed; only the rate did. So a figure recorded here describes a shop’s response to an occasional visitor, and says nothing about what it does to something polling it.
The result
Eleven of twenty returned a page with readable content. Four returned a page with none. Three refused outright, one returned a stub, and one would not answer at all.
The split between the first two is the finding, and I got it wrong on the first pass. Counting bytes alone, fifteen shops looked readable. Counting the text a reader would actually see — scripts and markup stripped out — four of those fifteen collapse. Byte size measures the response. It does not measure the content, and on a modern retail site those are no longer the same question.
- Readable — Flipkart 1,292,425 bytes / 22,267 visible characters · Myntra 448,381 / 20,616 · Pepperfry 650,900 / 15,668 · FirstCry 136,464 / 11,341 · Nykaa Fashion 1,130,447 / 10,992 · Lifestyle 592,679 / 5,805 · Westside 650,715 / 4,857 · Snapdeal 108,034 / 4,251 · Purplle 202,118 / 2,274 · Snitch 307,835 / 2,069 · Decathlon 333,570 / 704
- JavaScript shell — a 200, a full-looking response, and nothing in it. Croma 463,504 bytes / 216 visible characters · Bewakoof 250,459 / 230 · Tata CLiQ 14,349 / 46 · Urbanic 35,679 / 0. Tata CLiQ’s forty-six characters are “You need to enable JavaScript to run this app.”
- Refused — AJIO, HTTP 403, 366 bytes, titled “Access Denied” · Meesho, HTTP 403, 366 bytes, same title and same length · H&M India, HTTP 403, 391 bytes
- Stub — Amazon.in, HTTP 202, 2,008 bytes, no title, no product
- No answer — Nykaa, connection never completed inside 25 seconds. An earlier request the same day from the same machine had returned HTTP 403 instead.
Four different ways to say no
The failures are not variations on one behaviour. They are four distinct designs with four distinct consequences for anything consuming them.
An honest refusal is the 403. AJIO and Meesho return byte-for-byte identical 366-byte denials, which strongly suggests the same vendor product sitting in front of both. From a consumer’s point of view this is the best possible failure: it is unambiguous, it is cheap to detect, and code can act on it.
The stub is the dangerous one. Amazon.in’s 202 with two kilobytes is a success as far as any status-code check is concerned. A tool that trusts the status and parses what it got will extract a title from whatever boilerplate is in there and save an item with no price — and report success. Everything downstream then believes it has data. This is why a byte-size floor is worth having alongside the status check, and why ten kilobytes is not an arbitrary number: a real Indian retail product page in this sample is two orders of magnitude larger.
The JavaScript shell is the third, and it is not a refusal at all — which is what makes it slip past everything. Croma, Bewakoof, Urbanic and Tata CLiQ return a 200 and a response large enough to pass any size floor, containing no product because the page is assembled in the browser afterwards. A person sees a normal shop. A reader of the raw response sees a successful request with nothing in it. No status code is wrong, no threshold is breached, and there is still no data.
Inconsistency is the fourth shape, and the hardest to handle. Nykaa refused twice by two different routes within hours. A single measurement would have produced a confident, specific, wrong claim about how Nykaa behaves — which is exactly the trap, because one data point looks the same as a fact.
The borderline case, and what it turned out to be
On the first pass Tata CLiQ was the one I flagged as uncomfortable: 14,349 bytes passes a ten-kilobyte floor, but it sits far closer to the stub than to the median page in this sample, and a page that small from a large retailer deserves a second look rather than a pass.
The second look is where the JavaScript shell came from. Forty-six visible characters. Three other shops in the sample turned out to be doing the same thing behind much larger responses, which no size threshold would ever have caught — Croma sends almost half a megabyte to say nothing.
The general lesson is worth more than the specific shops, which will change. Any threshold converts a continuous signal into a binary one and creates a band where the honest answer is “technically fine, probably not”. Log the measurement, not only the verdict, so the borderline cases stay visible instead of being rounded away — and then go and look at the ones sitting in the band, because that is where the category you had not thought of is hiding.
What this does not show
It does not show that fifteen shops welcome automation. It shows that fifteen shops served one homepage request from a residential connection. Myntra and Flipkart both returned full pages here, and both are widely reported to refuse the same request from data-centre address space — which is the environment that matters if you are deploying something.
It does not cover product pages, which may be guarded differently from a homepage. It does not cover rate limiting, which only appears under load. And it is one client with one TLS fingerprint; a different client may well get a different answer, which is precisely what the Nykaa result demonstrates within a single day.
What it does establish is the shape of the problem, and that the shape is not uniform. Three of twenty refuse flatly, one answers misleadingly, one is unreliable, and the pattern does not follow company size or sector in any way I can see.
If you are building something that reads pages
Three things fall out of this that are worth more than the specific numbers, since the numbers will age.
- Check the content, not the status and not the size. A 2xx too small to be a page is a failure wearing a success; a 2xx large enough to pass a size floor and still empty is the same failure wearing a better disguise. Strip the markup and count what is left.
- Measure more than once before stating a behaviour as a fact. Nykaa gave two different refusals in one day; a single run would have produced a confident wrong answer.
- Read from the user’s browser where you can. Every shop that refused a server accepts a request from the tab the person already has open, because at that point it is a person browsing. A browser extension or bookmarklet sidesteps the entire category of problem rather than fighting it.