Three quarters of the clicks past our bot flag were one crawl
Up front, I build and run affiliate sites for other people as a paid service, so a click count is something I hand to someone else, and I am not neutral about it being right.
We log every outbound click ourselves instead of reading the network report back, one row per click carrying product, address, signature and time. That decision is what made the rest visible. On one UK tag the report handed me a count of 1,375 and nothing underneath it.
On the site I run myself that table holds 16,976 rows. The flag marked 11,155 of them and left 5,821. Bucketed by day, 4,336 of the survivors sit inside one three day window in August, reaching 674 products from 768 addresses. That is a crawl, and no total would ever have said so.
So the flag is not the number. How clicks fall across days, and how many addresses touched how many products, is what I read before quoting a total, and that is what the row level table is for.
If you keep your own table, what makes a row trustworthy to you?
Replies
A trutwothy row for me needs enough context to understand where it came from. Time and address are useful, but I would also want to see the pattern around that click.
@simran_kumar That is the right instinct, and it is what turned our own number over. No single row in that window looked wrong. What changed the reading was the neighbourhood around it: 4,336 arrivals inside one three day stretch, 674 products, 768 addresses, roughly six hits each. Nothing about any one of those rows is suspicious on its own.
So what each row carries is product, address, signature and time, and the reason for all four together is that they let the pattern be rebuilt afterwards instead of judged at write time. Judging at write time is where a flag falls short: among the rows ours had already cleared, one crawler signature still turns up 226 times.
Which is the practical version of what you are describing. The row is the evidence, the pattern is the verdict, and keeping those two apart is the only reason I am willing to hand a client a number at all.
My opinion is the flag is the wrong tool. A row only becomes trustworthy when I can rebuild the session behind it. One address touching 674 products in three days was never a visitor.