It’s a combination of normalized traffic and “reputation” with the crawlers in our person base. Crawlers can use non-unique keywords and phrases within the consumer-Agent - owning just their own personal name as https://prbookmarkingwebsites.com/story19629264/list-crawler-fundamentals-explained