Getintent Crawler 2017-10-03T12:49:21+00:00

Getintent Crawler

Hello webmasters,

To choose the best suited advertisement to show in ad slots of your website we have implemented the crawler bot that periodically downloads website pages to gather contextual information like keywords and content categories.

We care about your website resources. Our crawler adjusts frequency of visits and amount of information that it downloads to make sure that we do not strain resources of your website.

Our crawler respects robots.txt file. You can set desirable crawl delay or completely block our bot there.

Technical details

User-agent: Getintent Crawler (http://getintent.com/bot.html)

Support

For further questions please contact bot@getintent.com.