爬虫概述

ThinkBot

OwnThink is a search engine. Similar to other search engines, OwnThink uses a robot (ThinkBot) to discover and index pages on the public web. This index allows users to find, retrieve, and cite your content.

Below is information on ThinkBot and how you can set your site preferences to enable or limit access to your content.

Purpose

ThinkBot fetches pages on the public web, which enables your pages to become discoverable through OwnThink. Its purpose is search and retrieval: connecting people and applications to your content and sending them back to your site.

Identification

ThinkBot sends this user-agent:

1. Smartphone

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; ThinkBot/2.1; +https://www.ownthink.com/bot)

1. Desktop

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ThinkBot/2.1; +https://www.ownthink.com/bot) Chrome/W.X.Y.Z Safari/537.36

Its robots.txt product token is ThinkBot. User-agent strings can be spoofed, so don't trust the string alone.

Verification

The common crawlers generally crawl from the IP ranges published in the https://www.ownthink.com/docs/bot/ip.json object.

Respecting robots.txt

ThinkBot respects the Robots Exclusion Protocol. When choosing which rules to follow, it looks for a group addressed to ThinkBot. If you haven't provided one, it follows the rules you've set for major search engines; otherwise it follows the * group.

Give ThinkBot its own rules :

User-agent: ThinkBot
Allow: /
Disallow: /login

Removing pages from index

robots.txt controls crawling, not indexing. To remove a page already in Exa's index, use a noindex robots meta tag or an X-Robots-Tag: noindex header; ThinkBot drops the page after it next re-fetches it.

How ThinkBot behaves

Respects robots.txt

Checked before fetch, honoring the precedence above.

Signs requests

So you can confirm a request is genuinely us.

Crawls politely

Rate-limited per site to avoid putting load on your servers.

Public content only

It does not attempt to bypass logins, paywalls, or CAPTCHAs, and submits no forms.

Contact

If you have questions or need assistance, please reach out to help@ownthink.com. If possible, please reach out from an email that includes the domain you are contacting us about, which facilitates verification.