Skip to main content

Tomba's web crawler: what it is and how to block it

TombaPublicWebCrawler reads public web pages for business contact details. See what it collects, and how to slow it down or block it with robots.txt.

Written by Tomba.io Team

Tomba's web crawler, TombaPublicWebCrawler, reads public web pages to find business contact details, such as work email addresses. A crawler is a program that visits web pages automatically, as search engines do. Tomba's crawler follows robots.txt, so you can slow it down or block it on your website.

What is TombaPublicWebCrawler?

TombaPublicWebCrawler is the program Tomba uses to index business contact information that's publicly available, much like a search engine indexes web pages. It identifies itself to your website with this user agent:

Mozilla/5.0 (compatible; TombaPublicWebCrawler/3.0; +https://tomba.io)

The footer of the Tomba website links to the crawler's page as TombaBot.

What does Tomba's crawler collect?

Tomba's crawler visits publicly accessible pages only, such as company websites, press releases, news sites, public business directories and professional profiles. From them, it collects business contact information, such as professional email addresses. It never reads content behind a login.

How do I block Tomba's crawler?

To stop the crawler from visiting your website, add these two lines to your site's robots.txt file:

User-agent: TombaPublicWebCrawler

Disallow: /

The crawler reads your robots.txt before it visits any page, and it follows both Disallow and Allow rules.

How do I slow Tomba's crawler down?

To make the crawler wait between the pages it requests from your website, add these two lines to your robots.txt file:

User-agent: TombaPublicWebCrawler

Crawl-Delay: 10

Replace 10 with the minimum number of seconds you want between requests.

When do my robots.txt changes take effect?

Changes to robots.txt can take a while to be picked up, until the crawler's next scheduled visit. Check the file for mistakes: a syntax error can stop the crawler from reading it, and the crawler keeps its previous behaviour when it doesn't understand a rule.

How do I remove data Tomba already holds about me?

To have details about you removed from Tomba, use the Claim page. You don't need a Tomba account. See Remove your data from Tomba.

How do I contact Tomba about the crawler?

If you think Tomba's crawler is misbehaving on your website, or you have a question about it, email [email protected]. For the technical details, see the crawler page on the Tomba website.

Related articles

Did this answer your question?