Skip to main content

Known Bots

Crawler robots, also known as bots or spiders, are automated programs that search engines and other online services use to scan and index web pages. Knowing these robots and their IP addresses is very important for webmasters, developers, and IT professionals, because this information helps them recognize legitimate traffic, prevent unauthorized access, and improve their site's performance.

This guide collects a list of major providers such as Google, Meta (Facebook), Bing, Apple, OpenAI, and Yandex, along with links to official sources for verifying the robots' IP addresses. These sources are essential to detect and make sure the traffic of these robots is genuine and not spoofed by other malicious sources.

With this document, you can:

  • Verify the authenticity of crawlers accessing your website.
  • Block attacker IPs or add legitimate robots to the whitelist.
  • Improve your site's security by identifying and blocking unauthorized robots.
  • Stay up to date with the latest IP addresses published by search engines and online services.

Sources​

  • Google

    • Official link:

      https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#automatic
  • Meta / Facebook

    • Official link:

      https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
  • Bing

    • Official link:

      https://www.bing.com/toolbox/bingbot.json
  • Apple

    • Official link:

      https://search.developer.apple.com/applebot.json
  • OpenAI

    • Official link:

      https://platform.openai.com/docs/bots
  • Internet Archive

    • Unofficial link (Cloudflare Docs):

      https://developers.cloudflare.com/cache/troubleshooting/always-online/
  • Yandex

    • Official link (ASN):

      https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html?lang=en
    • Unofficial link (IP2Location):

      https://www.ip2location.com/free/robot-whitelist

      The Arvancloud CDN uses the IP2Location list.

  • Yahoo

    • Unofficial link (IP2Location):

      https://www.ip2location.com/free/robot-whitelist
  • Baidu

    • Unofficial link (IP2Location):

      https://www.ip2location.com/free/robot-whitelist
  • Ahrefs

    • Official link:

      https://api.ahrefs.com/v3/public/crawler-ip-ranges
  • Statuscake

    • Official link:

      https://www.statuscake.com/kb/knowledge-base/what-are-your-ips
  • Webengage

    • IPs:

      34.192.48.6
      54.82.121.36
      52.202.184.225
      35.154.107.85
      13.235.37.92
      13.234.183.246
      34.166.40.238
      34.166.74.125
  • Claude

    • Official link:

      https://platform.claude.com/docs/en/api/ip-addresses
  • Perplexity

    • Official link:

      https://www.perplexity.ai/perplexity-user.json
  • Copilot

    • Official link:

      https://learn.microsoft.com/en-us/copilot/security/plugin-ip-address

Configuration in the CDN​

To disable the whitelisting of these robots on your domain and configure the domain as you like, you can use the following API and the skip_global_whitelist field.

https://www.arvancloud.ir/api/cdn/4.0#tag/Firewall/operation/firewall.settings.update

For example:

curl --location --request PATCH 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/firewall/settings' \
--header 'authority: napi.arvancloud.ir' \
--header 'accept: application/json, text/plain, */*' \
--header 'authorization: API KEY' \
--header 'content-type: application/json' \
--header 'accept: application/json' \
--data '{"skip_global_firewall":true}'

Note that using skip_global_whitelist may cause problems for search engines and other robots when crawling your website.