Known Bots
Crawler robots, also known as bots or spiders, are automated programs that search engines and other online services use to scan and index web pages. Knowing these robots and their IP addresses is very important for webmasters, developers, and IT professionals, because this information helps them recognize legitimate traffic, prevent unauthorized access, and improve their site's performance.
This guide collects a list of major providers such as Google, Meta (Facebook), Bing, Apple, OpenAI, and Yandex, along with links to official sources for verifying the robots' IP addresses. These sources are essential to detect and make sure the traffic of these robots is genuine and not spoofed by other malicious sources.
With this document, you can:
- Verify the authenticity of crawlers accessing your website.
- Block attacker IPs or add legitimate robots to the whitelist.
- Improve your site's security by identifying and blocking unauthorized robots.
- Stay up to date with the latest IP addresses published by search engines and online services.
Sources
-
Google
-
Official link:
https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#automatic
-
-
Meta / Facebook
-
Official link:
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
-
-
Bing
-
Official link:
https://www.bing.com/toolbox/bingbot.json
-
-
Apple
-
Official link:
https://search.developer.apple.com/applebot.json
-
-
OpenAI
-
Official link:
https://platform.openai.com/docs/bots
-
-
Internet Archive
-
Unofficial link (Cloudflare Docs):
https://developers.cloudflare.com/cache/troubleshooting/always-online/
-
-
Yandex
-
Official link (ASN):
https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html?lang=en -
Unofficial link (IP2Location):
https://www.ip2location.com/free/robot-whitelistThe Arvancloud CDN uses the IP2Location list.
-
-
Yahoo
-
Unofficial link (IP2Location):
https://www.ip2location.com/free/robot-whitelist
-
-
Baidu
-
Unofficial link (IP2Location):
https://www.ip2location.com/free/robot-whitelist
-
-
Ahrefs
-
Official link:
https://api.ahrefs.com/v3/public/crawler-ip-ranges
-
-
Statuscake
-
Official link:
https://www.statuscake.com/kb/knowledge-base/what-are-your-ips
-
-
Webengage
-
IPs:
34.192.48.654.82.121.3652.202.184.22535.154.107.8513.235.37.9213.234.183.24634.166.40.23834.166.74.125
-
-
Claude
-
Official link:
https://platform.claude.com/docs/en/api/ip-addresses
-
-
Perplexity
-
Official link:
https://www.perplexity.ai/perplexity-user.json
-
-
Copilot
-
Official link:
https://learn.microsoft.com/en-us/copilot/security/plugin-ip-address
-
Configuration in the CDN
To disable the whitelisting of these robots on your domain and configure the domain as you like, you can use the following API and the skip_global_whitelist field.
https://www.arvancloud.ir/api/cdn/4.0#tag/Firewall/operation/firewall.settings.update
For example:
curl --location --request PATCH 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/firewall/settings' \
--header 'authority: napi.arvancloud.ir' \
--header 'accept: application/json, text/plain, */*' \
--header 'authorization: API KEY' \
--header 'content-type: application/json' \
--header 'accept: application/json' \
--data '{"skip_global_firewall":true}'
Note that using
skip_global_whitelistmay cause problems for search engines and other robots when crawling your website.