realrobotstxt.com valuation and analysis
Robots.txt Information
| Robot | Path | Permission |
| GoogleBot | / | ✔ |
| BingBot | / | ✔ |
| BaiduSpider | / | ✔ |
| YandexBot | / | ✔ |
Meta Tags
| Title | Robots.txt parser based on Google’s open source parser from |
| Description | Parse your robots.txt file the same way Google’s crawlers Distilled’s interpretation of how Google parses robots.txt files using a fork of their robust open source |
| Keywords | N/A |
Server Information
| WebSite | |
| Host IP | 172.67.156.191 |
| Location | United States |
Related Websites
| Site | Rank |
More to Explore
realrobotstxt.com Valuation
|
US$321,723
Last updated: 2023-05-12 19:30:41
realrobotstxt.com has Semrush global rank of 32,898,902. realrobotstxt.com has an estimated worth of US$ 321,723, based on its estimated Ads revenue. realrobotstxt.com receives approximately 37,122 unique visitors each day. Its web server is located in United States, with IP address 172.67.156.191. According to SiteAdvisor, realrobotstxt.com is safe to visit. |
Traffic & Worth Estimates
| Purchase/Sale Value | US$321,723 |
| Daily Ads Revenue | US$297 |
| Monthly Ads Revenue | US$8,910 |
| Yearly Ads Revenue | US$106,911 |
| Daily Unique Visitors | 2,475 |
| Note: All traffic and earnings values are estimates. | |
DNS Records
| Host | Type | TTL | Data |
| realrobotstxt.com. | A | 300 | IP: 172.67.156.191 |
| realrobotstxt.com. | A | 300 | IP: 104.21.13.158 |
| realrobotstxt.com. | AAAA | 300 | IPV6: 2606:4700:3036::6815:d9e |
| realrobotstxt.com. | AAAA | 300 | IPV6: 2606:4700:3032::ac43:9cbf |
| realrobotstxt.com. | NS | 86400 | NS Record: will.ns.cloudflare.com. |
| realrobotstxt.com. | NS | 86400 | NS Record: dawn.ns.cloudflare.com. |
| realrobotstxt.com. | TXT | 300 | TXT Record: google-site-verification=01eZ9dzikTJWwXW9ALCP2yte7OTOOapuVQO11PFhfdk |
HtmlToTextCheckTime:2023-05-12 19:30:41
| Robots.txt Parser Follow me on Twitter Parse your robots.txt file the same way Google’s crawlers do Choose a Googlebot, enter your robots.txt file in the text area and enter the path you’d like to check. Crawler Googlebot Googlebot Image Googlebot Video Googlebot News AdsBot AdsBot mobile AdSense Other Specify user agent (if "other" crawler selected): Robots.txt file User-agent: googlebot Disallow: /foo/ Path to check Parse You must ensure that the path you wish to check follows the format specified by RFC3986, since this library will not perform full normalization of those URI parameters. Only if the URI is in this format, will the matching be done according to the REP specification. This is exactly as per Google’s open source project . Why does this exist? The old Search Console robots.txt tester differs from real Googlebot behaviour and we expect to see it deprecated at some point. Google published an open source project containing the code their crawlers use to parse robots.txt |
HTTP Headers
HTTP/1.1 301 Moved Permanently
Date: Sun, 19 Dec 2021 21:15:35 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Sun, 19 Dec 2021 22:15:35 GMT
Location: https://www.realrobotstxt.com
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=vChsPxPsRCZu4ehx%2FfKvqwj8VBUWJZKdQWZzpgfIq%2B1P07PsagPphpRLbtbWfdSMcXlh%2BFaVYt7QX3tUVD3jEzQ5pUJhmpNp4ZE60qBRztnJomnEmhZytQj8eg9Oq7fo97Rn%2Bg%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 6c03a5a85a0b195d-EWR
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
HTTP/2 200
date: Sun, 19 Dec 2021 21:15:35 GMT
content-type: text/html; charset=utf-8
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GlsPvs5C%2BrspM8XVJHsXRCiWOVOK47mPHngDNpaQG6%2Bu5J1e9iwgFWyraa%2Fahz1%2BIK2fqb%2FCbfECZoZMnquU800foNL9i8qwCYSDfy3DH1WkUZLU3IRRzMotnZWZTrjFy%2BlImm%2BG9YY%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 6c03a5a8ac4c18a1-EWR
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
|
realrobotstxt.com Whois Information
Domain Name: REALROBOTSTXT.COM Registry Domain ID: 2455867289_DOMAIN_COM-VRSN Registrar WHOIS Server: whois.godaddy.com Registrar URL: http://www.godaddy.com Updated Date: 2021-11-17T18:57:33Z Creation Date: 2019-11-16T10:26:19Z Registry Expiry Date: 2022-11-16T10:26:19Z Registrar: GoDaddy.com, LLC Registrar IANA ID: 146 Registrar Abuse Contact Email: abuse@godaddy.com Registrar Abuse Contact Phone: 480-624-2505 Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited Domain Status: clientRenewProhibited https://icann.org/epp#clientRenewProhibited Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited Name Server: DAWN.NS.CLOUDFLARE.COM Name Server: WILL.NS.CLOUDFLARE.COM DNSSEC: unsigned >>> Last update of whois database: 2021-12-23T08:41:08Z <<< |