About the datadepths Robot
This is the information page on the datadepths robot. Please contact us if you have any additional questions, comments or concerns.
This robot was first launched in 2007 with the goal of analyzing website HTML & network data to find & highlight website problems & missteps. A website problem can be as simple as an HTML error -- or as complex as a poorly executed SEO strategy.
The automated datadepths robot does the following:
- 1. Request http://domain.com/robots.txt
- 2. If allowed, we proceed to request http://domain.com/
- 3. Process any redirects that may occur. Redirects are followed.
- 4. Process the resulting headers & html content returned during the request
The user controlled datadepths robot does the following:
- 1. Request http://domain.com/
- 2. Process any redirects that may occur. Redirects are followed.
- 3. In the event of a redirect, we either continue, or end the request.
- 4. If continuing, we process the resulting headers & html content returned during the request
- 5. Next all links returned from the HTML are processed & stored in a temporary database
- 6. Each level of links is assigned an internal crawl delay
- 7. Links are followed deeper into a site until either the algorithm limit (approximately 100 html pages) is reached, or the entire site has been visited
- 8. Results are displayed for the user
There are currently 284,328,692 domain names being tracked. This number varies and does not represent large groups of domains that we have internally blacklisted.
Please contact us with any questions, concerns, or problems.
Please see http://www.robotstxt.org/ for information on how to block the datadepths robot. Our user agent string for blocking is: datadepths
While we honor all robots.txt exclusion requests, we do not remove domain names from our database for statistical reasons.