If you have certain nginx/apache rules regarding Googlebot or other search engine crawlers,
to handle duplicate content issues, you can test them by imitating theirs User Agent request header signature with curl:
Enjoy..
to handle duplicate content issues, you can test them by imitating theirs User Agent request header signature with curl:
$ curl -I --header "User-Agent: Googlebot" http://yourwebisite.com/some-uri-to-be-handled-differently-with-bots
HTTP/1.1 302 Moved Temporarily
Server: nginx/0.6.26
Date: Mon, 18 Aug 2008 13:35:59 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Location: http://yourwebsite.com/some-redirected-specific-to-bot-uri
Enjoy..
