seo - How can I prevent Googlebot from crawling my Underscore client-side templates? -
in google webmaster tools, under crawl errors/other, we're seeing 400 error urls this:
/family-tree/<%=tree.user_url_slug%>/<%=tree.url_slug%>
this not real url, or url intended crawled. underscore/backbone template:
<script type="text/template" class="template" id="template-trees-list"> <% _.each(trees, function(tree) { %> <a href="/family-tree/<%=tree.user_url_slug%>/<%=tree.url_slug%>" rel="nofollow"> <%= tree.title %> </a> <% }); %> </script>
- why google crawling inside of
script
block? - why google ignoring
rel="nofollow"
attribute? - is there else can keep googlebot away our underscore templates?
update: i'm open using robots.txt if can find right pattern keep pages , block bad pages. example, want keep /surnames/jones/queries
while blocking /surnames/jones/queries/<%=url_slug%>
. have thousands this. looks googlebot may support basic patterns, though not full regexes.
update 2: well, doesn't @ root cause, , seems bit brittle long-term solution, tested in gwt following robots.txt patterns work:
user-agent: googlebot disallow: /*url_slug%%3e$ disallow: /*url_slug%%3e/$
just block these via robots.txt , you'll fine
Comments
Post a Comment