Webcrawler with yahoo.net

[Beery_Asst]

I just looked at my Downloads log file.

My website is http://www.USWingNuts.com .

Is there anyway to prevent webcrawlers from downloading files in the public area I have setup from my webstore?

[coastalrugs]

If it is publicly accessible, there is not much you can do to absolutely prevent download. But any proper spider should respect robots.txt directives.

You could likely add a robots.txt file with something like:
User-agent: *
Disallow: /*.zip$

This should prevent *.zip files. One could be added for pdfs or any other file type you want ignored.

For more info, see:
http://www.google.com/support/webmas...&answer=156449

I haven;t looked into this, but there may be a mechanism in SC to allow the modules to inject their own headers into the main page's output. If that is the case, it may be possible to have the download page add a NO-FOLLOW meta tag so spiders would not follow the actual download links (though they would also not follow any links back out of the page to the rest of your site)


Allow there could be solutions that are actually built into SC. I saw some solutions suggested of converting download links into POST requests instead of GET requests, requiring sessions for the user before being able to download, or things like these. Anything that a normal browser would do/have, but a spider ignores.

[Beery_Asst]

Thanks. I already had a robots.txt file that I had blocking a specific directory, but not this particular one and not the pdf files I had.

Thanks again.

Beery

[coastalrugs]

Just a note of clarification:
It is very likely that even if you had a directive preventing access to sc_data, the spiders will see the file as coming from:
...../store/store.php?downloads=1&id=4
or some derivative thereof.

A more narrow directive targeting this page might be in order. Hopefully the zip/pdf/etc... route is enough to squash the unnecessary downloads.

[Beery_Asst]

I'm going to continue following my logs for a few days and see if I have stopped the majority of the webcrawling. Thanks for the feedback.

Beery