Page cannot be crawled or displayed due to robots.txt?

What are some reasons that websites don’t want to be indexed by the search engines?
I just looked at a site called Box.com and noticed this message “Page cannot be crawled or displayed due to robots.txt.”
Can someone please explain? Thank you.
Answer #1
have a look at this one :
http://www.robotstxt.org/robotstxt.html
in short : the robots.txt does handle _all_ robots , not just google and msn, and believe it or not, there are tons of crawlers out there that have nothing good in mind …
Answer #2
Box hosts many files for many people and businesses me included and I’m glad they don’t let the robots in…
Answer #3
Box hosts many files for many people and businesses me included and I'm glad they don't let the robots in...
How come you’re glad that they don’t let robots in?
Answer #4
Box hosts many files for many people and businesses me included and I'm glad they don't let the robots in...
How come you're glad that they don't let robots in?
Isn’t it obvious? Clearly, He doesn’t want the links to his files to appear on Google, Due to privacy or anti-piracy concerns…
Personally, If I were to upload any “sensitive” material, I’d RAR it up with password protection and encrypted file names, But that’s just me.
Answer #5
Box hosts many files for many people and businesses me included and I'm glad they don't let the robots in...
How come you're glad that they don't let robots in?
Isn't it obvious? Clearly, He doesn't want the links to his files to appear on Google, Due to privacy or anti-piracy concerns...
Personally, If I were to upload any "sensitive" material, I'd RAR it up with password protection and encrypted file names, But that's just me.
The files I have stored on box are for members of an automotive forum to use… I don’t want them to have to mess with passwords and encryption to access the content… And also it’s none of Google’s or any other search engine’s business what I have stored there…
Answer #6
Box hosts many files for many people and businesses me included and I'm glad they don't let the robots in...
How come you're glad that they don't let robots in?
Isn't it obvious? Clearly, He doesn't want the links to his files to appear on Google, Due to privacy or anti-piracy concerns...
Personally, If I were to upload any "sensitive" material, I'd RAR it up with password protection and encrypted file names, But that's just me.
The files I have stored on box are for members of an automotive forum to use... I don't want them to have to mess with passwords and encryption to access the content... And also it's none of Google's or any other search engine's business what I have stored there...
I see, thank you for informing me. One question, would you find having this feature useful on a file sharing/hosting website? Features are regarding “Deleting/extracting files from .rar and .zip archives and being able to remove password from .rar archives (if you know the password)”. I was thinking of implementing these features:
1) View list of contents for your rar/zip archive.
2) Remove password from archive if you know it.
3) Delete files from archive.
4) Extract files from archive.
Also, if you wanted to share the that archive, users will see contents of archive on download page as well.
What do you think of those features and would you use it?
Answer #7

One question, would you find having this feature useful on a file sharing/hosting website? Features are regarding "Deleting/extracting files from .rar and .zip archives and being able to remove password from .rar archives (if you know the password)". I was thinking of implementing these features:
1) View list of contents for your rar/zip archive.
2) Remove password from archive if you know it.
3) Delete files from archive.
4) Extract files from archive.
Also, if you wanted to share the that archive, users will see contents of archive on download page as well.
What do you think of those features and would you use it?and what site would you like taken down for violating the DMCA?? you should read the DMCA.. and no, I wouldn’t use those features because it would cause the site to be taken down. you never want the contents known to the site. using those features would make the site know what is stored there, not good!. also, using the robots.txt file lessens problems like THIS. https://www.google.com/transparencyreport/removals/copyright/domains/.org/ google can not crawl WBB. these are from members here posting links to files on WBB on other sites