I think I see what you're saying, but I disagree that the robots.txt file should have any legal ramifications. Web site operators have many tools that they can use to limit traffic or protect data and they should make good use of those tools.

LinkedIn wants to make their data available publicly, except under certain conditions. In my opinion, if they can't find a technical solution, they should stop making the data available publicly.