The robots meta tag and robots.txt can be put to good effect if you have some pages with content that is unrelated to the general purpose of the site e.g. on this forum, I could place a robots metatag in the chat cat to prevent it being crawled so the search engines would only be able to see the more technical threads. This could be very handy if the chat threads became more abundant than the technical ones therefore reducing this site's relevancy to a search engine on the subject of ecommerce.

By default they will follow all links, but carefully choosing which pages to exclude from a SE's index can increase your chances of being found by the keywords of your choice. Apart from unrelated text you might want to consider disallowing system pages from being indexed for example account registration pages which in general have little content of value for a search engine.

Both the robots meta tags and robots.txt are about exclusion rather than inclusion but as my example above shows, excluding certain pages from being indexed can be beneficial.

There's a few nifty things you can do with robots.txt, this handy line can be added to it if you suffer problems with image theft.

User-agent: Googlebot-Image
Disallow: /

It won't stop image theft, but it will prevent your images from being displayed on a Google image search.

__________________
Lawrence