mynameisnobodyodyssea@googlemail.com wrote:
> it is a good idea to have a good robots.txt file

A robots.txt file serves a specific purpose. If you don't need one,
there is no reason to have one. I would guess that most site owners
don't have one, don't need one, don't want one, and don't even know what
one is.

> It is a good idea to have a proper robots.txt file with proper
> Disallow rules,

But only if there are portions of the site you want to disallow.

> even small sites can have URLs that are better blocked to bots,

Why don't we let the OP make that decision for himself.

> About XHTML and HTML, I think that an XHTML doctype is better,

<snip>

> obviously the general assumption is that 21st century websites use an
> XHTML doctype.

The problems with using an XHTML doctype have been discussed here in the
past. Most site are better off using HTML 4.01 strict. One can search
the archives of this group, or do a Google search to find out why in the
greater majority of cases, XHTML is a bad idea.