It is your website, and of course you can have on it
any XHTML errors or robots.txt file you want,
but, sorry to repeat,
it is a good idea to have a good robots.txt file
and valid (X)HTML, even if search engines and browsers
do their best to compensate for (X)HTML errors.

It is a good idea to have a proper robots.txt file
with proper Disallow rules,
even small sites can have URLs that are better blocked
to bots, large sites certainly have.

For example, you seem to have the same robots.txt file for
the http and https URLs,
maybe use the robots.txt to block the URLs you
have indexed now in Google with https
(just a suggestion).

About XHTML and HTML, I think that an XHTML doctype
is better, XHTML is newer than HTML and uses XML features
that make parsing by bots, or the use of
the DOM structure for example for AJAX, more reliable,
you do not need to escape closing angle brackets
within JavaScript (if you have those) because
you use CDATA to escape all of them at once, etc.
Also, as far as I know,
at the moment the MSN verification tag works only
in the XHTML format, closed with /> so
obviously the general assumption is that
21st century websites use an XHTML doctype.

About the background attribute for TD,
when I wrote that you could use the style attribute
instead, because the background attribute is deprecated,
you already use the style attribute for other td's in your home page,
(just to add that inline styling like that is not
usually a very good idea, better use an external CSS file).

Anyway, I just meant to be helpful,
the important thing is
if you are happy with the way that search engines
index your site in search results,
and with how people visit your site as it is now.

mynameisnobodyodyssea@googlemail.com wrote:

> It is your website, and of course you can have on it
> any XHTML errors or robots.txt file you want,
> but, sorry to repeat,
> it is a good idea to have a good robots.txt file
> and valid (X)HTML, even if search engines and browsers
> do their best to compensate for (X)HTML errors.
>
> It is a good idea to have a proper robots.txt file
> with proper Disallow rules,
> even small sites can have URLs that are better blocked
> to bots, large sites certainly have.
>
> For example, you seem to have the same robots.txt file for
> the http and https URLs,
> maybe use the robots.txt to block the URLs you
> have indexed now in Google with https
> (just a suggestion).
>
> About XHTML and HTML, I think that an XHTML doctype
> is better, XHTML is newer than HTML and uses XML features
> that make parsing by bots, or the use of
> the DOM structure for example for AJAX, more reliable,
> you do not need to escape closing angle brackets
> within JavaScript (if you have those) because
> you use CDATA to escape all of them at once, etc.
> Also, as far as I know,
> at the moment the MSN verification tag works only
> in the XHTML format, closed with /> so
> obviously the general assumption is that
> 21st century websites use an XHTML doctype.
>
> About the background attribute for TD,
> when I wrote that you could use the style attribute
> instead, because the background attribute is deprecated,
> you already use the style attribute for other td's in your home page,
> (just to add that inline styling like that is not
> usually a very good idea, better use an external CSS file).
>
> Anyway, I just meant to be helpful,
> the important thing is
> if you are happy with the way that search engines
> index your site in search results,
> and with how people visit your site as it is now.

I think it's fine for major search engines, such as the robots.txt file,
etc., but since it's not going to harm the ranking, it's not a bad idea
to add the disallow. I have nothing accessible over the web that I
want search engines to not index, so I really have no need for any
disallow paths, so I did add the empty one. Even an empty robots.txt
or the lack of one altogether is fine. Some of the first ranked web
hosts have no robots.txt at all, and other's just have blank one's.
However, I don't mind adding the blank disallow. I wasn't aware any
verification forms saw it as a problem with out it, and it's odd that
they do (I think they are broken and wrong, even if just logically),
but if any see it as a problem, I'd prefer to avoid it (even if it's
just an out of the way search engine), so again, I did add the empty
disallow. Thanks for that tip.

As for the CSS aspects, even though no major search engines have a
problem with the site that I can see, I will either have one of our
guys go through and convert it all to strict XHTML, because I can agree
with that this is only a good thing, for consistency (if nothing else).
However, I do agree that some of the top ranked sites I've mentioned
that even lack a robots.txt file altogether, do usually have 100%
compatible XML parser checks. Besides, using deprecated tags and CSS
in the file is a little crazy. I just had to do some quick patches when
I saw how it displayed in some browsers that don't use CSS in the same
way (I just need to find out what specific modifications are needed for
it to work in all of them well enough, or hire someone to do it).
Thanks for the tips.
--
Tim Greer, CEO/Founder/CTO, BurlyHost.com, Inc.
Shared Hosting, Reseller Hosting, Dedicated & Semi-Dedicated servers
and Custom Hosting. 24/7 support, 30 day guarantee, secure servers.
Industry's most experienced staff! -- Web Hosting With Muscle!

Scott Bryce <sbryce@scottbryce.com> wrote:

> mynameisnobodyodyssea@googlemail.com wrote:
>> it is a good idea to have a good robots.txt file
>
> A robots.txt file serves a specific purpose. If you don't need one,
> there is no reason to have one.

Not getting 404s is a good enough one for me.

--
John Bokma http://johnbokma.com/

AISE/AWW/SEO/web development forum: http://seo-expert-wiki.com/

Tim Greer <tim@burlyhost.com> wrote:

> As for the CSS aspects, even though no major search engines have a
> problem with the site that I can see, I will either have one of our
> guys go through and convert it all to strict XHTML,

I would make that HTML 4.01 strict. If you can't name a single reason why
you need XHTML, you most likely don't need it at all. It's so easy to do
XHTML wrong.

--
John Bokma http://johnbokma.com/

AISE/AWW/SEO/web development forum: http://seo-expert-wiki.com/

John Bokma wrote:

> Tim Greer <tim@burlyhost.com> wrote:
>
>> As for the CSS aspects, even though no major search engines have a
>> problem with the site that I can see, I will either have one of our
>> guys go through and convert it all to strict XHTML,
>
> I would make that HTML 4.01 strict. If you can't name a single reason
> why you need XHTML, you most likely don't need it at all. It's so easy
> to do XHTML wrong.
>

Certainly, I couldn't agree more. This is something to consider. When
it's all said and done, the important things are to work and look the
same (or as close as possible) across all browsers and even be text
browser friendly. JavaScript, CSS, etc. is all there just to enhance
it (if people have those options and have them enabled) and to still
not make things appear off across the browsers. Beyond that, it should
be pretty search engine friendly (which some things could benefit from
being changed anyway) and that's all that matters. It doesn't need to
be any single thing just for the sake of using it (if we don't need it
or want it).
--
Tim Greer, CEO/Founder/CTO, BurlyHost.com, Inc.
Shared Hosting, Reseller Hosting, Dedicated & Semi-Dedicated servers
and Custom Hosting. 24/7 support, 30 day guarantee, secure servers.
Industry's most experienced staff! -- Web Hosting With Muscle!