Top Left Text cha

Web & App Development

It recently came to my attention (via Google Webmaster Tools) that Joomla was having a negative impact on my website in Google Webmaster Tools.  Joomla, out-of-the-box, restricts Googlebot from accessing files that are now very important for them to be able to see.  Specifically the template files.  If Google can't see your style sheet, for example, their bot doesn't know that buttons are far enough from each other to be mobile friendly.  It can also cause Google to think that your pages' content runs off the side of the page.  Either of these, and probably more issues, can cause your site to be labeled 'not mobile friendly'.  Which will hurt your rankings in Google for mobile devices (which most people use).

From GWT:

How Googlebot sees this page

This page uses 29 resources which are blocked by robots.txt.

Does this screenshot look incorrect? Learn how to let Googlebot view the page correctly.

Show resources
  • http://tjohns.calclones.com/…css?v=3002
  • http://tjohns.calclones.com/…mobile.css
  • http://tjohns.calclones.com/…x1.min.css
  • http://tjohns.calclones.com/…finder.css
  • http://tjohns.calclones.com/…enuhck.css
  • http://tjohns.calclones.com/…maximenuck
  • http://tjohns.calclones.com/…ition1.css
  • http://tjohns.calclones.com/…siveck.css
  • http://tjohns.calclones.com/….3.js?v=12
  • http://tjohns.calclones.com/…jax.js?v=4
  • http://tjohns.calclones.com/…ls-core.js
  • http://tjohns.calclones.com/…ery.min.js
  • http://tjohns.calclones.com/…onflict.js
  • http://tjohns.calclones.com/…ate.min.js
  • http://tjohns.calclones.com/…js/core.js
  • http://tjohns.calclones.com/…ls-more.js
  • http://tjohns.calclones.com/…caption.js
  • http://tjohns.calclones.com/…kmobile.js
  • http://tjohns.calclones.com/…box-min.js
  • http://tjohns.calclones.com/…ipt.min.js
  • http://tjohns.calclones.com/…js/main.js
  • http://tjohns.calclones.com/…rap.min.js
  • http://tjohns.calclones.com/…mpleter.js
  • http://tjohns.calclones.com/…ore.min.js
  • http://tjohns.calclones.com/…imenuck.js
  • http://tjohns.calclones.com/…ymenuck.js
  • http://tjohns.calclones.com/…/style.css
  • http://tjohns.calclones.com/…mobile.css
  • http://tjohns.calclones.com/…-small.css
 

These styles need to be accessible to Googlebot for Google to understand that your site if mobile friendly.  

So I recommend removing some of the 'disallows' from the robots.txt file.  

Untouched robots.txt file:

# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

 
I would also get rid of all the comments at the top.  No point in making Googlebot crawl over them.  Here, in my opinion, is the way it should be edited to be:

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /installation/
Disallow: /logs/
Disallow: /tmp/

 












Comments   

0 #1 Tim 2015-08-12 02:36
I believe the Joomla team has realized how bad this is for SEO these days and has made the default robots file much friendlier to googlebot.
Quote

Add comment


Security code
Refresh

Leave your comments

Post comment as a guest

0
Your comments are subjected to administrator's moderation.
X