Sample Robots.txt file for Magento

We often get asked by SEO companies and clients for a more advanced robots.txt file to block directories and block problem bots, if you are experiencing slow speeds and high loads this can often be caused by agressive bots, remember bots cache the robots.txt file and it can take upto 4 weeks for them to refresh this, the best cause of action to block bots is add some additional confiigration in local.xml or Nginx / Apache, this is a little more technical so please get in touch if you have any concerns in that area.

Ok back to the point of this post, below is a sample robots.txt file we generated for a high traffic site with millions of product SKUs in the catalog, you can edit as you wish but this covers most imporatant parts, note there is a block on rogerbot, if you use SEO-Moz you may need to remove this from the disallow.

#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these “robots” where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/wc/robots.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Crawl-delay: 30

# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=

# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Harvest/1.5
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: humanlinks
Disallow:/
User-agent: ia_archiver
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density/0.9
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: lwp-trivial/1.34
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: Microsoft URL Control – 5.01.4511
Disallow:/
User-agent: Microsoft URL Control – 6.00.8169
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: MIIxpc/4.2
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: moget/2.1
Disallow:/
User-agent: mozilla/4
Disallow:/
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Disallow:/
User-agent: mozilla/5
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: ProPowerBot/2.14
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow:/
User-agent: RMA
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz/1.4
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl/UrlDispatcher
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: True_Robot/1.0
Disallow:/
User-agent: turingos
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebBandit/3.50
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: WebZip/4.0
Disallow:/
User-agent: Wget
Disallow:/
User-agent: Wget/1.5.3
Disallow:/
User-agent: Wget/1.6
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu’s
Disallow:/
User-agent: Xenu’s Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/

# SEO-related bots
User-agent: rogerbot
Disallow:/
User-agent: mj12bot
Disallow:/
User-agent: dotbot
Disallow:/
User-agent: ahrefsbot
Disallow:/

Tagged with: ,
Posted in Magento tips and guides

Upgrade from Magento Community to Enterprise

magento-enterprise-review-logoIn the past it has been well known that the upgrade path between Magento Community and Magento Enterprise has been difficult, well we are pleased to say we have now successfully managed to put in place a reasonably simple upgrade path from Magento Community to Enterprise.

The good news there is for merchants who already have a great looking and successful Magento Community store but looking to make the move to the next level and benefit from the great features Magento Enterprise has to offer is this can be done at a fraction of the cost of a full re-build.

We have a procedure that has been tested a number of times and dependent on the quality of the build of your community store can be a fast a reasonably pain free task.

Our approach takes 3 steps to upgrade.

Step 1 – A code review and analysis of your existing store, ensuring no core code changes were made and your templates are built to a good standard.
Step 3 – Upgrade and review all your modules to either 1.8 compatible or enterprise compatible versions where required, many community modules work fine in Enterprise.
Step 2 – Perform a community upgrade to Magento 1.8 and full testing to ensure all Magento 1.8 functionality is in place and correctly functioning.
Step 3 – By following our planned upgrade path which was designed along side Magento core team of expert consulting we will then do the upgrade to Magento Enterprise 1.13.x
Step 4 – The final steps are to configure your template and install the features of Enterprise.
Step 5 – Final testing and preparation for launch.

The process takes around 4 weeks if no major problems are detected in the code review and the upgrade to Magento 1.8 if you are not already running this version is a success, this can be affected by the number of modules and how old your version of community is.

If you would like more information on how you can make the move to Enterprise without the hassle of a full re-design and build process, call or email us for further information and we will be happy to do a free no obligation review of your existing platform.

Our last test upgrade went from Magento community 1.4 to Magento Enterprise 1.13.1 however the changes required in templates from versions this old are alot more than if you are already running the latest release.

Posted in Magento tips and guides

Heartbleed – Who’s affected?

Taken from www.theepochtimes.com

Last week, Heartbleed, the biggest security risk in the history of the Internet, hit our screens.

The bug is thought to have affected just over two thirds of all sites using an OpenSSL (an extension to the SSL that acts as a safeguard between the server and the user.)

Major online companies such as Google and Facebook have reported patching their systems and have recommended password changes to all their customers.

The threat became widely known last week, so most sites affected should have updated their systems by now to fix the issue.

It is worth pointing out that you should not change your password until it has been confirmed the website has updated, because until that happens, your new password is still vulnerable.

We are very pleased to report that none of our customer’s websites have been affected.

However, if you are concerned about your website, you can look on websites like this one to find out more.

You can find out more about the biggest companies that have been affected and who’s fixed the problem on Mashable.

Tagged with: , , ,
Posted in General 2J Design News and Updates

2J Developer Gets Perfect Score

Well it’s a massive well done to Mayur, the latest 2J developer to pass the Magento Certified Front End Developer exam.

Not only did he pass, which is an achievement in itself, but he managed an amazing 100% score too.

This means 2J Commerce now have 11 Certified Magento developers.

It’s exciting times here at 2J Commerce - keep your eyes peeled for more updates as more of our team are set to take the exams later this month.

Congratulations again Mayur – welcome to the Magento team!

commerce-footer

Tagged with: , ,
Posted in General 2J Design News and Updates

Two More Certified Magento Plus Developers

It’s another two passes for the Magento Developer Plus exam at 2J this week; bringing our team of certified Magento developers to ten.

Passing this is a massive achievement and definitely something to be proud of, so a big congratulations to our two new certified developers. Well done to both of you from everyone here at 2J.

Our 2J teams is growing fast, but we aren’t stopping there! We are really excited about the projects we have coming up in the next few months, but look out for more Magento news too, as three more developers sit the Magento Plus at the end of April. Good luck guys!

Our new badge for our certified Magento developers...

Tagged with: , ,
Posted in General 2J Design News and Updates

All you need: In Health

2J Commerce are happy to announce the launch of Irish company InHealth‘s new website.

inhealth.ie

Their new responsive design has been build on a Magento Enterprise platform, which accommodates their vast range of products and allows their customers to browse the site on any device as the site shrinks down to suit any platform.

Mobile size

Mobile size

Tablet size

Tablet size

We have really enjoyed working with InHealth and look forward to continuing to work with them in the future.

cropped-2j-logo.png

Tagged with: , , , ,
Posted in Latest project news from 2J Commerce

20 Years Ago…

Alarm Clock

Then

If we take a look at a typical day 20 years ago and compare it to present day, the changes mobile phones and tablets have made to most aspects of day to day life are staggering. 20 years ago, your battery-operated alarm clock would wake you up to begin your day. You’d drop the kids off at school, go to work, do the food shop and watch one of the five TV channels available after dinner.

Now, your mobile or tablet jingles a jaunty tune to get you out of bed. You can order the weekly food shop and have your evening meal delivered to your door already cooked, whilst playing candy crush or reading the last release all from one device. You can do all of this sat at home you probably furnished through an online retailer whilst choosing from over 200 channels on your HD Smart TV.

In an interview with Ecommerce Times, telecommunications analyst Jeff Kagan said: “Today when we leave the house in the morning we can’t forget our keys, wallet and smartphone. Tomorrow all we’ll have to remember is our smartphone.”

Americans on average now own four different digital devices and they engage in media content for more than 60 hours a week.

There’s an app for that…

Apps

Now

Smartphone owners use 86% their time on apps as opposed to only 14% using mobile browsing. Consumers spend almost 10 hours more using browsers and apps on a smartphone compared to 2012.

Mobile shopping

44% of Americans use their tablets for online shopping whilst watching TV. 87% of smartphone owners use their devices for shopping online. With consumers taking advantage of their ability to shop any time, anywhere there is a “huge opportunity” for retailers and brands to get online and expand their business.

As technology advances, mobile shopping becomes more popular, mostly for its ease. Principle Tirias Research analyst Jim McGregor  said: “The preferred devices are now mobile. As a result, the content, service, and method of accessing the information all needs to be targeted at those platforms.”

While these figures may focus on consumers across the pond, the UK is showing similar trends. The world has gone mobile, and it does generate in Ecommerce for business across the country.

This all equates to the fact that going on line is the way forward. With the reach of the Internet expanding, and the ever-speedier advancements in technology, it’s never been easier to sell, and buy online.

If you’re planning a website of any sort, then perhaps this is reason enough to go responsive.

contact our friendly sales team today!

Tagged with: , , ,
Posted in General 2J Design News and Updates, Uncategorized

The Internet Retailing Expo.

Internet Retailing Expo 2014

 2J are back at the NEC on the 26th and 27th March for the Internet Retailing Expo 2014. Here’s why you need to be there…

The high street looks completely different to how it did ten years ago. Woolworth’s has gone and Blockbuster stores are vacant, as more and more people are going online for information and products. This article: ‘Gyms, bars and bookies taking over high streets’ by the Guardian, talks about the changes to city centres, but it also highlights another message to retailers: online business is booming.

Instead of forking out for parking and petrol, customers are now browsing the shops on their mobiles. This is why an Ecommerce website has never been more important when it comes to future proofing your business.

2J can design, build and host your website, and we will be at IRX in March to show just what we can offer your online store.

The Internet Retailing Expo is an ideal place for retailers to learn ways of strengthening your online business – whether you’ve just started your company, or you want to boost your online presence, there’ll be plenty to see.

Along with hundreds of exhibitors, there are also free workshops, which provide training on customer service to SEO, and the whole event is free to attend. You can register here.

We’re really looking forward to taking part at IRX, so why not come and talk to us while you’re there? You’ll find us on the Magento pavilion.

http://www.internetretailingexpo.com/magento-pavilion/

See more at http://www.internetretailingexpo.com/magento-pavilion/

As Magento specialists we offer a professional service which includes in-depth training on using the Magento platform, so you can manage your website the way you want to.

To see more of what 2J can offer you, or to check out our latest launches, visit our website for more information.

cropped-2j-logo.png

Tagged with: , ,
Posted in General 2J Design News and Updates

Reducing options can increase conversions

Here’s a handy guide on how to increase conversions on your site, which proves the age old ‘less is more’…
How Reducing Options Can Increase Your Conversions
Courtesy of: Quick Sprout
Tagged with: , , ,
Posted in General 2J Design News and Updates

Mobile and tablet browsing on the up.

mobile-browsing

Ofcom’s latest Communications Market Report is out, and it’s yielded some not so surprising results…

As most will probably suspect, mobile internet use is rising, with 51% of people now owning a smartphone, and 49% of people now using their phone to access the internet.

It’s not just the mobile users either. 32% of tablet owners said their tablet was their most important device for internet browsing. It seems more and more people are starting to enjoy the benefits of browsing on the go.

Tablet and mobile browsing is also being used in the home. Most TV programmes are now ‘interactive’. Viewers feel more involved when they can tweet comments or vote for their favourite celebrities. 81% of tablet owners use their devices for media-meshing.

Over half of UK adults media-mesh (interact or talk about the programme they are watching online) regularly whilst watching TV. On top of these media-meshers, 49% of adults media multi-task whilst watching TV. Over a third of them said browsing the internet was their favourite multi-task.

Online Sales

The fact that there are now more ways for people to get online, so people can browse anywhere accumulates to the 10% increase in online sales between May 2012 and May 2013. The phenomenon of online retail means consumers can now buy products any time anywhere, so it’s no surprise people browse from the comfort of their own home with their feet up and the telly on rather than traipsing out in the rain.

Mobile Sites – Future proofing

What does this all mean for business? Probably the most important fact to come from this report is that more than 30% of webpage traffic came from mobiles and tablets last year. A mobile website is the best way to keep up with the advancements in technology and future proof your business.

The other option, which is becoming more popular, is to have a responsive design, which shrinks down to fit all device sizes. This saves retailers having multiple sites to suit their customers’ browsing platforms. To find out more about responsive designs, click here.

 

Tagged with: , , ,
Posted in General 2J Design News and Updates