R1Soft – CDP 2.0, CDP 3.0, cPanel Integration, Delays, and Poor Support (Updated)

R1SoftLet me start by saying that R1Soft, when it works, is an excellent solution that has on a few occasions saved us from partial or complete data loss in the event of an unexpected hardware failure or other unexpected data issue.  That is about the extent of what I have to say good about R1Soft which is extremely unfortunate.

My experiences with R1Soft formally began on March 12th, 2009 when we first obtained our trial R1Soft license.  I was in contact with David Wells from R1Soft after having faced an issue with some lost MySQL databases due to a mistake made by a technician when performing some maintenance on a server.  When discussing the issue with David he made it clear that with R1Soft backing up the server not only could we have restored those destroyed databases but we would also be protected against total data loss such as a catastrophic server failure.

We used the original trial license until March 30th, 2009 when we purchased the R1Soft Linux CDP starter pack for $500 including 5 Linux CDP agents, 1 MySQL addon, and 1 Archiving addon.  This was an excellent deal and up until this point R1Soft had lived up to every promise and we were very happy with the software.

David at R1Soft mentioned CDP 3.0 was coming out soon and mentioned, if memory serves, within the next quarter.  Promises were made such as faster backups, faster restorations, more reliable operation, cPanel integration, a more streamlined interface, and a lot of other features that surely anybody else running R1Soft 2.0 would love to see.  I’ll cut straight to the point – it was nearly 2 years before the 3.0 version of the R1Soft CDP was released in beta and even then it didn’t include all of the features that were promised and was even missing many of the key features of 2.0.

R1Soft 3.0 beta was released as a “standard edition” which only allowed backing up to the same server on either a secondary disk or network mounted storage.  There was no centralized backup server and when I looked into it no bare metal restoration – both features that had been available in 2.0 for over 2 years.  When the “enterprise edition” was finally released in beta, years after it was promised, it did not include bare metal restore or cPanel integration.  You are reading this correct – basic 2.0 features were totally absent from the 3.0 version of the R1Soft CDP even with it having been released nearly 2 years after it was originally promised.

With version 2.0 over the years and 3.0 the short period that we’ve been using it – we’ve always had strange issues that we’ve reported that have been entirely ignored.  We’ve had issues where 2.0 would cause kernel panics, would simply fail to back up, or would fail to restore that support either stated they could not replicate, that they claim don’t exist, or that they claim will be fixed in “the next release” or “sometime soon” that still happen.  We’ve had tickets where we gave them full and complete access to a server to diagnose and reproduce the issue, view and download logs, or anything else they needed to do and gotten back canned responses telling us how to do things on WINDOWS R1Soft backup servers or agents when we run exclusively Linux and made it clear in the ticket.

One issue that we, as well as several providers I network with, have faced with CDP 2.0 is when an end user starts a restoration and does not select the “overwrite files” option and “overwrite failed” errors cause the agent to fail on the restoration.  Now it wouldn’t be quite as bad if the agent just failed out and quit – but it actually fails “on” and consumes a full CPU core indefinitely until an administrator manually kills the restoration process.  There are several valid reasons to do a restoration without overwriting such as if you wish to replace any deleted files inside of a directory tree from a backup without overwriting any files not deleted – this error makes this impossible.  One provider I speak with opened a ticket about this on or around April 9th, 2009 and this issue has yet to be resolved.

Right now R1Soft 3.0 is extremely unstable on OpenVZ kernels, and has been since it has been released.  I know of approximately 5 kernel issues that R1Soft 3.0 has on various CentOS, CloudLinux, and OpenVZ kernels that cause the backups to either simply fail, or worse, for the entire server to hang or lock up.  Not only are R1Soft agent licensed extremely expensive, but they will take your production servers OFFLINE which is unacceptable.  I’ve gotten word that these issues have been resolved and will need to go through 10 business days of QA testing meaning, at minimum, 2 weeks before these serious kernel incompatibilities are resolved assuming all of their testing goes well.

With what R1Soft charges for new licenses, charges for maintenance, and the number of licenses they’ve sold, I simply do not understand how they cannot have a solid development team that can resolve issues in a timely fashion as well as building the new revisions of the software that have been promised.  Why does it take R1Soft 2 years to go from 2.0 to 3.0 when it was promised and why is the software so ridiculously unreliable, buggy, and incomplete when it finally makes it to the market?

Update 02/17/2011

R1Soft 3.0 has some “kernel” issues that occasionally causes a server under specific circumstances to lock up entirely forcing a reboot.  Their development team apparently has recently just finished finding and fixing all of these issues however the next step is to push those updates through “Quality Assurance” and then they’ll be available to those using the software.  While I am all for “QA”, I’m more for us not having to reboot servers twice a week due to the backup process causing the server to hang.

Last night the R1Soft process killed one of our servers at around 5 AM EST and unfortunately it occurred during a period of time that did not have staff coverage. The staff member who was supposed to be watching the process that decided to go awol which meant that bringing the server back online quickly fell onto my shoulders.  Coincidentally, knowing that we had staff coverage, I set my phone on silent for the first time in over two years and it just so happens that this is the night that R1Soft causes the server to hang up, while a staff member is not working like they should, and my phone is on silent.  I’m not trying to make excuses as the staff member should have been doing their job and that is definitely a failure on our part.

It is however a bad situation that should never have happened – if the R1Soft CDP Agent and Kernel Module would work like they’re supposed to the server never would have crashed and been offline.  While yes, the staff member certainly should have been doing their job which would have involved rebooting the server within minutes and avoiding extended downtime, ultimately the failure is due to the R1Soft Software.  I hope for R1Soft’s sake that no competitor brings a quality product to market although I dream about it every night.

It’s bad that a backup system that we rely on to protect our customers’ data also causes us to have to make sure we have somebody awake and watching in case the backup system takes one of our servers offline.  Hopefully R1Soft will have the fixes for these issues pushed out within the next two weeks although I won’t be holding my breath.

Share

8 comments

  1. I have been at webhosts where, trying to restore from a system failure took forever(hours and hours) because of R1Soft. When I first heard of this company I thought that it would be the next best thing since sliced bread but R1Soft seems to be a hit or miss with most providers.

    You never know if it will work when needed or have an issue out of the blue. I like the concept of what R1Soft has created and how me as an end user can restore my own files without having to contact support but even then the menus take a bit to load.

    I would really like to see this software develop to a very reliable and “stable” software that is cost effective for all webhosts and become mainstream.

    I think that the prices they charge are to high but then again, if a webhost is willing to invest in R1Soft CDP you know that they are trying to offer as much as they can to the end user.

  2. I was nervous when i saw my site down and even afraid about using mddhosting for starting my business as a reseller.
    With your clear explanation and explaning the reason why this happened with the assurance that it wont happen again im really happy for being a part of such a passionate entrepreneur’s hosting.
    Thank you Mike.

  3. I totally agree with you! We started using R1Soft to backup our web servers from July, 2010. Until now R1Soft CDP has caused 10+ crashes on 3 servers. Every time their support told us “it will be fixed in the next release” after a long time investigate. Sometimes they asked us to provide cdp logs although we already uploaded. I was wondering if they really understood the hosting industry. Once after we finally got “the next release”, we met a new bug on a server. I have never seen such a bad commercial software. R1Soft is lucky as there’s no second choice for their customers.

  4. Hi Mike

    Any experience on the last update they have released? We are on the fence and considering a purchase.

    Thanks.

    T

  5. A friend and I are in time of making a hosted backup solution, which simply download and compress the standard backups provided by cPanel. It is already working, but we want to make better systems before releasing it.

    We will maybe also make a self hosted version, but I am not sure on that part yet.

  6. There is absolutely no disputing the statements about 3.0, we tried to move to it at the hosting company I work for, and it was an absolutely horrible experience. I can say I’ve not seen the lockup/crash issues, but I’m also commenting on this quite a bit late. CDP2 has for us been stable, and saved our bacon more that a few times. We’ve had hardware failures in the CDP stack that have caused oh so many headaches, but that’s another story. How about some decent tape integration.

    R1: “Oh we have the archiver module it uses tar”.
    Customer: Oh that’s wonderful, how does it use tar?
    R1: OH! Well I’m glad you asked, you see, if your a user, you can download a tar.gz of your disk safe or files within it.
    Customer: Ok………so what about tape?
    R1: But don’t you see? IT USES TAR!
    Customer: did you know that as an admin I can’t use the archiver module on a cusomers disk safe so it’s really really really really useless to me?
    R1: But did you know that tar stands for TAPE ARchiver?
    Customer: I hate you just a little bit right now.

    (NOT AN ACTUAL CONVERSATION WITH R1)
    It just kind of seems this is how it’s looked at. I mean if your an enterprise backup solution, you should at least have some cgi scripts wrapping around mt/mtx and allow for archival of your disk safes. As many of us have learned, RAID’s fail, and backup to disk only, is just asking for the goddess of chaos to lean over and give you a kiss right between the cheeks with a redwood tree.

  7. Any host that takes hours to restore a backup does not have a sufficient network setup.
    At work all our hosting servers have both an internal and external network, both running at 1Gbit/s port speeds, and can backup at rates of 50-80MB/s easily across the mostly unused internal network.
    Also having a reasonable number of users per server (<100) helps to ensure that when the worst happens you can get things back up faster.
    With todays virtualisation technologies it's also increasingly easy to recover a dead server on to another physical box in far less time than before.

  8. @Iain Kay Our backups run over our private network which is gigabit end to end. The bottleneck with R1Soft (2.0, 3.0) is not the network, unfortunately.

    Now if we are talking cPanel backups themselves, you should be able to copy them on a gigabit network at roughly 100 megabytes/second so it shouldn’t take terribly long.

Leave a Reply

Your email address will not be published. Required fields are marked *