CubeBackup 2.6 released

We are proud to announce the release of CubeBackup version 2.6.  If you are using an old version of CubeBackup, you may have already received a prompt to upgrade. You can easily upgrade to the new version at any time by using the “Check for update” feature.

New features in CubeBackup 2.6

  • Support for secondary domains and domain aliases.

As a Google Apps administrator, you may manage one primary domain along with one or more secondary domains.  CubeBackup 2.6 makes it convenient to manage all backup users in one place, whether they are in a primary domain or secondary domain. Domain aliases are also well supported in the new version.   For example,  in the user management window, domain administrators can manage all users in one list, including user1@primary.com, user2@secondary.com  and  user3@alias.com.

  • Support multiple primary domains.

If you are a Google Apps partner or reseller, you may manage several Google Apps domains for your clients. The new version of CubeBackup allows you to backup multiple domains in one application.

  • Option to back up “Shared with me” files in Google Drive.

Old versions of CubeBackup always back up shared files by default, with no way to disable the feature. For some Google Apps domains, it may cause considerable data duplication and backups could potentially grow to unmanageable sizes. The new version allows you to choose whether or not to backup these shared files.

  • Option to backup “All mail” messages

The domain administrator now can choose whether or not to backup the “All mail” folder/label in Gmail.  Generally, if messages in “All Mail” folder are duplicates which already exist under other folders/labels, including them again would be a waste of time and space.  However, if the “All mail” folder is used to archive mail messages, while the “Inbox” or other labels are reserved for active messages,  the “All mail” option should, of course, be used.

  • Database performance tuning

SQLite database is used in CubeBackup to record the backup metadata. Performance and transactional security are both improved in the new version.

  • Bug fixed

A number of bugs have been fixed in the new version.

 

How to fetch users in secondary domains after the update?

Because secondary domain users were not visible in previous versions of CubeBackup, you need to refresh the user list in order to fetch the users in secondary domains (or domain alias) after the upgrade.

Open the menu item “Settings” -> “User management” , and click the Refresh button. All users, in both primary or secondary domains, will now show up in the users list.  Click the Save button to save all the settings.

user managment

 

From time to time, we receive emails from our customers asking new features. Thank you for all of your suggestions and feedbacks!   The most commonly asked question is:

What to expect in the future versions?

  • Parallel backup

This will be the most important feature in the next version of CubeBackup.   Parallel backup for multiple users can greatly accelerate backup speeds and should be available soon.

  • Google site backup

Google site data backup will also be added in future versions.

  • Restore backup data to another account

Restoring data to another account can be very useful in some circumstance. Look for this feature in future versions

  • Linux version

CubeBackup for Linux will be a service without UI,  and we are confident that Linux administrators will love it! Most features in the Linux version will be almost identical to those in the Windows version.

How to speed up backups for large organizations

Schools or large companies with hundreds of users can easily accumulate terabytes of Google Apps data, including tens of millions of Gmail messages. This presents a serious problem for backup strategies – the initial backup may take months or even longer!

One reason for this is that CubeBackup performs user backups one by one in a non-parallel manner. We plan to include parallel backup algorithm in the next version of CubeBackup, but in the meantime, there are still methods that can be used to greatly speed up large backups by working with the user data in parallel.

Before presenting some of these solutions, I’d like to give an introduction to how backup data is saved locally.  Simply put, all backup data for each user, including the actual Google Apps data and metadata used to record the backup progress for each user, are stored in that user’s own directory. For example, data for user “somone@yourcompany.com”  would be stored in the directory “someone@yourcompany.com” like this:

Folder structure for one user

Folders like “calendar”, “contacts”, “docs”, and “mail” are used to store the actual backup data for Google Apps. There is also a hidden folder named “_config”, which is used to record important metadata, like timestamps for the last backup, the cloud ID of the local backup files, etc. CubeBackup itself only stores general settings for the whole app, all user data, whether actual backup data or metadata to describe a user’s current status, are stored in that user’s own directory.

This design makes the backup data very portable, as well as bringing an extra benefit: several CubeBackup instances can run simultaneously while sharing the same local storage. This means that, for large companies with hundreds of employees, the best way to speed up the backup process is to split the users into separate CubeBackup instances on different computers. Let me illuminate this method with a specific example:

An Example

Assume there is a large Google Apps domain “example.com” with 1000 users, 200 of whom are located in the subdomain “sub.example.com”. There are also several organization units in the main domain (for example, 100 users in marketing 100 in dev, 50 in HR, etc.). The whole structure of the domain looks like this:

domain structure

For such a large domain, a complete initial backup may be so enormous that it would take months to back up all the data locally, which, of course, is unacceptable. In most cases, a professional local storage unit, such as a NAS or SAN, will be used to store the backup data. Usually, these devices have excellent write speeds and are capable of handling very high data throughputs. In this case, you can split the users into different CubeBackup instances running on separate computers, while setting the backup location for each instance to the same NAS/SAN device.  Since CubeBackup can be run on almost any computer, one powerful server and several low-speed desktop computers make a good combination for this strategy.

parallel backup

This parallel backup strategy can speed up the backup process tremendously. Since CubeBackup consumes very little memory and CPU resources, it can run in the background on almost any computer without interrupting the user. This makes it is very easy to add more computers to this kind of parallel backup system.

How many computers should be used to speed up the backup?

In most cases, the bottleneck of the parallel backup system lies in the speed of the NAS/SAN. Since most Gmail message backups are small files, the speed at which these small files can be written to local storage is the key factor in deciding how many computers can be added to the parallel system. Due to the wide spectrum in the performance of different storage devices, it is difficult to predict the optimal number of computers.  We suggest starting small, with 4 computers, and then adding more running instances to the parallel backup system to find the best combination.

Merge all users into the main server after a few backups

Because CubeBackup employs an incremental backup algorithm, subsequent backups after the initial full backup only store new or modified data, which is much less time-consuming.  Therefore, it shouldn’t be necessary to keep so many CubeBackup instances running after the first several backups; the main server should be sufficient to handle most of the load.
final structure

Another option for parallel backup:

In some cases, there is no NAS/SAN available within your company. Instead, each department is responsible for maintaining their own set of backup. In this situation, CubeBackup can be run directly on computers within the department, with backups split according to organizational units, like this:

OU backup

This method does not require high-performance local storage and can use existing PCs for the backup. However, since there is no central data storage, and local storage within each PC is not as stable or reliable as a NAS/SAN, maintenance of the backup data may become a problem.

We plan to add parallel backup and multi-domain support in the next version of CubeBackup.  If you have any suggestions for our product, please let us know!

 

CubeBackup 2.0 is released

Today we are shipping CubeBackup 2.0, a major release that has been in the works for 3 months. There are lots of bug fixes and feature improvements in the new version.

Can I continue to stay with CubeBackup 1.x?

No, you can’t!   Based on Google’s API deprecation statement  http://googleappsdeveloper.blogspot.com/2014/12/reminder-to-migrate-to-updated-google.html  and  https://developers.google.com/identity/protocols/OAuth_ref, several APIs used in CubeBackup 1.x will stop working after April 20, 2015.  As a result, CubeBackup 1.x  will no longer be functional after that date. It is important to upgrade CubeBackup to the new version in order to secure your Google Apps data.  We are sorry for the inconvenience, but this upgrade is not optional.

What’s new in CubeBackup 2.0?

  • Two-Legged OAuth 2.0 (OAuth2.0 service account) for domain-wide authorization.

CubeBackup 1.x uses 2-Legged OAuth1.0 for authorization. When CubeBackup 1.0 was in development, OAuth2.0 was not yet mature enough for Google Apps domain-wide authorization and hardly any code libraries could be found.  However, this situation has changed, and now CubeBackup has been completely moved to OAuth2.0, which offers superior security and flexibility.

  • Migration to the latest Google Apps APIs.

The following APIs are used in CubeBackup 2.0:

    1. Admin SDK Directory API
    2. Google Drive API  v2
    3. Gmail REST API 
    4. Google Contacts API v3
    5. Google Calendar API v3
  • Select users by Organization Unit.

With the new Google Admin Directory API, CubeBackup allows you to select backup users by their organization unit. This feature can be very useful for large companies who might wish to apply different backup policies on different departments.

  • Excellent multi-language support.

CubeBackup has been completely redesigned to support all languages. It can easily handle multiple languages in a single Google Apps domain now.

  • Backup shared files in the Google Drive “Shared with me” folder.

Shared files were not backed up in the old version of CubeBackup.  In CubeBackup 2.0, the “Shared with me” files are fully supported.

  • Fully backup the whole Gmail folder structure.

In CubeBackup 1.x, only messages in “Inbox”, “Sent Mail”, “Starred” and “All Mail” were backed up.  In the new version, the whole mailbox structure, including customized folders/labels, are all included in the backup.

  • Improved performance.

Code optimization and the adoption of the latest Google API has improved backup performance in CubeBackup 2.0.

  • Bug fixed

Many bugs have been fixed in the new version.

Steps to migrate to the new version.

  1. Uninstall CubeBackup 1.x.
  2. Download the latest version of CubeBackup from cubebackup.com.
  3. Install CubeBackup 2.0.

    During the install process, you might be asked to install .Net Framework 4.0 if it’s not preinstalled.   For Windows XP, Windows Server 2003 or Windows Server 2008 users,  KB2468871 hotfix may also be required in some cases. Please visit the installation page to get more information.

  4. Start CubeBackup, and follow these instructions for authorization and configuration.

    The OAuth2.0 Service account (2-Legged OAuth2.0) authorization can be complicated, so please follow the above instructions carefully.

Because significant changes have been made in CubeBackup 2.0, the backup data is not fully compatible with that of CubeBackup 1.x. Google Drive/Docs and Google Contacts backups are unchanged, but Gmail and Calendar backups are now handled in a slightly different way.

When specifying the backup location, there are two options:

A. Delete all data backed up by CubeBackup 1.x and use CubeBackup 2.0 to backup your Google Apps data from the beginning. This gives you a clean start, but may take longer for the first backup. (Recommended)

B. Keep the backup data of CubeBackup 1.x, and specify the same folder as the backup location for CubeBackup 2.0. Google Drive and Contacts data will be backed up incrementally based on the old data, but CubeBackup 2.0 will create a new backup for Gmail and Calendar data. The new data should overwrite the old backup without any problems. This method saves backup time but makes the backup folder a bit more messy.

If you have any problems or suggestions for CubeBackup, please do not hesitate to contact us:  support@cubebackup.com.   Thanks!

 

Price for educational organizations

Google Apps Logo NewRecently, we’ve received a lot of emails inquiring about the pricing for Non-Profit/Educational organizations. Since Google Apps is so popular among students and educators, these users are also very important to us at CubeBackup. We are pleased to offer a discount for Non-Profit/Educational organizations:

$99   USD/year      Up to 50 accounts

$499 USD/year      51-500 accounts

$999 USD/year      501-2000 accounts

$1999 USD/year     2001-5000 accounts

$2999 USD/year     more than 5000 accounts

These prices are heavily discounted and only available to non-profit or educational organizations.

Non-profit and education purchases can be made at https://www.cubebackup.com/pricing-siteprice.html .

Reseller Program: 

We are also looking for new members for our reseller program. If you are a Google Apps reseller/partner and are interested in introducing CubeBackup to your clients, please contact our sales team for more information.

CubeBackup is selected for Google Startup Launch

As a development team that focuses on Google platform, our products are heavily related to Google’s APIs.  We are glad to know that our team has been selected for Startup Launch by Google’s Developer Relations team.

Screen Shot 2014-12-05 at 4.16.27 PMGoogle Startup launch offers a series of events and training resources for startup teams, as well as free Google cloud service credit.   Google also sets up a Google+ community for startup launch teams.

It would be better if we can get venture capital support from Google startup launch 🙂

 

Automatic backup plan for Linux servers using rsync and crontab

Linux servers are widely used by lots of companies for hosting their websites, databases, or other services.  I personally likes Linux system much more than Windows Server, not only because Linux is free, but you get better performance and more powerful tools on Linux.  To set up a Linux server, sometime you don’t actually need a physical server,  subscribing a VPS from Linode, DigitalOcean or Amazon could be a better choice.  Though most cloud server providers have  backup or snapshot services (usually not free), it’s better to have your own backup plan.

The rsync command is a ideal tool for copying and synchronizing files and directories to a remote computer, while the crontab command is used to schedule jobs to be executed periodically.  Combinding these two commands, we can setup a light-weight and effective backup solution.

Due to the difference of Linux distributions, in this article, we use CentOS/Redhat system as an example to introduce how to setup up the backup plan.  However, before that, here are several questions to think about:

First,  what data should be backed up?

Generally, we only need to backup data important to us, like website pages, databases, configuration files and personal data.  It is generally not necessary to backup data such as Linux system files, installed software.

Here  are several directories that need to be taken care of:

  • /etc directory:  Though some files in this directory don’t need to backup, I wouldn’t bother to pick them out.  And since the total size of this directory is usually no more than 50 Megabyte, it would not hurt to back up the whole directory.
  • /home directory: This is the location for personal user data of all accounts (except root),  and obviously, the backup plan should cover this directory.  However, there is a problem: There are lots of cache data, log files, download software, or history records located in this directory. It is just meaningless to backup these data.  Rather than backing up the whole /home directory,  putting only specific sub-directories, such as /home/someone/gitdata, /home/anotherone/documents into your backup list would be a better choice.
  • /var/www directory:  This is the default directory for website files. (If your web files are located in other directories, find them out and put into your backup list ).
  • /var/spool/mail:  This is where the mail data is located, and definitely should be  backed up.
  • /var/lib/mysql:  This is the directory for holding the database data.  Include this directory in your backup list.

You may have other utilities or service data scattered in other directories which need to be backed up,  think carefully and find them out before you take action.

Second, fully backup or incremental backup?

If you want to fully backup all data listed above every time, you can compose a bash script to archive all files using Linux tar command and then send (scp) the tarball to the backup location. This method works well for backup in a local area network, but might not feasible for backing up data from a remote VPS to a local computer because hundreds or even thousands of megabytes are transferred between the two remote computers each time. It is a waste of bandwidth and disk storage.

Incremental backup, which employs the rsync utility in Linux, backs up only modified data each time. For most cases, this is a right choice due to its efficiency and cost-saving.

Where to store the backup data?

Generally, backup data should be stored on a remote computer, either another Linux VPS, or a Linux computer inside your company.

How to schedule an automatic backup plan?

Cron job is the best choice to schedule command to be executed periodically, for example, a backup script can be scheduled at midnight each day.

The following are the detailed instructions for making an automatic and incremental backup:

Host A: the active server with CentOS/Redhat system.

Host B: the backup service with CentOS/Redhat system.

I.)  Make sure rsync is installed on both Host A and Host B.  If it is not installed, install it using command:

yum install rsync.

II.) Login to host B using root account (Crontab requires a root user permission, though theoretically it can be done using a non-root account, but not easy) ,  and create a directory to hold the backup data:

mkdir -p  /var/ServerBackup

III.) Generate the SSH key pair :

ssh-keygen

Two files are generated in the /root/.ssh directory: id_rsa is a private key file, while id_rsa.pub is a public key file, which must to be copied to Host A:

scp /root/.ssh/id_rsa.pub  root@A.com:/root/id_rsa.pub

IV.)  Login to host A using root account, and attach the content in id_rsa.pub file to the authorized_keys file.  If /root/.ssh/authorized_keys file doesn’t exist in host A, execute the following commands to create it first:

 mkdir -p /root/.ssh
 chmod 700 /root/.ssh
 touch /root/.ssh/authorized_keys
 chmod 600 /root/.ssh/authorized_keys

To attach the public key generated in host B to authorized_keys file:

cat /root/id_rsa.pub >> /root/.ssh/authorized_keys

Now we can use scp or rsync to transfer data from host A to host B without password required.

V.)  Modify /etc/crontab file to schedule the execution of backup script.  Add this line to the end of the crontab file:

0 2 * * * root bash backup.sh     # the script file backup.sh is scheduled to be executed every day at 2:00AM.

The content of the backup.sh script is something like this:

#!/bin/sh

/usr/bin/rsync -avz -e "ssh -i /root/.ssh/id_rsa.pub"  root@A.com:/etc  /var/ServerBackup
/usr/bin/rsync -avz -e "ssh -i /root/.ssh/id_rsa.pub"  --exclude mysite/updraft  --exclude mysite/.cache    root@A.com:/var/www   /var/ServerBackup
........  (other similar commands)
/usr/bin/rsync -avz -e "ssh -i /root/.ssh/id_rsa.pub"  root@A.com:/var/lib/mysql   /var/ServerBackup

This script is just a sample and you can modify it based on your need.  You can use Linux man pages to get more usage of rsync.

Now you can rest easy without worrying about the data loss.

Introduce Google Apps development at GDG DevFest conference

jerry

Last weekend, I was invited as a lecturer in the Google Developer Group DevFest 2014 Beijing to introduce Google Apps development as well as our product CubBackup. DevFest is the biggest annual activity for GDG, where developers who are interested in Google’s technology can meet each other and learn new technologies together.

Though most of Google’s services are blocked in China, there are still lots of developers/fans of Android, Chrome, HTML5, Android Wear, etc. More than six hundred of developers came to GDG DevFest Beijing this year!   I am glad to know some excellent developers there and really impressed by them.  This is also my first time to meet some famous developers face to face whom I already know on Internet before.  Some young developers are teenagers in junior high school ( I knew nothing when I was in their age ).

Topics in this DevFest cover Android Ware, Material Design, Android Lollipop, Polymer, HTML5 and so on.  I learned a lot  in all the technical sessions, however, what impressed me most is the creative spirits there. All people are talking about technology trend, open source projects, start up a new business… Beijing is just like another Silicon Valley in Asian.

I introduced Google Apps and shared some of my experience on how to develop on Google Apps platform in this DevFest activities.  It was an amazing experience to communicate with so many Google developers face to face.

all

Why a backup solution for Google Apps is necessary?

It is said that Google’s service is so secure that massive data loss has never happened on Google’s cloud platform. Why do we still need a backup service?

The answer is that most data loss is due to user errors! Here are a few common scenarios:

  1. Employees might accidentally delete an important document or email without even noticing. This is an easy-to-make mistake, especially on mobile devices, where an unintentional gesture is sometimes all it takes.
  2. Disgruntled employees may be able to destroy important documents or data.
  3. Hackers can break into your cloud system and ruin your data.
  4. Google sometimes blocks accounts due to a policy violation. For example, an employee in your marketing department could accidentally send too many emails in a relatively short time and be automatically blocked.
  1. When employees leave the company, data loss can sometimes happen.
  2. Power outages, earthquakes, hurricanes and other disasters can cause data loss or make the internet inaccessible. A local backup is indispensable in such situations.

Though Google’s platform is very stable,  a  backup solution, such as CubeBackup, definitely adds more security to your business data.

Looking for investors for CubeBackup

Our team is currently seeking investors for CubeBackup.  After  making CubeBackup online, we need more financial support for marketing our product.   At the same time, developing the cloud-to-cloud version of CubeBackup is on our schedule. We are currently kind of short-handed and several more team members are needed.

Investment will make us move fast and realize business objectives more easily.  Not only money, but any resources can contribute to our marketing, customers, and products are welcomed.

If you would like to get further information about our product and our team, please feel free to contact jerry@cubebackup.com.

Windows+Office vs. Google Apps+Chromebook

Personally I don’t like most Microsoft’s products, even their most important one – Windows. After using MacBook Pro for several months, I found it’s hard to go back to the Windows platform : the shabby search capability and the awkward UI on Windows 8 often drive me mad. Honestly, Mac OSX is the best desktop Operating System that I ever met. With an excellent desktop UI plus Linux-compatible core, OSX is far user friendly, especially programmer friendly than Windows.

However, for MS Office, I must admit that it is one of the few excellent products made by Microsoft. Combined with Active Directory architecture, Server softwares like Exchange, SharePoint, it can handle almost all office work nicely and efficiently. Today, most companies use MS Office solution to handle their daily office work and  are happy with that. I don’t see any strong competitors of Microsoft in this field. ( Actually, there are a few other options, but none of them can really compete with Microsoft Office Suit.)

Things might change in the future. Microsoft now need to seriously consider the threat from Google Apps + Chromebook, which offers a lightweight office solution with much cheaper cost and better collaboration. Based on Google’s statement, there are already more than 5 million businesses on Google Apps, and the number is still increasing fast. At the same time, Chromebook has become the most popular notebook in Amazon. At least, I know several of my friends went Chromebook and has no plan to go back to Windows.

For most companies, Google Apps platform is already good enough to handle their office job nicely, and cheaper! Here I made a simple comparison between Microsoft office solution and Google Apps solution.

To setup the Microsoft Office environment:

  1.  A small data center, which requires a physical place with good security.
  2.  Several Windows Servers, like Domain Controller, Exchanges Server, Web Server, Database Server, Security Server …. They are expensive on both hardware and software!
  3. Desktops or laptops with Windows and Office.
  4. An IT group to maintain the above hardware and software.

This solution is a classical solution for most companies, and usually it works great. However, there is one problem: the expense! Though the cost may differ for different type and different size of companies,  it does cost a fortune to set all things up. Not every company can afford that. To lower the expense, some companies have no choice but unwisely lower the security and the maintenance cost, which may cause disastrous results in the future.
To setup a Google Apps working environment:

  1. 50 bucks for each employee each year.
  2. One Chromebook for each employee. The price of a Chromebook is much cheaper than that of a normal desktop or laptop.

See, Google’s cloud solution is A LOT cheaper!  Using Google’s solution, you can setup a similar office working environment for your company with much less money.
Some companies still worry about that the features of Google Apps cannot match Microsoft’s solution. However, based on my experience (might not right), this shouldn’t be the major concern. Google Apps covers most features, if not all, of Microsoft Office solution, with better collaboration capability. The biggest concern to move to Google Apps should be the migration cost, because this migration requires to retrain all employees for the new system and take the risk of data lost during the migration. For companies that have already invested in Microsoft solutions and have been accustomed to that environment, I don’t see any good reason to move to Google Apps.

Will Google Apps beat Microsoft Office? Maybe not in the next few years. However, it may happen in the future, with a great possibility.