TriCore Solutions Relies on Bacula Systems

Challenge
With TriCore solutions’ existing backup software, backup to tape was taking increasingly longer as its data center grew in size.

TriCore Solutions was growing month over month and needed to keep adding additional tapes and drives for its tape library. It was getting progressively more difficult to keep backup-to-tape time short enough to be completed in a timely manner so that the tape company could move them offsite.

TriCore Solution’s environment is mostly HPE-oriented. As an HPE partner, it has HP DL380’s, HP DL 460’s (for the enterprise servers), and an HP tape library with HP tape drives. All data backs up to an Infinidat SAN; Infinidat also being a TriCore Solution’s partner. Altogether, TriCore Solutions has well over a 1,000 servers and is building additional servers at a rapid month pace.

A large percentage of TriCore’s environment is Linux, however it also has many Windows systems, running MS SQL, Exchange, SharePoint, and Lync servers.

“We utilize several server systems. The core of what we do is Enterprise Business Suite hosting, as such we run applications including Oracle Enterprise Business Suite, PeopleSoft, SAP, and Business Intelligence solutions. Therefore, we have a lot of different applications that we have to backup,” said Matt Iavarone, manager, systems and storage, TriCore.

In addition to the difficulty of keeping backup-to-tape time within schedule parameters, there was another challenge: when backing up to tape, some of the larger database systems were taking too long and this situation was not optimum for some of TriCore Solution’s clients.

TriCore Solutions therefore decided to resolve that problem, as well as make it easier to migrate data from tape to disk, by backing up to disk first.

“The existing solution we had in place at that time would have been far too expensive to do that, costing over 250K to get a disk-based solution”, said Matt.

Solution
“We had always appreciated open source based solutions, such as Red Hat, Inc.,” said Matt. “When it came to looking for a backup to disk solution, we looked at Bacula. The entry costs for backing up to disk were also a reason to change to Bacula. “It was also about the scalability costs: we wanted to scale out to multiple backup servers and again, the existing solution would have been very costly.”

The evaluation period was a two-step process.

“First we looked at the community version of Bacula”, said Matt. “I was already familiar with it, some of the others on the team were, too. First, we did test backups to check that we could get through those backups quickly. Then, we figured out how to scale out the Storage daemons. We have well over 1,000 servers that we are backing up and we are adding new servers and storage all the time.  We wanted to make sure there was enough time in between each backup to be able to do things such as restart, or reload the Storage daemon, or add new storage. In addition we wanted items laid out in a way where we were free to do those kind of things, and still have the ability to continue to grow. When we saw there was a Bacula Enterprise version, with plugins available, we decided to purchase the Subscription.”

Evaluation of the Plugins took longer than anticipated.

“We now use the Bacula Oracle plugin, and that required a few months of testing. When it was time for the migration to Bacula, we wrote the scripts that collected data from the servers we had, we then created the configurations, the job files, and the client files for Bacula. It was an incredibly smooth process. We are testing the deduplication technology now, for data reduction, and it is working very well for us,” said Matt.

“We have several different types of data, and Bacula presents opportunities to become very efficient”, says Matt.  “For example, we have a good number of Oracle databases – up to 500, and using Bacula Systems’ Oracle plugin has saved huge amounts of disk space. Previously, we were backing up our Oracle databases with RMAN to disk and were keeping up to 3 days retention on disk. But with Bacula’s Oracle plugin, we have been able to save around 60 to 100TB of on-disk volume space for backups”.

“Migration from the previous backup solution to Bacula took about three weeks”, says Matt. “We did our POC’s and checked backups and restores across all of our different systems. Then we did a re-implementation of it and started adding Bacula to anything new that came into the environment. After we saw that it scaled up very well, we started migrating from the previous system.”

Result
“At this point, our cost savings with Bacula Enterprise Edition have been significant. Because the previous solution that we had was charging by disk volumes, it was more expensive. TriCore utilizes several licenses, with Bacula we were able to become more efficient and decrease costs,” said Matt.

“We were surprised at Bacula’s fast restore speeds, when we first did the testing. We were restoring groups of files in only one minute, where it used to take thirty minutes or an hour with the previous solution. We back up enormous amounts of data – and it goes really quickly; in just a few hours. Bacula’s compression is also really helpful, because it reduces network overhead. In addition, deduplication is saving us 60 to 80TB of space. We find Bacula Director really simple to use, and the Bacula Console is very straightforward,” said Matt.

“Life is easier because Bacula is open source-based,” said Matt. “We have run into a couple of situations where, if  we were working on a configuration and something is not working in the way we expect, we can look at the source and figure out exactly what that option should be doing. Also, we can look at different ways to utilize its flexibility: for example, we have just completed a Proof of Concept where, instead of going to tape with our disk backups, we migrate to Amazon Glacier. In the Bacula catalogue database, we customized a ‘comments table’ and modified it to what is called the ‘JSON data type’, so that we can take all our Amazon information, upload a volume from Amazon information and store it directly on the database.  With other vendors we have worked with in the past, their database was proprietary and closed and we would never have been able to do that. We also would have had to have two databases in place.”

“Bacula Systems Director makes it easy to access via a command line interface, and a database in the media table keeps track of all the on-disk volumes and the tape volumes that we use. That is where the comments column is, and we can use it for the Amazon data. So we have a process that, once a backup completes, there is an option in the configuration of the Job to run a script. It is that script that starts the process for us, updating the database for each step of the process. We maintain all the JSON data: when it is time to do a restore, we just kick off a download from Amazon”, says Matt.

“Now we have done the POC for backing up to Amazon, and we are almost ready to put it into production. We expect to put the majority of our daily backups over to Glacier. Our ‘full’ backups are typically over 200 TB, and copying all that data to tape can take two days to finish. When the ‘daily’ then starts the day after, it just gets queued up behind the full backups. And if a copy job fails, it could get further and further behind, and we would end up using more disk space. So taking the Daily’s and putting them in AWS will give us more opportunity to solve those issues.”

“Because we offer managed services, we get many special requests from customers. One example of this would be where we have clients that have their financial systems hosted by us. Typically, these customers have different financial year-ends, sometimes at the end of March or at the end of September. Bacula makes it easy for us to change data retention times according to these customers’ calendar needs. So we can make sure that the data is retained for as long as is necessary.“

 “Another thing is that Bacula’s scalability is great. We can add as many Bacula Storage daemons as we need, and there is little overhead with the Director. The Storage daemon takes the workload! We have added at least 1,000 servers to our environment, and during that time we have noticed that Bacula Enterprise Edition was always easy to manage. We are growing fast, and Bacula Enterprise Edition makes it easy for us to keep up with everything,” said Matt.