Early adopters say the expense and time spent to revamp a data center’s switching gear are well worth it; benefits include killer bandwidth and more flexibility
By Esther Shein |
Easy to implement
Like the IT executives at Walz Group, IT team leaders at GEHA believed that deploying a fabric model would not only meet the business requirements, but also reduce complexity, cost and staff needed to manage the data center. Bryan says the association also gained economies of scale by having a staff of two people who can manage an all-Ethernet environment, as opposed to needing additional personnel who are familiar with Fibre Channel.
“We didn’t have anyone on our team who was an expert in Fibre Channel, and the only way to achieve getting the claims processing system to be redundant and highly available was to leverage the Ethernet fabric expertise, which we had on staff,” he says.
Bryan says the association has been able to trim “probably a half million dollars of capital off the budget” since it didn’t have to purchase any Fibre Channel switching, and a quarter of a million dollars in operating expenses since it didn’t need staff to manage Fibre Channel. “Since collapsing everything to an Ethernet fabric, I was able to eliminate a whole stack of equipment,” says Bryan.
GEHA used a local managed services provider to help with setting up some of the more complex pieces of the architecture. “But from the time we unpacked the boxes to the time the environment was running was two days,” says Bryan. “It was very straightforward.”
And the performance, he adds, is “jaw-dropping.” A test they did copying a 4-gigabyte ISO file from one blade to another blade through the network, with the network and storage going through the same fabric, occurred in less than a second, “and we didn’t even see the transfer; I didn’t think it actually copied,” he says.
IT has now utilized the fabric for its backup environment with software from CommVault. Bryan says the association is seeing performance of about a terabyte an hour of throughput on the network, “which is probably eight to 10 times greater than before” the fabric was in place.
Today, all of GEHA’s production traffic is on the fabric, and Bryan says he couldn’t be more pleased with the infrastructure. He says scaling it out is not an issue, and is one of the major advantages with converged fabric and speed. GEHA is also able to run a very dense workload of virtual machines on a single blade, he says. “Instead of having to spend a lot of money on a lot of blades, you can increase the ROI on those blades without sacrificing performance,” says Bryan.
Laliberte says he sees a long life ahead for data center fabrics, noting that this type of architecture “is just getting started. If you think about complexity and size, and you have thousands of servers in your environment and thousands of switches, any kind of architecture change isn’t done lightly and takes time to evolve.”
Just as it took time for a three-tier architecture to evolve, it will take time for three tier to get broken down to two tier, he says, adding that flat fabric is the next logical step. “These things get announced and are available, but it still takes years to get widespread deployments,” says Laliberte.
Case study: Fabrics at work
When he used to look around his data center, all Dan Shipley would see was “a spaghetti mess” of cables and switches that were expensive to manage and error-prone. Shipley, architect at $600 million Supplies Network, a St. Louis-based wholesaler of office products, says the company had all the typical issues associated with a traditional infrastructure: some 300 servers that consumed a lot of power, took up a lot of space and experienced downtime due to hardware maintenance.
“We’re primarily an HP shop, and we had contracts on all those servers, which were from different generations, so if you lose a motherboard from one model, they’d overnight it and it was a big pain,” Shipley says. “So we said, ‘Look, we’ve got to get away from this. Virtualization is ready for prime time, and we need to get out of this traditional game.'”
Today, what Supplies Network has built in its data center is about as far from traditional as it gets. Rather than deploying Ethernet and Fibre Channel switches, the company turned to I/O Director from Xsigo, which sits on top of a rack of servers and directs traffic. All of the servers in that rack are plugged into the box, which dynamically establishes connectivity to all other data center resources. Unlike other data center fabrics, I/O Director offers InfiniBand, an open standards-based switched fabric communications link that provides high-performance computing.
“On all your servers you get rid of all those cables and Ethernet and Fibre switches and connect with one InfiniBand cable or two, for redundancy, which is what we did,” says Shipley. The cables are plugged into I/O Director. “You say ‘On servers one through 10, I want to connect all of those to this external Fibre Channel storage’ and it creates a virtual Fibre Channel storage network. So in reality, this is all running across InfiniBand, but the server … thinks it’s still connecting via Fibre Channel.”
The configuration means they now only have two cables instead of several, “and we have a ton of bandwidth.”
Supplies Network is fully virtualized, and has seen its data center shrink from about 20 racks to about four, Shipley says. Power consumption and cooling have also been reduced.
Shipley says he likes that InfiniBand has been used in the supercomputer world for a decade, and is low-cost and open, whereas other vendors “are so invested in Ethernet, they don’t want to see InfiniBand win.” Today, I/O Director runs at 56 gigabits per second, compared with the fastest Ethernet connection, which is 10 gigabits per second, he says.
In terms of cost, Shipley says a single port 10-gigabit Ethernet card is probably around $600, and an Ethernet switch port is needed on the other side, which runs approximately $1,000 per port. “So for each Ethernet connection, you’re looking at $1,600 for each one.” A 40-gigabit, single-port InfiniBand adapter is probably about $450 to $500, he says, and a 36-port InfiniBand switch box is $6,000, which works out to $167 per port.
Shipley says the company has now gotten rid of all of its core Ethernet switches in favor of InfiniBand.
“I was afraid at first because … I didn’t know much about InfiniBand,” he acknowledges, and most enterprise architectures run on Fibre Channel and Ethernet. “We brought [I/O Director] out here and did a bake-off with Cisco’s [Unified Data Center]. It whooped their butt. It was way less cost, way faster, it was simple and easy to use and Xsigo’s support has been fabulous,” he says.
Previously, big database jobs would take 12 hours, Shipley says. Since the deployment of I/O Director, those same jobs run in less than three hours. Migrating a virtual machine from one host to another now takes seconds, as opposed to minutes, he says.
He says he was initially concerned that because Xsigo is a much smaller vendor, it might not be around over the long term. But, says Shipley, “we found out VMware uses these guys.”
“What Xsigo is saying is, instead of having to use Ethernet and Fibre Channel, you can take all those out and put [their product] in and it creates a fabric,” explains Bob Laliberte, senior analyst at Enterprise Strategy Group. “They’re right, but when you’re talking about data center networking and data center fabrics, Xsigo is helping to create two tiers. But the Junipers and Ciscos and Brocades are trying to create that flat fabric.”
InfiniBand is a great protocol, Laliberte adds, but cautions that it’s not necessarily becoming more widely used. “It’s still primarily in the realm of supercomputing sites that need ultra-fast computing.”
Esther Shein is a freelance writer and editor. She can be reached at eshein@shein.net.
Read more about data center in Computerworld’s Data Center Topic Center.