As one of the world’s largest Internet commerce platforms, eBay demands extreme data center reliability; any downtime would impact transactions worth more than $2,000 a second. Thus, its global data center team must excel simultaneously in delivering uptime and flexibility, while keeping costs down – a truly complex challenge, requiring constant innovation to be successful.
REAL-TIME ENERGY OPTIMIZATION AND BEST PRACTICES IN A MISSION-CRITICAL DATA CENTER
Since its inception, eBay amassed a portfolio of 12 data center sites to keep up with its constantly-increasing compute load. Unsurprisingly, eBay’s data center operating costs are also increasing every year as business growth continues unabated.
In response, eBay created and began executing a four-year data center plan with an ambitious goal: to cut power costs in half, while simultaneously doubling compute performance. At the same time, this new infrastructure design would enable greater operational agility and increased reliability.
To achieve these goals, eBay would consolidate its assets into a smaller number of data centers, each designed from the ground up to be extremely energy-efficient without compromising uptime.
“A successful data center strategy assumes that computing demand will grow every year, yet enables this growth without proportionate increases in operational expense,” explains Dean Nelson, eBay’s Senior Director of Data Center Strategy and Operations. “We have to break the linear relationship between compute load and operational cost. This requires regular, disruptive leaps in our energy efficiency, computational power and server utilization profiles.”
In May 2010, eBay opened the flagship facility and cornerstone of its new data center strategy, Project Topaz. Located in South Jordan, Utah, Topaz is the single largest capital investment in eBay’s history, and will house more than a third of its worldwide server infrastructure. Despite deploying all the necessary redundancies required for a Tier IV data center, Topaz can still boast a Power Usage Effectiveness (PUE) metric of only 1.4.
MAXIMUM RACK EFFICIENCY ENABLED THROUGH BEST PRACTICES
By employing innovative energy-efficient mechanisms in its infrastructure – such as an optimized building design, water- side economizers, rainwater collection and dynamic controls throughout the cooling and electrical distribution subsystems – the Topaz facility provides power and cooling to the raised floor with remarkably minimal overhead. Complementing these facility measures, Topaz also implements many best practices on its raised floor space, working in tandem with the cooling plant to contribute to the site’s LEED Gold design. Five key design features of the Topaz raised floor space are particularly compelling for their value and applicability to all modern data centers:
Hot Aisle Containment
To achieve maximum cooling efficiencies, eBay employed a hot aisle containment design that had been very carefully designed, prototyped and measured prior to construction.
At their San Jose headquarters, eBay’s data center architecture team outfitted full-scale mock-ups of multiple containment designs – experimenting with different strategies (hot aisle vs. cold aisle containment), materials (containment doors vs. plastic sheathing) and dimensions. This design process determined an optimal balance of cost, flexibility and reliability prior to mass deployment. eBay found that hot aisle containment provided more generous buffer times than cold aisle containment in the event of a CRAC failure. The team deployed containment aisles comprising rows of 45U cabinets and partnered with their cabinet and containment vendor to ensure a modular design while still minimizing leakage.
400 V distribution to each cabinet
With multiple blade servers deployed in certain cabinets, eBay wanted any given rack to be capable of provisioning loads up to 17 kW. While this is achievable with traditional 208 V distribution, Topaz enjoys significant cost efficiencies by extending higher-voltage distribution all the way to each cabinet. By supplying 400 V power to each rack, eBay minimizes step-down transformation and line transmission loss while simultaneously reducing copper utilization. Specifically, eBay provides 400 V 3-phase power to two Raritan intelligent rack power distribution units (also known as “plug strips‘’ or “rack PDUs”) in each cabinet.
The rack PDU, with internal line-to-neutral (3-phase WYE) wiring, supplies each server with 240 V, single-phase power – well within the operating range of the power supplies of virtually any IT equipment. By eliminating unnecessary voltage transformations, 400 V power reduces energy costs by approximately 2-3 % vs. 208 V distribution; and approximately 4-5 % vs. 120 V distribution. As a global vendor of rack power distribution units, Raritan provides a very large range of product configurations that accept 400 V input. While 400 V distribution is relatively new in North America, Raritan has deployed 400 V systems where it is the nominal standard in Australia and in many European countries.
Detailed Server Power Instrumentation
Arguably the most important economic metric for eBay’s data center operations team is its aggregate cost per search. And because server CPU capabilities grow so quickly with each product generation, eBay replaces its servers in two-year refresh cycles, as performance-per-watt gains outweigh the capital costs of new assets.
With each server procured, however, eBay can only truly minimize its cost per search by factoring in all costs of running the server, not only capital expense. Notably, energy costs are the single largest operational expenses for any given asset.
To enable maximum savings, eBay deployed rack PDUs from Raritan that provide precise energy consumption data for every single power supply, of every single server. Accurate to IEC/ANSI “billing grade” standards, Raritan’s Dominion PX family of rack PDUs provides continuous sub +/- 1 % accurate kilowatt-hour information for each individual asset in the data center. Built with both IT and facility standards in mind, Dominion PX can pass this information upstream to eBay’s building management and asset tracking systems in real time, achieving what Green Grid terms PUE Category 3 (or PUE 3) monitoring.
“Dominion PX can provide me with the precision required to calculate my true operational costs for every server, down to the last penny”, says Nelson. “I can understand each vendor’s efficiency profiles and drive requirements that pay for themselves in the next server RFP.”
Granular Temperature Instrumentation
Having implemented many groundbreaking mechanisms in its cooling infrastructure, Topaz further enhances its cooling efficiencies – both in cost savings and in reliability – with a real time understanding of every server’s operating environment. While most data centers monitor set point and return temperatures, this data only approximates the actual server environment at the rack. And approximations require operators to build in safety margins and cooling headroom that are, by definition, wasteful.
In contrast, eBay knows the exact inlet and outlet temperature for every single cabinet in the Topaz data center. Specifically, Topaz adheres to ASHRAE recommendations for server environmental monitoring, measuring the cold aisle at the top, middle and bottom in addition to the hot aisle. Because this information can be monitored constantly, eBay can fine-tune its cooling parameters to optimize efficiency while ensuring instant alerts for any fault condition at the server level.
Typically, this level of environmental monitoring would be prohibitively expensive to deploy in every cabinet. However, the entire Dominion PX family of intelligent rack PDUs can provide this granularity of instrumentation with no additional infrastructure costs beyond the plug-and-play temperature sensors themselves. The existing network layer already provided by the Raritan rack PDUs is leveraged to measure and monitor humidity and temperature data.
Overhead Power Distribution Busway
Most data centers employ a maze of underfloor power cabling from PDU panel-boards to each cabinet. However, this design significantly impedes airflow supply and thereby compromises cooling efficiency. Because this often occurs under the raised floor, the inefficiency is easily overlooked and can be material.
Furthermore, a standard design that provides dedicated underfloor power “whips” for each breaker also consumes a significant amount of unnecessary and underutilized copper, while simultaneously limiting flexibility in commissioning new cabinets.
To solve this problem, Topaz uses an overhead power distribution busway to allow any cabinet to easily tap into a 400 A source of power. Two busses run alongside each row of cabinets, each supplying redundant power, thereby completely eliminating the need for underfloor whips. Receptacles can be added at any location along the row of cabinets in a matter of minutes by provisioning a breaker-protected junction box, instead of waiting several days for an electrician to provision additional drops.
FACTS & FIGURES
Founded in 1995, eBay connects a diverse and passionate community of individual buyers and sellers, as well as small businesses. With more than 90 million active users globally, eBay is the world’s largest online marketplace, where practically anyone can buy and sell practically anything.
eBay is a company that cares about improving reliability given that downtime costs $2,000/second or $120,000/minute. The large scale of eBay’s operations means they also need to keep equipment and electricity costs in check.
With compute loads – and operating costs – on the rise, eBay created and began executing a four-year data center plan with an ambitious goal:
- to cut power costs in half
- to double compute performance
- to gain greater operational agility and increased reliability
- Power Management
- PX™ intelligent rack PDUs
eBay developed a model for its data centers that makes their IT operations both highly reliable and 50% less expensive to operate than the data centers they are decommissioning.
In addition to minimizing faults and lowering operating costs, eBay’s South Jordan, Utah facility has achieved the U.S. Green Building Council’s Leadership in Energy & Environmental Design (LEED) Gold ranking. Just as important as the LEED Gold efficiency ranking and Tier IV reliability is the flexibility designed into the eBay data center. It is a data center that can adapt as requirements and equipment changes and, particularly important for eBay, as demand requires accommodation for future growth.
By implementing IT industry best practices as exemplified by eBay, even modest size data centers can improve their uptime, reduce costs and become more efficient using readily available equipment and straightforward techniques and processes.