Open heart surgery on the data center: Switchgear replacement in a live facility

數(shù)據(jù)中心的心臟手術(shù)：開關(guān)柜在線更換

By Mark Johns

U.S. Bank emerged during the 1990s from mergers and acquisitions among several major regional banks in the West and Midwest. Since then, the company continued to grow through additional large acquisitions and mergers with more than 50 banks. Today, U.S. Bancorp is a diversified American financial services holding company headquartered in Minneapolis, MN. It is the parent company of U.S. Bank National Association, which is the fifth largest bank in the United States by assets, and fourth largest by total branches. U.S. Bank’s branch network serves 25 midwestern and western states with 3,081 banking offices and 4,906 ATMs. U.S. Bancorp offers regional consumer and business banking and wealth management services, national wholesale and trust services and global payments services to over 15.8 million customers.

20世紀90年代，中部與中西部幾家大型地方銀行通過兼并，成立了美國合眾銀行。隨后，又接連并購了50多家銀行。現(xiàn)在，美國合眾銀行成為了一家多元化金融服務(wù)類控股公司，總部位于美國明尼蘇達州。美國合眾銀行是美國國家銀行協(xié)會的母公司，是全美資產(chǎn)規(guī)模排名第五、分支機構(gòu)數(shù)排名第四的銀行。美國合眾銀行擁有3081個辦事處和4906臺自動取款機，服務(wù)遍及中西部與西部25個州。美國合眾銀行為超過1580萬消費者、企業(yè)提供經(jīng)紀、理財、投資、抵押、信托和全球支付服務(wù)。

Rich in history, US Bancorp operates under the second oldest continuous national charter—originally Charter #24—granted during Abraham Lincoln’s administration in 1863. In addition, U.S. Bank helped finance Charles Lindbergh’s historic flight across the Atlantic. For sheer volume, U.S. Bank is the fifth-largest check processor in the nation, handling 4 billion paper checks annually at 12 processing sites. The bank’s air and ground courier fleet moves 15 million checks each day.

美國合眾銀行歷史悠久，擁有全美第二古老的營業(yè)執(zhí)照，是林肯政府于1863年頒發(fā)的。此外，還資助了查爾斯·林德伯格飛越大西洋這一歷史壯舉。美國合眾銀行是全美第五大支票商，12家網(wǎng)點每年要處理40億張支票業(yè)務(wù)，每天通過空運與陸運的支票數(shù)達到1500萬張。

Energy Park Site

Energy Park概況

U.S. Bank relies on its Energy Park site in St. Paul, MN, to support these operations. Energy Park comprises a 350,000-square-foot (ft²) multi-use building that houses the check production operations and 40,000-ft² data center, as well as support staff for both. Xcel Energy provides two 2,500-kilovolt-ampere (kVA) feeds to the data center and two 2,000-kVA feeds to the rest of the building.

美國合眾銀行依靠明尼蘇達州圣保羅的Energy Park數(shù)據(jù)中心維持業(yè)務(wù)運營。園區(qū)有一座35萬平方英尺的多功能大樓，運營其支票業(yè)務(wù)和4萬平方英尺的數(shù)據(jù)中心，并配備了后勤人員。Xcel Energy公司為數(shù)據(jù)中心提供雙路2500kVA電力供應(yīng)，為大樓其他業(yè)務(wù)提供雙路2000kVA供電。

The utility’s data center feeds supply power to two automatic throw over switches (ATO); each ATO feeds two transformers. Two transformers support the data center, and two other transformers support check production and power for the rest of the building, including offices and HVAC (see Figures 1-3).

通過將市電分配至2個自動雙投開關(guān)ATO，ATO再分別供給2臺變壓器。其中2臺變壓器給數(shù)據(jù)中心進行供電，另外2臺變壓器給支票業(yè)務(wù)及大樓其他部分提供電力，包括辦公與空調(diào)用電。見圖1～3。

Figure 1. Temporary stand-alone power plant

圖1 臨時發(fā)電站

Figures 2 and 3. Utility transfer and ATS

圖2和圖3 變壓器與ATS

A single UPS module feeds the check production area. However, two separate multi-module, parallel redundant UPS systems feed data center loads. Four N 1 1,500-kilowatt (kW) standby-rated engine generators backup the three UPS systems through existing switchgear distribution. The data center switchgear is a paralleling/closed-transition type, and the check production area switchgear is an open-transition type. The remaining office area space is not backed up by engine generators.

支票業(yè)務(wù)區(qū)由一臺單模塊UPS進行供電，2臺獨立多模塊UPS并列運行給數(shù)據(jù)中心進行供電。4臺1500kW（N 1）冗余發(fā)電機組通過開關(guān)柜給3臺UPS提供后備保障。數(shù)據(jù)中心開關(guān)柜采用短接型（先合后斷）切換開關(guān)，支票業(yè)務(wù)區(qū)開關(guān)柜采用非短接型（先斷后合）切換開關(guān)。辦公區(qū)沒有配置發(fā)電機。

Project Summary

項目概述

To ensure data center reliability, U.S. Bank initiated an Electric Modernization Project (data center electrical distribution). The project included replacing outdated switchgear and UPS systems, which were no longer supported by the manufacturer. In the project’s first phase, Russelectric paralleling switchboards were selected to replace existing equipment and create two separate distribution systems, each backed up by existing engine generators. Mechanical and UPS loads are divided between the two systems, so that either one can support the data center. Switchgear tie breakers increase overall redundancy. The facility benefits from new generator controls and new switchgear SCADA functionality, which will monitor and control utility or generator power.

為了保證數(shù)據(jù)中心可靠性，美國合眾銀行對數(shù)據(jù)中心進行了現(xiàn)代化配電改造，包括更換制造商不再維保的開關(guān)柜和UPS系統(tǒng)。首先，用羅斯電氣公司生產(chǎn)的重疊轉(zhuǎn)換開關(guān)柜替換現(xiàn)有設(shè)備，并建立兩套獨立配電系統(tǒng)，現(xiàn)有發(fā)電機作為備用電源提供保障。兩套配電系統(tǒng)為動力負載和UPS負載供電，任一配電系統(tǒng)都可支撐數(shù)據(jù)中心運行。通過開關(guān)柜內(nèi)聯(lián)絡(luò)開關(guān)，提高了供電的整體可靠性。新的發(fā)電機控制系統(tǒng)和新的開關(guān)柜SCADA系統(tǒng)將對市電和發(fā)電機供電進行監(jiān)控。

Since this project was undertaken in a live facility, several special considerations had to be addressed. In order to safely replace the existing switchgear, a temporary stand-alone power plant, sized to support all of the data center loads, was assembled in a parking lot just outside the building’s existing electric/switchgear room (see Figures 4-6). The temporary power plant consisted of a new utility transformer, powered from one of the utility’s ATOs, which supplies power to an automatic transfer switch (ATS). The ATS supplies power from either the utility feeds or the standby-rated engine generators to a new distribution switchboard to support data center loads. The switchboard was installed inside a small building to protect it from the elements. Maintenance bypass switches enable staff to work on the ATS.

該項目由于是在線改造，需要考慮到特殊情況。為了順利更換開關(guān)柜，在大樓配電室外的停車場位置搭建一個臨時的發(fā)電站，足以支撐整個數(shù)據(jù)中心全部負載（見圖4～6）。臨時發(fā)電站有1臺新變壓器，由現(xiàn)有的1路市電ATO進行供電，輸出引至ATS。ATS完成市電與發(fā)電機互投后，輸出至新配電柜給數(shù)據(jù)中心負載供電。配電柜安裝于一座小建筑物內(nèi)以免受到干擾。通過旁路檢修開關(guān)，運維人員可以對ATS進行維護。

Figure 4. Maintenance bypass switches were installed to allow for work on the ATS

圖4 安裝檢修旁路開關(guān)，以便檢修ATS

Figure 5 (Top) and 6 (Bottom). Switchboard was installed in a small building

圖5（上）和圖6（下） 小建筑物內(nèi)的開關(guān)柜

Each standby-rated engine generator has two sources of fuel oil. The primary source is from a bulk tank, with additional piping connected to the site’s two existing 10,000-gallon fuel oil storage tanks to allow for filling the bulk tank or direct feed to the engine generators (see Figure 7).

每臺發(fā)電機有兩路燃油源。現(xiàn)場有2個1萬加侖儲油罐通過油管連接大容量油箱與發(fā)電機，主用油路通過大容量油箱給發(fā)電機提供燃油，亦可通過儲油罐直接提供燃油（見圖7）。

Transferring Data Center Loads

數(shù)據(jù)中心負載遷移

U.S. Bank’s commissioning of the stand-alone power plant including testing the ATS, load testing the engine generators, infrared (IR) scanning all connections, and a simulated utility outage. Some additional cabling was added during commissioning to address cable heating due to excessive voltage drop. After commissioning was completed, data center loads were transferred to the stand-alone plant. This required providing temporary circuits for select mechanical equipment and moving loads away from four panelboards (two for mechanical equipment and two for the UPS), so that they could be shut down and re-fed from the temporary power plant. The panelboards were transferred one at a time to keep the data center on-line throughout all this work. The transfer work took place over two weekends.

發(fā)電站調(diào)試內(nèi)容包括ATS測試、發(fā)電機加載測試、紅外掃描全部電氣連接點，以及市電停電模擬。調(diào)試期間，通過外加電纜解決過度壓降引起的電纜發(fā)熱現(xiàn)象。調(diào)試結(jié)束后，搭建臨時電路，將2臺動力配電箱和2臺UPS配電箱轉(zhuǎn)移到臨時發(fā)電站，將數(shù)據(jù)中心負載遷移到發(fā)電站。一次轉(zhuǎn)移一個回路負載，保證數(shù)據(jù)中心在線運行。整個負載遷移工作用時兩個周末。

The mechanical loads were sequenced first in order to put load on the stand-alone plant to provide a stable power source when the UPS systems were cut over and brought on-line. Data center loads were transferred to engine-generator power at the beginning of each day to isolate the data center from the work.

按照順序先將動力負載轉(zhuǎn)移到臨時發(fā)電廠，以便UPS上線時能提供穩(wěn)定的電源供應(yīng)。每天一早，把數(shù)據(jù)中心負載轉(zhuǎn)移到發(fā)電機，動力負載進行隔離。

On the first Saturday devoted to the transfer process, the mechanical loads were rotated away from the first panelboard to be re-fed. Equipment requiring temporary power was cut over (see Figure 8). The isolated panelboard was then shut down and re-fed from the stand-alone plant. Once the panelboard was re-fed and power restored to it, equipment receiving temporary power was returned to its normal source. Mechanical loads were rotated back to this panelboard, so that the second panelboard could be shut down and re-fed. Data center loads were transferred back to utility power at the end of each day.

第一周周六，對第一臺配電盤動力負載進行遷移工作。對臨電負載進行轉(zhuǎn)移（圖8）。隨后，切斷隔離配電盤，再連接到發(fā)電站。當配電箱恢復(fù)供電，設(shè)備也就恢復(fù)正常供電。再將動力負載遷回該配電箱，以便對第二臺配電箱進行遷移作業(yè)。每天結(jié)束時，恢復(fù)數(shù)據(jù)中心市電供電。

The Sunday mechanical cut over followed the same sequence as Saturday, except the stand-alone power plant, with live load, was tested at the end of the day. This testing included having Xcel Energy simulate a utility outage to the data center, which the utility did with data center loads still on engine-generator power so as not to impact the data center.

第二天（周日），用同樣的方法對第二臺配電盤動力負載進行遷移操作。當天工作結(jié)束前，還對發(fā)電站進行了帶載測試。Xcel Energy公司模擬了數(shù)據(jù)中心市電停電，通過發(fā)電機以保證數(shù)據(jù)中心正常運行。

UPS were transferred the following weekend. On Saturday, the two UPS systems were transferred to engine-generator power and put into maintenance bypass so their primary power sources could be re-fed from the stand-alone power plant. At the end of the day, the two UPS systems went back on-line and transferred back to utility power. On Sunday, workers cut over the UPS maintenance bypass source. That day’s work concluded with additional testing of the stand-alone power plant, including another simulated utility outage to see how the plant would respond while supporting entire data center.

第二個周末對UPS負載進行了遷移。周六，將2臺UPS轉(zhuǎn)換到發(fā)電機供電回路，并將開關(guān)柜切換至維護旁路，以便將UPS主用電源轉(zhuǎn)換到發(fā)電站上。當天工作結(jié)束前，恢復(fù)2臺UPS市電供電。周日，將UPS切換至外部維護旁路。當天，最后還對發(fā)電站進行了測試，再次模擬了市電停電以觀察發(fā)電站的運行情況，數(shù)據(jù)中心完全由發(fā)電站供電。

Figure 7. Standby-rated engine generators have two sources of fuel oil

圖7 后備發(fā)電機組

Figure 8. Data center loads were transferred to the  temporary stand-alone power plant over the course of two weekends

圖8 數(shù)據(jù)中心負載遷移到臨時發(fā)電站

Cable Bus Installation

電纜橋架安裝

At the same time the stand-alone power plant was assembled and loads cut over to it, four sets of cable trays and cables were installed to facilitate dividing the UPS loads. These four sets of cable trays had to be run through office and production areas to get to the existing UPS room, which is a run of approximately 625 feet (see Figure 9). Each tray served one of the four primary and maintenance bypass UPS systems.

在組裝發(fā)電站、將負載轉(zhuǎn)移至發(fā)電站的同時，完成了4組橋架安裝與電纜敷設(shè)，以實現(xiàn)UPS負載均衡分配。4組UPS主路和旁路電纜橋架穿越辦公區(qū)與生產(chǎn)區(qū)到達UPS配電室，全長接近625英尺（見圖9）。

Figure 9. New cable buss ran about 625 feet through the facility

圖9 電纜敷設(shè)約625英尺

Switchgear and Generators

開關(guān)柜與發(fā)電機組

After the data center loads were transferred over to the stand-alone power plant, the old switchgear was disconnected from utility power so it could be disassembled and removed from the facility (see Figures 10 and 11). Then, the new switchgear was installed (see Figures 12 and 13).

數(shù)據(jù)中心負載全部遷移到發(fā)電站后，移除舊的開關(guān)柜（圖10和圖11），安裝新開關(guān)柜（圖12和圖13）。

The switchgear was designed for even distribution of loads, with an A (yellow) side and a B (blue) side (see Figure 14). Each side supports one of the two UPS systems, one of the two chillers with its pumps and towers, and half of the computer room cooling units.

開關(guān)柜采用A（黃色標牌）B（藍色標牌）兩路負載均衡設(shè)計（見圖14）。A/B路各給1套UPS和1套制冷單元（冷機及相關(guān)的泵、冷卻塔），以及一半機房空調(diào)進行供電。

Figure 10 and 11. Old switchgear was disassembled and removed from the facility

圖10圖11 舊開關(guān)柜拆除

After installation, portable load banks were brought in for commissioning the new switchgear. The engine generators also received a full re-commission due to the changes in the controls and the additional alarms.

開關(guān)柜安裝完成后，用便攜式假負載對新開關(guān)柜進行測試。由于更換了控制系統(tǒng)與新增報警功能，重新對發(fā)電機組進行全面調(diào)試。

Figure 12 and 13. New switchgear was installed

圖12圖13 新開關(guān)柜安裝

Figure 14. Switchgear supporting Yellow and Blue sides, equally dividing the critical load

圖14 雙路開關(guān)柜（黃藍牌）

After the new switchgear was fully commissioned, data center loads were cut over to the new switchgear following a similar transfer sequence as to the stand-alone power plant. The panelboards supporting mechanical and UPS equipment were again each cut over one panel at a time to keep the data center on-line, again requiring transferring data center loads to engine-generator power to isolate the data center throughout this work.

新開關(guān)柜全面調(diào)試結(jié)束后，采用負載遷移到發(fā)電站同樣的順序，將數(shù)據(jù)中心負載遷移到新開關(guān)柜。一次遷移1個配電回路，將動力配電箱和UPS配電箱切換到發(fā)電機組，隔離數(shù)據(jù)中心負載以保證其運行。

Figure 15 and 16. Upgraded engine-generator controls and alarming were installed, with panels installed in the Engineers Office

圖15圖16 更換后的控制系統(tǒng)與報警面板

As previously mentioned, upgraded engine-generator controls and alarming were installed as part of the project (see Figures 15 and 16). The older controls had to be upgraded to allow communication with the new switchgear. Upgraded alarm panels were installed in the Engineering Office. In addition, each switchboard has a SCADA screen with a workstation installed in the Engineering Office (see Figure 17). The project also included updating MOPs for all aspects of the switchgear operation (see Figure 18).

正如前面提到，本次改造包括發(fā)電機控制系統(tǒng)與報警的升級（圖15和圖16）。升級控制系統(tǒng)以實現(xiàn)與新開關(guān)柜的通信，新的報警面板安裝在工程部辦公室。每臺配電柜配一塊SCADA顯示屏，系統(tǒng)工作站安裝于工程部辦公室（圖17）。還對開關(guān)柜維修操作手冊進行了修改（圖18）。

Figure 17. New switchgear includes a new SCADA system

圖17 新開關(guān)柜SCADA系統(tǒng)

Figure 18. Updated MOPs for the switchgear

圖18 新版開關(guān)柜維修操作手冊

The overall project went well and was completed on time with no impact to the data center. Since this phase of the project was completed, we have performed a number of live load engine-generator tests, including a few brief utility power tests, in which the engine generators were started and supported transferred load. In each test, the new equipment performed great. Phase 2 of the modernization project is the replacement of UPS System 1, which is currently underway and anticipated to be completed later in 2014. Phase 3 is replacement of UPS System 2, scheduled for 2015

該項目整體進行很順利，按時完成了改造工作，沒有對數(shù)據(jù)中心運行造成影響。在本階段結(jié)束后，我們對發(fā)電機進行了帶載試驗，包括發(fā)電機待機狀態(tài)下的幾次市電測試。每次測試中，新設(shè)備運行非常好?，F(xiàn)代化配電改造的第二階段是更換UPS-1系統(tǒng)，目前系統(tǒng)在使用中，預(yù)計2014年晚些時候完成更換。第三階段是更換UPS-2系統(tǒng)，計劃于2015年完成更換改造。

Mark Johns is chief engineer, U.S. Bank IT Critical Facilities Services. He has more than 26 years data center engineering experience, completing numerous Infrastructure upgrade projects, including all commissioning, without interruption to data center operations. Mr. John’s long career prior to U.S. Bank includes working in a 7-story multi-use facility, which includes data center operations, check processing operations, and support staff.