Some sites have come back online, but Microsoft Bing Travel, Allrecipes.com, Big Fish Games and others remain unavailable as of 7 p.m. this evening, after a fire late last night shut down power at Seattle’s Fisher Plaza data center. Bing Travel had said previously that it expected to be back up around 5 p.m., but the message has since been edited to omit that reference.

Bing Travel is based on technology Microsoft acquired in its purchase of Farecast. We asked Microsoft when Bing Travel would come back online, why the service hadn’t been shifted previously to Microsoft data centers, and why a backup plan hadn’t kicked in.

"We are working with all the right folks down there to get it restored as quickly as possible. Unfortunately nothing more we can say at this point," Microsoft spokeswoman Whitney Burk responded via email.

In addition to those previously reported, another Fisher Plaza tenant seriously affected by the outage was Telecom Transport Management, a company founded by former McCaw Cellular executives that offers wireless backhaul services. Some of its representatives were among those at the scene talking with building officials this morning trying to figure out a workaround.

Bing Travel’s site describes the cause as a blown transformer. Here’s what Jim Clayton, vice president and general manager of Fisher Communications’ Seattle broadcast properties, told us when we were at the scene earlier today.

"We had a power surge last night, and the power surge overheated some of our switch gear, and the switch gear began to smoke, and when it smokes, the fire supression protocol takes over, and then it dumps water, so that really took the building offline, and because there was so much water there, even though our generator system did, in fact, work, it can’t be put online just because of safety issues until we’re certain that all of the water is out of the switch gear."

Read on for the statement issued late this afternoon by Fisher Communications, which owns the property.

Last night at approximately 11:10 pm, an incident in a garage-level electrical room disrupted power to Fisher Plaza East and knocked out the facility’s backup generation system. The electrical room is where Fisher Plaza East receives its power from Seattle City Light.

Fisher officials, CP Management and Egis Engineering have been coordinating with Seattle City Light to restore power to the building. Fisher is bringing in electrical generators to restore power to the building, at which time it can further assess the situation. The Company is working to restore normal service to its customers as soon as possible.

Fisher Plaza East houses data and network operations and co-location facilities for several companies. Fisher regrets any inconvenience these companies or their customers may have experienced as a result of this incident. …

The cause of the electrical fire remains under investigation, and the Company is working closely with Seattle City Light and outside experts to determine what caused this incident.

There were no injuries.

Previously: Seattle data center fire knocks out Bing Travel, other Web sites
How one CTO avoided a Web site disaster after data center fire

READ MORE and COMMENT, more 
and TV station forced to go old school after fire at Seattle’s Fisher Plaza  

Big web sites weren’t the only technological casualties after the fire and resulting power outage at Seattle’s Fisher Plaza today. KOMO-TV, which is housed at the building, was forced to go old-school on its afternoon broadcast. In an era of slick digital weather maps and over-the-top television graphics, it was pretty incredible to see.

READ MORE and COMMENT, more 

and How one CTO avoided a Web site disaster after data center fire  

Most Seattle geeks probably didn’t think they’d be spending a portion of their 4th of July holiday dealing with broken Web sites, back-up generators and damaged servers. But the small fire at the Fisher Plaza data center in downtown Seattle late last night knocked a number of sites offline for most of Friday, raising questions on TechFlash about how companies handle disaster planning and server co-location.

We actually first learned of the problem around 1 a.m. when Seattle-based Redfin posted a message on Twitter noting that their real estate site was offline because of problems at the data center. But by 4 a.m. Redfin’s site was back online, purring along whereas other sites struggled.

We asked Redfin CTO Michael Young how they avoided the catastrophic failure that other sites are experiencing today. Turns out, the company learned some important lessons after a similar electrical fire hit the same data center last June.

Here’s what Young told TechFlash today.

We were pretty embarrassed last June when Adhost had a similar electrical fire and took our site down for 8 hours (well into our core business hours) with brown-outs a day or two after that had us scrambling. ‘Fool me once, shame on you; fool me twice, shame on me’ resonated in our brains.

So by October 2008, we basically instituted a disaster avoidance plan where we had redundant-everything for our mission-critical databases, servers and networks in separate buildings.

When the problem happened last night, our beepers went off, we saw what looked like a major outage in one building, and were able to switch to the redundant systems.

Everything was up and running by 4am PST / 7am EST, well before our core business hours. We’re a startup, but we try to maintain high standards in our datacenter operations without spending too much money. The failover didn’t happen at the push-of-a-button, but the disaster planning paid off for us.

Young’s explanation is interesting given that many sites — including high-profile consumer-oriented sites such as AllRecipes, Bing Travel and Big Fish Games — have been offline most of the day.

I have a feeling there will be some high-level meetings with CTOs, IT administrators and co-location operators on Monday discussing some of the ways to make sure this doesn’t happen again.

I asked Young — who was up at 5 a.m. dealing with the situation — why other larger companies didn’t appear to have a similar plan in place.

"It’s hard to get every single point of failure," said Young. "And most people need to be burned once, like us."

[Flickr photo via Jamison_Judd]

READ MORE and COMMENT, more 

Seattle data center fire knocks out Bing Travel, other Web sites  

A fire last night at Seattle’s Fisher Plaza data center has knocked out service to some top Web sites, including Bing Travel and Authorize.net.

We first got word of the fire early this morning from the online real estate service Redfin, which suffered an outage last night. Redfin’s Web site was back up this morning, but we’ve noticed other sites that are experiencing problems. KOMO TV and radio broadcasts also were impacted, and we’ve noticed that Seattle’s AllRecipes.com was offline too. Reports are circulating that Verizon’s FIOS service, also is down, though it was unclear whether that was tied to the Fisher Plaza fire.

 A blown transformer appears to be the culprit, according to a message posted on the Web site of AdHost.

"Beginning at approximately 11:18 PM on July 2nd and continuing through the present time Fisher Plaza experienced a significant power event that required all power systems including street power, UPS, and Generator power to be completely shut down in Plaza East.

The event is ongoing and, at this point, we do not have a ETA for service restoration. Please note that Adhost’s Plaza West facility is not affected by this event. The Adhost phone system is not operational as a result of this event, but we are fully staffed and responding to emails sent to support@adhost.com .

Until further notice this will be the only conduit for communicating with Adhost. Please be patient with any requests sent into the Support email. We will process all requests as quickly as we can and we promise to answer every query.

If you have specific needs for assistance for powering up your devices, please send email to Support with specific instructions. We will do our best to accommodate your requirements. We apologize for the nature and extent of this event and are doing everything possible to restore service as quickly as possible."

Here’s the message from Bing Travel:

"A fire occurred at Fisher Plaza in downtown Seattle just after midnight on Friday morning. The blown transformer knocked out power to the entire building, which is home to the Bing Travel servers. We’re hard at work to restore service following this unexpected event."

There has also been a steady stream of discussion of the fire on Twitter.

And this isn’t the first outage at the Fisher Plaza data center, with service going offline last year after an electrical fire. The Fisher Plaza Web site also is down, and calls to the facility returned a busy signal.

However, a cached version of the site explains the facility this way:

"Fisher Plaza is the only mission-critical business community in the Northwest combining Class A office, data center, colocation and retail space with 21st century communications and media services."

We’ll be tracking the situation throughout the day, so let us know if there are updates to share.

UPDATE, 10:30 a.m.: TechFlash reporter Todd Bishop has arrived on the scene, where he reports that data center tenants are carrying server boxes out of the building. We’re asking tenants when the service may be restored, with some saying it could be early afternoon.

Reports from the scene are spotty. Some officials are telling tenants it may be the end of the day before service is restored.

Officials with the property management firm for Fisher Plaza declined to comment on the scene.

UPDATE, 11 a.m.: Bing Travel now is saying that it expects the travel service to be available again by 5 p.m. Pacific time today.

Does anyone have any technical information on why companies can’t switch to another co-location facility in order to keep Web sites running?

UPDATE, 11:20 a.m.: Verizon Wireless has generators on the scene to power its equipment.

Update, 11:27 a.m.: Here’s the alert sent out by Internap Network Services a short time ago to its data center tenants at Fisher Plaza.

At this time, there has been no significant change in status regarding the power outage at the SEF data center. Work is continuing on site to restore power as soon as possible.

The electrical vendor is onsite continuing to work with the electical switch gear at the SEF (Fisher) datacenter.

Customers in the SEF PNAP with IP connectivity continue to see their connectivity restored and stable; however, there may be further periods of degraded IP connectivity as the situation continues to develop.

Customers colocated in the SEF PNAP continue to see issues with a loss of power in some areas.

The focus of on-site personnel continues to be getting the rest of the generators online; however, we currently do not have an estimate as to when full generator capacity will be feeding the SEF facility.

We will continue to provide updates as they become available.

Update, 12:05 p.m.: Jon Davies, a spokesman for Verizon, tells TechFlash that the company’s FIOS and DSL service went down for about an hour at 9:43 a.m. today after batteries in the co-location facility ran out of juice. About 50,000 customers in Oregon and Washington lost Internet connectivity, said Davies.

Service was restored at about 10:45 a.m. through replacement batteries and the company is currently transporting a large generator to the building in order to make sure that Internet service continues to operate today.

Davies — who called the Fisher Plaza building a large "switching center" for the Northwest — said he had not heard when full power might be restored.

Todd Bishop photo of massive generator arriving at Fisher Plaza.

Update, 2:05 p.m.: Blogger Kyle Mulka has started a list of Web sites that have experienced outages due to the fire, including Bartell Drugs and Geocaching.com.

A few that he is missing are large Seattle Internet sites such as AllRecipes.com — which we discovered was offline early this morning as we atttempted to find a BBQ chicken recipe — and Big Fish Games — a Seattle company that operates a large downloadable game business. Here’s the message from Big Fish:

Update, 2:25 p.m.: Here’s the latest from hosting and colocation provider Internap which has a number of clients in the Fisher Plaza Building.

Two temporary generators are onsite at this time.  This repair is on schedule to be completed by late this evening, but we do not have a specific time yet. We will continue to provide updates as we get further information about the implementation of this plan.

Update, 4:45 p.m.: Here’s our latest report: "How one CTO avoided a Web site disaster after data center fire." We talked to Redfin CTO Michael Young to find out how the online real estate company got their site running shortly after the fire, whereas other companies are still down.

Update, 6 p.m.: TV station forced to go old school after fire at Seattle’s Fisher Plaza

Update, 7 p.m: Latest on data center ‘incident’

READ MORE and COMMENT, more