AFTERBURN REPORT 2011
When the Burning Man office moved downtown to Market Street in May 2011, Technical Support made sure the office and the on-site servers had as little downtime as possible. A great deal of rewiring was done at the new building to link our floors together. Equipment was carefully packed and prepared, and close planning with the transportation team ensured everything was moved quickly and safely. The move went better than expected, our servers saw minimal downtime, and the new office Internet was ready for business when office staff arrived after the weekend. Since then, a major upgrade has greatly increased bandwidth in the office, eliminating the slow performance we were sometimes seeing at the old location.
To better serve our very geographically distributed organization, a new web conferencing solution was selected and implemented. After a thorough analysis of options, Fuze Meeting was chosen to allow viewing of media, webcam and desktop images for meetings when our standard audio-only conference bridge would have been insufficient. We continue to test our usage and look to expand the service with additional accounts as needed.
Technical Support provided equipment and expertise to the DMV this year to help implement a major upgrade to their process and equipment, eliminating paper forms and streamlining the registration process with great success.
The accounting department successfully upgraded to new hardware workstations and new versions of productivity software. The accounting systems were scaled to accommodate remote access and new uses by the Human Resources team.
Looking forward to 2012, Technical Support will be continuing our hardware upgrade cycle for office laptops, upgrading our in-office backup server to increase capacity, and beginning the transition to Mac OS X version 10.7.
For 2011 we upgraded a small part of the Gerlach town wireless and backbone.
Next year, we are planning to upgrade the network at Black Rock Station, Burning Man's near-playa production facility.
We successfully tested the use of the San Francisco office VoIP phone system in the Gerlach office. Now, office personnel from BMHQ can bring and use their same phones and phone numbers while working in the Gerlach office. In the future, we hope to deploy a new phone system in Gerlach and Black Rock Station that will tie into the existing San Francisco office VoIP system.
We completed a massive cleanup of our facility in Gerlach that is used for safe storage of IT equipment. We inventoried our assets and e-cycled or donated old equipment. This cleanup will allow us to get ready for future upgrades and the acquisition of new machines. This purging was badly needed, as much of the older equipment was upwards of 12 years old.
As we continue to support the Nevada properties year around we also strive to maintain a more stable, speedy Internet connection at both Black Rock Station and in Gerlach. Gerlach's town wifi network still needs some upgrades from the old 2004 technology.
Black Rock Station experienced a long satellite outage from Wild Blue so we changed to Hughesnet. At that remote location, we only have satellite speeds and connectivity, which is not very good for VoIP. An upgrade of that network is in the plans for next year..
This has been quite a peaceful year at our datacenter. Since we moved to a different floor, we haven't experienced any more of the annoying power outages from our previous cage location. The challenge is to keep our power consumption at or below the limit imposed by our provider. One of our intelligent (remote controlled) power strips failed and had to be taken out for repairs, and there was a power surge when we installed it back, and that was the only outage we had all year, which lasted only a few minutes.
The mail and list system is performing flawlessly. We upgraded and replaced a couple servers that were getting old and cranky, and we are still performing OS upgrades on a pretty regular basis. We also started migrating some applications and websites to cloud-based hosting solutions. We are continuing to evaluate cloud-based solutions as servers or systems change over time.
We upgraded the ePlaya to the latest phpbb version, but we underestimated the load incurred by this application and first installed it on a server that turned out to be unable to support it, so we decided to move the ePlaya to Amazon EC2 entirely, which resolves the issue of scalability.
Black Rock City Operations
One spectacular change to Black Rock City operations was the upgrade of our network link from Gerlach to BRC. We now use FCC-licensed microwave equipment that uses frequencies which are completely different from what is used within BRC. This required more resources and work pre-event to test and setup, but once we had the connection completed and online, it turned out to be more reliable and better performing than our previous configurations. We kept the previous set up as a backup link, which we only used for the early setup and the last days of takedown.
The new network topography uses a triad of links, such that if any link fails, the connection to the default world Internet is rerouted within two or three seconds to bypass the failed link. This is achieved with redundant connections and high-end routers. In addition to all these upgrades, we started using more robust monitoring applications and network management tools.
One of the only real network problems we faced this year were power outages in the town of Gerlach during the event. This caused the BRC Internet to go down until we were able to get a generator in place at our facility in Gerlach. The plan for 2012 is to have a generator with failover capabilities as a contingency in case of main power failure.
Cat =^..^=, Eric Haugen, Brendan McKenna, and Chris Petrell