AFTERBURN REPORT 2008
The systems administration team made great leaps again this year towards better reliability and availability of the Burning Man information systems and successfully overcame this year's challenges.
Upon returning from the 2007 event, our lead systems administrator was faced with a hack on the server that hosts the website, the image gallery, and the eplaya. After a thorough audit, the server was reinstalled, along with all applications from the backup, and no data was lost. Lack of installation documentation about some specifics on the web server delayed the restoration of the image gallery. Measures were taken to organize applications and data in a different way, to ensure that this wouldn't happen again, or at least to reduce impact of a potential repeat incident. Software upgrades and patches were also installed to prevent similar hacks from happening.
In the fall, a number of servers were moved to a new datacenter co-location facility. The move was from one United Layer location to another one of their facilities in San Francisco. This move also helped to accommodate systems growth planned for the winter. Our team visited several facilities before making a choice. Uptime and the services at the new location are satisfactory, and no significant outage has happened, with one brief exception, when the power for the cage had to be upgraded to accommodate growth, and our T1 connection was re-routed from the old location.
Over the winter, the systems administration team performed a hardware upgrade of the mail server. Upgrades of other servers were also planned but were delayed because of hardware compatibility issues and missing parts that were difficult to get. A dedicated streaming server was added as well.
In spring 2008, the team focused on getting ready for the event, by providing fast service to the high volume of requests from other parts of the organization, and by attending Geek Week to get all the desert hardware ready.
Nevada & Playa Operations
In 2008 we had a few significant changes to our Nevada and Playa systems. In April we changed the relationship for our microwave connection in Nevada. We now have a contract directly with the network engineer, rather than using a third-party provider. Another major decision was to purchase and implement a new “backbone” microwave link from Gerlach to Black Rock City. We planned on testing this new connection during our yearly Geek Week event, but the equipment got damaged during shipping and sadly we did not get the gear in time for pre-event testing. Working with our ISP network engineer, we installed, tested, and ran the equipment on time for the event, and it worked flawlessly. We were able to pull a full 15 megs, even though we only have 10 megs to the internet. All around everyone was very happy with this gear, and the connection it provided for our infrastructure needs. We will continue to use it year after year from now on, for a very robust and solid internet connection. Because the gear worked so well and we are not anticipating any significant changes to our connection, we will probably discontinue our Geek Week testing.
We hosted another very successful Geek Week. This year we had twice the participation in the event, and even though there was extreme heat and dust storms, it was still a successful testing effort and all had a fun time. Since our new gear did not arrive in time, we set up our backup link and tested all the other nodes we use for Black Rock City. We set up a grid for power and spread out trailers to act as various locations in Black Rock City. Testing went well, and many people stayed online for the week.
After years of experiencing a degrade in the Starband service, it was decided to change to VelaSat for our satellite ISP at Black Rock Station. The new system was installed in early August and worked great initially. A few months after the event, it seems that the connection needs to be realigned. In the future we hope to run internet to the ranch from our main link in Gerlach. This project has had a few challenges over the year and we will continue work on this in 2008.
Another major change and one reason for a very successful network on the playa was the erection of a temporary 60’ tower in Gerlach. In the past we have more or less “winged it”, with small short masts attached to buildings, but we have grown to the point where we need a stable platform high in the air on both ends of our main wireless link. Ideally, in 2008 we will install a more permanent tower in Gerlach for communications, internet, and radio needs going forward.
One more project is the wireless network that Burning Man provides for the town of Gerlach. As the event was ramping up, several of the local nodes starting having problems. This could have been due to the heat in that July (upwards of 117 degrees one day); it also could have been because of an overload of users, dust, or the age of the machines. A quick swap of machines helped one node, but shut down another location that is not used as much. Over the winter of 2008-2008 the nodes will be repaired or replaced, and some engineering will take place on a possible re-design of the whole town wifi network. Traffic shaping will most definitely be added to the mix, as currently there is none in place. These decisions may have to wait however for other projects to be designed and completed.
In conclusion, this year had its ups and downs, but this was the best year to date. We strive to improve things every year and work out the problems as they occur, but overall it’s always blast.
Cat =^..^=, Linux Demoness
Chris Petrell (Taz), Nevada Technical Lead