AFTERBURN REPORT 2006
The Systems Administration team worked diligently in 2006 to significantly improve the overall reliability and efficiency of the Burning Man information systems, through the implementation of more reliable software and better procedures, and getting much-needed, new hardware.
In the spring of 2006, email services were moved to Postfix, greatly improving email services, reliability, and spam filtering. The mailing lists were transitioned from EZMLM to Gnu Mailman; consequently, list management and configuration (for more than 360 lists!) are more flexible and reliable. The transition to the new mail systems did not go as smoothly as it could have due to a combination of team member transitions, team member availability, complexity of making many changes at once, and general communications. A great deal was learned by the Systems Administration team that reinforced the mission-critical nature of keeping technical operations flowing as smoothly as possible and using rigorous processes to test changes. In the end the email services are better than ever and many team leads have been trained to manage their own lists, which provides more flexibility and reduces the number of list administration requests.
Personnel changes included the lead administrator departing for the summer to work as a wildfire firefighter and the transition happened during the email services upgrade. A new lead administrator came on board to help with the transition in progress. Her first task was to pick up the pieces and ensure the transition to Mailman and Postfix was smooth. Once the migration was completed and the dust settled, extensive training and personalized customer service were provided.
Over the summer of 2006, the backup system for all Burning Man servers was overhauled and has been working flawlessly since. In July, the firewall at BMHQ had a hard drive failure, and the team toiled overnight to build a temporary solution with parts of donated, antiquated, dusty machines that were breaking faster than Linux install could run. It was decided to invest in a new machine and backup parts to use in case of another failure for our strategic firewall or network devices.
Nevada & Playa Operations
One of the first, and most important, System Administration tasks in the remote facilities of Nevada is ensuring the reliability of our Internet connection backbone and the dependent network components, including the office, Black Rock Saloon, Black Rock Station network, Gerlach town wireless, and the event networks.
In March, the playa network team hosted the first ever Playa Wireless Summit at the San Francisco offices. The summit brought together many of the various departments, art projects, and theme camps with the plans to use wireless broadcasting technologies for the event. The goal was to increase the chance of success for projects by identifying, communicating, and collaborating with one another, ahead of time, as related to gear, bandwidth, and channel usage. The result was a closer working relationship and increase in shared knowledge for everyone. It also laid the groundwork for our first playa network testing exercise over the summer. It appears that an increasing number of sophisticated technology projects are attempted each year, so the summit will continue to be held on an annual basis.
July was Systems Administration’s first Geek Week, during which participant and staff geeks gathered on the playa to rehearse the digital performance planned for the event: test and tuning of the microwave connection with Gerlach, bandwidth, simulations of various entities that require network and internet access during the event, like ticket booths for gate and airport, accounting, emergency services, Department of Mutant Vehicles, shipping and receiving, Department of Public Works (DPW), to name a few.
In 2006 both our Internet Service Provider (ISP) and our backbone network technology changed. We shut down our frame-relay T1 in the spring and changed to a local microwave provider located in Nevada. The change was a little rough at first, but over time we worked out the bugs and got our new 3meg link working well, essentially doubling our Internet speed in the process. Currently our production facility gets its Internet connection through a Satellite based provider. In 2007 we hope to run our connection in Gerlach to the ranch via wireless.
One of the primary goals of Geek Week was to test the traffic that is shaping the combined network with the Internet At Burningman (IBM) project. Previously, IBM had their own Internet feed using a satellite-based pipe. This year we combined our networks and switched to a 10meg terrestrial based microwave transported network. The tests were carried out successfully despite the July heat, and except for bandwidth issues that had to be resolved later, Systems Administration were confident that the event infrastructure could rely on solid technology. This technology testing week will be formalized and will occur again in 2007, and whenever there are changes (like exact city location, new network gear) which may impact the functionality of the on-playa network.
Systems Administration event setup went very well this year. In fact, the setup itself was the best to date. Proper planning and setup ensured that everything went like clockwork. Systems Administration had use of a 60 foot boom truck, which saved the day by significantly reducing DPW heavy equipment assistance. Systems Administration has 2 main data links to Gerlach, one microwave (5.8) and one WiFi (2.4) connection. When it came time to set up the main data links from Gerlach to the Network Operations Center (NOC), Systems Administration learned that size really does matter -- there were a bunch of problems getting the full 10meg up and running -- largely due to the elevation of the antennas as the 30 foot NOC tower doubled in height to 60 feet. Systems Administration added about 10 more feet to the connection in Gerlach and still the problems existed. 3 things may have been the problem: first, the shaking that can occur with a not-so-solid tower in Gerlach; second, is the elevation (also in Gerlach); third, the size of the dishes. In 2007 Systems Administration plans to make several changed to the main link; have the ISP assist with the backbone link; quadruple the size of the dishes; and install a new, higher, more solid tower in Gerlach.
The goal for 2007 is to improve the reliability of equipment and the availability of Systems Administration services. Big steps have been taken already, but as demands for technology services grow (watch for a streaming media server), so do needs for faster and more reliable hardware, support procedures, and enhanced features.
Catherine =^..^= (Cat), Sys Admin Team Lead
Chris Petrell (Taz), Nevada Technical Operations