AFTERBURN REPORT 2009
The Burning Man Systems Administration Team went through a lot of changes and implemented a lot of improvements in 2009, at the headquarters as well as on playa. With a new office, new development platform, playa network traffic shaping and more, it was definitely an exciting year, opening the way for bigger things.
San Francisco Office and Datacenter
The Burning Man headquarters moved to another building in May. It was only after the lease paperwork had been signed that we realized that we needed to rewire the whole building for data and voice. Thankfully, our core team and some rock star volunteers came together, hung out in walls and ceilings, and the new wiring was ready in time for the arrival of computers and staff. Moving also required reconfiguring office servers and the firewall; the opportunity was also taken to implement a VoIP phone system and to change and upgrade our internet connectivity, which improved part of our systems and costs.
The biggest project accomplished was the launch of the development server, Furnace, which is a Linux server, set up with several Xen instances, each replicating one of the production servers where various websites and applications live. Developers and volunteers from various departments can now work on their department projects and websites directly on the development server, with a full development environment adapted to their needs, and a test platform identical to the production server. This allows more flexibility for departments and developers, and minimizes risk to the production servers. Several departments have already taken advantage of the opportunity of this new platform, and many more are starting new projects or moving existing projects to this shared workspace.
Operating system and backup software were upgraded on some servers to make cross platform backups possible. All data is now backed up in San Francisco and Gerlach, creating redundant data storage in case of a major disaster. A dedicated backup server for the Accounting Department was added in the datacenter. The mail services and the mail server are doing well, despite a constant increase in load.
The Accounting Department experienced challenging growth of IT, in that more people use it more often, which means that bandwidth and uptime were sometimes a challenge. Most of the workings of the department are confidential, but it is safe to say that new methods and hardware were used, as well as remote tools … and some lessons were learned as well.
Nevada and Playa operations
One of the objectives of our Nevada technical operations is to provide fast and reliable Internet to Burning Man's Nevada Properties, and to the infrastructure in Black Rock City during the event. Each year, the speed, reliability and efficiency of the network is improved by research and implementation of new technologies, and by taking in account the feedback gathered from various departments.
In 2009, a two-stage traffic shaping setup was implemented, with the first stage at the main hub in Gerlach, and the second at the Network Operations Center (NOC) in Center Camp. The shaping this year prioritized voice traffic, certain departments, and maximum throughput that each location was allowed when the load was high. The free wireless network provided to the town of Gerlach experienced some reduced speeds during the week of the event, as the bandwidth they are used to having to themselves all year is shared with all of Black Rock City during that time.
Some inexpensive hardware was purchased and integrated with our older network gear. Sector antennas were used a bit more to replace some of our omni antennas, to avoid the shadowing problems of omni antennas.
In the past, some locations in Center Camp were hard wired to our NOC via individual trenches, using the cheapest CAT5e cable available. This year, coordinated in conjunction with our newly engineered in-house power grid, and in an effort to ensure safety and efficiency, we experimented with installing shielded Ethernet cable in the same trenches used for power distribution. Shielded CAT5e and an experimental run of donated military-grade fiber-optic cable were used for this purpose, and both worked great!
Additional wiring was added to the Gerlach Office, which in 2010 will likely be used to extend the VoIP phone system that was installed in the San Francisco Office. In the meantime, the PBX from the previous San Francisco office was moved to Gerlach as a backup for the aging and unreliable phone system there. Most of the technical equipment that was stored in the office has now been moved to the new shower building.
The Black Rock Saloon was converted from a wired connection to a microwave connection, and more VoIP phones and computers were added in it for staff members to use. No significant changes were implemented for the town Wifi or Black Rock Station in 2009, though each requires a fair amount of technical support and troubleshooting throughout the year.
Things keep getting better! This year’s major changes really helped make for great improvements in the network, and this is a trend that will hopefully continue every year.
Cat =^..^= , Brendan McKenna, and Chris Petrell