The GeoCities Cage at Exodus Communications
The following pictures were taken with my [cheap] digital camera at the GeoCities cage in Exodus Communication's "Wyatt" data center on 5/7/1999.
This is what the inside of the cage looks like. Nearly all of the machines
in the cage are Sun Ultra 2s. Attached to nearly all of these boxes are
dual sets of MTI JBODs ("Just a Box Of Disks") (either 2500s or 8300s).
The newer JBODs (the 8300) make up the majority of the storage arrays in the
cage.Nearly all of the (15+?) isles in the cage look identical to this; There are nine shelves per rack -- Each rack holds three U2/dual JBOD clusers. As you can see, there all raised floors throughout the entire data center. At the far end of this isle, you can see the chain-link fence which separates our cage from Exodus's other customer cage(s). |
The hardware is arranged in the racks so that the machines are installed
face-to-face and back-to-back. This is a view of an isle of machines that
are back-to-back. Holes in the raised flooring and in the ceiling above
pipe cooled air through all of the hosts in the racks.Most of the cables you see in this picture are SCSI, network (front and back end), console, and power. With roughly 120 clusters and 12TB of storage space, lots of cable is needed. Although it looks like nothing but spaghetti, it's actually fairly well managed... |
These are a little closer pictures of some of the U2/JBOD clusters. These
are MTI 8300 JBOD units, housing twelve 9G drives each. They are striped
and mirrored in a software
RAID-10 configuration using
Veritas Volume Manager.
|
Since early 1998, GeoCities's site architechture has been such that if one
of the web servers went down, the data contained on the local disks (JBODs)
(right) would be unavailable to the outside world. We have begun moving
towards high-capacity and high-availability "Network Appliance" hardware file
servers (left), which can be mounted to any number of web servers, via NFS.
While this also does not completely remove the "Single Point of Failure" from
the site, average uptime has been increased due to the increased internal fault
tolerance inherently built into the NetApp filers, such as redundant power
supplies, data striping and parity checking (RAID-4), and the "Write
Anywhere File Layout" ("WAFL") FS.
|
Each GeoCities server is multi-homed, so that it exists on two LANs. One of
which is the "public" (or "front-end") connection, which is used to serve
internet service requests to the outside world. Each machine also has access
(through a second NIC) to a "private" (or "back-end") network, which is used
for inter-machine communication. This private network is primarily used for
NFS traffic. There are four Cisco Catylist 5500 switches installed in the
GeoCities cage, which handle all of the network traffic (both public and
private) for the site. As you can imagine (and see from these pictures),
this equates to a whole bunch of ethernet cables. Cable management gets
increasingly difficult to grasp each time a new box is added to the mix.
|
However, when there's time to plan and it's done right, it can also be quite
nice. This is the other extreme, and its a testament to the anality of the
Exodus staff. Is this some of the most beautiful cable management you've
ever seen, or what??!!?The above picture are some of the newer patch panels, connecting the hosts to the Cat 5500 switches. The above-right picture are two of GeoCities's newest switches, Geo-C and Geo-D. To the right, you can see a rear-view of some of the patch panels. The blue bundles of ethernet cable run up to the top of the racks, where they run out (in 8" thick bundles of cable) across the cage to the various individual patch panels in each rack. |
Silicon Valley was built on top of five major fault lines, so Earthquake
preparedness is a big deal in the area. This shot was taken looking straight
up from within one of the rack isles. The two thick metal bars are welded to
the tops of the racks, and prevent them from moving independently of each other.
Then, there are four 1/2" thick steel rods that bolt them all together and
prevent any particular rack from swaying too far one way or the other.
|