SYSTEMS ENGINEER (Yahoo! Mail Engineering, Oncall Manager)   --   Oncall Manager for all Yahoo! Mail systems. Organized and led regularly scheduled training for all members of the various production oncall teams in the US and India. Wrote and updated documentation and runbooks for oncall participants. Managed and scheduled production oncall resources for Sunnyvale and Bangalore teams, including weekly "hand-off" meetings, shift reports, summaries, and metrics for upper management teams. Daily short- and long-term capacity management and performance analysis and application profiling. Wrote and managed scripts in Perl and shell to crawl and analyze/repair/manage production user data, to collect and compile statistics, and to monitor the system at varied levels of granularity. Compiled and wrote postmortem reports for internal and external parties for production issues and outages. Direct communication channel between Yahoo! Mail SE teams in Sunnyvale and Bangalore, Production Operations (NOC, SA, Network Engineering, Storage Operations, Procurement, Kernel Development, Change Management, etc.), vendors and partners (SBC, BT, Rogers, etc.). Led daily and weekly CAP meetings for various system components.

SYSTEMS ENGINEER (NOC Tier II Team, Lead)   --   Technical Lead of Tier I and Tier II teams. Responsible for configuration and implementation of Netcool ObjectServers, ELMd license managers, NT Event List and Unix Desktop clients, and syslog probes (including all filters and configuration files). Development and maintenance of scripts and tools to integrate Netcool clients with Knowledge Base (i.e. Runbooks), documentation web sites, and RT3 ticketing system. Development of tools to troubleshoot and resolve common alerts received by NOC via Netcool. Evaluation, specification, installation/configuration, and population of Knowledge Base for Tier I and Tier II group, to be used both domestically and internationally. Technical writing to streamline processes within the NOC. SQL reporting on asset management, ticket tracking, and change management databases, to gather statistics on system downtime, resource capacity planning (devices and supporters), and employee and system performance. Managed projects to specify and build-out infrastructure for new Network Operations facilities in Sunnyvale and Bangalore, and performed international travel, as appropriate. Built tools to implement Change Management process for Yahoo! Operations, and chair weekly Change Management meetings. Specify and recommend access control and DVR systems for use in existing and new co-location facilities, and maintenance of all access control devices (including HID/Mifare badging systems).

TECHNICAL YAHOO! (Senior Site Operations Engineer)   --   Installation, configuration, and maintenance of Yahoo!'s "master clone" images and JumpStart installation for Sun Solaris (versions 2.5.1 through 8.0). Management of Yahoo!'s two secure vault co-location facilities in Santa Clara, where all banking and credit card transactions took place throughout the Yahoo! Network. Managed multiple projects to expand the primary vault installation, and later the build-out of Yahoo!'s new 8,000 square-foot general purpose data center and 700 square-foot finance vault in Santa Clara (including all related security equipment, including Loronix DVRs and NexWatch's "Pro-Watch" system). Installation, configuration, and maintenance of various types of servers and storage devices, from Compaq, HP, Sun, Dell, Cisco, Juniper, Foundry, Rackable Systems, Spectra-Logic, etc. Finally, partially responsible for implementing, maintaining, and reporting on Yahoo!'s asset tracking and hardware database (using MySQL and PHP), which is relied on by both the Operations and Finance departments for inventory control purposes for all hardware in the production network worldwide.

TECHNICAL YAHOO! (System Administrator)   --   Administration of production and test-environment Sun Sparc/Solaris and Intel FreeBSD and Linux servers, co-located in various data centers around the world. Primary systems administration contact for all Sun hardware/ Solaris OS issues for all properties at Yahoo!, including Yahoo! Maps, Yahoo! Yellow Pages, Yahoo! Quotes, Yahoo! Groups, Yahoo! BillPay, Yahoo! Billing, Yahoo! Mail, and Yahoo! PayDirect. Other duties included automation of routine tasks using Bourne shell and Perl scripts, DNS administration, NFS, Apache, Sendmail, SSH, FTP, Veritas Volume Manager, AMANDA, Solaris JumpStart, as well as some Cisco IOS and Foundry / Alteon load balancer and ACL configuration maintenance. Also responsible for maintaining Network Appliance filers throughout the Silicon Valley, and shared administration responsibilities of enterprise-wide ticketing system, using "Request Tracker" software.

SYSTEM ADMINISTRATOR   --   Administration of approximately 175 production and test-environment Sun Sparc servers, in both the Santa Clara and Marina del Rey sites. Primary responsibilities included automating tasks using Perl and Bourne-shell scripts, DNS, NFS, NIS+, Informix database connectivity, scheduling and performing hardware and OS installs/upgrades, Sendmail and Qpopper configuration and compilation, Veritas installation and support, configuration of Cisco switches and routers, and automating OS installs/upgrades using Solaris JumpStart.

OPERATIONS MANAGER   --   Managing the workloads and schedules of the Site Operations staff (5-7 operators). Duties included hiring new operators, writing a complete "Operations Manual" which included written procedures for maintenance and monitoring techniques, evaluation and recommendation of new monitoring and tracking software. I also planned, orchestrated, and coordinated two major site architecture upgrades.

OPERATIONS LEAD   --   Shift scheduling, installing and customizing BMC's PATROL product, gathering and presenting the site's vital statistics (traffic, member growth rates, etc.), and forming/documenting procedures for use by the Site Operators.

WEB SITE OPERATOR   --   Monitoring the site, auditing backups, writing new tools and scripts to assist in monitoring and restarting custom applications on the site, gathering site statistics and generating daily reports (using Excel) for Senior Management.