Sr. Director of Online Operations
icix
Duties Included:
- Budgeting and planning -- Ushered in significant cost savings in multiple areas
- Project architecture (collaboration with Engineering)
- Puppet configuration work including authoring several modules (see github.com/boinger for open source work)
- System automation testing and simplified developer environment creation with Vagrant
- Application metrics gathering & visualization via statsd -> graphite/pencil
- Engineering work on Ops-related daemons and tools
- Data center hardware spec and migration planning
- AWS & Joyent integration
- Implemented Logstash, Elasticsearch cluster (w/ Kibana) for all system and application logs
- Agile (scrum) development cycles with engineering team as projects required
- Engineering (both prototyping and production code) of several geographically-distributed application pieces
- Implementation of Range cluster data management system
- Team management and hiring
- Managed 3rd-party security consulting, policy coordination, and auditing
- Lead coordinator of customer technical RFIs
|
Dec 2012 -- Dec 2014
South San Francisco, CA |
VP of Operations
XDN (acquired by FortiNet)
Duties Included:
- Extensive overhaul of a [mostly] manually-managed platform
- Introduction of best-practices:
- Ops-driven software releases
- Comprehensive runbooks & project documentation
- Metrics-driven monitoring initiative
- Sales calls (both conference phone calls and on-site visits)
- Monitoring expansion, including end-to-end API & external monitoring (Nimsoft/Watchmouse)
- Spec and purchase hardware platform for Cassandra-based statistics engine
- Capital Expenditure & Operational Expenditure budgeting
- OpEx reduction (elimination of over-provisioned server resources & improved operational efficiency resulting in significantly higher ROI)
- Overhaul of customer support systems & procedures, resulting in far fewer support issues and greater customer satisfaction and retention
- Disaster Recovery/Business Continuity Planning
- Personnel planning and hiring
- Chef system management automation (including AWS nodes)
- Data center overhaul & cleanup
- Oversight of KVM automation project (taken from ad hoc manual builds to simple Chef stanza spawns new VMs)
- CrowdDirector (Global Server Load Balancer) administration
- Extensive shell (Bourne and bash) scripting
|
June 2012 -- Dec 2012
San Mateo, CA |
Manager of Operations -- Services
Mozilla
Duties Included:
- Manager of operations team for Mozilla's hosted product division "Services", which includes Firefox Sync, BrowserID/Persona, and various backend API layers.
- Established best-practices in many facets of operations that were adopted into other divisions of the company
- Review and scouting for next data center location
- Spec and purchase internal cloud platform for rapid project deployment
- Evaluated and implemented several new hardware vendors
- Involved in platform architecture decisions
- Capital Expenditure budgeting
- Disaster Recovery/Business Continuity Planning
- Personnel planning and hiring
- Puppet system management
- Enhanced system build automation
- Data center overhaul and reconfiguration
- Operations-related blogging
- Zeus Traffic Manager (load balancer) administration
- Logstash administration
- Extensive shell (Bourne and bash) scripting
|
September 2010 -- June 2012
Mountain View, CA |
Sr. Services Engineer - Advertising services production engineering team
Yahoo!, Inc
Duties Included:
- Primary point-of-contact for next generation ad-serving system
- Close work with developers on projects at all stages of project (from early code discussion through launching live product)
- Comprehensive documentation written for several projects
- Extensive Perl scripting, including enhancements and bugfixes to widely-used internal tools
- Expanded system metrics and graphing (internal rrd-based system)
- Considerable bash scripting
- Coordinated work and hand-offs with overseas teams
- Rotating on-call duty (covers all projects the team handles)
|
September 2009 -- September 2010
Santa Clara, CA |
Infrastructure Manager - [started as Senior Systems Engineer]
LiveOps, Inc
Duties Included:
- Manager of four teams: Systems Engineers, Database Administrators, Operations Tools Developers and Lab Engineering (7 full-time employees, various contractors)
- Initially also managed Network Engineers (2 people) before spinning them off as an independent team
- Passed PCI DSS (v1.1 and v1.2) compliance audit
- Reviewed, scouted and selected data center location
- Re-architected (in conjunction with Network Manager) systems infrastructure and built out new, high-density data center (and stayed in budget)
- Expanded platform to two new data centers
- Gracefully shut down three old, outmoded data center installations, migrating functionality to other locations
- Evaluated and implemented several new hardware vendors
- Architected platform hardware overhaul
- Architected and implemented office/lab consolidation and move (involving 300+ servers)
- Architected Virtualized QA Lab (Citrix XenServer & VMLogix LabManager), providing a flexible 'cloud' infrastructure
- Oversaw MySQL upgrade (70+ node replication tree)
- Capital Expenditure budgeting
- Re-engineering and implementation of Nagios-based monitoring system with custom front-end
- Disaster Recovery/Business Continuity Planning
- Part of Hadoop/MapReduce architecture team. Oversaw implementation.
- Distributed Splunk implementation
- System metrics and graphing via custom tools
- Implementation of Puppet system management
- F5 BigIP/LTM (load balancer) administration
- Some F5 3DNS/GTM (geographic load balancer) administration
- Extensive shell (Bourne and bash) scripting
- Part of the management team driving Scrum [Agile derivative] in Operations
|
February 2007 -- September 2009
Santa Clara, CA |
Manager - [started as Senior Unix/Linux Administrator]
Trading Technologies, Inc
Duties Included:
- Manager of Unix Administration, Internal Development and Monitoring Teams
- Lead Unix (Solaris) and Linux server administration and engineering/design
- Engineering, planning and on-site (physical) deployment of data centers (located in Chicago, New York, London, Frankfurt, Tokyo and Singapore)
- Extensive remote server administration
- Security planning and implementation (vulnerability assessment and policy delineation)
- Passed SAS 70 (Type II) audit [on the first attempt]
- System hardening
- Extensive system automation
- VMWare Infrastructure 3/ESX planning, implementation, and administration
- Wrote a centralized system tracking system in PHP using a MySQL database back-end
- Distributed system monitoring with [modified] Nagios (with heavily modified APAN and custom scripts) - several Nagios patches accepted into the main source tree
- System metrics and graphing with MRTG (customized Cacti GUI on the front end)
- NetFlow collecting and graphing with flow-tools (Cacti GUI on the front end)
- Cacti administration and hacking (source patches accepted into the main source tree)
- RRDTool administration and hacking (source patches accepted into the main source tree)
- System forensics planning, implementation and administration (crash analysis, user action auditing)
- qmail planning, implementation and administration
- BIND/DNS engineering, implementation, and administration (with encrypted transactions between master and slaves)
- Extensive shell (Bourne, bash, and ksh) scripting
- Perl coding/hacking
- sed & awk coding
- squid administration (including http and https acceleration)
- Apache administration and performance tuning
- PHP scripting of internal admin interfaces
- Oracle server administration and database design
- PL/SQL scripting
- MySQL and PostgreSQL server administration and database design
|
August 2003 -- February 2007
Chicago, IL |
Unix Administrator
Transora/ECPG (now 1SYNC)
[+]
Duties Included:
- Unix (Solaris) and Linux server administration and engineering/design
- Remote Unix/Linux server administration
- Solaris Jumpstart design and implementation (traditional method and Flash Archive)
- Security planning and implementation (vulnerability assessment and policy delineation)
- System hardening
- High-availability and fail-over planning, implementation and administration (VCS Clustering, content switching, and low-tech "warm" failover solutions)
- Extensive system automation via scripting/cron
- System forensics planning, implementation and administration (crash analysis, user action auditing)
- iPlanet/Netscape LDAP Server administration
- qmail planning, implementation and administration
- BIND/DNS engineering, implementation, and administration (with encrypted transactions between master and slaves)
- custom build of OpenSSH (CVS core with hacks)
- OpenSSH/OpenSSL rollout planning and implementation
- Extensive shell (Bourne, bash, and ksh) scripting
- Perl coding/hacking
- sed & awk coding
- scotty/tkined design/implementation (OSS SNMP monitoring tools)
- squid administration (including http and https acceleration)
- iPlanet/Netscape Enterprise Server administration and performance tweaking
- Apache administration
- Siteminder administration
- BEA WebLogic administration
- ht://Dig administration
- ATG administration
- i2 Discovery administration
- AMSE administration
- Ariba administration
- Syncra administration
- Veritas Volume Manager and NetBackup administration
- PHP scripting internal admin interfaces
- Unix skills training
- Scripting training
|
June 2001 -- August 2003
Chicago, IL |
Systems and Network Administrator (Externally Contracted)
Taos Mountain, Inc.
[+]
Duties Included:
With TechTV (now G4):
- Unix (Solaris) and Linux server administration and engineering/design
- Unix team management
- Two "lit" data center moves
- Vignette StoryServer administration
- iChat ROOMS and BOARDS Administration
- Extensive shell (Bourne, bash, and tcsh) scripting
- Apache web server administration and custom compilation
- Elaborate mod_rewrite configuration in apache
- Remote Unix/Linux server administration
- Internet/Intranet strategy consultation
- Hardware recommendation, ordering, installation and troubleshooting (Sparc and Intel)
- Veritas Volume Manager and Clustering
With Audiobase (no longer in business):
- Solaris server administration and engineering
- Extensive shell (Bourne, bash, and tcsh) scripting
- Remote Solaris server administration
- System build/config automation (scripted, not Jumpstarted for internal reasons)
- Security auditing
|
April 2000 -- May 2001
San Francisco, CA |
VP/Web Systems Administrator
Bank of America (formerly NationsBank)
[+]
Duties Included:
- Unix (Solaris and HP/UX) and Linux server administration
- NT server administration
- Web server administration - Netscape Enterprise Server
- Remote server administration (Unix and NT)
- Internet/intranet strategy consultation
- Netscape Proxy Server administration
- Netscape Communicator administration (configuration, distribution, etc. v4.04-4.7)
- Developer tool integration (NetObjects TeamFusion, NetDynamics)
|
September 1998 -- April 2000
Chicago, IL |
Independent Writer
Chicago Tribune
Duty was:
Writing a column which consisted of instructional articles for relative novices to accomplish various technical tasks under Linux for the monthly Sunday insert magazine called "Silicon Prairie".
Some of the article titles:
- "Connecting Linux via Modem to an ISP"
- "File Sharing with a MacOS Computer"
- "Connecting a PalmOS Device to a Linux Desktop"
|
September 1999 -- Summer 2000
Chicago, IL |
Interactive Developer / Network & Systems Administrator (Contract)
Ethos Interactive (now Eviciti)
[+]
Duties Included:
- Network (LAN) and system management/maintenance (server and client -- Windows and Mac)
- Server administration (NT)
- Hardware installation and troubleshooting (Windows and Mac)
- Intranet design and programming
- Internet strategy consultation
- Database design, management and interfacing (MS Access)
- Software evaluation and implementation
|
January 1998 -- March 1998
Cincinnati, OH |
Internet Developer / Interactive Designer / Webmaster
Millennium Marketing, Inc. (now Celestia)
[+]
Duties Included:
- Network (LAN) design, engineering, and management (Windows, Mac, and Linux)
- ISDN maintenance (Netopia 435 Router)
- Hardware installation and troubleshooting (Mac and Windows)
- Software evaluation, recommendation, implementation, troubleshooting and desktop support
- Database design / management (MySQL and mSQL)
- Web conceptualization, design, and programming (Linux- and NT-hosted)
- Distance learning systems development (web-based training)
- Site maintenance (DNS acquisition, daemon maintenance, etc.)
|
April 1997 -- January 1998
Cincinnati, OH |
Senior Programmer
Unius, Inc. (no longer in business)
[+]
Duties Included:
- Web design and programming
- Site maintenance (DNS acquisition, server maintenance, etc.)
- Network (LAN) management and maintenance (Windows and Mac)
- Hardware installation and troubleshooting (Mac and Windows)
- Technology implementation (video feeds, multimedia servers, etc.)
- Database management (Informix, mSQL, Butler, some Oracle)
- Software evaluation, recommendation, implementation, troubleshooting and desktop support
|
May 1996 -- March 1997
Cincinnati, OH |
| Independent/Contract Work
[+]
Duties/Projects Include:
- Co-founder, sole programmer and system administrator of ServiceSheet, Inc.
- Occasional programmer for several open source projects.
- Server engineering/design and administration (Linux, Sun/Solaris, NT, Netware and Mac)
- Distance learning system development (web-based training)
- Network (LAN) and system management/maintenance (Server and Client -- Linux, Mac, and Windows)
- Hardware installation and troubleshooting (Linux, Mac, and Windows)
- Internet/Intranet strategy consultation
- PERC Social Psychology research interface in Tcl/Tk for Win32 systems.
- Database design, management and interfacing (MySQL, PgSQL)
- Apache/Tomcat configuration
- Large-volume qmail implementation (notably not used for SPAMming)
|
~1993 -- Present |