< SKILLS >

< CERTIFICATIONS >

Hurricane Electric IPv6 certified Sage.

< EXPERIENCE >

3/2002 - present
Chicago, Illinois

Provide UNIX software and hardware solutions to companies in the Chicago area, including Acxiom, CME Group, SymphonyIRI, and Alliance IT, on contract and subcontract basis. Specialize in quickly understanding undocumented systems, fixing production outages, legacy modernization, performance analysis and long-term support.

Sysadmin
  • Deployed VMs running RHEL, Centos, and Ubuntu and managed them with Puppet; created custom Puppet modules. Used augeas to modify config files.
  • Set up nagios and RRDtool to perform system and service monitoring.
  • Configured encrypted rsyslogd for secure logging in the cloud.
  • Created SSL cert management scripts.
  • Wrote Perl framework to audit Linux systems for firmware, drivers and patches.
  • Repaired and standardized RPM environment on Red Hat Enterprise Linux servers at IRI.
  • Solved ext3fs corruption caused by EMC Powerpath on Linux.
  • Implemented QLogic HBA failover and network install of RHEL.
  • Researched Red Hat Satellite Server and implemented custom channels and profiles.
  • Rebuilt and packaged in-house Perl modules for upgrade to Perl 5.6.
  • Built and standardized sitewide Perl 5.8 environment and modules.
  • Introduced centralized documentation repository for UNIX team at IRI.
  • Upgraded Apache installation and rebuilt third-party and custom modules written by client. Solved several Oracle issues related to mod_perl. Provided documentation on process to client and trained sysadmins.
  • Designed a SQL-based Apache log analyzer and bulk loading tool.
  • Migrated an undocumented in-house web app through Java and OS upgrade.
Architecture
  • Investigated slow tape duplication from RHEL and Solaris Netbackup media servers, and Sun VTL to T10KB drives at Acxiom. Conducted a health check, comparing kernel, HBA, network, filesystem, SAN, and Netbackup config to best practices. Wrote custom code to gather and graph device performance stats using RRDtool; benchmarked virtual tape storage. Demonstrated VTL and SAN storage to be the bottleneck.
  • Performed feasibility study on changes needed to convert batch jobs to a just-in-time architecture.
  • Conducted architecture review and audit of UNIX software and hardware in Internet-facing and batch environments, including firmware and drivers, patches, boot disks, NFS, NIS and NTP.
  • Corrected systemic NTP sync issues by rearchitecting IRI NTP environment.
Performance
  • Worked with several teams to troubleshoot a network issue causing slow WAN transfers at Acxiom.
  • Alleviated network link saturation during backups, fixing an application throughput issue.
  • Improved system's backups from 36 hours to 9 hours by redesigning network.
  • Reverse-engineered a legacy, closed-source middleware application and created a replacement with Perl and open-source tools, alleviating high-load issues for IRI clients.
  • Wrote software to collect and graph gigabytes of performance data.
  • Analyzed large NFS environment and identified bottleneck causing recurrent client timeouts. Recommended NFS filesystem reorganization and parallel servers, fixing the problem.
  • Conducted feasibility study and benchmarking of jumbo Gigabit Ethernet frames and NFS.
  • In-depth performance analysis of several systems, including Oracle DB servers.
  • Analyzed performance of IRI applications running on HP-UX 11i, using PerfView, glance, sar, tusc and tcpdump to examine system and NFS behavior.
Optimization
  • Saved three weeks of batch processing time for Pepsi by rewriting shell/C++ app.
  • Reduced shell script runtime from 50 hours to 25 seconds.
  • Improved Perl web app execution time from 1 minute to 1 second.
  • Saved 60% disk space and 50% time in C++ application.
  • Improved shell script runtime from 60 minutes to 5 minutes, alleviating client timeouts.
  • Diagrammed a complex undocumented batch workflow, enabling optimization of the bottlenecks.
  • Achieved 300% speed improvement by identifying a severe disk bottleneck in a C application at IRI, rewriting the algorithm and instituting kernel parameter changes.
Outages
  • Diagnosed a network issue causing repeated client-facing outages at IRI.
  • Traced webserver lockups to database locking issue.
  • Mitigated outage risk after NFS server crash by injecting new code into closed-source binary.
  • Fixed lock table overflow causing production outage by writing tools to map NFS locks to filenames and processes, utilizing UNIX kernel internals.
  • Diagnosed and fixed out-of-memory condition in client application causing production outage, via kernel parameter changes.
  • Tracked down and fixed internal server errors in Tomcat application.
  • Eliminated crashes and leaks in legacy C app, a UNIX-to-mainframe job dispatcher.
Front-end
  • Built a web app in Perl which moves UC4 scheduler jobs between environments at CME Group, targeted at internal users. It accepts change request information and multiple XML file uploads, performs XML transforms and validation using standard modules, and emails the results.
    • Application interfaces with Oracle and LDAP.
    • Local database caching with SQLite and failover to ensure high availability.
    • Progressive Javascript and AJAX with jQuery. Cross-browser compatible.
    • Installer safely deploys application to production, QA, DR and development systems.
  • Created a web interface to an API documentation system, wrote a jQuery plugin for incremental search, and added proxy caching with nginx. CSS3 media queries and progressive enhancement are used to scale from desktop to mobile (iPhone, iPad) while emphasizing cross-browser compatibility.
Back-end
  • Rearchitected 10-year-old C and Perl-based login and load balancing app to allow multiple user sessions. Replaced adhoc databases with a SQL database.
  • Wrote a Java program which interfaces with UC4 job scheduler.
  • Ported 32-bit HP-UX business app to Linux and added 64-bit support. Migrated authorization from UNIX/NIS to PAM, enabling use with LDAP. Fixed buffer overflows and memory leaks.
  • Rearchitected the management of IRI's NIS environment by replacing a collection of shell, Perl and C code with a modular Perl-based solution, fixing security and file corruption issues. Also overhauled accompanying Visual Basic user interface for internal clients.
3/2000 - 1/2002
Rosemont, Illinois

Supported large HP environment including D, K, N, and V class servers. Implemented critical MC/ServiceGuard cluster and highly customized scripts. Audited and tested existing clusters. Upgraded production server from 10.20 to 11.00 in minimal time, via cold install and data restore; documented procedure for team. Assessed impact of security vulnerabilities. Deployed SSH widely for secure administration and auditing.

1/1999 - 2/2000
Elk Grove Village, Illinois

Worked with a team of architects and administrators to design and manage one of the largest HP-UX environments in the Chicago area. Technical support contact for Operations Control Center, comprising 8 servers and 100+ workstations.

Involved with implementation of Business Resumption Project, comprising 6 servers, 85 workstations and 200+ users. Worked with developers to port and test applications. Performed physical setup and OS config; UI design; NFS/NIS setup; and documentation. Site was activated successfully for one month during August 1999 fire. Upgraded site as requirements grew. Received award for outstanding performance.

< EDUCATION >

B.S., Mathematics and Computer Science; minor, East Asian Languages and Cultures; University of Illinois at Urbana-Champaign, January 1999. GPA: 3.86/4.0.