Provide UNIX software and hardware solutions
to companies in the Chicago area, including
Acxiom,
CME Group,
SymphonyIRI, and
Alliance IT, on contract and subcontract basis.
Specialize in quickly understanding undocumented systems, fixing
production outages, legacy modernization, performance analysis
and long-term support.
| Sysadmin |
- Deployed VMs running RHEL, Centos, and Ubuntu and managed them with
Puppet; created custom Puppet modules. Used augeas to modify config files.
- Set up nagios and RRDtool to perform system and service monitoring.
- Configured encrypted rsyslogd for secure logging in the cloud.
- Created SSL cert management scripts.
- Wrote Perl framework to audit Linux systems for firmware, drivers and patches.
- Repaired and standardized RPM environment on Red Hat Enterprise Linux servers at IRI.
- Solved ext3fs corruption caused by EMC Powerpath on Linux.
- Implemented QLogic HBA failover and network install of RHEL.
- Researched Red Hat Satellite Server and implemented custom channels and profiles.
- Rebuilt and packaged in-house Perl modules for upgrade to Perl 5.6.
- Built and standardized sitewide Perl 5.8 environment and modules.
- Introduced centralized documentation repository for UNIX team at IRI.
- Upgraded Apache installation and rebuilt third-party and custom modules
written by client. Solved several Oracle issues related to mod_perl.
Provided documentation on process to client and trained sysadmins.
- Designed a SQL-based Apache log analyzer and bulk loading tool.
- Migrated an undocumented in-house web app through Java and OS upgrade.
|
| Architecture |
- Investigated slow tape duplication from RHEL and Solaris Netbackup
media servers, and Sun VTL to T10KB drives at Acxiom. Conducted a health check,
comparing kernel, HBA, network, filesystem, SAN, and Netbackup config to
best practices. Wrote custom code to gather and graph device performance
stats using RRDtool; benchmarked virtual tape storage. Demonstrated VTL and
SAN storage to be the bottleneck.
- Performed feasibility study on changes needed
to convert batch jobs to a just-in-time architecture.
- Conducted architecture review and audit of UNIX software and hardware
in Internet-facing and batch environments, including
firmware and drivers, patches, boot disks, NFS, NIS and NTP.
- Corrected systemic NTP sync issues by rearchitecting IRI NTP environment.
|
| Performance |
- Worked with several teams to troubleshoot a network issue causing
slow WAN transfers at Acxiom.
- Alleviated network link saturation during backups, fixing an application throughput issue.
- Improved system's backups from 36 hours to 9 hours by redesigning network.
- Reverse-engineered a legacy, closed-source middleware application and
created a replacement with Perl and open-source tools, alleviating
high-load issues for IRI clients.
- Wrote software to collect and graph gigabytes of performance data.
- Analyzed large NFS environment and identified bottleneck
causing recurrent client timeouts. Recommended NFS
filesystem reorganization and parallel servers, fixing
the problem.
- Conducted feasibility study and benchmarking of jumbo Gigabit Ethernet frames
and NFS.
- In-depth performance analysis of several systems, including Oracle DB servers.
- Analyzed performance of IRI applications running on HP-UX 11i,
using PerfView, glance, sar, tusc and tcpdump to examine system and NFS
behavior.
|
| Optimization |
- Saved three weeks of batch processing time for Pepsi by rewriting shell/C++ app.
- Reduced shell script runtime from 50 hours to 25 seconds.
- Improved Perl web app execution time from 1 minute to 1 second.
- Saved 60% disk space and 50% time in C++ application.
- Improved shell script runtime from 60 minutes to 5 minutes, alleviating client timeouts.
- Diagrammed a complex undocumented batch workflow, enabling optimization of the bottlenecks.
- Achieved 300% speed improvement by identifying a severe disk
bottleneck in a C application at IRI, rewriting the algorithm and
instituting kernel parameter changes.
|
| Outages |
- Diagnosed a network issue causing repeated client-facing outages at IRI.
- Traced webserver lockups to database locking issue.
- Mitigated outage risk after NFS server crash by injecting new code into closed-source
binary.
- Fixed lock table overflow causing production outage by writing tools
to map NFS locks to filenames and processes, utilizing UNIX kernel internals.
- Diagnosed and fixed out-of-memory condition in client application
causing production outage, via kernel parameter changes.
- Tracked down and fixed internal server errors in Tomcat application.
- Eliminated crashes and leaks in legacy C app, a UNIX-to-mainframe job dispatcher.
|
| Front-end |
- Built a web app in Perl which moves UC4 scheduler jobs between
environments at CME Group, targeted at internal users. It accepts
change request information and multiple XML file uploads, performs
XML transforms and validation using standard modules, and emails the
results.
- Application interfaces with Oracle and LDAP.
- Local database caching with SQLite and failover to ensure high availability.
- Progressive Javascript and AJAX with jQuery. Cross-browser compatible.
- Installer safely deploys application to production, QA, DR
and development systems.
- Created a web interface to an API documentation system, wrote a jQuery
plugin for incremental search, and added proxy caching with nginx. CSS3
media queries and progressive enhancement are used to scale from desktop
to mobile (iPhone, iPad) while emphasizing cross-browser compatibility.
|
| Back-end |
- Rearchitected 10-year-old C and Perl-based login and load balancing app to
allow multiple user sessions. Replaced adhoc databases with a SQL database.
- Wrote a Java program which interfaces with UC4 job scheduler.
- Ported 32-bit HP-UX business app to Linux and added 64-bit support.
Migrated authorization from UNIX/NIS to PAM, enabling use with
LDAP. Fixed buffer overflows and memory leaks.
- Rearchitected the management of IRI's NIS environment by replacing a
collection of shell, Perl and C code with a modular Perl-based
solution, fixing security and file corruption issues. Also overhauled
accompanying Visual Basic user interface for internal clients.
|