Mike Schuh: Technical Solutions

During the course of my career, I have provided solutions to many problems. Some of them are described below, and a few include source code.
Note: at the moment(May 2005), this page is a "work in progress".

The following are described in greater detail below:
These lines will be replaced with links when the corresponding descriptions are completed.
  • speeding up software configuration management process
  • debugging infinite loop when TAB was pressed on "intelligent terminal"
  • verifying completion of build process
  • automating portion of check reconciliation process
  • sendmail queue growing to abnormal length
  • mail loops offsite
  • bogus DNS server
  • /tmp filling up
  • monitoring health of multiple systems
  • bug in sendmail/firewall interaction
  • setting up syslog for multiple systems
  • monitoring 9-1-1 calls (call monitor)
  • reporting daily call activity
  • setting up DNS update process
  • monitor for SS7 messages
  • GPS clock failure
  • DGPS network bug
  • /opt_filling_up
  • runaway process using >90% of cpu

    Problem:/opt_filling_up
    Customer:XYPOINT NOC
    Techniques/languages used:Perl
    Problem details: A NOC staff member reported that /opt on one of their monitoring systems was at 98% of capacity. By itself, this statistic meant little. However, shortly after my arrival at XYPOINT, I had written a script which daily logged the amount of space consumed on each of the productions systems, including this one. From this, I extracted the relevant numbers for the /opt partition for the previous week and discovered that the it had been growing at about 8 MB per day. When I ran find /opt -type f -mtime -1 -ls the largest file that I found was only about 1 MB (and, incidently, related to another heretofore unknown problem).
    Solution:I wrote a Perl script to search for files that had been opened (created) and then unlinked. Doing so causes the file to be "removed" from its directory even though it continues to consume space. It took me about an hour to develop and run the script. When I did, I found a single file that was over 350 MB long. It appeared to be a log file from a systems monitoring program. Killing that program immediately freed up the disk space.

    Debriefing this with my manager later, we observed that I probably was the only employee of the company who could have solved this: the other sysadmin did not have any programming background, and it was unlikely that the developers were knowledgable about UNIX file system peculiarities.

    More recently (May 2002), Brian Hatch has written an online article[March, 2005: dead link, sorry, I'll look for the article] about files that are open but unlinked.
    Back to the top

    Problem:monitor for SS7 messages
    Customer:
    Techniques/languages used:
    Problem details:
    Solution:

    Back to the top


    Problem:
    Customer:
    Techniques/languages used:
    Problem details:
    Solution:

    Back to the top

    To Mike Schuh's home page
    Mail (but not spam) is welcome to:     schuh AT farmdale D0T com

    Thank you for the visit.