Someone didn't appreciate the existence of queueing systems like PBS and rolled their own queueing system that is supposed to take care of removing dead jobs from an app server. So I had to look for the old files that were causing the problem and see if I could delete them. Thankfully this is really easy with a simple find command.
find ./ -type f -mtime +10
Switching the -type to d will do the same for directories. Unfortunately there is no clever hack for beating people who do this kind of thing over the internet.