Resubmitting incomplete jobs to IonMan

One feature of the JobHunter workflow is that you can resubmit a failed sample run without starting over. It will resume right where it left off. Here's how you do it.

Using resubmit (command-line)

This method is best when the only thing that was broken was the last step, and you know how to fix it. If it is anything more complex, see the manual guide.

1. Fix whatever broke with your rejected sample.
2. Run resubmit on the reject job directory:

resubmit /data/PRD/jobs/rejects/0823h-trc-username-2005-03-20-1236

All resubmit does is remove the condor logs, the frank.flg, and move the directory to the input folder.

Manually

1. Go to your rejected or previously run sample
2. Fix whatever input files you need to get the sample to run again
3. YOU MUST REMOVE condor/joblog.txt - this is the file that records the success or failure of your job. If it is there, it will move the job back to output or rejects without running anything.
4. Move the sample folder, for instance, yeas3, to /data/PRD/jobs/input

If you want JobHunter (runjob, to be specific) to re-run a particular program, you must remove the output files for that program. If re-running this program will change the results of programs that run afterwards, you must delete the results of those programs as well.

For instance, if you want to run go3d, you must remove it's peak file. Since the peak file is used to run drift_calib, which in turn is used to generate make_dta and mascot results, you must delete the output of those programs too.

These are the files you must remove if you would like a section to be re-run:

Example: Re-running with a mascotsearch

Say you forgot to include the mascotsearch input files, ms-mak2.inp and ms-mak3.inp, but the job ran well otherwise. Just do this:

• copy ms-mak2.inp and ms-mak3.inp into the sample directory
• remove condor/joblog.txt
• move the sample directory to /data/PRD/jobs/input

Example: Re-running go3d & pp3d

If I had a bad results with go3d and pp3d, and wanted to restart my sample from there, I would remove:

• condor/joblog.txt
• *.peak
• *.abc
• DTA-*
• ms-*

Then move the folder into /data/PRD/jobs/input