Did you try running both situations multiple time (e.g. deleting the logo file between run's) to ensure the hdd disk access is not making a difference?
The first time you run comskip on a new file the file will be read into memory. Next run will go much quicker as the disk is no longer the bottleneck
Will depend of course on file size and available memory
Yes, I have tried running both situations several times, and the problem is consistent. It doesn't seem like it's hammering the hdd, so I don't think it's that. Just watching the Comskip process in Task Manager, it looks like it uses 13 threads across all cpu cores while it's doing the logo detection and then drops down to five threads on one cpu core when it slows down. When running it with the logo file already there it uses 13 threads across all cores from the start and continues like that for the whole run.
I've tried this on both H264 and MPEG2 files and it's the same, but MPEG2 is much faster (as expected) running at around 450fps when it uses all cores.