Build 89

kerryklarson · Post by **kerryklarson** » Wed Mar 28, 2007 1:41 pm

I just ran across a real problem episode. The program is a little more difficult because it is an ATSC 16:9 broadcast whose material is 4:3 pillarboxed. However, I'm pretty sure that's not the problem. There is complete disagreement between ComskipGUI and VideoRedo about the frame numbering at the cutpoints. VRD shows that only frame 18451 is 'in the black' of the cut points that Comskip detected. When viewed in ComskipGUI, all cut points are 'in the black' except 23412. We're not talking about a few frames here. It's dozens or more. I think this might be due to timing errors in the source material that aren't being accounted for. Have set max_repair_size=0 in *.ini and zero discontinuity checkpoints in VRD. Log files are uploaded as ftp://athome.kaashoek.com/Kerry/b89/Pussycat Dolls Present_ The Search for the Next Doll-(She Dances Like a Drag Queen)-2007-03-27-0.zip. Final frame in VRD is 107725. Any thoughts?

Post by **erik** » Wed Mar 28, 2007 4:10 pm

Looking at the timing.csv file there are at least 85 missing frames compared to the PTS so when max_repair_size=0 the PTS based videoredo will at the end of the file 85 frames off compared to the comskip numbering.
Also when max_repair_size=0 the preview in comskip (which is using the PTS to reposition) will be increasingly wrong as the comskip analysis has been done on actual present frames while the review repositioning is done on PTS.
PTS says there are 107730 frame, actually there are 107645 frames
Will look further into it.
Could you also test with max_repair_size=200 and see if comskip review agrees with comskip analysis?

kerryklarson · Post by **kerryklarson** » Wed Mar 28, 2007 10:59 pm

Could you also test with max_repair_size=200 and see if comskip review agrees with comskip analysis?

I fear that I'm now massively confused (once again). Files are uploaded as ftp://athome.kaashoek.com/Kerry/b89/Pussycat Dolls Present_ The Search for the Next Doll-(She Dances Like a Drag Queen)-2007-03-27-0.MRS200.zip.

Unexpectedly (to me), max_repair_size=200 significantly improved the cut points. Only 84702/84526 was far off from a black transition. I thought it had been established that max_repair_size=0 was needed for VRD. See viewtopic.php?t=334&postdays=0&postorder=asc&start=15 and http://www.videoredo.net/msgBoard/showt ... 530&page=3.

ComskipGUI transitions appear to be in the black with both max_repair_size=200 or max_repair_size=0 when using the *.csv file.

I did notice one other thing. VRD frame references have always been one digit less than the CS *.txt file references (presumably due to the initial frame difference, 0 vs 1). For this video, VRD is two behind, whether the timeline was corrected or not: 18465/18467, 18450/18452. VideoRedo_Offset=0 setting.

Post by **erik** » Thu Mar 29, 2007 6:57 am

You are right about the previous conclusion on max_repair_size=0 for VRD.

There may be a difference between "missing frames" and "corrupt frames"
I don't count missing or corrupt. VRD may not count missing.
Will look into this

kerryklarson · Post by **kerryklarson** » Sat Mar 31, 2007 1:10 pm

From viewtopic.php?t=365:

erik wrote:Comskip uses timing information for the analysis of the commercials.
This information has to be accurate for a high quality analysis.
The timeline repair enures this timing information is correct.
Disabling the timeline repair by setting max_repair_size=0 will degrade the commercial recognition performance.
It is best to use a commercial cutter or player that also interprets and repairs the timeline.

This is really problematic. I went back to the original American Idol episode (ftp://athome.kaashoek.com/Kerry/b77/Ame ... -0-.77.zip) that led to the conclusion that max_repair_size=0 was needed. I then ran that video through VRD's QuickstreamFix as recommended here: http://www.videoredo.net/msgBoard/showt ... 530&page=3. I then ran Comskip on the 'fixed' video with max_repair_size=400. The wrong cut points were all there again, virtually unchanged from the original run. To top it off, there were still multiple PTS errors displayed in the output window. So it appears that some errors require no correction and other types of errors require correction.

I have uploaded the log files of the latest run to ftp://athome.kaashoek.com/Kerry/b89/Ame ... -0.QSF.zip.

Post by **erik** » Sat Mar 31, 2007 1:19 pm

I am discussing how QSF does its repair on the VRD forum.
Hope they can explain what VRD and QSF are actually doing so I can generate the correct VRD input.
For the moment the info is rather fuzzy.

kerryklarson · Post by **kerryklarson** » Sat Mar 31, 2007 9:41 pm

I reconfirmed the max_repair_size=0 produces perfect detection on this video. ftp://athome.kaashoek.com/Kerry/b89/American Idol-2007-02-20-0.QSF.MRS0.zip

Post by **erik** » Sat Mar 31, 2007 10:06 pm

I received the licence key so I can do some testing on corrupt TS files.
Its clear that a real good solution will be extremely difficult.
For the best analysis max_repair_size should be set to 40 or larger (200). This will ensure the timing of the commercials is maintained.
However quickstreamfix does not use the PTS to repair the timing of the video but only for the synchronization of audio and video so a repaired chunks of video will not be reinserted by QSF resulting in incorrect cut points.
One solution is to use the PTS for the timing analysis but the real frame count for the comemrcial list generation. And now how to do this without using vast amounts of memory or slowing down comskip.......

Post by **erik** » Sat Mar 31, 2007 10:15 pm

kerryklarson wrote:I reconfirmed the max_repair_size=0 produces perfect detection on this video. ftp://athome.kaashoek.com/Kerry/b89/American Idol-2007-02-20-0.QSF.MRS0.zip

This video has 143 single frame video errors spaced evenly over the file causing only minor timing errors, each small enough to fall in the detection range so there should be no impact on detection accuracy. Setting max_repair_size=0 is the best setting for this kind of small errors.
only when you record from a bad broadcast will multi frame video burst errors there will be an impact on timing analysis.
Maybe you could try to optimize your antenna configuration to eliminate these errors.
The "pussycatdols" recording has twice an error where 12 frames are missing leading to wrong timing analysis inside a commercial, therefore the results of the analysis where better with max_repair_size=200. But then the output was very inconsistent with VRD
Yet no good solution......

kerryklarson · Post by **kerryklarson** » Sat Mar 31, 2007 10:55 pm

erik wrote:I am discussing how QSF does its repair on the VRD forum.

I'm following. I just posted comments/questions there. I posted the VRD-log.txt file to your FTP (Kerry/Build 89).

You are correct that Pussycat Dolls does have some significant dropouts. These are noticable when watching. These will occur from time to time even with a good signal due to random interference.

For the best analysis max_repair_size should be set to 40 or larger (200). This will ensure the timing of the commercials is maintained.

It seems the opposite for me, because most of my recordings are of good quality. If I do this, I'm back to where I was with almost all of the cut points being late. It's only the occasional bad video that seems to need the repair.