Why are these blocks not commercials?

[yankinoz]

Hi - I'm rather new to comskip (only been messing with it for a week now) and think I've got most of it figured out. It's doing a great job of finding the breaks between show (in Australia) and commercial but sometimes it's leaving in blocks of commercial - but I can't figure out why it won't mark them as commercial. In the example below, blocks 28 and 34 should be commercial - anyone know why they are not?

I'm using comskip79_48

Here's an example, first my ini file:

Code: Select all

detect_method=255			;1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 255=all
validate_silence=1			; Default, set to 0 to force using this clues if selected above.
validate_uniform=1			; Default, set to 0 to force using this clues (like pure white frames) if blackframe is selected above.
validate_scenechange=1		; Default, set to 0 to force using this clues if selected above.
verbose=10				;show a lot of extra info, level 5 is also OK, set to 0 to disable
max_brightness=60      			;frame not black if any pixels checked are greater than this (scale 0 to 255)
test_brightness=40      		;frame not pure black if any pixels checked are greater than this, will check average brightness (scale 0 to 255)
max_avg_brightness=25			;maximum average brightness for a dim frame to be considered black (scale 0 to 255) 0 means autosetting
max_commercialbreak=600 		;maximum length in seconds to consider a segment a commercial break
min_commercialbreak=25			;minimum length in seconds to consider a segment a commercial break
max_commercial_size=300			;maximum time in seconds for a single commercial or multiple commercials if no breaks in between
min_commercial_size=5   		;mimimum time in seconds for a single commercial
min_show_segment_length=200 	; any segment longer than this will be scored towards show.
non_uniformity=500
max_volume=100				; any frame with sound volume larger than this will not be regarded as black frame
min_silence=20				; Any deep silence longer than this amount  of frames is a possible cutpoint
ticker_tape=0				; Amount of pixels from bottom to ignore in all processing 
punish=0					; Compare to average for sum of 1=brightness, 2=uniform 4=volume, 8=silence, 16=schange, set to 0 to disable
punish_threshold=1.3		; Multiply when amount is above average * punish_threshold
punish_modifier=2			; When above average * threshold multiply score by this value
intelligent_brightness=0 		; Set to 1 to use a USA specific algorithm to tune some of the settings, not adviced outside the USA
logo_percentile=0.92			; if more then this amount of logo is found then logo detection will be disabled
logo_threshold=0.75
punish_no_logo=1			; Default, set to 0 to avoid show segments without logo to be scored towards commercial
aggressive_logo_rejection=0
connect_blocks_with_logo=1		; set to 1 if you want successive blocks with logo on the transition to be regarded as connected, set to 0 to disable
cut_on_ar_change=1			; set to 1 if you want to cut also on aspect ratio changes, set to 0 to disable
delete_show_after_last_commercial=0	; set to 1 if you want to delete the last block if its a show and after a commercial
delete_show_before_or_after_current=0	; set to 1 if you want to delete the previous and the next show in the recording, this can lead to the deletion of trailers of next show
remove_before=0				; amount of seconds of show to be removed before ALL commercials
remove_after=0				; amount of seconds of show to be removed after ALL commercials
shrink_logo=5				; Reduce the duration of the logo with this amount of seconds
after_logo=0		; set to number of seconds after logo disappears comskip should start to search for silence to insert an additional cutpoint
padding=0
ms_audio_delay=5
max_repair_size=40			; Will repair maximum 40 missing MPEG frames in the timeline, set to 1 to disable repairing for players that don't use PTS. 
delete_logo_file=0			; set to 1 if you want comskip to tidy up after finishing
output_debugwindow=1   ; Only for tuning, to be replaced later
non_uniformity=1000         ; Only for tuning, to be replaced later
max_volume=1000         ; Only for tuning, to be replaced later
output_framearray=1			; create a big excel file for detailed analysis, set to 0 to disable
output_videoredo=0
output_womble=0
output_mls=0			; set to 1 if you want MPeg Video Wizard bookmark file output
output_cuttermaran=0
output_mpeg2schnitt=0
output_mpgtx=0
output_dvrcut=0
output_zoomplayer_chapter=0
output_zoomplayer_cutlist=0
output_edl=0
output_edlx=0
output_vcf=0
output_btv=1				; set to 1 if you want Beyond TV chapter cutlist output
output_projectx=0			; set to 1 if you want ProjectX cutlist output (Xcl)
output_demux=0				; set to 1 if you want comskip to demux the mpeg file while scanning
sage_framenumber_bug=0
sage_minute_bug=0
live_tv=0					; set to 1 if you use parallelprocessing and need the output while recording
live_tv_retries=1			; change to 8 when using live_tv in BTV
cuttermaran_options="cut=\"true\" unattended=\"true\" muxResult=\"false\" snapToCutPoints=\"true\" closeApp=\"true\""
mpeg2schnitt_options="mpeg2schnitt.exe /S /E /R25  /Z %2 %1"
dvrcut_options="dvrcut \"%s.dvr-ms\" \"%s_clean.dvr-ms\" "

Now, from the debug window, Block 28 and Block 34 are commercial. Comskip defined the breakpoints perfectly and they are both shorter than the 200 second min_show_length. From the block list, I don't see why either is marked as show:

Code: Select all

 25:--    1   1   0  42300  42674  9.00   14.960  0.10   0 1.30            S     b 1082+ 0.00 1338+ 95-  0.000   108 NONE      
 26:--    7   1   6  42675  44179  9.00   60.160  0.22   0 1.30            S     b 1289+ 0.00 1298+ 95-  0.000   128 NONE      
 27:++   11   7   4  44180  64311  0.00  805.240  0.05   0 1.30        EL    v  ub 319- 0.93 1065- 98+  0.000    31 NONE      
 28:++    5   5   0  64312  66313  1.00   80.040  0.19   0 1.30                  b 1470+ 0.00 1319+ 960  0.000   147 NONE      
 29:--    1   1   0  66314  67813  9.00   59.960  0.09   0 1.30            S     b 663- 0.00 1321+ 94-  0.000    66 NONE      
 30:--    1   1   0  67814  68188  9.00   14.960  0.13   0 1.30            S     b 981+ 0.00 1459+ 97+  0.000    98 NONE      
 31:--    2   1   1  68189  68564  9.00   15.000  0.06   0 1.30            S v   b 1045+ 0.00 1190- 93-  0.000   104 NONE      
 32:--    7   1   6  68565  69568  1.50   40.120  0.21   0 1.30           N      b 1218+ 0.00 1416+ 95-  0.000   121 NONE      
 33:++   14   6   8  69569  85627  0.00  642.320  0.08   0 1.30        EL    v   b 324- 0.90 1000- 960  0.000    32 NONE      
 34:++    8   8   0  85628  88251  1.00  104.920  0.65   0 1.30              v   b 2016+ 0.00 1383+ 97+  0.000   201 NONE      
 35:--    1   1   0  88252  88937  2.48   27.400  0.23   3 1.30          C     a   2199+ 0.00 1485+ 960  0.000   219 NONE      

The Fuzzy Scoring ignores both:

Code: Select all

Combining Blocks 20 thru 21 result in strict standard commercial length of 150.00 with a tolerance of 0.600000.
Block 21 score:	Before - 1.50	After - 2.25	Combined count - 2
Block 20 score:	Before - 3.38	After - 5.06	Combined count - 3
Block 21 has logo.
Block 21 score:	Before - 2.25	After - 0.02
Block 22 has strict standard length for a commercial.
Block 22 score:	Before - 1.00	After - 9.00
Block 23 has strict standard length for a commercial.
Block 23 score:	Before - 1.00	After - 9.00
Block 24 has strict standard length for a commercial.
Block 24 score:	Before - 1.00	After - 9.00
Block 25 has strict standard length for a commercial.
Block 25 score:	Before - 1.00	After - 9.00
Block 26 has strict standard length for a commercial.
Block 26 score:	Before - 1.00	After - 9.00
Block 27 has logo.
Block 27 score:	Before - 1.00	After - 0.01
Block 27 has excess length.
Block 27 score:	Before - 0.01	After - 0.00
Block 29 has strict standard length for a commercial.
Block 29 score:	Before - 1.00	After - 9.00
Block 30 has strict standard length for a commercial.
Block 30 score:	Before - 1.00	After - 9.00
Block 31 has strict standard length for a commercial.
Block 31 score:	Before - 1.00	After - 9.00
Block 32 has non-strict standard length for a commercial.
Block 32 score:	Before - 1.00	After - 1.50
Block 33 has logo.
Block 33 score:	Before - 1.00	After - 0.01
Block 33 has excess length.
Block 33 score:	Before - 0.01	After - 0.00
Combining Blocks 35 thru 36 result in strict standard commercial length of 29.88 with a tolerance of 0.200000.
Block 35 score:	Before - 1.00	After - 1.50	Combined count - 1
Block 36 score:	Before - 1.00	After - 1.50	Combined count - 1
Combining Blocks 36 thru 40 result in strict standard commercial length of 20.20 with a tolerance of 0.240000.
Block 36 score:	Before - 1.50	After - 1.80	Combined count - 2
Block 37 score:	Before - 1.00	After - 1.20	Combined count - 1
Block 38 score:	Before - 1.00	After - 1.20	Combined count - 1
Block 39 score:	Before - 1.00	After - 1.20	Combined count - 1
Block 40 score:	Before - 1.00	After - 1.20	Combined count - 1
Combining Blocks 35 thru 36 result in strict standard commercial length of 29.88 with a tolerance of 0.200000.
Block 36 score:	Before - 1.80	After - 2.70	Combined count - 3
Block 35 score:	Before - 1.50	After - 2.25	Combined count - 2
Combining Blocks 37 thru 43 result in strict standard commercial length of 29.76 with a tolerance of 0.240000.
Block 37 score:	Before - 1.20	After - 1.37	Combined count - 2
Block 38 score:	Before - 1.20	After - 1.37	Combined count - 2
Block 39 score:	Before - 1.20	After - 1.37	Combined count - 2
Block 40 score:	Before - 1.20	After - 1.37	Combined count - 2
Block 41 score:	Before - 1.00	After - 1.14	Combined count - 1
Block 42 score:	Before - 1.00	After - 1.14	Combined count - 1
Block 43 score:	Before - 1.00	After - 1.14	Combined count - 1
Combining Blocks 36 thru 40 result in strict standard commercial length of 20.20 with a tolerance of 0.240000.
Block 40 score:	Before - 1.37	After - 1.65	Combined count - 3
Block 39 score:	Before - 1.37	After - 1.65	Combined count - 3
Block 38 score:	Before - 1.37	After - 1.65	Combined count - 3
Block 37 score:	Before - 1.37	After - 1.65	Combined count - 3
Block 36 score:	Before - 2.70	After - 3.24	Combined count - 4
Block 42 has non-strict standard length for a commercial.
Block 42 score:	Before - 1.14	After - 1.71
Combining Blocks 37 thru 43 result in strict standard commercial length of 29.76 with a tolerance of 0.240000.
Block 43 score:	Before - 1.14	After - 1.31	Combined count - 2
Block 42 score:	Before - 1.71	After - 1.96	Combined count - 2
Block 41 score:	Before - 1.14	After - 1.31	Combined count - 2
Block 40 score:	Before - 1.65	After - 1.88	Combined count - 4
Block 39 score:	Before - 1.65	After - 1.88	Combined count - 4
Block 38 score:	Before - 1.65	After - 1.88	Combined count - 4
Block 37 score:	Before - 1.65	After - 1.88	Combined count - 4
Block 44 has strict standard length for a commercial.
Block 44 score:	Before - 1.00	After - 9.00
Combining Blocks 37 thru 44 result in strict standard commercial length of 59.80 with a tolerance of 0.200000.
Block 44 score:	Before - 9.00	After - 10.13	Combined count - 1
Block 43 score:	Before - 1.31	After - 1.47	Combined count - 3
Block 42 score:	Before - 1.96	After - 2.20	Combined count - 3
Block 41 score:	Before - 1.31	After - 1.47	Combined count - 3
Block 40 score:	Before - 1.88	After - 2.12	Combined count - 5
Block 39 score:	Before - 1.88	After - 2.12	Combined count - 5
Block 38 score:	Before - 1.88	After - 2.12	Combined count - 5
Block 37 score:	Before - 1.88	After - 2.12	Combined count - 5
Combining Blocks 35 thru 44 result in strict standard commercial length of 89.68 with a tolerance of 0.160000.
Block 44 score:	Before - 10.13	After - 11.14	Combined count - 2
Block 43 score:	Before - 1.47	After - 1.62	Combined count - 4
Block 42 score:	Before - 2.20	After - 2.42	Combined count - 4
Block 41 score:	Before - 1.47	After - 1.62	Combined count - 4
Block 40 score:	Before - 2.12	After - 2.33	Combined count - 6
Block 39 score:	Before - 2.12	After - 2.33	Combined count - 6
Block 38 score:	Before - 2.12	After - 2.33	Combined count - 6
Block 37 score:	Before - 2.12	After - 2.33	Combined count - 6
Block 36 score:	Before - 3.24	After - 3.56	Combined count - 5
Block 35 score:	Before - 2.25	After - 2.48	Combined count - 3
Block 45 has strict standard length for a commercial.
Block 45 score:	Before - 1.00	After - 9.00
Block 46 has strict standard length for a commercial.
Block 46 score:	Before - 1.00	After - 9.00
Block 47 has logo.
Block 47 score:	Before - 1.00	After - 0.01
Block 47 has excess length.
Block 47 score:	Before - 0.01	After - 0.00
Block 48 has strict standard length for a commercial.
Block 48 score:	Before - 1.00	After - 9.00
Block 49 has strict standard length for a commercial.
Block 49 score:	Before - 1.00	After - 9.00
Block 50 has strict standard length for a commercial.
Block 50 score:	Before - 1.00	After - 9.00
Block 51 has strict standard length for a commercial.
Block 51 score:	Before - 1.00	After - 9.00
Block 52 has strict standard length for a commercial.
Block 52 score:	Before - 1.00	After - 9.00
Block 53 has logo.
Block 53 score:	Before - 1.00	After - 0.01
Block 53 has excess length.
Block 53 score:	Before - 0.01	After - 0.00

I can zip and send you the whole logs if you want. This isn't a totally isolated example for me - I've seen this happen with several programs.

Thanks
Rob

[erik]

This is indeed strange because they both have no logo so they should be scored towards commercial.
You correctly set
punish_no_logo=1

Could you mail me the ini file you use, the .csv file, the .log file and the list of segments that are scored wrongly (or better a corrected .txt file)?

Keeping max_volume at 1000 seems rather high. Try to lower that value to say 500.

[yankinoz]

wow - that was quick!

max_volume is set to 100 further up in the file, it's my understanding the the first one is the one that is used. Is that right?

I've emailed the full log and ini files and, well everything except the dvr-ms file!

[yankinoz]

interesting: I just ran the edited version of the same program (which included the two unwanted commercial blocks - but the rest had been removed) and comskip identified both correctly as commercials.

hmm I wonder if double processing is a valid strategy...

[erik]

The problem is in the large news segment without logo you recorded before the actual show.

If the total fraction of the recording that has logo drops below 50% then logo is no longer used to remove parts that do not have a logo because it is assumed the a ratio worse then 50% show and 50% commercial means that the logo detection is unreliable.

So if you would have recorded less of the news than everything would have been OK.

That also explains why the second run correctly detects the two commercial block leftovers.

Comskip only uses the FIRST occurrence of a setting in the ini file.

[yankinoz]

If only Australian TV could broadcast late night television at or near the advertised time! but of course we can't control that here!

Now I have some questions:

By FIRST occurrence, you mean from the top down, right?

In the debug window I'm getting a black bar (not gray, but black) which I thought meant the logo was being used. Blocks 27 and 33 both have 'L' listed which I thought indicated that the logo was used to score these blocks as show. Is this correct?

I have another example below, a shorter show that has kept two entire commercial breaks - block 14 (clearly because it exceeded min_show_segment_length) and blocks 16 & 17 - these two are shorter and should (I think) be marked as commercial. This particular show did not have enough logo and in the debug window the logo detection shows as a gray bar and there are no blocks listed with an L

From this (same ini file as above) and you tell why 16 & 17 are show? (I'll email the full logs to you seperately)

Code: Select all

  #     sbf  bs  be     fs     fe    sc      len   scr cmb   ar                cut bri  logo  vol  sil corr stdev        cc
  0:--    5   0   5      1    884  1.50   35.320  0.16   0 1.30           N      b 1003- 0.00 1374+ 98+  0.000   100 NONE      
  1:--    6   6   0    885   3639  1.50  110.160  0.14   1 1.30          C    s    1079+ 0.00 1018- 91-  0.000   107 NONE      
  2:--    7   1   6   3640   3892 13.50   10.080  0.15   1 1.30          C S     b 545- 0.00 793- 94-  0.000    54 NONE      
  3:++   13   6   7   3893   6182  1.00   91.560  0.10   0 1.30                  b 330- 0.00 962- 960  0.000    33 NONE      
  4:++   25   8  17   6183   6705 99.99   20.880  0.38   0 1.30            S    ub 1896+ 0.00 1064- 93-  0.000   189 NONE      
  5:++   18  18   0   6706  16143  0.02  377.480  0.16   0 1.30        E      s    1413+ 0.00 1138+ 98+  0.000   141 NONE      
  6:++   12   1  11  16144  18703  1.00  102.360  0.12   0 1.30                  b 1088+ 0.00 1174+ 98+  0.000   108 NONE      
  7:--   12  12   0  18704  19339 18.00   25.400  0.14   0 1.75       A    S  s    1108+ 0.00 1106+ 92-  0.000   110 NONE      
  8:--    3   1   2  19340  20091  9.00   30.040  0.08   0 1.30            S     b 468- 0.00 1044- 94-  0.000    46 NONE      
  9:--    2   2   0  20092  21213  9.00   44.840  0.20   0 1.30            S     b 499- 0.00 1186+ 95-  0.000    49 NONE      
 10:--    1   1   0  21214  21588  9.00   14.960  0.08   0 1.30            S  s    733- 0.00 1156+ 960  0.000    73 NONE      
 11:--    1   1   0  21589  21964  9.00   15.000  0.21   0 1.30            S  s    1130+ 0.00 1166+ 93-  0.000   113 NONE      
 12:--    3   1   2  21965  22214 18.00    9.960  0.39   0 1.30            S     b 2672+ 0.00 1317+ 960  0.000   267 NONE      
 13:++   13   2  11  22215  28584  0.01  254.760  0.12   0 1.30        E         b 1127+ 0.00 1085+ 98+  0.000   112 NONE      
 14:++   20  11   9  28585  33599  0.01  200.560  0.14   0 1.30        E         b 945- 0.00 1025- 94-  0.000    94 NONE      
 15:++   26   9  17  33600  42002  0.01  336.080  0.12   0 1.30        E         b 1149+ 0.00 1097+ 97+  0.000   114 NONE      
 16:++   17  17   0  42003  43893  1.00   75.600  0.15   0 1.30               s    1310+ 0.00 1121+ 94-  0.000   131 NONE      
 17:++    1   1   0  43894  47271  1.00  135.080  0.10   0 1.30               s    781- 0.00 859- 92-  0.000    78 NONE      
 18:--    3   1   2  47272  47517 18.00    9.800  1.03   0 1.30            S     b 4780+ 0.00 1372+ 97+  0.000   478 NONE      
 19:-+    3   3   0  47518  59342  0.01  472.960  0.09   0 1.30        E           894- 0.00 1097+ 98+  0.000    89 NONE 

[erik]

That is where your silence detection had to e better tuned

set it to
max_volume=500
and
min_silence=10

[yankinoz]

I'm starting to understand a bit better now. The 'score' starts at 1.0 and if comskip can detect some reason that a block may or may not be commercial, it will raise or lower the score (higher means commercial, lower means show.) In my second case, the two segments in question both scored 1.00 so were left in. I tried setting the global threshold to .99 and it took them out - but that screwed up a bunch of other recordings all in test folders of course. global_threshold is back to 1.05 - I see the logic in leaving something in if you can't tell for sure it's commercial.

Erik - I think what you were saying before is that with less than 50% logo the punish logo parameter stops working, but logo will still be used to detect show. Is this right?

[yankinoz]

i think we just posted at the same time. I'll try the settings you suggest.

update new silence settings work great on recording number two. I'll be testing them on other recordings.

[erik]

I think what you were saying before is that with less than 50% logo the punish logo parameter stops working, but logo will still be used to detect show. Is this right?

Indeed