By Erik.
A short guide to tuning Comskip
Setting up the test environment.
Starting the tuning: the black
frame detection.
Using aspect ratio based cutting
Final commercial list generation
Automated setting of certain
parameters
Using the optimized Comskip.ini
Verifying Comskip output during
normal execution
Before the tuning of Comskip
can start it is essential to understand the basic operation of Comskip. Comskip
reads and decodes a recorded mpeg file based on instructions Comskip reads from
a so called ini file. The recording is analyzed on various characteristics such
as black frames, silences and changes in aspect ratio.
Based on this information
Comskip segments the recording in blocks and using heuristics, together with
additional information such as the presence of logo, the scene change rate,
Close Captioning information and other information sources Comskip tries to
determine what blocks of the recording are to be characterized as commercials.
This process can only be
successful under a number of conditions. The information to be used for the
segmentation must be available in the recording. When, for example, the
recording contains insufficient or too many black frames the segmentation
process has a high chance to fail. It is therefore essential to instruct
Comskip how to recognize the various segmentation information sources to ensure
an optimal result.
However, Comskip may not be
able to deliver acceptable results on certain recordings from certain
broadcasters because the required information is simply not available in the
broadcast. The purpose of the tuning is therefore to extract the maximum of
information from the recording to enable Comskip to do the best possible job.
A simple strategy is to
install Comskip and use the comskip.ini file available in the download in the
PVR system and let the PVR do its recording. Afterwards, when watching the
recording enhanced by the information from Comskip the results can be judged
and if not as required some changes to the ini file can be made. The impact of
these changes can be judged after the next recording is done.
This strategy works if only
limited adaptations are required. It is not the right approach to get the best
results for specific broadcasting styles.
It is therefore recommended
to use a more elaborate strategy to optimize Comskip.
Create a separate folder and
copy the files from the downloaded Comskip zip file into this directory.
Then copy a number of
relevant recordings into this directory and rename them to appropriate short
names.
There should now be the
following files in the folder. Comskip.exe, comskip.ini, comskip.dictionary and
a number of mpeg files. Lets name them for this guide as short.mpg, long.mpg,
etc....
One mpeg files should preferably
not be very big, about 10 minutes, and contain a mix of show and commercial.
This we will call short.mpg.
A second mpeg should be a full recording of about one hour containing a typical
sequence of show segments mixed with commercials. This will be called long.mpg.
Now edit the comskip.ini
file. Comskip is rather picky on how you format the ini file. No spaces before
of after the '=' are allowed. Multiple occurrences of any setting can exist
anywhere in the file but only the first will be used. There is no way to
'comment out' a setting.
The following settings
should be present when starting the tuning, each on a separate line:
verbose=10
detect_method=47
delete_logo_file=0
intelligent_brightness=0
output_framearray=1
output_debugwindow=1 ; Only for tuning, to be replaced later
non_uniformity=1000 ;
Only for tuning, to be replaced later
max_volume=1000 ;
Only for tuning, to be replaced later
Now we need to run Comskip
on the first mpg file. For the first part of the tuning we will use short.mpg.
To run Comskip on the mpeg we create a shortcut icon on the desktop to
Comskip.exe. Drag and drop short.mpg on the Comskip icon and watch the processing
being done.
First a console window opens
and some initial logging appears. Then the debug window opens. To understand
the content of the debugwindow please read debugwindow description file.
The frame number in the top
left corner should start at 1 and increment with jumps till the last frame
number.
Once Comskip reaches the
final frame there will be some more logging in the console window and the
debugwindow will jump to the first frame.
Comskip can now be
terminated by closing the Comskip console window.
The test folder will contain
a number of new files, each with its specific content.
The file short.log contains
the logging output of the last comskip run.
The file short.txt contains
the start end ending frame numbers of the identified commercials, if any.
The file short.csv contains
information on brightness, volume, etc... one line per frame. The first line
contains the descriptive name of the information. The last figure on the first
line is the frame rate times hundred as read from the mpeg file.
The file short.logo.txt
contains the logo recognition mask if a logo has been found.
Now use your favorite editor
or text file viewer and open the log file short.log.
The first part of the log
file lists some essentials of the comskip run. Then there is information on the
frame per frame scanning starting with the line
First Frame
After all frames has been
scanned there is a line
Finished scanning
file. Starting to build Commercial
List.
After this line the global
processing with closing frame groups and cleaning up information.
Then the recording is
segmented in blocks starting with a line such as
Creating block 0
From 1 (1) to 350 (348) because of Blackframe with 0 head and 2 tail
Then some additional small
cleaning up and information generation and the Fuzzy Logic scoring starts with
Fuzzy scoring of the
blocks
After the scoring there is
the information on the detected commercials starting with
Initial Commercial List
And finally the list of
blocks with their details starting with
Block list after
weighing
Five parameters determine
the black frame detection process. They are
max_avg_brightness=20
test_brightness=40
max_brightness=60
non_uniformity=1000
max_volume=1000
intelligent_brightness=0
The first parameter to look
at is max_avg_brightness, in general test_brightness should be 20 above
max_avg_brightness and max_brightness should be again 20 above test_brightness.
A frame is a candidate black
frame when the average of the brightness of its pixels if below
max_avg_brightness. It is therefore required to identify the black level in the
recording and set the max_ avg_brightness somewhat above this level. To do this
look in the log file at the list of black frames just below
Black Frame List
There should be lines such
as
14
16776 16 35 77
Where 16776 is the frame
number, 16 the brightness, 35 the uniformity and 77 the volume. If there are no
black frames the max_avg_brightness should be increased with steps of 5,
together with test_brightness and max_brightness. Rerun Comskip on short.mpg
after each increase.
Look at the minimum
brightness in the black frame list that should occur a number of times, and set
max_avg_brightness to that level plus five. Rerun Comskip to verify the correct
setting. Use the debugwindow to verify the correctness. Use the ‘n’ key to jump
to each next black frame and use the cursor keys to scan to the right to verify
if a number of the frames behind the current position are indeed black.
Be sure to set
test_brightness and max_brightness to respective 20 and 40 above
max_avg_brightness.
When one single pixel in a
frame has brightness above max_brightness then that frame will not be regarded
as black. When more then 5 pixels on a line have a brightness above
test_brightness the frame will not be regarded as black.
The detection of the black
bars used for the aspectratio calculation uses the same settings so when
short.mpg contains parts with different aspectratios is easy to see in the
debugwindow if the processing is correct.
Sometimes there is something
inside the recording that needs to be ignored. One possibility is to ignore a
border around the picture. This is controlled by
border=10
The default border is 10.
Another option is to ignore
only a part at the bottom of the video. This is useful when the broadcast
contains tickertape at the bottom. The controlling parameter specifies the
number of pixels at the bottom to ignore as:
ticker_tape=120
The default is 0, the
setting can be checked in the debugwindow because the part of the video that
will be cancelled out is shown as black.
Black frames with a volume
above max_volume will be removed from the black frame list before the final
black frame list is shown. This is shown in the log file with lines such as
391 - Removing black
frame 41319, from black frame list because volume 1655 is more than 600,
brightness 16, uniform 1
Volume levels of –1 signal
that Comskip was unable to match the mpeg sound decoding with the picture
decoding for that specific frame so these volumes should be ignored.
Key is to identify the
volume level that is regarded as silence. This is never at zero but could be
anywhere between 10 and 500 depending on the broadcast
Some sequences of subsequent
black frames in the black frame list possibly show the volume going down,
remain for some frames at a low level and then again going up.
Below you can find a typical
sequence from the black frame list
1076 165202 16 148 178
1077 165203
16
141 104
1078 165204
16
148 97
1079 165205
16
110 90
1080 165206
16
105 90
1081 165207
16
95 90
1082 165208
16
78 90
1083 165209
16
72 90
1084 165210
16
38 90
1085 165211
16
48 90
1086 165212
16
4 78
1087 165213
16
0 78
1088 165214
15
0 75
1089 165215
15
0 75
1090 165216
16
0 75
1091 165217
15
0 75
After frame 165214 the
final black level is reached with brightness of 15, uniform of 0 and volume of
75.
The lowest volume level that
reoccurs on more then one frame sequence is the silence level. The max_volume
level should be set at about four times the silence level. There may be black
frame sequences that do not show the low volume level of silence but remain at
substantial higher volume levels. These sequences should not be regarded as
having silence and the max_volume level can be set below these levels to
effectively remove them from the black frame list.
Below you will find an
example from the black frame list where the frames are perfectly black
(brightness is 16 and uniformity is below 5) but the sound level is too high to
be silence.
193 82467
16
3 537
194 82468
16
1 537
Acceptable max_volume levels
could be in the range of 250 up to 1000.
Again the debugwindow can be
used to visually verify the alignment of the black frames and the silences. Use
the ‘a’ key to jump to the next block boundary and use the cursor key to verify
if the frames behind the current position indeed do have a black frame with
silence.
Black frames with low
brightness and silence do normally also have a low level of non_uniformity,
possibly as low as zero. The non_uniformity level is used to recognize frames
with uniform brightness, as these could also be block separators. To set the
non_uniformity level look again at the black frame list just like what was done
for the max_volume but now for the uniformity.
Acceptable levels for
non_uniformity could be in the range of 250 up to 1000. However if you record
on low quality or have a very high noise level you may have to go up to 2000.
Start Comskip with the
adjusted setting on long.mpg and manually verify the black frame detection by
using the 'n' and the 'p' keys to step through the detected black frames using
the debugwindow.
When no black frame is
detected between certain segments use the arrow keys to step frame by frame
through the transition. Look at the detected values behind the frame number in
the upper left corner and verify that processing done as expected.
For those that have access
to spreadsheet software such as excel there is also another way to find the
right level.
After Comskip has been run
on short.mpg there appears a short.csv file in the test folder. Now open this
file with your spreadsheet software.
The first part of the
spreadsheet looks like this
frame |
brightness |
scene_change |
logo |
uniform |
sound |
minY |
MaxY |
ar_ratio |
goodEdge |
isblack |
2500 |
1 |
0 |
0 |
1 |
0 |
-1 |
10 |
566 |
130 |
495 |
0 |
|
2 |
64 |
0 |
1 |
27177 |
-1 |
10 |
566 |
130 |
495 |
0 |
|
3 |
64 |
480 |
1 |
27259 |
1618 |
10 |
566 |
130 |
495 |
0 |
|
4 |
64 |
475 |
1 |
27327 |
1482 |
10 |
566 |
130 |
495 |
0 |
|
5 |
64 |
475 |
1 |
27310 |
2759 |
10 |
566 |
130 |
495 |
0 |
|
6 |
64 |
475 |
1 |
27285 |
1026 |
10 |
566 |
130 |
495 |
0 |
|
7 |
64 |
480 |
1 |
27353 |
2291 |
10 |
566 |
130 |
495 |
0 |
|
8 |
64 |
475 |
1 |
27365 |
2771 |
10 |
566 |
130 |
495 |
0 |
|
Now select the cell just
below brightness and sort the sheet in ascending order on brightness (in excel
click the a/z button on the toolbar or select data/sort from the menu). The
frames with the lowest brightness will appear at the top of the list and it is
easy to see what the black level is. Do the same for the volume (below the word
sound) to find the silence level.
To facilitate this process
one can download a large Comskip.xls files to be used as a template for more
detailed analysis. Copy the short.csv from the open spreadsheet tab (select the
whole sheet and then Copy) and Paste it on the input_data sheet (select cell A1
and Paste). Now it is possible to use the autofilter on the input_data sheet
and to look at the two graphs on the other sheets.
Comskip can, next to black
frame based cut points, also generate cutpoints when the aspect ratio changes.
This optional feature is controlled by
cut_on_ar_change=1
If this is set to 1 Comskip
will segment the recording also on changes on aspect ratio. Whether this is a
good thing largely depends on broadcast style and personal preferences. Cutting
on aspect ratio changes is automatically disabled when insufficient logo is
found and will only happen in segments where no logo is found.
The presence of logo on a
certain segment of the recording guides the heuristics towards interpreting
that segment as show. It is therefore important to have good logo detection.
Logo detection is a very
complicated process with a large number of tunable parameters. It is therefore
advised not to modify most of these parameters.
Only when there are multiple
slightly different logo’s in the broadcast and Comskip fails to see the
difference between them it is advised to set
aggressive_logo_rejection=1
Some shows have multiple
logos. When all logos are at the top or at the bottom of the screen there is no
problem. However, when the logo's are on diagonal sides of the screen Comskip
will not accept the logo as valid. A possible way to solve this is to set
subtitles=1
This will instruct Comskip
not to search for logos at the bottom of the screen.
When Comskip is started and
it cannot find a logo file such as short.logo.txt it will start to scan for a
logo after 1000 frames. You can change this by setting give_up_logo_search to
another value such as
give_up_logo_search=2000
In the debugwindow you can
see the appearance of pixels that are candidate for the logo mask. The maximum
fraction of the screen size that Comskip accepts as a valid logo is defined by:
logo_max_percentage_of_screen
=0.1
This value should be
acceptable in most cases but can be increase to 0.35 if needed.
Once the logo has been
identified Comskip saves the learned logo edge pixels and start again from the
beginning. This can be disabled for slightly faster processing but this is not
recommended.
When Comskip has learned the
logo it will track the presence of logo as shown in the logo rectangle and in
the black bar in the Frame bar at the bottom of the debugwindow. The fraction
of edge pixels that should match the mask is defined by
logo_threshold=0.70
When too many false
positives are detected this value can be increase in steps of 0.05 till
0.95.
When one logo block was
found and after that a long time of no logo, Comskip assumes it has learned a
wrong logo and restarts its learning from well within the recording. When after
this a logo is found Comskip will restart for a second time.
The following setting
controls the deletion of the logo file
delete_logo_file=0
When set to 1 the learned
logo file will be deleted after processing is finished.
When no logo is found after
about 12 minutes in the show Comskip will give up logo detection and assumes no
logo at all.
The amount of logo over time
will tell Comskip how to interpret the presence of logo. If more then 50% logo
is found Comskip assumes logo to be present on show only. If less then 32% logo
is found but more then 15% Comskip assumes logo to be present on commercials
only and the logo logic is reversed.
If the amount of logo
detected is above a certain threshold defined by
logo_percentile=0.92
it is assumed there is
always logo and the logo heuristics will also be disabled.
When the logo detection is
assumed to be robust it can be used to suppress false black frames from being
detected. This is enabled with the following setting
connect_blocks_with_logo=1
This will connect two blocks
into one block if there is logo present at the end of the first block and at
the beginning of the second block. An example of a line signaling the deletion
of a block due to this connection process is
Joining blocks 20 and 21 at frame 149986 because they both have a logo.
It is also possible to
create additional pseudo blackframes some seconds after a logo disappears or
before a logo appears The following line in the ini file will create a pseudo
blackframe 6 seconds after the logo disappears and 8 seconds before the logo appears.
after_logo=6
before_logo=8
Sometimes the logo continues
too long and only disappears during the start of the commercial. This can be
solved by shrinking the duration of the logo by using
shrink_logo=10
The effect on the processing
is that the logo is assumed to appear 10 seconds later than reality and to
disappear 10 seconds before reality. As
long as there are no black frames just before the end of the show there is no
need to be very precise on this setting
Using all the information
gathered from the recoding Comskip will generate a so call Block List. This
block list is directly after the black frame list.
An example of a block list:
Creating
block 0 From 1 (1) to 2074 (2071) because of Blackframe with 0 head and 3 tail
Creating
block 1 From 2075 (2078) to 2339 (2339) because of Blackframe with 3 head and 0
tail
Creating
block 2 From 2340 (2341) to 2509 (2508) because of Blackframe with 1 head and 1
tail
Creating
block 3 From 2510 (2511) to 2830 (2827) because of Blackframe with 1 head and 3
tail
Creating
block 4 From 2831 (2834) to 3651 (3650) because of Aspectratio with 3 head and
1 tail
Creating
block 5 From 3652 (3653) to 5279 (5276) because of Blackframe with 1 head and 3
tail
Creating
block 6 From 5280 (5283) to 5813 (5810) because of Blackframe with 3 head and 3
tail
Creating
block 7 From 5814 (5817) to 6266 (6265) because of Aspectratio with 3 head and
1 tail
Creating
block 8 From 6267 (6268) to 6890 (6889) because of Aspectratio with 1 head and
1 tail
Creating
block 9 From 6891 (6892) to 10306 (10306) because of Blackframe with 1 head and
0 tail
The reason for segmentation
to the next block is indicated as Blackframe or Aspectratio
After the logo processing
certain blocks may have been connected as described above leading to a reduced
block list. The final block list can be found at the very end of the log file.
Block
list after weighing
----------------------------------------------------
#
sbf bs be fs fe
sc len scc
scr cmbnd ar
cut bri logo
stdev
0)+
3 0 3 1 2074
0.18 82.920 0
0.00 0 1.30
b 61 0.83 120
1)-
3 3 0 2075 2339
1.78 10.560 0 0.00
2 1.30 b 101 0.00
98
2)-
2 1 1 2340 2509
1.78 6.760 0
0.00 2 1.30
b 86 0.00 108
3)-
4 1 3 2510 2830
1.78 12.800 0
0.00 2 1.30
a b 82 0.00
63
4)-
4 3 1 2831 3651
2.00 32.800 0
0.00 0 1.58
a 124 0.00
142
5)-
4 1 3 3652 5279
1.00 65.080 0
0.00 0 1.30
b 125 0.00 138
6)-
6 3 3 5280 5813
1.00 21.320 0
0.00 0 1.30
b 109 0.00 218
7)-
4 3 1 5814 6266
1.00 18.080 0
0.00 0 1.30
a 145 0.00
97
8)-
2 1 1 6267
6890 3.00 24.920
0 0.00 0
1.69 a 77
0.00 117
9)+
1 1 0 6891 10306
0.32 136.600 0
0.00 0 1.30
a b 69 0.69
142
The letters below ‘cut’
describe again the reason for the segmentation: a=aspectatio, b=blackframe,
u=uniform frame, s=scene change. If you see more ‘u’ then ‘b’ below ‘cut’ then
your black frame detection may need adjustment. See above on how.
The start (fs) and the end
(fe) frame numbers can be used for the visual verification in the debugwindow.
Use the ‘a’ key to skip quickly to the end of each block. Use the ‘b’ key to
skip to the beginning of each block.
Remember that Comskip will
differentiate between show and commercials based on this block list. If certain
blocks combine both show and commercial there is no way for Comskip to split
them again after the creation of the block list. Only when additional black
frames or aspect ratio changes are detected Comskip can further split the blocks.
Now the most important
parameters for the creation of the block list have been set it is time to tune
the heuristics. Comskip uses a heuristics or Fuzzy Logic scoring process to
determine if a block is part of a show or part of a commercial. First ensure
Comskip has been run on long.mpg and it has generated long.csv.
The most important
parameters guiding the heuristics are
min_show_segment_length=140
max_commercial_size=120
min_commercial_size=4
The amount of score increase
or decrease is controlled by the following parameters
length_strict_modifier=3.0
length_nonstrict_modifier=1.5
combined_length_strict_modifier=2.0
combined_length_nonstrict_modifier=1.25
logo_present_modifier=0.01
excessive_length_modifier=0.01
ar_wrong_modifier=2.0
min_schange_modifier=0.5
max_schange_modifier=2.0
schange_threshold=80
cc_commercial_type_modifier=4.0
cc_wrong_type_modifier=2.0
cc_correct_type_modifier=0.75
dictionary_modifier=1.05
When the scoring process is
done Comskip will generate the list of commercials using this parameter
global_threshold=1.05
Any block that scores (below
‘sc’ in the final block list) above this threshold will be regarded as
commercial, any block scoring lower will be regarded as show. Then Comskip
combines the blocks into the commercial list and does one additional check
using the following parameters
max_commercialbreak=600
min_commercialbreak=10
Any commercial that is
shorter then min_commercialbreak
or longer then max_commercialbreak will
be discarded from the list.
The final result is visible
in the debug windows as the gray bars in the Frame bar and will be output the
.txt file. The blocks that are part of the show will be indicated with a ‘+’ in
the block list.
Although it is possible to
do a complete rerun of Comskip after changing some setting it is possible to do
it somewhat quicker. One the .csv file is generated Comskip can read the .csv
file instead of the .mpg file. This will eliminate the long mpeg decoding
process so Comskip will be much quicker. Dragging and dropping a file like
long.csv on the Comskip desktop icon can do this. The debugwindow will show the
first frame of the mpeg once the processing is finished and in the Frame Bar
the details can be seen. Using the various keys the result can be quickly
reviewed.
Please remember that this can
only be done for changes to the heuristics parameters and the max_volume
setting. Any changes to the brightness and logo detection parameters requires a
full rerun of Comskip.
Fine tuning for certain
broadcast styles.
Some recordings will contain
after the final commercial the beginning of the next show. This is more often
happening when post-padding is applied. To automatically delete this set the
following parameter
delete_show_after_last_commercial=1
As a consequence the final
commercial segment at the end of the recording is allowed to be much longer
then max_commercialbreak
The reverse is also
available to automatically delete the part of the show before the first
commercial set
delete_show_before_first_commercial=1
Both above algorithms will be
automatically disabled if the commercial is not within the outside 15% of the
recording
When logo detection is
enabled Comskip also automatically recognizes a special recording sequence: a
show block of a previous show, optionally a single short commercial block of
length less then min_commercialbreak and then the first show block of the intended show.
Comskip can also detect the reverse of this pattern at the end of the show. The
single show blocks before the first real show block and after the last real
show block will be deleted automatically.
Also a single long show block without logo before or after the other
show blocks will be deleted automatically. This behavior can be disabled with
delete_show_before_or_after_current=0
In certain circumstances the
commercials should be smaller as detected by Comskip. This can be done by using
the following line in the ini file
padding=6
The will set the start of
the detected commercial 6 seconds later and sets the end of the commercial 6
seconds earlier. The debugwindow will show this by marking the first and last 6
seconds of the commercial as lighter gray.
For certain broadcast
condition such as typical in the USA or Canada some tuning has been automated.
To use this set
intelligent_brightness=1
max_volume=0
The scoring threshold can
also be automatically adjusted by setting
global_threshold=0
Now Comskip will automatic
try to determine a number of settings based on the exact fraction of show
present in the recording and the expected amount of black frames. This is
determined by
score_percentile=0.71
black_percentile=0.0073
These fractions should match
exactly you broadcast conditions and any show with a lot of very dark or black
frames or less then 1-
score_percentile fraction commercial
will mess up the automated process. They are therefore not recommended in
countries with less strictly regulated broadcast conditions.
Unless you know what you are
doing it is not recommended to use this automated approach.
Before the optimized
Comskip.ini can be used some changes have to be made.
During normal use the
debugwindow should not be shown, no .csv should be generated, the logo file
should be deleted and no or only limited logging should be done.
These are the recommended
settings.
verbose=0
delete_logo_file=1
output_framearray=0
output_debugwindow=0
During normal execution you
may want to verify the quality of the generated .txt file listing the detected
commercials. A simple way to do this is to drag and drop the generated .txt
file on the Comskip.exe. This will open the debugwindow and show the in the
frame bar the location of the detected commercials. Using the mouse to click on
the frame bar or by using the keyboard as described in the debugwindow guide
one can quickly review the correctness of the generated data. It is even
possible to edit the list of commercials and to save the new list.
Pressing F1 will show the
available command keys.
Comskip runs on “Idle”
priority allowing other programs to execute normally. The Disk IO by Comskip
can be rather high so it may be useful
to reduce the throttle the execution speed of Comskip. The main parameters for
this with their default values are
play_nice_sleep=10
play_nice_start=-1
play_nice_end=-1
The value of play_nice_sleep is the number of milliseconds Comskip will suspend execution after each
frame processed. This will be done starting from play_nice_start till play_nice_end, both in 24 hour time format. From 7 in the morning till eleven in the
evening will be
play_nice_start=700
play_nice_end=2300