Global Threshold Adjustments? / My INI

To discuss the tuning of comskip and for posting your ini files
Post Reply
pcon2009
Posts: 20
Joined: Wed Sep 28, 2016 5:30 pm

Global Threshold Adjustments? / My INI

Post by pcon2009 »

I was wondering if anyone else has adjusted the global_threshold parameter in the process of tuning their INI file? I know it says not to change this value in the documentation... BUT, I have been making a lot of tweaks over the last few weeks, and I *think* I am finally getting close to something that works on a wide variety of different programming... However, my global threshold is currently at 1.79 because I was getting some show pieces (NOT commercials) marked as commercial, albeit with a rather low score, and 99% of my commercials are scoring rather high (at least 2.0, often 10+). I will post my .INI file as it is currently, in case anyone wants to have a look at it, make adjustments to suit their needs etc. Basically, I am just looking for any feedback anyone has; if you DO use this, let me know how it works for you.

NOTE 1: I record OTA programming in the USA, and have no idea how this may or may not work for cable/satellite or non-USA programming.

NOTE 2: A couple settings are specific and may need adjusted for your situations, including thread_count (set to 6, as my PC is hexacore, set to 4 if you have quadcore, etc) and possibly some others. Also, I use captions and tend to modify my dictionary as needed, though this shouldn't make a huge difference as it is mostly a "backup" method and just inflates the already-above-threshold scores of commercials.

Code: Select all

detect_method=59 ;1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 128=cutscenes, 255=all
validate_silence=1 ; Default, set to 0 to force using this clues if selected above.
validate_uniform=1 ; Default, set to 0 to force using this clues (like pure white frames) if blackframe is selected above.
validate_scenechange=1 ; Default, set to 0 to force using this clues if selected above.
verbose=10 ;show a lot of extra info, level 5 is also OK, set to 0 to disable
max_brightness=60 ;frame not black if any pixels checked are greater than this (scale 0 to 255)
test_brightness=40 ;frame not pure black if any pixels checked are greater than this, will check average brightness (scale 0 to 255)
max_avg_brightness=25 ;maximum average brightness for a dim frame to be considered black (scale 0 to 255) 0 means autosetting
brightness_jump=200 ;
max_commercialbreak=600 ;maximum length in seconds to consider a segment a commercial break
min_commercialbreak=4 ;minimum length in seconds to consider a segment a commercial break
max_commercial_size=140 ;maximum time in seconds for a single commercial or multiple commercials if no breaks in between
min_commercial_size=4 ;mimimum time in seconds for a single commercial
non_uniformity=500 ; Set to 0 to disable cutpoints based on uniform frames
max_volume=150 ; any frame with sound volume larger than this will not be regarded as black frame
min_silence=30 ; Any deep silence longer than this amount  of frames is a possible cutpoint
ticker_tape=0 ; Amount of pixels from bottom to ignore in all processing
punish=1 ; Compare to average for sum of 1=brightness, 2=uniform 4=volume, 8=silence, 16=schange, set to 0 to disable
punish_threshold=2 ; Multiply when amount is above average * punish_threshold
punish_modifier=4 ; When above average * threshold multiply score by this value
intelligent_brightness=0 ; Set to 1 to use a USA specific algorithm to tune some of the settings, not adviced outside the USA
logo_percentile=0.98 ; if more then this amount of logo is found then logo detection will be disabled
logo_threshold=0.75 ;
punish_no_logo=1 ; Default, set to 0 to avoid show segments without logo to be scored towards commercial
aggressive_logo_rejection=0 ;
connect_blocks_with_logo=1 ; set to 1 if you want successive blocks with logo on the transition to be regarded as connected, set to 0 to disable
logo_filter=0 ; set the size of the filter to apply to bad logo detection, 4 seems to be a good value.
delete_show_after_last_commercial=1 ; set to 1 if you want to delete the last block if its a show and after a commercial
delete_show_before_or_after_current=1 ; set to 1 if you want to delete the previous and the next show in the recording, this can lead to the deletion of trailers of next show
remove_before=0 ; amount of seconds of show to be removed before ALL commercials
remove_after=0 ; amount of seconds of show to be removed after ALL commercials
shrink_logo=2 ; Reduce the duration of the logo with this amount of seconds
after_logo=0 ; set to number of seconds after logo disappears comskip should start to search for silence to insert an additional cutpoint
padding=1.75 ;
ms_audio_delay=0 ;
volume_slip=20 ;
skip_b_frames=0 ; Set to 1 to force Comskip to skip frames for higher processing speed.
hardware_decode=0 ; Set to 1 to enable hardware accelerated video decoding, only available in donator version
max_repair_size=400 ; Will repair maximum 200 missing MPEG frames in the timeline, set to 0 to disable repairing for players that don't use PTS.
disable_heuristics=0 ; bit pattern for disabling heuristics, adding 1 disables heristics 1, adding 2 disables heristics 2, adding 4 disables heristics 3, 255  disables all heuristics
delete_logo_file=0 ; set to 1 if you want comskip to tidy up after finishing
output_framearray=1 ; create a big excel file for detailed analysis, set to 0 to disable
output_data=1 ; create a dump of the user data channel, used for CC and XDS (such as V-Chip info). Can be use together with output_framearray to remote debug CC decoding
output_videoredo=0 ; The old videoredo format
output_videoredo3=0 ; The new videoredo v3 format.
output_womble=0 ;
output_mls=0 ; set to 1 if you want MPeg Video Wizard bookmark file output
output_cuttermaran=0
output_mpeg2schnitt=0
output_mpgtx=0
output_dvrcut=0
output_zoomplayer_chapter=0
output_zoomplayer_cutlist=0
output_edl=1
output_dvrmstb=0 ; Set to 1 if you're running DVRMS-Toolbox
output_edlx=0
output_vcf=0
output_bsplayer=0
output_btv=0 ; set to 1 if you want Beyond TV chapter cutlist output
output_projectx=0 ; set to 1 if you want ProjectX cutlist output (Xcl)
output_avisynth=0
output_vdr=0 ; set to 1 if you want XBMC to skipping commercials
output_demux=0 ; set to 1 if you want comskip to demux the mpeg file while scanning
sage_framenumber_bug=0
sage_minute_bug=0
live_tv=0 ; set to 1 if you use parallelprocessing and need the output while recording
live_tv_retries=4 ; change to 16 when using live_tv in BTV, used for mpeg PS and TS
dvrms_live_tv_retries=300 ; only used for dvr_ms
standoff=0 ; change to 8000000 when using live_tv in BTV
cuttermaran_options="cut=\"true\" unattended=\"true\" muxResult=\"false\" snapToCutPoints=\"true\" closeApp=\"true\""
mpeg2schnitt_options="mpeg2schnitt.exe /S /E /R25  /Z %2 %1"
avisynth_options="LoadPlugin(\"MPEG2Dec3.dll\") \nMPEG2Source(\"%s\")\n"
dvrcut_options="dvrcut \"%s.dvr-ms\" \"%s_clean.dvr-ms\" "
windowtitle="Comskip - %s"
two_pass_logo=1
maxbright=1
noise_level=5
global_threshold=1.79
thread_count=6
delete_show_before_first_commercial=1
length_strict_modifier=3
length_nonstrict_modifier=1.5
combined_length_strict_modifier=2
combined_length_nonstrict_modifier=1.25
ar_wrong_modifier=10
excessive_length_modifier=0.01
dark_block_modifier=0.3
min_schange_modifier=0.5
max_schange_modifier=2
logo_present_modifier=0.1
border=10
give_up_logo_search=1020
delay_logo_search=120
logo_max_percentage_of_screen=0.1
logo_percentage_threshold=0.38
edge_level_threshold=5
edge_radius=2
edge_step=2
before_logo=0
min_black_frames_for_break=1
ar_delta=0.1
min_show_segment_length=210
cc_commercial_type_modifier=1.75
cc_wrong_type_modifier=1.75
cc_correct_type_modifier=0.8
fps=29.97
logo_fraction=.25
ignore_side=0
cut_on_ar_change=1
variable_bitrate=1
always_keep_first_seconds=0
always_keep_last_seconds=0
output_default=1
output_plist_cutlist=0
output_tuning=0
output_training=0
added_recording=4
use_existing_logo_file=0
EDIT 1: Changed "padding" to 2.5 as Comskip was removing a small amount of the show, and I prefer (as I think most people do) to occasionally have a couple seconds of commercials rather than occasionally losing a couple seconds of show.

EDIT 15-OCT-16: Made some changes including disabling scene change detection (method 123 now, was 127) decreasing min black frames for break from 3 back to 1 (one recording had only 1 black frame between end of a commercial break and the show starting again), increasing logo_threshold_percentage to .45 (up from .35) and increasing min_show_segment_length to 210 (observed a commercial block of length ~190 in a recording) as well as possibly a few other small tweaks since last posting.

EDIT 19-OCT-16: INI updated, and started new list of results based on the new INI. New INI includes changes listed from period between 15-OCT and today, as well as a couple other small tweaks.
Last edited by pcon2009 on Thu Oct 20, 2016 5:58 am, edited 8 times in total.
WhiteKnight
Posts: 19
Joined: Wed Jun 29, 2016 7:30 pm

Re: Global Threshold Adjustments? / My INI

Post by WhiteKnight »

Don't suppose you could provide a short list of success and failures??? I don't mind testing, it isn't necessary for me to duplicate your results though.
pcon2009
Posts: 20
Joined: Wed Sep 28, 2016 5:30 pm

Re: Global Threshold Adjustments? / My INI

Post by pcon2009 »

WhiteKnight wrote:Don't suppose you could provide a short list of success and failures??? I don't mind testing, it isn't necessary for me to duplicate your results though.
I will start keeping a better list of what seems to be working and what doesn't. Unfortunately I haven't been keeping a running list of what didn't work because I've been making adjustments on the fly until it did work. I do have several episodes saved (unprocessed recordings) for future testing, so when I have some time I will go back through and post the results.
pcon2009
Posts: 20
Joined: Wed Sep 28, 2016 5:30 pm

Re: Global Threshold Adjustments? / My INI

Post by pcon2009 »

Using the .ini file above, and forcing myself NOT to make any changes until I run a good sample of shows through it (rather than my habit of tweaking it ever so slightly after every show), here is what I am finding (shows organized by network, with date of recording YYYMMDD):


ABC
  • Designated Survivor 20161012 - Partial Success (Comskip removed most but not all of the commercials. Commercials that remain are due to network using logo during commercials for this and other shows.)
NBC
  • Timeless 20161003 - 100% Success (This did NOT have pre/post 2 minutes I usually add to my recording and the ini was tuned specifically with this one before posting my last edits above)
  • The Good Place 20161013 - FAIL (Comskip cut out the end of show before like it should and worked well for all commercial breaks UNTIL the last commercial break was cut out along with the final piece of show after that break, jumping right into the start of the next show.)
FOX
  • Family Guy 20161016 - FAIL (Comskip removed the commercials as it should, but one small block of show was randomly removed in the middle, not even adjacent to commercials. Will need to investigate this further, but initial review shows extremely poor logo detection. Adjusting delay_logo_search brought back the portion of the show, but lost the first couple of minutes due to presence of a different logo. This needs more testing.)
  • The Simpsons 20161016 - 100% Success (Everything seemed to work on this episode, DESPITE logo not being present at all, and logo detection method being disabled. Oddly, I remember a previous episode of The Simpsons failing due to a lack of logo. For some reason, FOX does not seem to apply logo to this show, so this may be hit or miss, but at least this recording worked correctly.)
CBS
  • Survivor - 100% Success? - (Last time I tested on this show, it worked perfectly. I don't recall what settings may have been adjusted, so will need to review this again sometime later this week after a new episode airs. Typically, this show has very good logo presence/detection and nice lengthy black frame cut points, so hasn't caused any issues for me yet.)
Last edited by pcon2009 on Mon Oct 17, 2016 6:21 am, edited 3 times in total.
pcon2009
Posts: 20
Joined: Wed Sep 28, 2016 5:30 pm

Re: Global Threshold Adjustments? / My INI

Post by pcon2009 »

As I have been testing, I have worked out some adjustments to my INI that will fix some of the noted failures above. I am listing them here because if anyone intends to use my INI, I would recommend trying these changes if you have any similar problems. After each different recording is tested, I have been switching my INI back to what I show in the first post, so once I am able to test a multitude of shows, making adjustments as necessary and compiling them here, I will apply ALL of the following changes, and re-test to make sure one fix didn't break another.
  • logo_percentage_threshold=0.38; Down from .45 (Fixed the missing final segment from The Good Place)
  • cc_commercial_type_modifier=1.75 ; Down from 16 (Was set very high because commercial type CC seemed accurate, until some incorrect detection of commercial type CCs were found in a show segment of The Good Place, resulting in too high scores)
  • cc_wrong_type_modifier=1.75; Down from 8 (Same as above, just in case this one is incorrectly detected as well)
  • detect_method=59; Changed from 123 (Disabling Silence method, fixing the above Family Guy issues - extra cutpoints were being generated in the middle of the show, which due to the poor albeit sufficient logo detection, was causing some of the blocks without logo to be removed)
  • punish_modifier=4;Up from 2 (Fixes Designated Survivor issues with commercials that had logo. Even though commercials had logo, they were also too bright using punish=1, so setting this to 4 bumped the blocks above threshold into commercials.)
Sufreak
Posts: 20
Joined: Tue Sep 13, 2016 2:17 am

Re: Global Threshold Adjustments? / My INI

Post by Sufreak »

I had the same issue with The Simpsons from 10/16/2016. I ran it wiht the default ini file, and I think it worked much better. I have to watch it to tell, but judging by time, it seems better.
pcon2009
Posts: 20
Joined: Wed Sep 28, 2016 5:30 pm

Re: Global Threshold Adjustments? / My INI

Post by pcon2009 »

Here is a new list of rest results, as I have decided to go ahead and make the changes I previously mentioned since it seems like they are working better. (New list started as of today, 19-Oct-2016. The INI file posted in the first message is also updated as of today.)

NOTE: This list contains results from two different INI iterations, so each result will be marked with a (1) or a (2). I made adjustments after looking at WhiteKnight's INI, which is where I started (2). However, based on some of these results, I will be doing another new version with more adjustments very soon, so I will probably be starting another new list soon.

CBS
  • The Big Bang Theory 17-Oct-2016 (1) - 90% Success (Several commercials remain, but they are previews for other CBS shows and as such show the same logo, so at the moment, I am not worried about trying to get rid of these as they aren't a particular bother to me if everything else works.)
  • Bull. 18-Oct-2016 (1) - 75% Success (Removed some, not all commercials, but did not lose any show bits. Some of this is due to network promo spots with logo, but also had a few spots with no logo that were not removed, will need more investigation.)
  • Survivor 19-Oct-2016 (1) - 99% Success (Removed all commercials, except for a short one right before the "scenes from our next episode", which were also kept.)
NBC
  • Timeless 17-Oct-2016 (1) - 90% Success (Several commercials remain, all previews for other NBC shows. Same issues as Big Bang Theory on CBS)
  • The Good Place 20-Oct-2016 (2) - FAILURE(I am still losing the last showblock from this show, and also missed a commercial or two. Anytime I lose any bit of show, I can't consider it even a partial success.)
Post Reply