Did I request INI tuning help correctly?

To discuss the tuning of comskip and for posting your ini files
Post Reply
Columbus
Posts: 7
Joined: Sun Jan 27, 2008 9:26 am

Did I request INI tuning help correctly?

Post by Columbus »

Hi Erik, hi everyone,

9 days ago I followed the procedure listed at "How to ask for help" at viewtopic.php?f=5&t=34 and posted two ZIP attachments and a covering email to erik@kaashoek.com.

Perhaps I should have posted my query here first, as a courtesy?

(If so, I apologise, and will paste the contents of the email below, just to be sure. I'll also re-send the email right now, in case it got lost somehow.)

It's just that after 9 days I would have thought some kind of response might be expected, and my MPGs which need Comskipping are piling up!

I'm NOT complaining about this brief delay, though. Far from it! I'd simply thought I would save all the other forum readers the tedium of reading my long, boring, covering email by dealing directly with you (as per the "how to ask for help" instructions.)

But who knows, perhaps some readers may find it helpful too.

I look forward to a reply at your earliest convenience, Erik, and reading any suggestions you can make regarding my INI settings. Keep up the great work!

Columbus
Fremantle, Western Australia


Here's the paste of that January 28 email:

Hi Erik,

let me add my voice to the chorus: "well done on a great piece of software, I hope it makes you rich one day, you deserve it!"

I too would appreciate a bit of guidance regarding fine-tuning my INI file ... it's working pretty well now, but I'm a bit baffled as to how to fine-tune it better from here, and I have one additional, unusal problem I'm hoping you can help with.

I've endeavoured to follow your instructions at "How to ask for help" at viewtopic.php?t=34... please note I've attached TWO zips, containing two job samples using the identical INI file. The INI is fairly closely based on your "More aggressive Comskip INI" at viewtopic.php?f=2&t=427&p=1884&hilit=ag ... +ini#p1884 , but not identical.

• One show is a 4:3 black-bordered "Futurama" episode. Comskip's accuracy with this one is pretty good.
• The other is a 16:9 widescreen "Mythbusters" episode, Comskip is having trouble with one of the ad breaks on this one.

Beginning / ending accuracy of both isn't exactly right either, although the "Futurama" one is mostly darn close.

I've included the comskip.ini, the .csv, .log and .ref files as instructed, and I've carefully edited the REF files to indicate my desired frame cut locations. I've also included the logo.txt and original (unedited) .txt files as Comskip generated them.

The reason for including both the TXT and REF files?

Well, I've included both the TXT and edited REF files so you can directly compare them and see another, particular dilemma I have: a constant problem Comskip has with ALL detections on ANY recordings of mine is this: there is always a spurious "ad break" detected where none exists, at the VERY END of each show; the very last few seconds in fact.

(In other words, both TXT file examples show an entire, EXTRANEOUS FINAL LINE, for an ad-break that never exists, with a duration varying from about 5 to about 140 frames only, compared to the REF files, which are corrected and what I would LIKE Comskip to generate, with your help.)

I have no idea what causes this, and no experimenting I've done with the INI variables seems to enforce my simple desire: "do not detect a spurious ad break under 5 seconds duration AT THE VERY END OF THE RECORDING, IT IS NOT AN AD BREAK."

Can you think of anyway of "hard-wiring" my INI to enforce this? I can manually edit the lines out, but that defeats my near-flawless automation of this whole process. (Note shows here, such as "Futurama" often have a final, single 60 seconds of program content, immediately before this bogus final, ad detection.)

I wonder if any other users suffer from this problem? Would you like me to post about it in your forum to find out?

I'm a contributor (in a small way) to Paul Ahb / Paul Bartlett's very handy WSMPP project ( 'Web Scheduler Module Post Processor' ) ... which is basically a clever collection of VBS scripts and batch files interfacing with several open source video processing modules (e.g. ProjectX) which are highly customisable and can be made to do ANYTHING to your video files with ANY command-line addressable software.

Have you heard of it? Comskip has become a valued component of it, and I was one of its earliest supporters, and one of the first to have a go at INI tuning.

Basically, these are the steps I take:

• record SD DVB-TV in .TS (full Transport Stream) mode using Web Scheduler
• trim the head and tail of the resultant TS file with Mpg2Cut2 ( a fault tolerant, GOP level, binary MPG2 cutter )

then, using WSMPP:

• demux to MPG2 with ProjectX (thus fixing timeline glitches / errors and extracting perfect .SUB subtitles)
• remux back to MPG2 with mplex1
• run Comskip on the new MPG2 (command line) generating EDL (for MPlayer) and BCF (for BSPlayer)
• convert to very high quality MP3 / DivX using DGIndex, WaveGain, lame, AviSynth, avs2avi, DivX Pro 6.8 and finally DivXMux

This all works beautifully, although it's taken a long time to perfect.

The flexibility of AviSynth scripting, of course, basically means any conceivable cropping / processing / resizing / title overlaying etc can be done, and the resultant AVI DivX file can make use of your Comskip ad break files (i.e. EDLs with MPlayer, or my favourite: BCFs with BSPlayer.) The end result is very slick indeed ... when Comskip gets its break detection correct!

(And yes, I could even use Comskip to create an AviSynth file for hard-cutting of the output DivX file, but I can't do that 'cos I wish to use the previously-extracted subtitles with their hard-wired timeline, without having to permanently burn them into the final AVI, also: the ad-break detection isn't 100% reliable as of yet.) Now a utility that could re-synch the subtitle's timeline to match a comskip edit list on the fly ... hmm, that would be something!

I only mention all that in case it's of interest to you, or might explain how the errant final, bogus 5 second "ad breaks" are being detected ... is it to do with all this timeline manipulation / correction? I can't see how, but I mention all these boring details just in case.

I'm hoping, ultimately, that the one INI file can cover all my channels' quirks ... I've selected the two attached samples as very representative of the free-to-air DVB-TV we have here in Perth, Western Australia (although they're very different program types, aspect ratios and from different channels). The logos' positions, sizes and durations are typical of usage here.

So, in short, I'm making two requests:

• Please, can you tweak my INI settings, or point out where my current settings are failing?
• Please, is there any way of nuking that errant last few seconds ad-break entry?

Along with everyone else, I appreciate the hard work you've put into this wonderful tool, which is so close to perfect, it's tantalising!

Erik, If I can help you with anything at anytime, don't hesitate to ask. I await your response with bated breath.

Cheers from Fremantle, Western Australia.

Columbus

(70mm cinema projectionist, film / tv post-production technician for many years, general-purpose I.T. guy, MCP, CISCO CCNA Academy graduate, corporate network engineer)

P.S. URLs of software referred to above:

Web Scheduler ( see http://forums.dvbowners.com/index.php?showforum=28/ and http://www.bluebit.com.au/ )
Mpg2Cut2 ( see http://www.geocities.com/rocketjet4/ )
WSMPP project ( 'Web Scheduler Module Post Processor' ) at http://forums.dvbowners.com/index.php?s ... topic=6888

Attachments were:
--------------------------------------------------------------------------------
4ACV01 Futurama COMSKIP test.zip 582.0 kb
--------------------------------------------------------------------------------
(01-21) SBS - Mythbusters COMSKIP test.zip 1163.8 kb
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Did I request INI tuning help correctly?

Post by erik »

Your mail has never reached my inbox.
Could you send separate mails containing the two attachments then I will have a look
Columbus
Posts: 7
Joined: Sun Jan 27, 2008 9:26 am

Re: Did I request INI tuning help correctly?

Post by Columbus »

Thanks for the prompt reply, Erik.

Well, that explains the 9 day silence! I resent the email and attachments as I stated I would above, and I've just this minute received, and replied to, the test email you sent me. Perhaps my emails are being filtered out at your end as SPAM or something? Perhaps the attachments are too large or something?

For the record, I BCC'd myself with both of the attachment emails, and I successfully received them both, so they are "being sent OK", for what that's worth. (Note, due to my personal PC being in pieces at present, I'm using a standby PC and my ISP's webmail service; but it seems to work OK.)

I look forward to further communications. Is there a "backup" email address you can offer, or is there a mechanism by which I can upload the relevant attachments directly here, via PM or something? If you haven't received my re-sent email with attachments, please advise me, and I'll try sending them from another PC from here with its native email program ... just in case my ISP's webmail service is fussy about big attachments.

Cheers.
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Did I request INI tuning help correctly?

Post by erik »

As mentioned in announcements there where some hosting problems causing loss of email.
I did not yet receive any mails with attachments from you
Columbus
Posts: 7
Joined: Sun Jan 27, 2008 9:26 am

Re: Did I request INI tuning help correctly?

Post by Columbus »

Hi again Erik,

thanks for the offer of your FTP space. That's what I used in the end ... I've just successfully uploaded both ZIPS and a covering RTF of my email to a new folder named "Columbus" at your FTP server.

Thanks again for your patience in this matter; I have no idea where the email problems were occurring, but at least you have those files now. I'm sure looking forward to your feedback regarding them, and any tuning tips you can share.

Best wishes,

Columbus
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Did I request INI tuning help correctly?

Post by erik »

Thank you so much for your input.
I finally have been able to solve the long outstanding "Why is there a short commercial at the end of the recording" bug

I change one setting in your ini file to make processing more robust

max_commercial_size=125

Please download comskip build 114 and test.
If you find build 114 to be OK I will make the general announcement
Columbus
Posts: 7
Joined: Sun Jan 27, 2008 9:26 am

Re: Did I request INI tuning help correctly?

Post by Columbus »

Thanks for that, Erik.

I've downloaded build 114 and am about to test it on a few recordings, especially on those two sample ones I submitted earlier (the Mythbusters and Futurama episodes). I'll report back later.

One tiny confusion ... I'm not quite sure what you mean by this:
I change one setting in your ini file to make processing more robust
max_commercial_size=125
max_commercial_size=125 was the value I was already using; and I see it's the default in the INI in build 114 too. Are you saying I should change it to something else, or that you've done something in build 114 that will make that value more robust?

If I'm to "change that one setting" what should I change it to? Perhaps I've misunderstood you there.

In any case, I'll give 114 a spin with that value and report back later.

Thanks again!
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Did I request INI tuning help correctly?

Post by erik »

In your ini file it is set to 58 and I think 125 should be a better value for max_commercial_size
Columbus
Posts: 7
Joined: Sun Jan 27, 2008 9:26 am

Re: Did I request INI tuning help correctly?

Post by Columbus »

Thanks for looking at those files Erik, but actually, I'm afraid you're mistaken about the reading of that max_commercial_size value in my INI file.

As I said above, I already have that value you're referring to in my INI, and it's been there for several recent versions:

max_commercial_size=125

I think the "58" you're referring to is actually the line two below that one; perhaps you glanced a bit too quickly, or parallax kicked in? That value is indeed:

min_show_segment_length=58

(The reason I'd lowered this value from its default of 125 is because there is often a last bit of Futurama, with a duration of almost exactly 60 secs, immediately before the very end of the show, and 58 secs seems like a reasonable target to catch a segment of exactly 60 seconds in duration.) Do those mathematics sound right?

Is this perhaps what you were thinking about? That changing the min_show_segment_length value, that you might be able to obtain better detection rates? Maybe you meant to type "min_show_segment_length" instead of "max_commercial_size" ? Was this a simple typo or pasting error?

Other than that, I'm afraid I have no other suggestions to make.

I'm now a bit confused again. Please: which value are you referring to, and what changes do you recommend?

I've just run those two same episodes through Build 14, and I'm afraid the output is virtually exactly the same as before; specifically, the same bogus, redundant final line is still being produced, and the same ad commencements being utterly missed. In other words: no improvement at all. (The Futurama test has a few frames different, but only on the last bogus line, and the Mythbusters one is *precisely* the same as before.)

Shall I upload the latest CSVs to prove this point, or will you just take my word for it?

This is most frustrating; I'd hoped the detailed CSVs would provide you with a push in the right direction, but I'm afraid we've accomplished little.

Would you mind please, Erik, casting another careful eye over those ZIPs I uploaded, to see if there are any other clues we may have missed? In the meantime, I'm afraid I have to report that version 14 seems to be no improvement at all.

In particular: the Mythbusters missed commencement errors are *exactly* as before, i.e. missing the commencement of several ad breaks entirely. Are there no clues in its CSV or TXT at all, compared to my desired REF file values?

I appreciate your effort, but I'm afraid no improvement can be reported.

Any other ideas? Surely, my ZIPs must yield more clues? The errors are very consistent.

I'm happy to upload the whole darn MPGs if that will help (3.4 GB in total) but I bet you won't want me to do that!

I'm very grateful for the effort you're putting in here. That last bogus line bug is very stubborn, isn't it? Please keep the suggestions coming. Are there any other values we can tweak? The LOGO parameters, for instance? Or padding?

Cheers, Columbus.
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Did I request INI tuning help correctly?

Post by erik »

You where right about
min_show_segment_length=58
it was indeed that value that I suggested to increase

It is no problem having a show segment in the order of 60 seconds or even less when the min_show_segment_length is set to 125
Multiple clues will be used to determine if a segment is show or commercial.
Did you try the value 125? Did it cause problems?

When you use the value of 58 and there are commercial breaks of 60 seconds or more these will be regarded as show. That is not what you want.

With the value 125 both examples you did send me deliver good results.

Reference mythbuster
FILE PROCESSING COMPLETE 85706 FRAMES AT 2500
-------------------
27258 30266
53939 57700
72609 76005

Detected mytbuster
FILE PROCESSING COMPLETE 85706 FRAMES AT 2500
-------------------
27260 30260
53940 57696
72611 75998
85705 85706


Reference futurama
FILE PROCESSING COMPLETE 45513 FRAMES AT 2500
-------------------
12695 17570
27803 31945
39642 44021

Detected futurama
FILE PROCESSING COMPLETE 45513 FRAMES AT 2500
-------------------
12690 17565
27804 31936
39637 44016
45512 45513

When you do want to use the value of 58 you have to make one change
add
excessive_length_modifier=1
to your ini file.
This prevents commercial breaks with a length larger then 58 seconds to be scored towards show.
This setting may cause other problems so you should test

Hope all this info does help.
Columbus
Posts: 7
Joined: Sun Jan 27, 2008 9:26 am

Re: Did I request INI tuning help correctly?

Post by Columbus »

OK Erik,

I finally found time to give the new INI setting (min_show_segment_length=125) a go today, and it worked splendidly. I got precisely the same results as you did, with version 14. I'm most impressed ... and with such differing program material ... and with no errors ... very impressive.

Who'd have thought one little change could make such a difference, but it seems to. I'm now about to run a batch of coversions, on a whole lot of program material, using the new setting, wish me luck.

I'll report back with my results, but I'm very optomistic since both the test files we've discussed gave such super accurate results with that new value.

Thanks, Erik, I'm delighted. The stubborn last bogus line problem is still with us, I see. Any ideas about how to handle that? Just manually deleting that last spurious line, I guess? It doesn't make too much difference in practice I suppose, but it sure would be nice to nail that problem too.

But I'm not complaining; I'm very happy with the accuracy we're now achieving, and will now go and put it through some serious testing.

Cheers
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Did I request INI tuning help correctly?

Post by erik »

Good to hear!
Is the last line with the one frame commercial bothering you?
Its there only when the recording does not end with a commercial because some other users needed to have a last entry with the last frame number.
You can delete it easily.
The problem with the small commercial (couple of seconds, not one frame) being found should be solved now.
Just make sure you tune the min_commercial_break_at_start_or_end setting to the right value/
Columbus
Posts: 7
Joined: Sun Jan 27, 2008 9:26 am

Re: Did I request INI tuning help correctly?

Post by Columbus »

Hi Erik, hi everybody,

well, the latest settings are looking really good! I've processed a lot of recorded shows in the last week, but only examined precise ad breaks versus Comskip detection for seven of them, varying from "Tomb Raider" the movie to a b&w documentary on post-war Japan.

Of these seven recordings I took at random, I got the following results:

43 commercial detections in all, all were perfectly correct within a few frames except for 2 which were fooled by a subtly inserted mini ad, and only one outright minor "error", which cropped about 7 seconds of a movie. One out of 43!

A pretty astounding result; and I would consider that one "error" within tolerance ... it's a 7 seconds error, which is less than 10 seconds, which might seem to be a reasonable target of error for this kind of thing.

I'll go and check another six or seven shows at my next opportunity, and report back if I get wildly differing results ... but for now, this is very close to a practical 100% accuracy rate and I am very happy with this version and the settings we've got going. Congratulations!

So for people in Perth, Western Australia, give this INI, version 14 and these settings a go; I'm now getting the best results I've ever achieved with Comskip, over quite a period of time, and several versions.

Thanks very much, Erik, I appreciate very much the feedback effort you made; I'll report back with any developments, but in the meantime: well done.

Cheers.
Post Reply