EDL files incorrect in eyetv?

[webster242]

I recently upgraded to eyetv HD and upgraded comskip to the donator version (running osx 10.7.5). I've tried marking at least half a dozen files, and only one has produced commercials marked correctly. I am trying to understand why. Some of them, there were no markers that showed up in the recording and in others, there was just a marker at the end. In reviewing the log files (verbose=10), it seems that at least several of the attempted recordings identified commercials. I'm OK with the idea that it may not have found commercials for certain channels, but I'm trying to figure out why it didn't properly mark the commercials for the recordings it found commercials in. In the recording that worked, it seemed that the values in the .edl file were reasonable and were time values since start of recording (is that correct?). The edl file of a program that seems to not have worked, but that the log said worked correctly, looks like this:

87204.25 87429.58 0
88056.60 88267.08 0
88784.03 89009.52 0
89257.43 89467.91 0
89853.83 90079.32 0
90211.86 90256.57 0


and the log file said this about the commercials:

---------------------
Final Commercial List
---------------------
1 - start: 5828 end: 12581 [ 2: 11] length: 0:03:45.32
2 - start: 31373 end: 37681 [ 13: 24] length: 0:03:30.47
3 - start: 53174 end: 59932 [ 26: 35] length: 0:03:45.49
4 - start: 67362 end: 73670 [ 37: 46] length: 0:03:30.47
5 - start: 85236 end: 91994 [ 48: 59] length: 0:03:45.49
6 - start: 95966 end: 97306 [ 61: 62] length: 0:00:44.71

I'd appreciate any ideas on where to start looking to understand if I have something wrong in an .ini file or what my problem might be.

Thanks

[erik]

Set

output_framearray=1
in comskip.ini
run comskip on the recording.
Then drag and drop the generated .csv file onto ComskipGUI.exe
a debug window will appear and you can use it to understand how the detection went.

read the documentation on how to navigate and interpret

[webster242]

So, I followed your instructions. It looks like the everything went fine with detection (or it's not the problem anyway). It seems that the problem is a very large initial PTS value. All the times in the generated .edl file are biased by the initial (large) PTS value. There are no issues with audio or video PTS jumps in the log file. It seems that the method of compensating for this (after reading all the forums I can) is to set max_repair_size to something non-zero. I've run it set to 200, but it doesn't fix the problem. The initial PTS value is 94096.405056. Is max_repair_size actually the value I want to be testing? Does the max_repair_size need to be greater than the initial PTS value?

[chico]

I recently upgraded to eyetv HD and upgraded comskip to the donator version (running osx 10.7.5). I've tried marking at least half a dozen files, and only one has produced commercials marked correctly.

For years EyeTV and comskip have worked together (almost) flawlessly for me. But recently strange things have been occurring, including .edl files that look like they've detected actual commercials (number and length apparently correct) but which are incorrectly placed in the video. I suspected the marker-setting function in the markCommercials.py script, so I re-wrote it in Applescript, but no luck. I haven't had time to dig really deep into the problem yet because I travel a lot and my media center mac stays at home, but what I've tested so far hasn't revealed any real clues. I've tried comskipping the .mpg file under Windows (instead of Wine on the mac) with the same .edl being generated (no surprise there).

If you make any progress on this, would you please post here? I really miss reliable EyeTV/comskip operation.

ps. Another strange EyeTV+comskip occurrence I've encountered lately is comskip trying to skip EyeTV .mpg files (in the EyeTV package) without making any headway and running forever, until manually quitting ( viewtopic.php?f=5&t=1486 ). I'm suspecting something odd in the way EyeTV creates the .mpg wrapper, but no firm evidence yet. Perhaps the EyeTV program is somehow responsible for the incorrectly set markers too?

[erik]

Can one of you manually create an .edl file and test if eye-tv is still correctly skipping?

One possible reason is the "very large initial PTS" causing a PTS overflow that libavcodec fails to process.

Could one of you test if ffplay (part of ffmpeg) is able to playback the recording?

[Phillie14586]

Are you trying to run comskip during the recording? EyeTV changes the name of the file when the recording is completed. Erik has stated that from version 81_055 and later comskip closes and reopens the file periodically. Previous versions kept the file open. The problem with this is comskip looses the connection when the name gets changes. Initially I would get strange placement of the commercial markings. Now it just stops and I get no .edl file. If you run comskip from your RecordingDone script instead of the RecordingStarted it works fine. I have tried a number of versions of comskip earlier than 81_055 and still have the problem. Someone else has 81_049 working during the recording. I set that up to run with last nights recordings but have not had a chance to see how it worked. I am running on a mid 2010 Mac Pro with OS10.6.8.

[chico]

Are you trying to run comskip during the recording?

In my case, I only run comskip after EyeTV is finished recording, via "RecordingDone". I also have a delay inserted in the script so that comskip (via MarkCommercials) doesn't start immediately, just in case EyeTV is still futzing with the file.

Re-running the comskip process additional times (by manually invoking "RecordingDone") generates the same .edl files, with the same misplaced commercials, so I don't think EyeTV is messing with the file during the comskip process, at least in my case.

[erik]

Can you set
verbose=10
and send me a zip file with the generated log file and a text file stating where you would like to have seen the commercials marked?

[chico]

Hello Erik - Some interesting things learned from testing:

I took a recent episode of "Breaking Bad" recorded by EyeTV (version 3.61/7120) from U.S. cable channel AMC.

I first determined and marked the correct commercial breaks by hand (accurate +/- 1 second using the EyeTV built-in editor):

Code: Select all

actual .mpg breaks determined visually
in seconds		    in h:mm:ss.s:
0000.00	0035.01	0:00:00	0:00:35   (previously on…)	
0408.01	0528.01	0:06:48	0:08:48   (commercial break)		
1185.01	1406.01	0:19:45	0:23:26   (commercial break)		
1763.01	2070.10	0:29:23	0:34:30   (commercial break)		
2809.01	3105.01	0:46:49	0:51:45   (commercial break)		
3834.10	3834.10	1:03:54	1:03:54   (credits to end)

Then I comskipped the EyeTV-generated .mpg file:

Code: Select all

.mpg breaks
in seconds		    in h:mm:ss.s:	difference from actual:
0131.23	0166.40	0:02:11	0:02:46	131.23	131.39
0539.21	0659.99	0:08:59	0:11:00	131.20	131.98
1316.65	1537.34	0:21:57	0:25:37	131.64	131.33
1894.86	2201.00	0:31:35	0:36:41	131.85	130.90
2940.80	3236.63	0:49:01	0:53:57	131.79	131.62
3834.16	3965.53	1:03:54	1:06:06	000.06	131.43

Comskip correctly caught the breaks but incorrectly marked the breaks in the .edl file. Notice that there is a constant offset of 131.x seconds, except for the final break.

Then I exported from EyeTV to an H.264 .mp4 file (without compression):

Code: Select all

.mp4 breaks
in seconds		    in h:mm:ss.s:	difference from actual:
0000.00	0035.30	0:00:00	0:00:35	000.00	00.29
0372.61	0493.46	0:06:13	0:08:13	-35.40	-34.55
1029.96	1135.23	0:17:10	0:18:55	-155.05	-270.78
1386.25	1511.28	0:23:06	0:25:11	-376.76	-558.82
2123.96	2266.96	0:35:24	0:37:47	-685.05	-838.05
2719.32	2849.68	0:45:19	0:47:30	-1114.78	-984.42

Notice that comskip correctly caught the first break, but incorrectly placed all subsequent breaks. With this file, the breaks were not offset by a constant amount, but rather a negatively increasing offset.

I also calculated the offset differences between the original .mpg and exported .mp4 files, just to see if anything popped out:

Code: Select all

(.mpg - .mp4 offsets)			
in seconds		    in h:mm:ss.s:
0131.23	0131.10	0:02:11	0:02:11
0166.60	0166.53	0:02:47	0:02:47
0286.69	0402.11	0:04:47	0:06:42
0508.61	0689.72	0:08:29	0:11:30
0816.84	0969.67	0:13:37	0:16:10
1114.84	1115.85	0:18:35	0:18:36

My own workflow is to first record using EyeTV, then run comskip on the recorded .mpg file (in the EyeTV package), then transcode the EyeTV .mpg file to a .m4v file using Handbrake (command line version) for viewing in XBMC. I use the comskip-generated .edl file to both mark the recording in EyeTV and to allow XBMC to automatically skip commercials. I don't usually view the program in EyeTV, I just glance at the markers visually to see if they are correct.

I tried to run comskip on the Handbrake-generated .mkv file but it failed, giving the following error message:

Frame Rate set to 90000.000 f/s
Format changed to [1024 : 576]
Could not allocate memory for frame array

Wow, 90,000 fps! I see that someone had previously reported a similar issue with EyeTV and Handbrake on these forums, but there has been no apparent resolution.


I'm admittedly no expert on video file formats, and comskip itself seems like wonderful black magic to me, but I'd guess there's something in the EyeTV output files that is providing incorrect or misleading information to comskip. Since comskip and EyeTV used to work very well together, my next step will be to run these same tests with an older version of EyeTV and see if the results are different.

I can send the zipped log files from the .mpg, .mp4 and .m4v comskips, if you can enable your ftp server, Erik, or let me know an alternate way to send them.

Hopefully this infomation is helpful. Please let me know if there is further testing that might be helpful.

Thanks….chico

[chico]

One possible reason is the "very large initial PTS" causing a PTS overflow that libavcodec fails to process.

Could one of you test if ffplay (part of ffmpeg) is able to playback the recording?

Just tested the Breaking Bad .mpg file with ffplay/windows, plays ok. However, this file may or may not have the issue that webster242 was reporting.

[erik]

@Chico

I openend the FTP port so you can upload to my ftp server

[Phillie14586]

Chico,
You know XBMC will play the EyeTV .mpg files just fine. You don't need to convert them to .m4v unless you are wanting to save some space. I have a RecordingDone script that populates another folder with an alias to the original .mpg with the correct file name and structure so XBMC will scrape the data correctly.

[chico]

Chico,
You know XBMC will play the EyeTV .mpg files just fine. You don't need to convert them to .m4v unless you are wanting to save some space. I have a RecordingDone script that populates another folder with an alias to the original .mpg with the correct file name and structure so XBMC will scrape the data correctly.

Thanks, Phillie. Yes, I know the Handbrake compression is not necessary for viewing. But I travel constantly and carry a portable drive with all my video. Plus I record things I know my family and friends might choose to watch at some unknown time in the future. I don't necessarily watch immediately after recording. So the compression becomes necessary for reasonable disk space usage. My system has been working well until the recent past when something broke the comskip part of the workflow. I'm trying everything I can to figure out where the glitch is...

BTW, I think I recognize your username as one of the many sources of code for my own EyeTV RecordingDone script. My script is highly modded but I wouldn't be surprised if some of what I've got came from one of your postings way back. If not you, someone else with a similar username (Philliexxxx....) If so, thanks for the inspiration

[Phillie14586]

Glad to hear the script has been helpful. I can't take credit for all of it, that script had made its rounds well before I got hold of it.

I got another HD instead of trying to compress everything. I record too many TV shows then watch at my leisure and delete after watching. I have tried using handbrake on some files with mixed results. All my recordings have had the commercials removed with EyeTV. Sometimes with handbrake the audio is off and other times it is just fine. It got me thinking why am I using handbrake when EyeTV can export compressed files. Next time I need compressed files I might try EyeTV. Have you tried using EyeTV instead of handbrake?

I think something has unexpectedly changed with the environment that comskip needs to run in on a Mac. It was working just fine until this summer with version 80.031 when I started getting errors with Wine. Any later version works with Wine but comskip will fail once eyeTV changes the name after recording. Running comskip after the recording is finished seems to be the only way I can get it to work.

[chico]

Glad to hear the script has been helpful. I can't take credit for all of it, that script had made its rounds well before I got hold of it.

I have tried using handbrake on some files with mixed results. All my recordings have had the commercials removed with EyeTV. Sometimes with handbrake the audio is off and other times it is just fine. It got me thinking why am I using handbrake when EyeTV can export compressed files. Next time I need compressed files I might try EyeTV. Have you tried using EyeTV instead of handbrake?

I find EyeTV's compression is ok but takes longer, creates larger files, and creates lesser quality video than handbrake. It's fine for most purposes, but I incorporated HandbrakeCLI with custom presets into my RecordingDone script, and use different profiles for different program sources (for example, less quality for The Daily Show, greater quality for movies). Very happy with it.

I think something has unexpectedly changed with the environment that comskip needs to run in on a Mac. It was working just fine until this summer with version 80.031 when I started getting errors with Wine. Any later version works with Wine but comskip will fail once eyeTV changes the name after recording. Running comskip after the recording is finished seems to be the only way I can get it to work.

Yes, I didn't notice it immediately because of the way I watch. I started noticing some files weren't skipping correctly, others caused comskip to hang, but it didn't dawn on me initially that it might be a system issue. When I started looking closer, I realized that something (comskip, EyeTV, my python, xterm, wine, or Mac OS installation?) was causing different behavior than in the past. Hence my postings here and elsewhere. Others have noticed the same issues, but no one seems yet to have definitively identified the issue. Current thought is something is different with EyeTV output. Thorough testing is hard when I'm traveling so much. Anyway, bit-by-bit I think we're getting closer to figuring out what broke.