Problem with .edl file generation

[BrettB]

I've been doing some testing with the latest version of NPVR (the replacement for GBPVR). And, I've found a few recordings where when NPVR loads the timeline for the recording with the comskip information from the .edl file, all of the commercial breaks are in the first half of the timeline. It appears that the comskip break points have just been compressed from the full length into the first 1/2 proportionally.

I looked at the .txt file version, and it shows:

Code: Select all

FILE PROCESSING COMPLETE 116567 FRAMES AT 5994
-------------------
39724	 44554
50945	 55481
60658	 67173
77053	 82475
87366	 91708
106783	 116568

Which would seem like the breaks are spread throughout the recording.

But, the .edl file version shows:

Code: Select all

662.73	743.31	0
849.93 925.61	0
1011.98	1120.67	0
1285.50	1375.96	0
1457.56	1530.00	0
1781.50	1944.74	0

Which would seem to only be about 30 minutes of the 60 minute recording. (And would seem to correspond to the way that NPVR is showing the breaks in its timeline.)

I will email you a .zip file with the log and the timing.csv file.

Any suggestions about how to get the break points at the correct positions in NPVR would be greatly appreciated.

[erik]

On certain broadcasts the type of video is changing.
From 29.97 Hz to 59.94Hz
or from 25Hz to 50Hz.
The EDL uses the fps at the end of the recording do the generation. However it is possible the commercial segment at the end is broadcasted in a different fps.
Could you check if this could be happening.
Will have a look at the files you mailed later this week.

[BrettB]

On certain broadcasts the type of video is changing.
From 29.97 Hz to 59.94Hz
or from 25Hz to 50Hz.
The EDL uses the fps at the end of the recording do the generation. However it is possible the commercial segment at the end is broadcasted in a different fps.
Could you check if this could be happening.

I think that this (or some variation of it) is happening. In the log file, it shows:

Code: Select all

Printing AR cblock list
-----------------------------------------
Block: 0	Start:      1	End:    924	AR_R: 1.32	Length: 0:00:15.39, [1920x1088] minX=250, minY=  1, maxX=1678, maxY=1088
Block: 1	Start:    925	End:   1306	AR_R: 1.77	Length: 0:00:06.35, [1920x1088] minX=234, minY=130, maxX=1686, maxY=950
Block: 2	Start:   1307	End:   1604	AR_R: 1.60	Length: 0:00:04.95, [1920x1088] minX=298, minY=130, maxX=1598, maxY=950
Block: 3	Start:   1605	End:   2038	AR_R: 1.77	Length: 0:00:07.22, [1920x1088] minX=234, minY=130, maxX=1686, maxY=950
Block: 4	Start:   2039	End:  39748	AR_R: 1.32	Length: 0:10:29.11, [1920x1088] minX=242, minY=  1, maxX=1678, maxY=1088
Block: 5	Start:  39749	End:  40649	AR_R: 1.77	Length: 0:00:15.01, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 6	Start:  40650	End:  41548	AR_R: 1.32	Length: 0:00:14.98, [1920x1088] minX=242, minY=  1, maxX=1686, maxY=1088
Block: 7	Start:  41549	End:  44561	AR_R: 1.77	Length: 0:00:50.25, [1920x1088] minX=250, minY=130, maxX=1678, maxY=958
Block: 8	Start:  44562	End:  50971	AR_R: 1.32	Length: 0:01:46.92, [1920x1088] minX=242, minY=  1, maxX=1670, maxY=1088
Block: 9	Start:  50972	End:  51839	AR_R: 1.77	Length: 0:00:14.46, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 10	Start:  51840	End:  51872	AR_R: 1.12	Length: 0:00:00.53, [1920x1088] minX=354, minY=  1, maxX=1566, maxY=1088
Block: 11	Start:  51873	End:  52772	AR_R: 1.32	Length: 0:00:14.99, [1920x1088] minX=234, minY=  1, maxX=1686, maxY=1088
Block: 12	Start:  52773	End:  55496	AR_R: 1.77	Length: 0:00:45.42, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 13	Start:  55497	End:  60684	AR_R: 1.32	Length: 0:01:26.53, [1920x1088] minX=242, minY=  1, maxX=1678, maxY=1088
Block: 14	Start:  60685	End:  61583	AR_R: 1.77	Length: 0:00:14.98, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 15	Start:  61584	End:  62033	AR_R: 2.39	Length: 0:00:07.49, [1920x1088] minX=  1, minY=146, maxX=1920, maxY=950
Block: 16	Start:  62034	End:  63882	AR_R: 1.77	Length: 0:00:30.83, [1920x1088] minX=250, minY=130, maxX=1678, maxY=958
Block: 17	Start:  63883	End:  64283	AR_R: 1.32	Length: 0:00:06.67, [1920x1088] minX=234, minY=  1, maxX=1686, maxY=1088
Block: 18	Start:  64284	End:  65381	AR_R: 1.77	Length: 0:00:18.30, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 19	Start:  65382	End:  68073	AR_R: 1.32	Length: 0:00:44.89, [1920x1088] minX=234, minY=  1, maxX=1678, maxY=1088
Block: 20	Start:  68074	End:  68222	AR_R: 1.77	Length: 0:00:02.46, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 21	Start:  68223	End:  77085	AR_R: 1.32	Length: 0:02:27.84, [1920x1088] minX=234, minY=  1, maxX=1678, maxY=1088
Block: 22	Start:  77086	End:  78034	AR_R: 2.39	Length: 0:00:15.81, [1920x1088] minX=  1, minY=138, maxX=1920, maxY=966
Block: 23	Start:  78035	End:  82485	AR_R: 1.77	Length: 0:01:14.24, [1920x1088] minX=234, minY=138, maxX=1686, maxY=958
Block: 24	Start:  82486	End:  88289	AR_R: 1.32	Length: 0:01:36.81, [1920x1088] minX=242, minY=  1, maxX=1670, maxY=1088
Block: 25	Start:  88290	End:  89239	AR_R: 1.77	Length: 0:00:15.83, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 26	Start:  89240	End:  91035	AR_R: 1.32	Length: 0:00:29.94, [1920x1088] minX=234, minY=  1, maxX=1686, maxY=1088
Block: 27	Start:  91036	End:  91915	AR_R: 1.77	Length: 0:00:14.66, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 28	Start:  91916	End: 106656	AR_R: 1.32	Length: 0:04:05.91, [1920x1088] minX=242, minY=  1, maxX=1670, maxY=1088
Block: 29	Start: 106657	End: 106784	AR_R: 1.77	Length: 0:00:02.11, [1920x1088] minX=  1, minY=  1, maxX=1920, maxY=1088
Block: 30	Start: 106785	End: 109397	AR_R: 1.32	Length: 0:00:43.57, [1920x1088] minX=234, minY=  1, maxX=1686, maxY=1088
Block: 31	Start: 109398	End: 116568	AR_R: 1.77	Length: 0:01:59.61, [1280x 720] minX=  1, minY=  1, maxX=1280, maxY=720

Which I think means that the resolution is changing from 1920x1088 to 1280x720 right at the end. So, a change in frame rate wouldn't be surprising.

It also shows:

Code: Select all

	---------------------
	Final Commercial List
	---------------------
1 - start:  39724	end:  44554	[     6:    10]	length: 0:01:20.58
2 - start:  50945	end:  55481	[    12:    16]	length: 0:01:15.67
3 - start:  60658	end:  67173	[    18:    26]	length: 0:01:48.69
4 - start:  77053	end:  82475	[    28:    34]	length: 0:01:30.45
5 - start:  87366	end:  91708	[    36:    42]	length: 0:01:12.43
6 - start: 106783	end: 116568	[    46:    54]	length: 0:02:43.24

Logo fraction:              0.6302      
Maximum volume found:         9257
Average volume:                758
Sound threshold:               500
Silence threshold:             100
Minimum volume found:            0
Average frames with silence:    74
Black threshold:                25
Minimum brightness found:       15
Minimum bright pixels found:     0
Minimum dim level found:         0
Average brightness:            673
Uniformity level:              500
Average non uniformity:        286
Maximum gap between logo's:    132
Suggested logo_threshold:   0.8250
Suggested shrink_logo:	         5
Max commercial size found:      51
Dominant aspect ratio:      1.3200
Score threshold:            1.0500
Framerate:                  59.94
Total commercial length:    0:07:07.84
Block list after weighing
----------------------------------------------------
  #     sbf  bs  be     fs     fe    sc      len   scr cmb   ar                   cut bri  logo   vol  sil corr stdev        cc
  0:++    1   0   1      1    821   0.00   13.680  0.03   1 1.32  6     {   LC       b   600- 0.00  791+ 88+  0.000    60           
  1:++    1   1   0    822    928   0.00    1.768  0.21   3 1.32  6     {   LC     a     429- 0.00  695- 73-  0.000    42           
  2:++    1   1   0    929   1288   0.00    5.989  0.06   2 1.77  6     { A LC     a     519- 0.00  707- 78+  0.000    51           
  3:++    1   1   0   1289   1599   0.00    5.172  0.00   2 1.60  6     { A LCN    a     147- 0.00  688- 75+  0.000    14           
  4:++   16   1  15   1600   2022   0.00    7.040  0.88   2 1.77  6     { A LC     a b  7803+ 0.00  892+ 81+  0.000   780           
  5:++   41  16  25   2023  39723   0.00  628.962  0.04   0 1.32           EL      a b   315- 0.98  753- 77+  0.000    31           
  6:--   25  25   0  39724  40648  12.00   15.415  0.44   0 1.77          A L  S   a b  4077+ 0.00  920+ 93+  0.000   407           
  7:--    1   1   0  40649  41549   6.00   15.015  0.04   0 1.32            L  S   a     344- 0.00  774+ 85+  0.000    34           
  8:--    1   1   0  41550  43356  12.00   30.130  0.06   0 1.77          A L  S     b   437- 0.00  656- 70-  0.000    43           
  9:--    1   1   0  43357  44248  18.00   14.865  0.33   1 1.77          A LC S     b  3832+ 0.00  885+ 90+  0.000   383           
 10:--    7   1   6  44249  44554   9.00    5.088  0.60   1 1.77          A LCN    a b  4349+ 0.00  824+ 91+  0.000   434           
 11:++   33   6  27  44555  50944   0.01  106.590  0.03   0 1.32            L      a b   316- 0.79  748- 71-  0.000    31           
 12:--   28  28   0  50945  51822   9.00   14.631  0.27   2 1.77          A LC     a    1570+ 0.00  589- 73-  0.000   157           
 13:--    1   1   0  51823  52773   4.50   15.849  0.08   2 1.32            LC     a     949+ 0.00  853+ 90+  0.000    94           
 14:--    2   1   1  52774  54873  18.38   35.018  0.08   4 1.77          A LCN      b   674+ 0.00  657- 70-  0.000    67           
 15:--    3   2   1  54874  55171  41.34    4.955  0.40   6 1.77          A LCN      b  1966+ 0.00  681- 78+  0.000   196           
 16:--   11   1  10  55172  55481  24.00    5.155  0.13   4 1.77          A LCN    a b  1112+ 0.00  849+ 89+  0.000   111           
 17:++   37  10  27  55482  60657   0.01   86.336  0.01   0 1.32            L      a b   187- 0.86  636- 63-  0.000    18           
 18:--   27  27   0  60658  61583  12.00   15.432  0.17   0 1.77          A L  S   a    1665+ 0.00  884+ 86+  0.000   166           
 19:--    1   1   0  61584  62034   4.00    7.508  0.28   0 2.39          A L      a    1611+ 0.00 1090+ 97+  0.000   161           
 20:--    6   1   5  62035  62938  16.00   15.065  0.03   1 1.77          A LC S    ub   286- 0.00  696- 69-  0.000    28           
 21:--   21   6  15  62939  63758  15.19   13.664  0.13   5 1.77          A LC      ub  1087+ 0.00  780+ 92+  0.000   108           
 22:--   15  15   0  63759  63832  15.19    1.218  0.55   5 1.77          A LC       b   972+ 0.00  584- 64-  0.000    97           
 23:--    1   1   0  63833  63899   9.00    1.101  0.90   4 1.77          A LC     a    4149+ 0.00  985+ 88+  0.000   414           
 24:--    1   1   0  63900  64284   4.50    6.406  0.24   4 1.32            LC     a    1233+ 0.00  881+ 96+  0.000   123           
 25:--   25   1  24  64285  65357   9.00   17.885  0.30   4 1.77          A LC     a b  2952+ 0.00  868+ 87+  0.000   295           
 26:--   24  24   0  65358  67173   6.00   30.280  0.06   0 1.32            L  S    u    654- 0.00  851+ 93+  0.000    65           
 27:++    1   1  33  67174  77052   0.00  164.765  0.03   0 1.32           EL      a     299- 0.86  721- 72-  0.000    29           
 28:--   33  33   0  77053  78045   5.22   16.550  0.35   2 2.39          A LC     a    1793+ 0.00 1082+ 95+  0.000   179           
 29:--    1   1   0  78046  81139   5.22   51.602  0.17   2 1.77          A LC       b  1069+ 0.00 1010+ 92+  0.000   106           
 30:--    2   1   1  81140  81440   7.84    5.005  1.14   2 1.77          A LCN      b  7079+ 0.00  993+ 90+  0.000   707           
 31:--    2   1   1  81441  81531   5.22    1.502  1.11   2 1.77          A LC       b  6450+ 0.00  765+ 85+  0.000   645           
 32:--    1   1   0  81532  82006   9.29    7.908  0.97   4 1.77          A LC       b  6616+ 0.00 1003+ 92+  0.000   661           
 33:--    4   1   3  82007  82290   9.29    4.721  0.92   4 1.77          A LC       b  6059+ 0.00 1032+ 98+  0.000   605           
 34:--   12   3   9  82291  82475   9.29    3.070  0.88   4 1.77          A LC     a b  3805+ 0.00 1078+ 93+  0.000   380           
 35:++   33   9  24  82476  87365   0.02   81.565  0.02   2 1.32            LC       b   226- 0.92  562- 51-  0.000    22           
 36:--   26  25   1  87366  87884  12.50    8.642  0.38   6 1.32            LC       b  2400+ 0.00  720- 72-  0.000   240           
 37:--   11   2   9  87885  88133   5.56    4.137  0.37   4 1.32            LC       b  2690+ 0.00 1052+ 97+  0.000   269           
 38:--   10  10   0  88134  88307   5.56    2.886  0.32   4 1.32            LC     a    2626+ 0.00  942+ 90+  0.000   262           
 39:--   16   1  15  88308  89175  16.00   14.464  0.10   6 1.77          A LC       b   837+ 0.00  518- 52-  0.000    83           
 40:--   15  15   0  89176  89253  16.00    1.285  0.76   6 1.77          A LC     a    3551+ 0.00  805+ 71-  0.000   355           
 41:--    3   1   2  89254  91032   8.00   29.663  0.12   6 1.32            LC     a b   761+ 0.00  762+ 82+  0.000    76           
 42:--    2   2   0  91033  91708   9.00   11.261  0.56   4 1.77          A LC       b  3644+ 0.00  936+ 93+  0.000   364           
 43:++   10   1   9  91709  91899   0.50    3.170  0.36   4 1.77      2 { A LC     a b  1439+ 0.00  800+ 87+  0.000   143           
 44:++   34   9  25  91900 106606   0.00  245.345  0.02   0 1.32           EL      a b   237- 0.98  649- 66-  0.000    23           
 45:++   26  25   1 106607 106782   0.50    2.920  0.29   0 1.77      2 { A L      a b  3215+ 0.00  899+ 82+  0.000   321           
 46:--    4   2   2 106783 108588   8.00   30.113  0.04   1 1.32            LC S     b   458- 0.00  837+ 88+  0.000    45           
 47:--    2   2   0 108589 109381   6.00   13.213  0.03   3 1.32            LC     a     649- 0.00  978+ 97+  0.000    64           
 48:--   37   1  36 109382 109569  12.00    3.120  0.65   3 1.77          A LC       b  1468+ 0.00  931+ 76+  0.000   146           
 49:--   41  36   5 109570 110813  12.00   20.737  0.36   0 1.77          A L  S     b  1484+ 0.00  886+ 72-  0.000   148           
 50:--   47   5  42 110814 111415  12.00   10.027  0.13   0 1.77          A L  S    u    358- 0.00 1283+ 76+  0.000    35           
 51:--   45  43   2 111416 113945   9.00   42.192  0.12   2 1.77          A LC      u    577- 0.00  787+ 38-  0.000    57           
 52:--   19   3  16 113946 115032   9.00   18.118  0.50   2 1.77          A LC      u   2031+ 0.00 2007+ 80+  0.000   203           
 53:--   47  16  31 115033 116449   9.00   23.624  0.03   2 1.77          A LC      u    275- 0.00  548- 39-  0.000    27           
 54:--   31  31   0 116450 116568   9.00    1.969  0.06   2 1.77          A LC       b   243- 0.00  231-  4-  0.000    24           

116568 Frames Processed

Does that tell you if the frame rate changed?

[BrettB]

On certain broadcasts the type of video is changing.
From 29.97 Hz to 59.94Hz
or from 25Hz to 50Hz.
The EDL uses the fps at the end of the recording do the generation. However it is possible the commercial segment at the end is broadcasted in a different fps.
Could you check if this could be happening.

I think that this (or some variation of it) is happening. In the log file, it shows:
...
Does that tell you if the frame rate changed?

Bump.

[erik]

Indeed, a problem,
Same could happen when during a recording the commercial is in another fps.
Maybe I have to rewrite comskip to use timecodes i.s.o. frame numbers.
Will be some serious effort....
Not to be done soon.

[BrettB]

Indeed, a problem,
Same could happen when during a recording the commercial is in another fps.
Maybe I have to rewrite comskip to use timecodes i.s.o. frame numbers.
Will be some serious effort....
Not to be done soon.

As a short-term work-around, would it be possible for you to keep track of either the average or median fps throughout the entire recording, and then use that for the converstion to edl? And maybe output a warning in the log file if the fps isn't constant throughout the entire recording? Those wouldn't be perfect, but would probably be better than what it does right now.

[erik]

will consider both suggestions

[BrettB]

will consider both suggestions

Hi, Erik!

I was wondering if you had a chance to think about a work around for the changing fps rate yet.

With the latest version of NPVR, I'm getting quite good results with Comskip and US broadcast shows. However, I still occassionally have this problem where the conversion to .edl is off and all of the commercials get offset into the 1st have of the show.

[erik]

yes, I though about it and had a look at the source.
Next to that there is the problem of the jumping PTS.
Some broadcasts have a non continues PTS and video players simply us the PTS to jump instead of building a correct timeline (as Comskip does)
The only good solution is do a completely rewrite of all time related use of frame numbers to time_codes based on the PTS
This will take some time, and time is not abundantly available.....
So be patient