Comskip Support Forum

Comskip is a free commercial detector, browse the forum for more information
It is currently Tue Aug 21, 2018 3:29 am

All times are UTC [ DST ]




Post new topic Reply to topic  [ 12 posts ] 
Author Message
 Post subject: Enigma2/Dreambox/Vu+
PostPosted: Tue Sep 18, 2012 6:45 pm 
Offline

Joined: Tue Sep 18, 2012 6:05 pm
Posts: 3
I had read a bit that there is some possiblility to use Comskip with Enigma2. Enigma2 is used on Dreambox and some other Linux boxes like Vu+. But I have not been able to figure out if Comskip already can generate the ".cuts" file that Enigma2 is using. I wrote a small Perl-script that make a .cuts file from the .mdl file and that works well enough, but it would be nice to have builtin output directly from Comskip. I have searched this forum a bit for enigma2 and dreambox but with no result. Maybe one of the already existing formats is the same? The format Enigma2 is using looks like this;

Binary file in the same location as the .ts file with the extension .ts.cuts, note that the .ts ending is not removed and .cuts is just appended to the filename. File contains sequence of;

bigendian 64 bit PTS followed by bigendian 32 bit Type

For the files I have experimented with, PTS is scaled integer with standard 90000 Hz
Type field is:
type 0 = in
type 1 = out
type 2 = mark
type 3 = last play position

Would it be possible to add that output format if it does not already exist?

There seem to exist some third party programs that can convert .mdl to .cuts, but I have not tested any of them as I dont realy need their other functions.

(I have earlier experimented a bit with the standard version of Comskip with good results and have now decided to become a donor and try it on HD recordings which I am testing just now :-)


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Wed Sep 19, 2012 8:02 am 
Offline
Site Admin

Joined: Sun Aug 21, 2005 3:49 pm
Posts: 3195
Is PTS always starting at zero?


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Wed Sep 19, 2012 4:44 pm 
Offline

Joined: Tue Sep 18, 2012 6:05 pm
Posts: 3
Yes, the PTS values in .cuts start at 0 and are continuous for each recording. After some searching, I found documentation on the file formats used:

https://github.com/libo/Enigma2/blob/master/doc/FILEFORMAT

(That is from an old tree, but looking at ex. OpenPli sources is has not been updated lately. Currently there is one more type of file, .ts.sc, for each recording, that is not documented in the link. That file seem to be created to speed up fast rewind/forward.)

One thing I forgot to mention about the .cuts; When doing a fresh recording the .cuts will contain automatically created marks for each program start. So if I add some extra buffer time before and after one program the .cuts will have one mark at start of the program and one at the end. It might be good to merge with those existing marks. Or maybe not :-) The quality of those marks are not very good for commercial channels, at least not in Sweden. So, not important!

Also, it may be nice to have a choice whether to use in and out marks at each commercial, or normal marks. If you add in and out those parts are not shown at all when I play a recording, so that is absolutely preferred. But for debugging it might be nice to see what is happening. But there is a cuts editor in the receiver so its easy to check anyway. So, also not important!

Thank you for your nice work!


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Tue Jul 30, 2013 9:31 pm 
Offline

Joined: Tue Sep 18, 2012 6:05 pm
Posts: 3
Hi Erik!

Have you had any chance to implement .cuts output for Enigma2?

Thanks!


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Fri Aug 09, 2013 4:39 pm 
Offline
Site Admin

Joined: Sun Aug 21, 2005 3:49 pm
Posts: 3195
Did have a short holiday, just back


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Thu Aug 04, 2016 10:45 pm 
Offline

Joined: Thu Aug 04, 2016 10:41 pm
Posts: 1
Just to ask:
Is that feature (actually) implemented ? And if "yes"... how do i activate it?

Thanks for all responses.

PS: If not, is there a possible work-around available?


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Fri Aug 05, 2016 7:21 am 
Offline
Site Admin

Joined: Sun Aug 21, 2005 3:49 pm
Posts: 3195
Its not implemented.
YOu can take one of the pts output formats and convert is using a shell script


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Sun Apr 15, 2018 2:54 pm 
Offline

Joined: Sun Apr 15, 2018 2:49 pm
Posts: 3
Hi there,

I am currently trying to convert the cut markers from the comskip .txt file to the enigma2 .cuts format (mentioned in the first post). Is someone already having a shell script for that purpose? I am struggling with the binary file format of the .cuts files.

Thanks and kind regards

Felix


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Sun Apr 15, 2018 5:57 pm 
Offline
Site Admin

Joined: Sun Aug 21, 2005 3:49 pm
Posts: 3195
If enigma is using time codes you better use the edl output
Its in seconds since start


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Sun Apr 15, 2018 11:58 pm 
Offline

Joined: Sun Apr 15, 2018 2:49 pm
Posts: 3
Enigma2 does not use the time codes in seconds, so no need to use .edl. With massive reverse engineering I managed to deconstruct the .cuts file format and build a PHP script to convert the comskip plist (because XML is easy to read in PHP) to .cuts.

I will try to recode this as a python script, as PHP has a poor CLI SAPI.

If you are interested, I can provide the format algorithm to you, if you want to implement the .cuts format into comskip.

I'll post links in a few days.

Thanks for your awesome work!


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Mon Apr 16, 2018 6:36 am 
Offline
Site Admin

Joined: Sun Aug 21, 2005 3:49 pm
Posts: 3195
Please do post the format description


Top
 Profile  
Reply with quote  
 Post subject: Re: Enigma2/Dreambox/Vu+
PostPosted: Mon Apr 16, 2018 7:46 pm 
Offline

Joined: Sun Apr 15, 2018 2:49 pm
Posts: 3
Hi Erik,

I published my small tool on github, I hope it’s okay to link it here: https://github.com/felixrupp/comskipCutsConverter

To the format:

.cuts is a binary Textfile, constructed as follows (and asCatMan already stated above):

One Big Endian 64bit unsigned long long (the PTS), followed by one Big Endian 32bit unsigned int (the type).

Reading the .cuts file is in C-code is demonstrated in the Enigma core (https://github.com/openatv/enigma2/blob/fff9a170b68b8ba5d44b3a046b51b63f2ad9b8c6/lib/service/servicemp3.cpp#L3470):
Code:
FILE *f = fopen(filename.c_str(), "rb"); // Read bytewise

if (f)
{
   while (1)
   {
      unsigned long long where;
      unsigned int what;

      if (!fread(&where, sizeof(where), 1, f))
         break;
      if (!fread(&what, sizeof(what), 1, f))
         break;

      where = be64toh(where);
      what = ntohl(what);

      if (what < 4)
         m_cue_entries.insert(cueEntry(where, what));
   }
   fclose(f);
   eDebug("[eServiceMP3] cuts file has %zd entries", m_cue_entries.size());
}
else
   eDebug("[eServiceMP3] cutfile not found!");

The two functions used here, be64toh() and ntohl() are used to decode the big endian byte order to the machine’s byte order.

be64toh(): Converts a big endian 64bit unsigned long long to machine byte order 64bit.
ntohl(): Converts a big endian 32bit unsigned int to machine byte order 32bit.

In PHP or Python these conversions can be done by using the unpack() (to decode) and pack() (to encode) functions with according options.

To write the .cuts file, the following can be done in C-code:

Code:
FILE *f = fopen(filename.c_str(), "wb"); // Write bytewise

if (f)
{
   signed long long where = 0;
   guint what = 0;

   for (std::multiset<cueEntry>::iterator i(m_cue_entries.begin()); i != m_cue_entries.end(); ++i)
   {
      if (where == i->where && what == i->what)
         /* ignore double entries */
         continue;
      else
      {
         where = htobe64(i->where);
         what = htonl(i->what);
         fwrite(&where, sizeof(where), 1, f);
         fwrite(&what, sizeof(what), 1, f);
         /* temorary save for comparing */
         where = i->where;
         what = i->what;
      }
   }
   fclose(f);
   eDebug("[eServiceMP3] cuts file has been write");
}

htobe64(): Converts unsigned long long from 64bit machine byte order to 64bit big endian
htonl(): Converts unsigned int 31bit machine byte order to 32bit big endian

This is a code snippet from my tool where the both values are being converted (packed) in PHP:
Code:
$where = pack("J", intval($cutsEntry[0])); // Format value 'J' means: convert to unsigned long long (always 64 bit, big endian byte order)
$what = pack("N", intval($cutsEntry[1])); // Format value 'N' means: convert to unsigned long (always 32 bit, big endian byte order)

fwrite($fileHandle, $where, (64 / 8)); // 64bit / 8 = 8 Byte, because fwrite needs the parameter to be represented as bytes not bits
fwrite($fileHandle, $what, (32 / 8)); // 32bit / 8 = 4 Byte, because fwrite needs the parameter to be represented as bytes not bits

As CatMan stated in his post, these types of markers are known by Enigma2’s .cuts format (https://github.com/libo/Enigma2/blob/master/doc/FILEFORMAT):
type 0 = in (only for editing the files)
type 1 = out (only for editing the files)
type 2 = mark (most likely only for viewing the files and jumping)
type 3 = last play position (only set by the player-frontend)

Type 2 is basically the most interesting marker for me, because, that’s the one you will need to skip over commercials when using the playback plugin (in my case EnhancedMovieCenter -> EMC).
Type 0 and 1 are only interesting if the user has a plugin for actually editing the recorded files on his Enigma2 set-top-box, like CutlistEditor and/or moviecut, so I decided to only write out Type 2 markers for jumping over the commercials while viewing them for now.
In the end it would be nice to have all three types combined. So we need a type 0 (in) at frame 0 and when a commercial starts, we set a type 1 (out) and a type 2 (marker). Commercial ends, we set a type 0 (in) and a type 2 (marker), and so on.

As stated before, I used the .plist output by comskip, because of the easy handling of XML in PHP, as the base for converting the values into the .cuts format. My tool also takes the default .cuts files from Enigma2 and merges the markers (because Enigma2 sets up markers at the beginning of the show and the end).

I hope that this brings some light into how .cuts format is build up. Would love to hear/see a native .cuts output in comskip!

If not, I would be happy to get feedback on my small tool, so I can cover all necessary functions you all need.


Regards,

Felix


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group