Enigma2/Dreambox/Vu+

Here you can ask your questions on how to use Comskip for the detection of commercials. Also questions on how to remove commercials are welcome
Post Reply
CatMan
Posts: 3
Joined: Tue Sep 18, 2012 6:05 pm

Enigma2/Dreambox/Vu+

Post by CatMan »

I had read a bit that there is some possiblility to use Comskip with Enigma2. Enigma2 is used on Dreambox and some other Linux boxes like Vu+. But I have not been able to figure out if Comskip already can generate the ".cuts" file that Enigma2 is using. I wrote a small Perl-script that make a .cuts file from the .mdl file and that works well enough, but it would be nice to have builtin output directly from Comskip. I have searched this forum a bit for enigma2 and dreambox but with no result. Maybe one of the already existing formats is the same? The format Enigma2 is using looks like this;

Binary file in the same location as the .ts file with the extension .ts.cuts, note that the .ts ending is not removed and .cuts is just appended to the filename. File contains sequence of;

bigendian 64 bit PTS followed by bigendian 32 bit Type

For the files I have experimented with, PTS is scaled integer with standard 90000 Hz
Type field is:
type 0 = in
type 1 = out
type 2 = mark
type 3 = last play position

Would it be possible to add that output format if it does not already exist?

There seem to exist some third party programs that can convert .mdl to .cuts, but I have not tested any of them as I dont realy need their other functions.

(I have earlier experimented a bit with the standard version of Comskip with good results and have now decided to become a donor and try it on HD recordings which I am testing just now :-)
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Enigma2/Dreambox/Vu+

Post by erik »

Is PTS always starting at zero?
CatMan
Posts: 3
Joined: Tue Sep 18, 2012 6:05 pm

Re: Enigma2/Dreambox/Vu+

Post by CatMan »

Yes, the PTS values in .cuts start at 0 and are continuous for each recording. After some searching, I found documentation on the file formats used:

https://github.com/libo/Enigma2/blob/ma ... FILEFORMAT

(That is from an old tree, but looking at ex. OpenPli sources is has not been updated lately. Currently there is one more type of file, .ts.sc, for each recording, that is not documented in the link. That file seem to be created to speed up fast rewind/forward.)

One thing I forgot to mention about the .cuts; When doing a fresh recording the .cuts will contain automatically created marks for each program start. So if I add some extra buffer time before and after one program the .cuts will have one mark at start of the program and one at the end. It might be good to merge with those existing marks. Or maybe not :-) The quality of those marks are not very good for commercial channels, at least not in Sweden. So, not important!

Also, it may be nice to have a choice whether to use in and out marks at each commercial, or normal marks. If you add in and out those parts are not shown at all when I play a recording, so that is absolutely preferred. But for debugging it might be nice to see what is happening. But there is a cuts editor in the receiver so its easy to check anyway. So, also not important!

Thank you for your nice work!
CatMan
Posts: 3
Joined: Tue Sep 18, 2012 6:05 pm

Re: Enigma2/Dreambox/Vu+

Post by CatMan »

Hi Erik!

Have you had any chance to implement .cuts output for Enigma2?

Thanks!
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Enigma2/Dreambox/Vu+

Post by erik »

Did have a short holiday, just back
MM1970
Posts: 1
Joined: Thu Aug 04, 2016 10:41 pm

Re: Enigma2/Dreambox/Vu+

Post by MM1970 »

Just to ask:
Is that feature (actually) implemented ? And if "yes"... how do i activate it?

Thanks for all responses.

PS: If not, is there a possible work-around available?
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Enigma2/Dreambox/Vu+

Post by erik »

Its not implemented.
YOu can take one of the pts output formats and convert is using a shell script
MrZapadoodle
Posts: 3
Joined: Sun Apr 15, 2018 2:49 pm

Re: Enigma2/Dreambox/Vu+

Post by MrZapadoodle »

Hi there,

I am currently trying to convert the cut markers from the comskip .txt file to the enigma2 .cuts format (mentioned in the first post). Is someone already having a shell script for that purpose? I am struggling with the binary file format of the .cuts files.

Thanks and kind regards

Felix
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Enigma2/Dreambox/Vu+

Post by erik »

If enigma is using time codes you better use the edl output
Its in seconds since start
MrZapadoodle
Posts: 3
Joined: Sun Apr 15, 2018 2:49 pm

Re: Enigma2/Dreambox/Vu+

Post by MrZapadoodle »

Enigma2 does not use the time codes in seconds, so no need to use .edl. With massive reverse engineering I managed to deconstruct the .cuts file format and build a PHP script to convert the comskip plist (because XML is easy to read in PHP) to .cuts.

I will try to recode this as a python script, as PHP has a poor CLI SAPI.

If you are interested, I can provide the format algorithm to you, if you want to implement the .cuts format into comskip.

I'll post links in a few days.

Thanks for your awesome work!
erik
Site Admin
Posts: 3368
Joined: Sun Aug 21, 2005 3:49 pm

Re: Enigma2/Dreambox/Vu+

Post by erik »

Please do post the format description
MrZapadoodle
Posts: 3
Joined: Sun Apr 15, 2018 2:49 pm

Re: Enigma2/Dreambox/Vu+

Post by MrZapadoodle »

Hi Erik,

I published my small tool on github, I hope it’s okay to link it here: https://github.com/felixrupp/comskipCutsConverter

To the format:

.cuts is a binary Textfile, constructed as follows (and asCatMan already stated above):

One Big Endian 64bit unsigned long long (the PTS), followed by one Big Endian 32bit unsigned int (the type).

Reading the .cuts file is in C-code is demonstrated in the Enigma core (https://github.com/openatv/enigma2/blob ... .cpp#L3470):

Code: Select all

FILE *f = fopen(filename.c_str(), "rb"); // Read bytewise

if (f)
{
	while (1)
	{
		unsigned long long where;
		unsigned int what;

		if (!fread(&where, sizeof(where), 1, f))
			break;
		if (!fread(&what, sizeof(what), 1, f))
			break;

		where = be64toh(where);
		what = ntohl(what);

		if (what < 4)
			m_cue_entries.insert(cueEntry(where, what));
	}
	fclose(f);
	eDebug("[eServiceMP3] cuts file has %zd entries", m_cue_entries.size());
}
else
	eDebug("[eServiceMP3] cutfile not found!");
The two functions used here, be64toh() and ntohl() are used to decode the big endian byte order to the machine’s byte order.

be64toh(): Converts a big endian 64bit unsigned long long to machine byte order 64bit.
ntohl(): Converts a big endian 32bit unsigned int to machine byte order 32bit.

In PHP or Python these conversions can be done by using the unpack() (to decode) and pack() (to encode) functions with according options.

To write the .cuts file, the following can be done in C-code:

Code: Select all

FILE *f = fopen(filename.c_str(), "wb"); // Write bytewise

if (f)
{
	signed long long where = 0;
	guint what = 0;

	for (std::multiset<cueEntry>::iterator i(m_cue_entries.begin()); i != m_cue_entries.end(); ++i)
	{
		if (where == i->where && what == i->what)
			/* ignore double entries */
			continue;
		else
		{
			where = htobe64(i->where);
			what = htonl(i->what);
			fwrite(&where, sizeof(where), 1, f);
			fwrite(&what, sizeof(what), 1, f);
			/* temorary save for comparing */
			where = i->where;
			what = i->what;
		}
	}
	fclose(f);
	eDebug("[eServiceMP3] cuts file has been write");
}
htobe64(): Converts unsigned long long from 64bit machine byte order to 64bit big endian
htonl(): Converts unsigned int 31bit machine byte order to 32bit big endian

This is a code snippet from my tool where the both values are being converted (packed) in PHP:

Code: Select all

$where = pack("J", intval($cutsEntry[0])); // Format value 'J' means: convert to unsigned long long (always 64 bit, big endian byte order)
$what = pack("N", intval($cutsEntry[1])); // Format value 'N' means: convert to unsigned long (always 32 bit, big endian byte order)

fwrite($fileHandle, $where, (64 / 8)); // 64bit / 8 = 8 Byte, because fwrite needs the parameter to be represented as bytes not bits
fwrite($fileHandle, $what, (32 / 8)); // 32bit / 8 = 4 Byte, because fwrite needs the parameter to be represented as bytes not bits
As CatMan stated in his post, these types of markers are known by Enigma2’s .cuts format (https://github.com/libo/Enigma2/blob/ma ... FILEFORMAT):
type 0 = in (only for editing the files)
type 1 = out (only for editing the files)
type 2 = mark (most likely only for viewing the files and jumping)
type 3 = last play position (only set by the player-frontend)

Type 2 is basically the most interesting marker for me, because, that’s the one you will need to skip over commercials when using the playback plugin (in my case EnhancedMovieCenter -> EMC).
Type 0 and 1 are only interesting if the user has a plugin for actually editing the recorded files on his Enigma2 set-top-box, like CutlistEditor and/or moviecut, so I decided to only write out Type 2 markers for jumping over the commercials while viewing them for now.
In the end it would be nice to have all three types combined. So we need a type 0 (in) at frame 0 and when a commercial starts, we set a type 1 (out) and a type 2 (marker). Commercial ends, we set a type 0 (in) and a type 2 (marker), and so on.

As stated before, I used the .plist output by comskip, because of the easy handling of XML in PHP, as the base for converting the values into the .cuts format. My tool also takes the default .cuts files from Enigma2 and merges the markers (because Enigma2 sets up markers at the beginning of the show and the end).

I hope that this brings some light into how .cuts format is build up. Would love to hear/see a native .cuts output in comskip!

If not, I would be happy to get feedback on my small tool, so I can cover all necessary functions you all need.


Regards,

Felix
Post Reply