Handbrake/H.264: "Could not allocate memory for frame array"

Here you can ask your questions on how to use Comskip for the detection of commercials. Also questions on how to remove commercials are welcome
Post Reply
Prof Yaffle
Posts: 7
Joined: Sat Jan 26, 2013 2:14 pm

Handbrake/H.264: "Could not allocate memory for frame array"

Post by Prof Yaffle » Sat Jan 26, 2013 2:39 pm

Bought the donator version a couple of days ago so I could process H.264 files, and I've failed with every one since!

I run my files through Handbrake beforehand to fix .TS errors, get rid of unwanted audio streams, shrink the file, etc. I do this with MPEG-2 and H.264 broadcasts. But every one I try to drop into comskip fails with the same error:

Code: Select all

Frame Rate set to 90000.000 f/s
Format changed to [704 : 574]
Could not allocate memory for frame array
Obviously, the "format changed" alters depending on the resolution, but I'm not surprised comskip is choking if it's trying to malloc enough for a 90k fps file unless it's only a few seconds of CGA resolution :shock: ! VLC tells me that the file is 25 fps; mkvinfo confirms the same. Sho shurely shome mishtake shomewhere?

Sample file info:

Code: Select all

(MKVInfo) + EBML head
(MKVInfo) |+ EBML version: 1
(MKVInfo) |+ EBML read version: 1
(MKVInfo) |+ EBML maximum ID length: 4
(MKVInfo) |+ EBML maximum size length: 8
(MKVInfo) |+ Doc type: matroska
(MKVInfo) |+ Doc type version: 2
(MKVInfo) |+ Doc type read version: 2
(MKVInfo) + Segment, size 387374560
(MKVInfo) |+ Seek head
(MKVInfo) | + Seek entry
(MKVInfo) |  + Seek ID: 0x11 0x4d 0x9b 0x74 (KaxSeekHead)
(MKVInfo) |  + Seek position: 387367910
(MKVInfo) | + Seek entry
(MKVInfo) |  + Seek ID: 0x15 0x49 0xa9 0x66 (KaxInfo)
(MKVInfo) |  + Seek position: 4358
(MKVInfo) | + Seek entry
(MKVInfo) |  + Seek ID: 0x16 0x54 0xae 0x6b (KaxTracks)
(MKVInfo) |  + Seek position: 4429
(MKVInfo) | + Seek entry
(MKVInfo) |  + Seek ID: 0x1c 0x53 0xbb 0x6b (KaxCues)
(MKVInfo) |  + Seek position: 387357922
(MKVInfo) |+ EbmlVoid (size: 187)
(MKVInfo) |+ EbmlVoid (size: 4096)
(MKVInfo) |+ Segment information
(MKVInfo) | + Segment UID: 0x4c 0x75 0x80 0xc3 0xc6 0x75 0x93 0x46 0x93 0xff 0x00 0xfc 0xde 0xa8 0x42 0xfe
(MKVInfo) | + Muxing application: libmkv 0.6.5
(MKVInfo) | + Writing application: HandBrake 0.9.8
(MKVInfo) | + Timecode scale: 1000000
(MKVInfo) | + Duration: 1607.632s (00:26:47.632)
(MKVInfo) |+ Segment tracks
(MKVInfo) | + A track
(MKVInfo) |  + Track number: 1 (track ID for mkvmerge & mkvextract: 0)
(MKVInfo) |  + Track UID: 1812890191
(MKVInfo) |  + Track type: video
(MKVInfo) |  + Lacing flag: 0
(MKVInfo) |  + Codec ID: V_MPEG4/ISO/AVC
(MKVInfo) |  + CodecPrivate, length 43 (h.264 profile: Main @L3.0)
(MKVInfo) |  + Default duration: 40.000ms (25.000 frames/fields per second for a video track)
(MKVInfo) |  + Default flag: 1
(MKVInfo) |  + MinCache: 1
(MKVInfo) |  + Video track
(MKVInfo) |   + Pixel width: 704
(MKVInfo) |   + Pixel height: 574
(MKVInfo) |   + Display width: 1024
(MKVInfo) |   + Display height: 574
(MKVInfo) | + A track
(MKVInfo) |  + Track number: 2 (track ID for mkvmerge & mkvextract: 1)
(MKVInfo) |  + Track UID: 571209515
(MKVInfo) |  + Track type: audio
(MKVInfo) |  + Lacing flag: 0
(MKVInfo) |  + Codec ID: A_AC3
(MKVInfo) |  + Language: eng
(MKVInfo) |  + Default flag: 1
(MKVInfo) |  + Audio track
(MKVInfo) |   + Sampling frequency: 48000
(MKVInfo) |   + Channels: 2
(MKVInfo) |+ Cluster
comskip.ini is the default. I tried switching output_framearray on to see if there was anything in the resultant csv, but no file is ever generated, the process borks long before that.

I have 4Gb of RAM on a Win7 64 machine with about 1.5Gb free after startup (don't you love Windows' memory use... :-) ) - so plenty of free physical RAM. Automatic paging, currently set to 6Gb on a disc with 200Gb+ free - so it's not a real memory limitation beyond the amount comskip *thinks* it needs.

Interestingly, it's something about how comskip reads the Handbrake file, since I can process the original recording - it just doesn't fit my workflow, and I'd prefer to comskip things as a last step so I'm working on the real file-to-be-watched.

Thoughts appreciated!

Cheers....

erik
Site Admin
Posts: 3297
Joined: Sun Aug 21, 2005 3:49 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by erik » Sat Jan 26, 2013 5:01 pm

It must be something special in the handbrake output format you selected.
Can you test:
- If the generated file can be played with ffplay.exe (google for the pre-build wwindows exe)
- If comskip can process the file when handbrake instead generates plain program stream or transport stream mpeg2 video/audio (DVD output) (NTSC mpeg 2 video and MPEG2 audio, either VGA or QVGA)

Prof Yaffle
Posts: 7
Joined: Sat Jan 26, 2013 2:14 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by Prof Yaffle » Sat Jan 26, 2013 6:45 pm

Thanks for the quick response.

1. Yes - the version of ffplay I have plays it fine.

Code: Select all

ffplay version N-48513-gcaee85b Copyright (c) 2003-2013 the FFmpeg developers
  built on Jan  6 2013 16:16:53 with gcc 4.7.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libg
sm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --e
nable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --e
nable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-amrwben
c --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-
libxvid --enable-zlib
  libavutil      52. 13.100 / 52. 13.100
  libavcodec     54. 86.100 / 54. 86.100
  libavformat    54. 59.106 / 54. 59.106
  libavdevice    54.  3.102 / 54.  3.102
  libavfilter     3. 32.100 /  3. 32.100
  libswscale      2.  1.103 /  2.  1.103
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, matroska,webm, from 'C:\Users\Ian\Desktop\Finished\Power Rangers - 19x
21.mkv':
  Metadata:
    title           : Power Rangers Super Samurai
    creation_time   : 2013-01-25 18:35:41
  Duration: 00:26:47.70, start: 0.000000, bitrate: 2195 kb/s
    Stream #0:0(áeng): Video: h264 (Main), yuv420p, 704x574 [SAR 16:11 DAR 512:28
7], 25 fps, 25 tbr, 1k tbn, 180k tbc (default)
    Stream #0:1(eng): Audio: ac3, 48000 Hz, stereo, fltp, 224 kb/s (default)
Frame changed from size:0x0 to size:704x574   0KB sq=    0B f=0/0
  32.52 A-V:  0.020 fd= 301 aq=   19KB vq=   45KB sq=    0B f=0/0
You can see that it's getting the right frame rate, although that "Frame changed from..." might be relevant - except comskip gets the right frame size as you can see in the above error. And I think that shows on everything I play, so it's probably just the video stream initialising and thus a red herring.

As an aside, I can also play the files in XBMC, SMplayer, DivX player and even Windows Media Player. Those applications that give file information all identify the stream as 25fps, assuming that's where things are going wrong.

2. Not entirely sure of your second question, since that goes beyond my knowledge of Handbrake. Some samples, though, to see if this is what you mean:

a. Regular/Normal profile, no changes from HB defaults (i.e. x.264 encoder) - fails with the error
b. Regular/Normal profile, but using ffmpeg MPEG-4 - succeeds (but it's no longer AVC1, so the quality is awful)
c. Regular/Normal profile, but using ffmpeg MPEG-2 - succeeds (ditto MPEG-4)
d. Devices/Universal profile, back to x.264 - fails

I normally use default Normal, but with minor tweaks (decomb set to default instead of off; mkv instead of mp4; audio streams set to auto passthru - nothing that should really damage the video stream). That's where the error originally showed up, therefore.

From the above, it seems that anything HB outputs in H.264 causes comskip to choke, even if other applications can understand the streams okay.

Prof Yaffle
Posts: 7
Joined: Sat Jan 26, 2013 2:14 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by Prof Yaffle » Sat Jan 26, 2013 7:01 pm

A QVGA x.264 encode chokes as well, although that'd still need about (90,000*320*200*24 bits=) 257Mb per uncompressed second of video...

Conversely, a gig of RAM for a half-hour programme would leave about 6 bytes per frame at 90k fps... monochrome microdot, anyone?

(E&OE and mathematical finger failures)

erik
Site Admin
Posts: 3297
Joined: Sun Aug 21, 2005 3:49 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by erik » Sat Jan 26, 2013 9:02 pm

I tried handbrake
input 1920x1080 h.264
Output mkv, x264 video 25 constant fps aac audio passthrough, no problem.

This is what comskip tells
Input #0, matroska,webm, from '...file.mkv':
Duration: 00:00:44.96, start: 0.000000, bitrate: 815 kb/s
Stream #0:0(eng): Video: h264 (Main), yuv420p, 320x184, SAR 1:1 DAR 40:23, 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
Stream #0:1: Audio: ac3, 48000 Hz, stereo, fltp, 640 kb/s (default)

Prof Yaffle
Posts: 7
Joined: Sat Jan 26, 2013 2:14 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by Prof Yaffle » Sat Jan 26, 2013 9:27 pm

Interesting... any idea how we can get more debug information out of comskip, perhaps, to see what it's seeing? I've just gone back through over a year of re-encodes - normal profile, high profile, performed on maybe three different PCs over that time - and anything since early 2012 seems to fail. I wonder if I moved to a different version of Handbrake then - and whether the problem is with how they write the frame rate, or how you read it. It's only comskip that seems to see this issue, though, since *everything* else reads these files.

Or is there any way to over-ride the frame rate detection, and tell comskip that it's 25fps?

It's not the source, since I can replicate the issue on both DVD rips and OTA recordings - the only constant is the version of comskip I'm using, which I only installed two days ago (81_052_donators).

Input #0, matroska,webm, from 'C:\Users\Ian\Desktop\Finished\Power Rangers - 19x21.mkv':
Metadata:
title : Power Rangers Super Samurai
creation_time : 2013-01-25 18:35:41
Duration: 00:26:47.69, start: 0.000000, bitrate: 2195 kb/s
Stream #0:0(eng): Video: h264 (Main), yuv420p, 704x574 [SAR 16:11 DAR 512:287], 25 fps, 25 tbr, 1k tbn, 180k tbc (default)
Stream #0:1(eng): Audio: ac3, 48000 Hz, stereo, s16, 224 kb/s (default)
Frame Rate set to 90000.000 f/s
Format changed to [704 : 574]
Could not allocate memory for frame array

C:\Users\Ian\Downloads\comskip81_052_donators>

Prof Yaffle
Posts: 7
Joined: Sat Jan 26, 2013 2:14 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by Prof Yaffle » Sat Jan 26, 2013 9:35 pm

Hmmm....

http://forum.videohelp.com/threads/3388 ... nt-convert

Let me try to force the frame rate and see if that works. There's something about how HB writes it combined with how you read it that's failing - while the media players seem to get around this (presumably because they're calculating the frame rate differently), the fault may well be with HB...

Let me do some more poking.

Prof Yaffle
Posts: 7
Joined: Sat Jan 26, 2013 2:14 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by Prof Yaffle » Sat Jan 26, 2013 9:49 pm

FWIW, it seems that 90000 is the magic number defined by the MPEG standard as a timebase. Handbrake uses this as "ticks", so it works in units of 90ms, which is suspicious. The only comskip source I can find (the EyeTV version and the recent Linux port) also use this as the internal timebase.

Other applications have problems with this, although not all - as I said, all the players seem to be okay, but many of the editors seem to choke. It also may be related to using variable frame rate instead of constant frame rate - I'll test it, but maybe over-riding VFR (the HB default) will fix the issue - and it should make little difference to normal compression, I think VFR only really helps with video streams that have a lot of pretty static information in them, like anime (or "cartoons", as they were called when I was a boy :wink: ).

You wouldn't be presuming (maybe even requiring) CFR, would you?

erik
Site Admin
Posts: 3297
Joined: Sun Aug 21, 2005 3:49 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by erik » Sat Jan 26, 2013 10:33 pm

Can you try CFR?

Prof Yaffle
Posts: 7
Joined: Sat Jan 26, 2013 2:14 pm

Re: Handbrake/H.264: "Could not allocate memory for frame ar

Post by Prof Yaffle » Sat Jan 26, 2013 11:25 pm

Yes, it just takes me a while to finish the previous HD encode :-)

CFR works, so comskip is failing on VFR files. I think that 90k is the *peak* frame rate, perhaps coming in as a default from Handbrake, perhaps just set to say "here's a really high number, just to make you aware that anything could come at you".

I can't comment as to why VLC et al read the correct rate - indeed, why mkvinfo does as well, since that's clearly just asking the file and not playing a sample to work it out itself (for example) - and even doing that would fail, because VFR would likely encode the opening credits of most films as a low frame rate since they don't change as much as the main scenes later on. So there must be other ways to work out the framerate of a file.

Anyway, that's the problem, and that's the solution: change Handbrake to use CFR. I leave it to you to wonder whether your code needs any thought as well.

Thanks, Erik.

Post Reply