Handbrake stalls on 2nd pass (VideoToolbox H265)

[eupatoire]

Problem Description

As mentioned in the title: in several circumstances and settings, when H265 VideoToolbox is selected as encoder (either 8 or 10 bits) with a 2-pass encoding, the 1st pass is carried out normally but the 2nd pass generates the same error message, over and over.
On the other hand, with identical settings, x265 does the job (though using much more computer resources).
Same problem with 1.72 or 1.7.3.
Various sources are TV programs (H264 in .mt2s container).

Activity Log, Crash Log or any other details

[19:41:22] macgui: Handbrake Version: 1.7.3 (2024021100)
[19:41:22] Compile-time hardening features are enabled
[19:41:22] hb_init: starting libhb thread
[19:41:41] macgui: Automation: permission granted for com.apple.systemevents.
[19:41:41] Compile-time hardening features are enabled
[19:41:41] hb_init: starting libhb thread
[19:41:41] macgui: fr.handbrake.HandBrakeXPCService scanning specifically for title: 1
[19:41:41] CPU: Unknown
[19:41:41]  - logical processor count: 8
[19:41:41] hb_scan: path=/Volumes/.../NHK World-Japan - A Cat's-eye View Of Japan (Kanagawa Miura Sea Breeze) - 16-07-2023 04h53 08m (1226).m2ts, title_index=1
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:333: failed opening UDF image /Volumes/.../NHK World-Japan - A Cat's-eye View Of Japan (Kanagawa Miura Sea Breeze) - 16-07-2023 04h53 08m (1226).m2ts
disc.c:437: error opening file BDMV/index.bdmv
disc.c:437: error opening file BDMV/BACKUP/index.bdmv
[19:41:41] bd: not a bd - trying as a stream/file instead
libdvdread: DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav: vm: vm: failed to read VIDEO_TS.IFO
[19:41:41] dvd: not a dvd - trying as a stream/file instead
[19:41:41] file is MPEG Transport Stream with 192 byte packets offset 4 bytes
[19:41:41] Found the following PIDS
[19:41:41]     Video PIDS :
[19:41:41]       0x6e0 type H.264 (0x1b) (PCR)
[19:41:41]     Audio PIDS :
[19:41:41]       0x6b8 type AAC (0xf)
[19:41:41]     Subtitle PIDS :
[19:41:41]     Other PIDS :
[19:41:41] stream id 0x6b8 (type 0xf substream 0x0) audio 0x6b8
[19:41:43] scan: decoding previews for title 1
[19:41:43] file is MPEG Transport Stream with 192 byte packets offset 4 bytes
[aac @ 0x120e0a900] Sample rate index in program config element does not match the sample rate index configured by the container.
[aac @ 0x120e0a900] channel element 1.13 is not allocated
[19:41:43] scan: audio 0x6b8: aac, rate=48000Hz, bitrate=128000 Unknown (HE-AAC, 2.0 ch, 128 kbps)
[19:41:43] stream: 28 good frames, 0 errors (0%)
[19:41:43] using bitstream PAR 4:3
[19:41:43] scan: 10 previews, 1440x1080, 25.000 fps, autocrop = 0/0/0/0, aspect 16:9, PAR 4:3, color profile: 1-1-1, chroma location: left
[19:41:43] libhb: scan thread found 1 valid title(s)
[19:41:43] macgui: fr.handbrake.HandBrakeXPCService scan done
[19:41:43] macgui: fr.handbrake.HandBrakeXPCService started encoding NHK World-Japan - A Cat's-eye View Of Japan (Kanagawa Miura Sea Breeze) - 16-07-2023 04h53 08m (1226).mp4
[19:41:43] macgui: fr.handbrake.HandBrakeXPCService with preset Perso H265 VTB
[19:41:43] Starting work at: Wed Feb 14 19:41:43 2024

[19:41:43] 1 job(s) to process
[19:41:43] Starting Task: Analysis Pass
[19:41:43] work: skipping crop/scale filter
[19:41:43] job configuration:
[19:41:43]  * source
[19:41:43]    + /Volumes/.../NHK World-Japan - A Cat's-eye View Of Japan (Kanagawa Miura Sea Breeze) - 16-07-2023 04h53 08m (1226).m2ts
[19:41:43]    + title 1, start 00:02:05.00 stop 00:07:07.00
[19:41:43]  * destination
[19:41:43]    + /Users/JPB/Movies/TV/NHK/A cat's eye view of Japan/NHK World-Japan - A Cat's-eye View Of Japan (Kanagawa Miura Sea Breeze) - 16-07-2023 04h53 08m (1226).mp4
[19:41:43]    + container: MPEG-4 (libavformat)
[19:41:43]  * video track
[19:41:43]    + decoder: h264 8-bit (yuv420p)
[19:41:43]      + bitrate 200 kbps
[19:41:43]    + filters
[19:41:43]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[19:41:43]      + Decomb (mode=63)
[19:41:43]      + Framerate Shaper (mode=1:rate=27000000/1080000)
[19:41:43]        + frame rate: 25.000 fps -> constant 25.000 fps
[19:41:43]      + Sharpen (lapsharp) (y-strength=0.05:y-kernel=isolap:cb-strength=0.05:cb-kernel=isolap)
[19:41:43]      + Format (format=p010le)
[19:41:43]    + Output geometry
[19:41:43]      + storage dimensions: 1440 x 1080
[19:41:43]      + pixel aspect ratio: 4 : 3
[19:41:43]      + display dimensions: 1920 x 1080
[19:41:43]    + encoder: H.265 10-bit (VideoToolbox)
[19:41:43]      + preset:  quality
[19:41:43]      + profile: auto
[19:41:43]      + level:   auto
[19:41:43]      + bitrate: 2500 kbps, pass: 1
[19:41:43]      + color profile: 1-1-1
[19:41:43]      + chroma location: left
[19:41:43]  * audio track 1
[19:41:43]    + decoder: Unknown (HE-AAC, 2.0 ch, 128 kbps) (track 1, id 0x6b8)
[19:41:43]      + bitrate: 128 kbps, samplerate: 48000 Hz
[19:41:43]    + AAC Passthru
[19:41:43] file is MPEG Transport Stream with 192 byte packets offset 4 bytes
[19:41:43] sync: expecting 7575 video frames
[19:41:43] encvt_Init: encoding with output bitrate 2500 Kbps
[19:41:43] encvt_Init: com.apple.videotoolbox.videoencoder.ave.hevc
[aac @ 0x120e0aa50] Sample rate index in program config element does not match the sample rate index configured by the container.
[19:41:46] sync: first pts audio 0x6b8 is 0
[19:41:46] sync: first pts video is 1649
[20:30:09] sync: reached video pts 27181886, exiting early
[20:30:09] sync: reached audio 0x6b8 pts 27183360, exiting early
[20:30:10] hb_ts_stream_decode - eof
[20:30:10] hb_ts_stream_decode - eof
[20:30:10] reader: done. 299 scr changes
[20:30:13] work: average encoding speed for job is 2.529052 fps
[20:30:13] comb detect: heavy 5754 | light 412 | uncombed 1176 | total 7342
[20:30:13] decomb: deinterlaced 11508 | blended 824 | unfiltered 1176 | total 13508
[20:30:13] vfr: 7551 frames output, 5963 dropped and 6 duped for CFR/PFR
[20:30:13] vfr: lost time: 0 (0 frames)
[20:30:13] vfr: gained time: 0 (0 frames) (0 not accounted for)
[20:30:13] stream: 11961 good frames, 0 errors (0%)
[20:30:13] aac-decoder done: 11213 frames, 0 decoder errors
[20:30:13] h264-decoder done: 11960 frames, 0 decoder errors
[20:30:13] sync: got 7342 frames, 7575 expected
[20:30:13] sync: framerate min 12.383 fps, max 25.000 fps, avg 24.311 fps
[20:30:13] Starting Task: Encoding Pass
[20:30:13] work: skipping crop/scale filter
[20:30:13] job configuration:
[20:30:13]  * source
[20:30:13]    + /Volumes/.../NHK World-Japan - A Cat's-eye View Of Japan (Kanagawa Miura Sea Breeze) - 16-07-2023 04h53 08m (1226).m2ts
[20:30:13]    + title 1, start 00:02:05.00 stop 00:07:07.00
[20:30:13]  * destination
[20:30:13]    + /Users/JPB/Movies/TV/NHK/A cat's eye view of Japan/NHK World-Japan - A Cat's-eye View Of Japan (Kanagawa Miura Sea Breeze) - 16-07-2023 04h53 08m (1226).mp4
[20:30:13]    + container: MPEG-4 (libavformat)
[20:30:13]  * video track
[20:30:13]    + decoder: h264 8-bit (yuv420p)
[20:30:13]      + bitrate 200 kbps
[20:30:13]    + filters
[20:30:13]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[20:30:13]      + Decomb (mode=63)
[20:30:13]      + Framerate Shaper (mode=1:rate=27000000/1080000)
[20:30:13]        + frame rate: 25.000 fps -> constant 25.000 fps
[20:30:13]      + Sharpen (lapsharp) (y-strength=0.05:y-kernel=isolap:cb-strength=0.05:cb-kernel=isolap)
[20:30:13]      + Format (format=p010le)
[20:30:13]    + Output geometry
[20:30:13]      + storage dimensions: 1440 x 1080
[20:30:13]      + pixel aspect ratio: 4 : 3
[20:30:13]      + display dimensions: 1920 x 1080
[20:30:13]    + encoder: H.265 10-bit (VideoToolbox)
[20:30:13]      + preset:  quality
[20:30:13]      + profile: auto
[20:30:13]      + level:   auto
[20:30:13]      + bitrate: 2500 kbps, pass: 2
[20:30:13]      + color profile: 1-1-1
[20:30:13]      + chroma location: left
[20:30:13]  * audio track 1
[20:30:13]    + decoder: Unknown (HE-AAC, 2.0 ch, 128 kbps) (track 1, id 0x6b8)
[20:30:13]      + bitrate: 128 kbps, samplerate: 48000 Hz
[20:30:13]    + AAC Passthru
[20:30:13] file is MPEG Transport Stream with 192 byte packets offset 4 bytes
[20:30:13] sync: expecting 7342 video frames
[20:30:13] encvt_Init: encoding with output bitrate 2500 Kbps
[aac @ 0x120e1e110] Sample rate index in program config element does not match the sample rate index configured by the container.
[aac @ 0x120e1e110] channel element 1.13 is not allocated
[20:30:16] sync: first pts audio 0x6b8 is 0
[20:30:16] sync: first pts video is 1601
[20:30:18] VTCompressionSession: hb_vt_compression_output_callback called error
[20:31:18] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:31:18] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:31:18] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:32:16] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:32:16] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:32:16] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:33:14] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:33:14] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:33:14] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:34:13] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:34:13] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:34:13] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:35:11] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:35:11] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:35:11] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:36:09] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:36:09] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:36:09] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:37:07] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:37:07] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
[20:37:07] VTCompressionSession: hb_vt_compression_output_callback sample buffer is NULL
Etc.

What Operating System are you running?

macOS 14.3.1

What version of HandBrake are you running?

1.7.3 (2024021100)

Where did you download HandBrake from?

handbrake.fr

[galad87]

Does it always happen on the same files? Can you share one of such files to test?

[eupatoire]

I was not able to reproduce the problem with extracts of the faulty file (extracts obtained with tsMuxer).
Since it's too big to be uploaded here, I've put it on WeTransfer.
Size is ~ 280 MB, and the link is valid for 7 days from now (2024-02-15):
https://we.tl/t-P84b9hFVjy

[galad87]

Thanks, I'll test it. Does it work without EEDI2 Bob (there is no reason to use an hardware encoder together with EEID2, but this is another story).

[eupatoire]

On the same file, without filters HandBrake also halts at the beginning of the 2nd pass (same error message).

As for deinterlacing, since I'm new to —and naïvely experimenting on— VTB, I did not realize that VTB deinterlaces the input files. I've just found this link:
https://developer.apple.com/documentation/videotoolbox/kvtdecompressionproperty_deinterlacemode_verticalfilter
(On the other hand, x264-x265 really needs deinterlacing; that's why I used it…)

[galad87]

It doesn't deinterlace automatically. I was asking because I couldn't reproduce the issue M1 Mac mini without EEDI2, and the EEDI2 test is still running.

[eupatoire]

I agree: since my previous message I've tested VTB H265 without EEDI2 and, alas, the result is flawed indeed.
In case it helps: I use a M3 iMac (2023).

[twalker314]

I agree: since my previous message I've tested VTB H265 without EEDI2 and, alas, the result is flawed indeed.

Flawed as in it has interlacing artifacts, but it doesn't hang, correct?

[eupatoire]

Sorry for having been too vague: yes, even without any filter, Handbrake still hangs on these intractable source files. The 2nd pass stops at its beginning and the same message is issued again and again (about VTCompressionSession).
I've found a way, however, to circumvent the problem: using tsMuxer to (split and) remux beforehand —to .m2ts or .ts— the segment I need to encode. With the resulting file as source, Handbrake does the job normally (both passes).
This doesn't explain why Handbrake choked on the first place; is it too picky about some slight defects of the source file (which don't affect VLC)?

[twalker314]

What if you simply remux the whole thing with tsMuxer, does it still hang?

Also I’ve only just realized you’re using point-to-point encoding, perhaps galad missed it too, which might explain why he couldn’t reproduce it?

[twalker314]

I'm curious to see if I can reproduce it on an M2 Mini, can you please share the sample again?

[eupatoire]

I've not tried yet to remux the entire file.
Here it is again:
https://www.swisstransfer.com/d/b2ed7ac3-f340-4f91-abd8-1ea3d9e4d321
I've never used that service before; they say the file is available for 30 days (~ March 25).

[twalker314]

If they delete it too fast, I have a rough idea of where their offices are, I can complain in person 😂

[galad87]

Oh right, I didn't notice it was a point-to-point encode.