Mux adding frames to start of video

[marketing3]

Describe the issue:

• Mux appears to be adding extra frames to the beginning of videos, which is problematic for looping

What’s the URL of the record or model in question?

• The asset in the media library is here: Loading...
• the mp4Url provided by the API gives a Mux url of this: https://stream.mux.com/6V48g3boltSf5uQRB8HnelvtPglzZzYu/high.mp4

Do you have any sample code you can provide?

• This isn’t code related, per se, but if you look at the asset in the library, and the asset at the Mux URL, it’s obvious that one or more frames have been added to the beginning (I don’t have a video editor handy to get it exactly, but it’s obvious).

What do I have to do either in the upload or in the query to not have this happen?

[roger]

Hey @marketing3,

Welcome to the forum!

Could you please explain what you mean by extra frames…? What exactly are you seeing that shouldn’t be there? At a glance, the two videos look identical to me in several ways:

• Visually, comparing them side-by-side
• In duration, if you download both files and look at their metadata
• Using a frame extractor like https://frame-extractor.com/, I could not see any visual difference in the first few hundred milliseconds

What are the visual symptoms you’re noticing that makes you think there are extra frames? Do they just look slightly different? If so, could it just be recompression artifacts you’re seeing? MP4 (or rather, H.264) is a lossy codec to start with, and Mux transcodes it into different qualities & bitrates to better accommodate users on different devices & connections. That shouldn’t modify the frame rate per se, but individual frames (of the same timecode) can look slightly different because they’re being lossily recompressed from an already-lossy initial upload.

Or if by added frames you mean that there’s some delay before the video starts playing, that might be because the MP4 is cached on your computer and the other version is still buffering…? The high.mp4 isn’t the original version, by the way, but a recompressed version (to a slightly bigger filesize).

Lastly, how are you serving/watching the video? Normally we don’t suggest serving either of the MP4s directly (either from datocms-assets.com or stream.mux.com) but to use a HLS adaptive-bitrate stream instead (which would serve a .m3u8 playlist to be handled on the frontend by a special video player like one of ours). More about that here: How to stream videos efficiently — Stream videos efficiently — DatoCMS

---

Does any of that help? If I’m misunderstanding, could you please clarify and maybe take some screenshots/recordings to better demonstrate what you’re noticing?

[marketing3]

Hi!

I first noticed the problem when just watching the loop in my dev environment. There’s a flash of white every time the frame loops. After much digging into Stackoverflow and other forums, it seems several people experienced similar problems when they had extra frames in their video. That seemed unlikely, because the video worked fine previously (I’ll come back to this), but I figured I’d look. Sure enough, the video as downloaded from the media library has the starting frame clearly visible at time zero, whereas here’s a screenshot of the direct Mux URL, where you can see that I’ve let it run for a few fractions of a second beyond 0, and it’s still black (in another fraction of a second, the first frame image shows up):

Those few fractions of a second where the first frames are blank from Mux result in a flicker when the video loops.

Coming back to “how it worked previously,” I’m not inherently opposed to moving to .m3u8 — the main thing I’m doing here is moving away from the react-datocms VideoPlayer to the native html5 video component (I’m working in NextJS). The primary reason for doing this is that the required Mux player is too heavy, and the whole process of loading and starting to play the streamed image is way too slow. (To see the current, slow version, you can clear your caches and visit https://www.idap-global.com. Unfortunately, I can’t show you a live version of the native video player, as I’m still mid-development, but trust me, it’s near instantaneous on my dev server, vs. the few second delay you get on the live site).

Thanks in advance for any help/advice!

[roger]

I see what you mean now! Thank you, the screenshot was really helpful.

Hmmmmm… this might be a a codec issue with Safari and QuickTime in particular? Chrome and Firefox on my Mac do not have this issue (I believe they use their own video renderers?). I bet it affects iPhone too, but I don’t have one to test with right this moment…

It might have to do the particular way Mux is re-encoding the file? Let me speak to their support team on this and get back to you as soon as I can!

I’m not a codec expert, but at first glance, it looks like the Mux version is missing a keyframe at 0.00 seconds that the original has:

Original:

> ffprobe -loglevel error -select_streams v:0 -show_entries packet=pts_time,flags -of csv=print_section=0 original.mp4 | awk -F',' '/K/ {print $1}' 

0.000000
3.003000
6.006000
9.009000
12.012000

Mux version:

ffprobe -loglevel error -select_streams v:0 -show_entries packet=pts_time,flags -of csv=print_section=0 high.mp4 | awk -F',' '/K/ {print $1}'

0.085000
4.989900
9.994900

As a workaround, if you add a start time to the URL, it seems to skip the initial fractions of a second and go straight to the first keyframe: https://stream.mux.com/6V48g3boltSf5uQRB8HnelvtPglzZzYu/high.mp4#t=0.085 (does that make a difference for you at all? It seems to work on my computer)

I’ll report this to Mux and see what they say. Thank you for the report! I’ll write back as soon as I can.

---

Separate from the frames issue though, can I recommend (IMHO only) that you see if there’s any tweaking you might be able to do the <VideoPlayer/> component, perhaps by adding preload='auto' to its props to see if that helps at all?

If you just use the raw high.mp4 as your video source, you’re forcing all your visitors to download the full-res video just for an animated hero background, and as much of it as they can, even if they scroll right past it right after a second.

If the preload doesn’t work, you MIGHT be able to use the .m3u8 as the source, but it’s not clear to me whether (as of 2024) that works correctly in modern browsers: streaming - Playing m3u8 Files with HTML Video Tag - Stack Overflow

Otherwise, Mux has a lightweight polyfill: Mux-flavored HTML video tag | Mux

Or Video.js might be quicker to load? https://videojs.com/

If any of those solutions are acceptably fast for you, they should be much better than serving the MP4 directly.

[roger]

Just a little more debugging in Safari to test that keyframe theory out:

https://www.datocms-assets.com/12345/12345-your-originalvideo.mp4#t=0,0.084 shows the tree as expected.

https://stream.mux.com/6V48g3boltSf5uQRB8HnelvtPglzZzYu/high.mp4#t=0,0.083 shows a blank since it stops before the first keyframe

But then https://stream.mux.com/6V48g3boltSf5uQRB8HnelvtPglzZzYu/high.mp4#t=0,0.085 (just 0.002 seconds later) shows the tree again.

Chrome and Firefox must be using the first keyframe automatically, but not Safari / Apple decoders =/ I’ve created a ticket with Mux and will see what they say. Thanks for your patience!

[marketing3]

Thanks! Also, FYI, I’ve confirmed that the problem doesn’t exist for me on Chrome, and it does exist on iOS Safari, so looks like you’re right. Will look forward to the response from Mux. I’ll also look at some other options with VideoPlayer.

[roger]

Hi @marketing3,

We heard back from Mux, and apparently this was a bug with an older version of their encoder. Your video was originally uploaded last year, but if you download and upload it again (either as a new file or replacing the original), it should be fixed.

For example, I uploaded it to a demo project, and the new Mux re-encoding of it doesn’t have that problem anymore in either Safari or QuickTime:

https://stream.mux.com/IPXo00wyFSoOjSAqDNXXCxTQdaD72y4Vt/high.mp4 (please only use that for testing and not in your real website, because it’s just a demo that will be deleted soon; you can do the same thing in your real project)

Please let us know if that helps.

Thank you and sorry for the inconvenience!