This is a follow-up to Video, Mobile, and the Open Web. As promised there, OS-based H.264 support for the HTML5 <video> element has already landed in Gecko, and it just released this week in Firefox Beta for Android. Firefox OS (B2G to the early adopters!) also supports H.264 from the HTML5 <video> element.
The challenge remains working through OS decoders on the various desktop OSes. Here’s where we are (thanks to roc, cdouble, and cpearce):
- Bug 794282, to enable GStreamer in official Linux builds.
- Bug 799315, to use Windows Media Foundation on Vista and newer Windows releases. This would provide H.264/AAC/MP3 support.
- Tracking bug 799318 for the above two and the missing Mac OS X bug, plus the Windows XP solution described next.
- The idea for Windows XP is to use Flash. According to roc, “we believe it may be possible to use Flash unmodified. Modern Flash has APIs to let us inject compressed data to the media engine without going through their networking layer, and we can recover rendered video frames.”
So, hard work still to-do ahead of us, but nothing that we can’t overcome (knock on wood).
We are taking the fight for unencumbered formats to the next battlefront, WebRTC, also as promised. More on that front later.
As always, the dev-media list (mailman “subscribe” protocol; also a Google Group) is a fine place to discuss any of this.
/be
that’s an awful solution for XP users, and they are already most likely to be using old/slow computers, where Flash already performs very poorly.
surely it would be preferable, by everyone included, to use native codecs (if installed), or even ship one (ffmpeg) with firefox, or even as a separate download.
and no, that doesn’t leave XP users any more exposed to security vulnerabilities, because using flash is currently the most insecure thing we do on the web..
Why waste resources engineering some complicated one-off solution for an ancient OS like XP, which is going to be irrelevant in just over a year anyway? In April 2014 XP will be dead – and good riddance!
So the non-legacy Windows solution runs fully on the GPU (DXVA2) just like in IE9/10?
@Tom: you may be right. Authors write explicit Flash fallback via embed-inside-video, anyway. But some of our key developers believe that Mozilla should make one last try at a uniform HTML5-only authoring experience.
@Asbjørn: XP will linger for quite a while, unsupported, especially in parts of the world not flush with cash for new PCs and tablets.
@Hera: yes, that’s the goal.
/be
Ffmpeg is no solution since the majority of Firefox users live in countries that accept the legality of software patents, making its use illegal.
Using Flash as a way to piggy-back off of the fees that Adobe has already paid is a great idea. I personally have about a dozen computers, running XP, and I will never pay to upgrade them to a future Windows, and will keep them running as long as possible. Windows 7 is unfamiliar feeling and throws away the decades of experience I have in knowing settings and how to customize things, and Windows 8 is far worse in terms of being even more unfamiliar.
Android phones have enough market share that they could change the way websites are done, the same way that when iPhone didn’t support Flash, almost all websites switched to h.264 to work on iPhone. If Android phones removed Flash and h.264 and supported only WebM, they have enough market share that websites would quickly support it. Firefox alone does not have enough market share to force that change.
OO, another question,
If you will use the OS to provide codecs – does this mean that Firefox will support all Media Foundation codecs installed? Or is this not possible due to some inherent limitations HTML5 video specification?
Brendan, you didn’t answer part of Tom Jones’ question. Can we make XP use platform codecs if they’re installed? I find it hard to imagine that the majority of XP users haven’t had an h264 codec installed by something on their system already (if not pre-installed by the OEM). But I’ve never seen any data about it either. Just a lot of “XP users can’t see h264 quotes”.
@Hera: We’re trying to get MP3 and AAC Audio covered by OS decoders, in addition to H.264 A/V among de-facto standards.
@WesJ: Microsoft never shipped an H.264 decoder in WinXP, I’m told. It may have come to some users via optional software, e.g. Flash. I think roc will comment in more detail.
To answer Tom’s question about FFMPEG, Mozilla is still against paying any royalties for encumbered formats that have become de-facto standards.
/be
Hera: we will restrict the formats to a whitelist — initially MP3 and MP4 with AAC/H.264. We’re already doing this on Android. There are a few reasons to restrict the exposed codecs to the minimum set needed
— security; less attack surface
— testability; fewer combinations to test
— interoperability; limiting the number of codecs used on the Web increases the likelihood that Web content will work everywhere
The last one is the most important.
Wesj: Microsoft has never shipped an H.264 codec in Windows XP AFAIK. Some users will have downloaded DirectShow codec packs but we can’t rely on that (and we shouldn’t encourage users to download those packs because they’re a big malware channel). AFAIK the most widely distributed H.264 codec on XP is in Flash, so that’s what we’ll look into.
Surely we could just give XP users a menu item (firefox button-options-purchase h.264 codec) where users could pay via paypal/credit/debit card say $2 or so for a h.264 licence to enable h.264 or to download it from mozilla’s server if mpeg-la don’t allow you to ship h.264 disabled.
Requiring flash sounds like a bad idea in my opinion. One of the main objectives of html5 is to kill flash, now quite a number of xp firefox users will have to keep it installed.
When do you think h.264 will be ready for firefox users, maybe you could ship firefox 18 with it disabled and allow us to enable it in about:config so that we can all give it a lot of testing before it is enabled for the masses.
What is happening with Daala and VP9? Have you asked google if they will join forces with you and just make a single codec otherwise we will have the same issue that we had with h.264 vs webm, we need a single next gen video codec.
I have a suggestion for a new html5 standard, to allow the browser to detect which apu/gpu you have (it would require you to give permission). That way if you have an apu or gpu that can hardware decode Daala then websites like youtube etc could send you the daala video, if not it would send you h.264. Otherwise next gen video codecs will require users to manually enable these like on youtube.com/html5 and those codecs may never take off as a result.
@Sam: you are looking at it wrong, IMHO. Flash and XP are both dying, we are not prolonging either appreciably by using one for H.264 in FF on the other. Flash is very widely distributed on XP, and it’s probably not going away sooner than when the XP boxes in question will go away.
Daala is a way out still. I’ll let roc and others respond at greater length.
/be
Sam, Firefox users on XP aren’t going to pay for H.264. They’ll just download Chrome for free and use that, if the Web author hasn’t already supplied Flash fallback.
Keeping Flash alive on XP is not really an issue. It’s already there and supported by browsers. There aren’t going to be new Windows XP users installing Flash because Firefox needs it.
The next-gen free codec issue is being worked on. I don’t want to say much about it here, but we are organizing a group at the IETF to standardize it.
maybe you are right about flash.
I like that you are getting a standard video ietf stardard that sounds great 🙂 Would it not be better to postpone deciding which codec to use for WebRTC until the next gen video codec is complete and to just use that codec? Otherwise everyone will be using the high bandwidth webm/h264 for many years to come.
You guys could do with adding WebP support or telling firefox users which functionality you are waiting for until you add WebP support as it is nearly a great format, just problems with black and white poor compression and a few other niggles left.
! This comment system exposes the last person’s e-mail address.
@Roc
Will there be some ability to override the whitelist similarly to your GPU blacklist? (to use 3rd party h264/AVC codecs and/or avoid Flash) :3
Is there a chance of Flash and Firefox bundle? (It seems to work for IE & Chrome)
@Hera: always a chance of something, we’re looking at all possibilities…
/be
test comment.
i have been using window xp , this now is giving me huge problem in watching video,and on other hand it’s now being very slow, getting hard to harder to bear such slowness long, this solution comes to me as a boon, please let me know how i can install this, if possible it come to with browser up date, it will be very easier for me to up load such system with existing system, please help me out in this regard. look forward to your support .
Why fight h.264 in webrtc? The story there is exactly the same as the video tag, except more so. Mobile devices already have built in support for h.264. Furthermore, pre-existing video phone already use h.264. Preventing h.264 is just going to hinder webrtc adoption on mobile, and hinder its integration with pre-existing video phone hardware.
The end result is people ship desktop and mobile apps and the web loses out.
“To answer Tom’s question about FFMPEG, Mozilla is still against paying any royalties for encumbered formats that have become de-facto standards.”
and yet, FirefoxOS devices, that you partner on, will play MP4 out of the box? magic? disingenuous?
anyway, i didn’t say you necessarily need to ship ffmpeg, just link to it and let users install it themselves.
and while there are not a lot of XP users without flash, from my personal experience, one of the top three things i suggest to users to keep themselves safe on the internet is to: 1) use firefox, 2) disable/remove java and 3) disable/remove flash, if they can.
this just makes it harder to do so than the (ffmpeg) alternative..
Hi guys, is there a chance that FLAC will be supported at some stage? I definitely do no expect it to be a priority, but it is the free (and patent unencumbered?) audio codec of choice for many “enthusiasts”.
@Brendan: H.264 audio was not good for low-latency (n-way realtime), wherefore Opus. H.264 video has a better shot, but not a “lock” yet, on WebRTC video. As many people pointed out in comments on my blog post this spring, VP8 decoding h/w support is not too different from H.264 video decoding h/w support.
So a lot of the problem here, as in many things, is timing, and WebRTC is next year’s problem, not this year’s (H.264 from HTML video). Don’t give up yet!
@Tom: why are you throwing “disingenuous” around? Firefox OS, like Firefox on Android, must be able to play H.264 out of the box to gain any market share. That’s “this year’s problem” and we’ve solved it for the user to even have a choice of our products.
But this does not mean Mozilla or our downstream distributors pay patent royalties in either case (FFOS/FF-on-Android). We’re consistent on this. We lost a battle, we won’t pay _rentiers_, we are working for the long haul, on both tech and legal fronts.
In this light your idea of making users download FFMPEG or anything else (the DirectShow malware vector @roc mentioned above) is not going to help much. Most users don’t care, won’t know, and won’t lift a finger — and rightly so. It’s not the user’s problem to solve. We need something that just “makes it work” on XP.
@DBCooper: not sure about FLAC, it would have to be lower priority. Maybe @roc or one of his crew can comment.
/be
so we might have h.264 support in Firefox (desktop) stable by the end of the year?
WebP would make webpages smaller and therefore loads faster, that is more important than flac. I’m not sure is flac is completely patent free either, could be a small risk adding it.
@Sam: the work for desktop is not scheduled yet, maybe @roc will attempt a guess. We’re working on it, not sure how long it will take on all OSes, but it’ll get done.
On WebP, Jeff Muizelaar’s blog post from last year:
https://muizelaar.blogspot.com/2011/04/webp.html
was against implementing. Has WebP changed in ways that overcome the (solid IMHO) reasons Jeff gave to not implement?
/be
“But this does not mean Mozilla or our downstream distributors pay patent royalties in either case (FFOS/FF-on-Android). ”
i don’t understand, you mean FirefoxOS devices will have h264 support without your partners (OEMs?) paying MPEG-LA? how did you manage that?
anyway, i still think your statements here _are_ disingenuous. you can:
1) admit you lost (one) battle, and you can
2) be pragmatic enough to ship h264 support where it benefits you (to enable you to gain marketshare and spread the mozilla message — which i wholeheartedly support)
3) but you can’t be pragmatic enough to benefit your _users_ on XP, in a market where you are already established and don’t feel the pressure to compromise on your values..
and as i said, relying on flash, with weekly security issues, is (imho) just as big of a security risk, if not bigger, than recommending ffmpeg (to the very small minority of users that don’t have some other codec installed).
btw, i have great respect for both you and the whole mozilla mission (i’m a decade-long supporter in several different ways), so my comments aren’t coming out of hate, but out of prioritizing users over everything else.
hope i didn’t offend you.. 🙁
Yes WebP has made many improvements, the comments on the link you posted highlight many, so do the ones here: https://bugzilla.mozilla.org/show_bug.cgi?id=600919
Maybe you could get Jeff Muizelaar to re-assess WebP for you (assuming he is a mozilla employee)
Someone did a mini review of a recent WebP build a few months ago here: https://blog.yoav.ws/2012/07/Images-Can-we-have-less
I think the only real problem is poor compression on black and white images. Maybe you could have someone contact google to ask them when they will fix that as WebP is ready now. It will be especially helpful for mobile firefox given slow 3g speeds and data caps.
@Tom: if you want to dispute something in earnest, best not to impute bad motive. “Disingenuous” has in its first meaning “insincere or calculating”. How about we disagree on definitions or premises? Or perhaps one of us is simply mistaken?
On to the substance of your comment. You wrote “I don’t understand, you mean FirefoxOS devices will have h264 support without your partners (OEMs?) paying MPEG-LA? how did you manage that?”
No, I wrote — and you quoted me — that “this does not mean Mozilla or our downstream distributors pay patent royalties….” Someone has paid, but not Mozilla for its open source built on OS-level codecs, and not any of our downstream open source distros (Linux, mainly).
We agree on your (1) and (2), but your (3), “you can’t be pragmatic enough to benefit your _users_ on XP, in a market where you are already established and don’t feel the pressure to compromise on your values” wants us to pay a royalty, when instead we can use Flash where Adobe has already paid.
Why do you think we should pay in case (3) when we avoided it in (2)? What’s “disingenuous” about treating Flash the same as the OS or other provisioned lower-level software?
/be
When i wrote that black and white images had poor compression i mean’t black+red, here is the bug: https://code.google.com/p/webp/issues/detail?id=98
Thats 1 of just 20 bugs that is on google’s webp bug tracker. You should ask google when they think those bugs that have been accepted will be finished as that would be a great time to add it to firefox.
(Sorry for the late comment, this just showed up in my planet feed for some reason)
Are there any thoughts on allowing embedders / xulrunner apps to add decoders? That is, in contexts that don’t touch the web (so that holding the web isn’t a concern). This doesn’t necessarily have to be done by the owners of the current code, but they would need to indicate that they won’t reject the possibility before useful work can be started…
@BE: are you suggesting that distributing FFMPEG would involve Mozilla paying royalties for FFMPEG? Am I interpreting that correctly? I am not a patent or licensing lawyer, just an ordinary pleb, but isn’t FFMPEG free? I am honestly not sure! Are you referring to the software patent issue BarryM raised? Is the threat of software patent claims on FFMPEG the (potential) royalty payments that Mozilla is worried about?
Please note, this is just a point of clarification question. I have no desire to challenge the logic or decision-making explained in this post. I’m just curious whether there are legal issues with the FFMPEG option.
@Asbjørn: MS relented once re XP’s end-of-life. If 8 flops, as it might well do, who is to say MS will not relent again? It’s still popular obvious reasons: Vista was a disaster; 7 is not markedly different or better and 8 is a tablet OS. So the upgrade path for XP users is 7 and nada. Why bother? Most XP users might as well try Linux instead. XP is tried and true, there’s no reason to rubbish it. I’ll use it until I have absolutely no choice and I’d like to thank Mozilla for their pragmatism in supporting XP through hardware acceleration that IE refused to offer (borrowed from Google or not) and this H.264 support decision.
@BE: It sounds as though these policy decisions have been made ahead of testing the XP solution. If performance issues arise when developing the Flash piggy-back option, is it possible that Mozilla may revisit the FFMPEG option? I only mention this because at present Flash is a performance problem in my experience. Would a built-in codec like FFMPEG produce better performance than the Flash piggy-back option? Perhaps that is one for cpearce, cdouble, roc or others?
BarryM, most countries use Firefox now, and software patent are only legal in USA and Japan as far I know, so only a really little part of the world. Firefox was only the most used web client on Europe where software patent are still legally rejected and have really near zero chance to be legalized at least in next decade.
Popolon: I wish software patents were “illegal” in Europe, but it is not true. See https://en.wikipedia.org/wiki/Software_patents_under_the_European_Patent_Convention for details.
Anyway, international businesses must worry about the US system, which is badly broken, and that sets a bar.
/be