[Also posted at hacks.mozilla.org.]
I wrote The Open Web and Its Adversaries just over five years ago, based on the first SXSW Browser Wars panel (we just had our fifth, it was great — thanks to all who came).
Some history
The little slideshow I presented is in part quaint. WPF/E and Adobe Apollo, remember those? (Either the code names, or the extant renamed products?) The Web has come a long way since 2007.
But other parts of my slideshow are still relevant, in particular the part where Mozilla and Opera committed to an unencumbered <video> element for HTML5:
- Working with Opera via WHATWG on <video>
- Unencumbered Ogg Theora decoder in all browsers
- Ogg Vorbis for <audio>
- Other formats possible
- DHTML player controls
We did what we said we would. We fought against the odds. We carried the unencumbered HTML5 <video> torch even when it burned our hands.
We were called naive (no) idealists (yes). We were told that we were rolling a large stone up a tall hill (and how!). We were told that we could never overcome the momentum behind H.264 (possibly true, but Mozilla was not about to give up and pay off the patent rentiers).
Then in 2009 Google announced that it would acquire On2 (completed in 2010), and Opera and Mozilla had a White Knight.
At Google I/O in May 2010, Adobe announced that it would include VP8 (but not all of WebM?) support in an upcoming Flash release.
On January 11, 2011, Mike Jazayeri of Google blogged:
… we are changing Chrome’s HTML5 <video> support to make it consistent with the codecs already supported by the open Chromium project. Specifically, we are supporting the WebM (VP8) and Theora video codecs, and will consider adding support for other high-quality open codecs in the future. Though H.264 plays an important role in video, as our goal is to enable open innovation, support for the codec will be removed and our resources directed towards completely open codec technologies.
These changes will occur in the next couple months….
A followup post three days later confirmed that Chrome would rely on Flash fallback to play H.264 video.
Where we are today
It is now March 2012 and the changes promised by Google and Adobe have not been made.
What’s more, any such changes are irrelevant if made only on desktop Chrome — not on Google’s mobile browsers for Android — because authors typically do not encode twice (once in H.264, once in WebM), they instead write Flash fallback in an <object> tag nested inside the <video> tag. Here’s an example adapted from an Opera developer document:
<video controls poster="video.jpg" width="854" height="480"> <source src="video.mp4" type="video/mp4"> <object type="application/x-shockwave-flash" data="player.swf" width="854" height="504"> <param name="allowfullscreen" value="true"> <param name="allowscriptaccess" value="always"> <param name="flashvars" value="file=video.mp4"> <!--[if IE]><param name="movie" value="player.swf"><![endif]--> <img src="video.jpg" width="854" height="480" alt="Video"> <p>Your browser can't play HTML5 video. </object> </video>
The Opera doc nicely carried the unencumbered video torch by including
<source src="video.webm" type="video/webm">
after the first <source> child in the <video> container (after the first, because of an iOS WebKit bug, the Opera doc said), but most authors do not encode twice and host two versions of their video (yes, you who do are to be commended; please don’t spam my blog with comments, you’re not typical — and YouTube is neither typical nor yet completely transcoded [1]).
Of course the ultimate fallback content could be a link to a video to download and view in a helper app, but that’s not “HTML5 video” and it is user-hostile (profoundly so on mobile). Flash fallback does manage to blend in with HTML5, modulo the loss of expressiveness afforded by DHTML playback controls.
Now, consider carefully where we are today.
Firefox supports only unencumbered formats from Gecko’s <video> implementation. We rely on Flash fallback that authors invariably write, as shown above. Let that sink in: we, Mozilla, rely on Flash to implement H.264 for Firefox users.
Adobe has announced that it will not develop Flash on mobile devices.
In spite of the early 2011 Google blog post, desktop Chrome still supports H.264 from <video>. Even if it were to drop that support, desktop Chrome has a custom patched Flash embedding, so the fallback shown above will work well for almost all users.
Mobile matters most
Android stock browsers (all Android versions), and Chrome on Android 4, all support H.264 from <video>. Given the devices that Android has targeted over its existence, where H.264 hardware decoding is by far the most power-efficient way to decode, how could this be otherwise? Google has to compete with Apple on mobile.
Steve Jobs may have dealt the death-blow to Flash on mobile, but he also championed and invested in H.264, and asserted that “[a]ll video codecs are covered by patents”. Apple sells a lot of H.264-supporting hardware. That hardware in general, and specifically in video playback quality, is the gold standard.
Google is in my opinion not going to ship mobile browsers this year or next that fail to play H.264 content that Apple plays perfectly. Whatever happens in the very long run, Mozilla can’t wait for such an event. Don’t ask Google why they bought On2 but failed to push WebM to the exclusion of H.264 on Android. The question answers itself.
So even if desktop Chrome drops H.264 support, Chrome users almost to a person won’t notice, thanks to Flash fallback. And Apple and Google, along with Microsoft and whomever else might try to gain mobile market share, will continue to ship H.264 support on all their mobile OSes and devices — hardware-implemented H.264, because that uses far less battery than alternative decoders.
Here is a chart of H.264 video in HTML5 content on the Web from MeFeedia:
And here are some charts showing the rise of mobile over desktop from The Economist:
These charts show Bell’s Law of Computer Classes in action. Bell’s Law predicts that the new class of computing devices will replace older ones.
In the face of this shift, Mozilla must advance its mission to serve users above all other agendas, and to keep the Web — including the “Mobile Web” — open, interoperable, and evolving.
What Mozilla is doing
We have successfully launched Boot to Gecko (B2G) and we’re preparing to release a new and improved Firefox for Android, to carry our mission to mobile users.
What should we do about H.264?
Andreas Gal proposes to use OS- and hardware-based H.264 decoding capabilities on Android and B2G. That thread has run to over 240 messages, and spawned some online media coverage.
Some say we should hold out longer for someone (Google? Adobe?) to change something to advance WebM over H.264.
Remember, dropping H.264 from <video> only on desktop and not on mobile doesn’t matter, because of Flash fallback.
Others say we should hold out indefinitely and by ourselves, rather than integrate OS decoders for encumbered video.
I’ve heard people blame software patents. I hate software patents too, but software isn’t even the issue on mobile. Fairly dedicated DSP hardware takes in bits and puts out pixels. H.264 decoding lives completely in hardware now.
Yes, some hardware also supports WebM decoding, or will soon. Too little, too late for HTML5 <video> as deployed and consumed this year or (for shipping devices) next.
As I wrote in the newsgroup thread, Mozilla has never ignored users or market share. We do not care only about market share, but ignoring usability and market share can easily lead to extinction. Without users our mission is meaningless and our ability to affect the evolution of open standards goes to zero.
Clearly we have principles that prohibit us from abusing users for any end (e.g., by putting ads in Firefox’s user interface to make money to sustain ourselves). But we have never rejected encumbered formats handled by plugins, and OS-dependent H.264 decoding is not different in kind from Flash-dependent H.264 decoding in my view.
We will not require anyone to pay for Firefox. We will not burden our downstream source redistributors with royalty fees. We may have to continue to fall back on Flash on some desktop OSes. I’ll write more when I know more about desktop H.264, specifically on Windows XP.
What I do know for certain is this: H.264 is absolutely required right now to compete on mobile. I do not believe that we can reject H.264 content in Firefox on Android or in B2G and survive the shift to mobile.
Losing a battle is a bitter experience. I won’t sugar-coat this pill. But we must swallow it if we are to succeed in our mobile initiatives. Failure on mobile is too likely to consign Mozilla to decline and irrelevance. So I am fully in favor of Andreas’s proposal.
Our mission continues
Our mission, to promote openness, innovation, and opportunity on the Web, matters more than ever. As I said at SXSW in 2007, it obligates us to develop and promote unencumbered video. We lost one battle, but the war goes on. We will always push for open, unencumbered standards first and foremost.
In particular we must fight to keep WebRTC unencumbered. Mozilla and Opera also lost the earlier skirmish to mandate an unencumbered default format for HTML5 <video>, but WebRTC is a new front in the long war for an open and unencumbered Web.
We are researching downloadable JS decoders via Broadway.js, but fully utilizing parallel and dedicated hardware from JS for battery-friendly decoding is a ways off.
Can we win the long war? I don’t know if we’ll see a final victory, but we must fight on. Patents expire (remember the LZW patent?). They can be invalidated. (Netscape paid to do this to certain obnoxious patents, based on prior art.) They can be worked around. And patent law can be reformed.
Mozilla is here for the long haul. We will never give up, never surrender.
/be
[1] Some points about WebM on YouTube vs. H.264:
- Google has at best transcoded only about half the videos into WebM. E.g., this YouTube search for “cat” gives ~1.8M results, while the same one for WebM videos gives 704K results.
- WebM on YouTube is presented only for videos that lack ads, which is a shrinking number on YouTube. Anything monetizable (i.e., popular) has ads and therefore is served as H.264.
- All this is moot when you consider mobile, since there is no Flash on mobile, and as of yet no WebM hardware, and Apple’s market-leading position.
You missed out some victories:
* MPEG-LA was forced to become specific about some of it’s future licensing plans and the threat of future fees for free content was lifted
* A competitive free video codec now exists, and hasn’t yet been sued out of existence despite high-profile usage in Firefox, Chrome, Skype, Youtube etc. This used to be thought crazy and/or impossible.
* The IETF is in the final stages of developing a class-leading, royalty-free “internet audio codec” which is ideal for WebRTC and the web.
* Many other minor milestones i can’t think of right now
I’ve supported royalty-free codecs from the beginning, and believe that in the long run royalty-free codecs are inevitable but it’s gone much faster and better than I ever hoped, in large part thanks to Mozilla, Google & Xiph (amongst others) working together. Kudos to all of you.
I truly believe you could have won this round if any one of Microsoft, Apple or Adobe had got on board with WebM. Again, that’s amazingly close and better than I’d have expected.
I support the semi-open web.
This assumption that mobile will take over desktop is very annoying. So far mobile phone web tends to deliver a lo-fi experience that takes the web back 15 years to the beginning. ULs blown up to full screen as menus, linear content, two-finger-punching text-based interactivity, that sort of thing. Tablets might change this but please can we start to look beyond the estimation of total devices predicted to be sold and start looking at the number of mobile devices that actually use the web, then the quality of experience on those devices? Does anyone think desktop-based computing ever really go away? Or perhaps will it remain constant or grow into TV-based computing? I live in a first-world country, Recently on the train home, I tried to access the most trivial of web content on a ‘smart’ phone with the full alphabet-soup of connectivity standards. GPRS, 3G, HSDPA, you name it! All I got was endless nothing and eventually a timeout. We’re getting an National Broadband Network (NBN), not a National Broadband Mobile Network. I can’t see the desktop web falling away too fast in this country at least.
Thanks for the update, Brendan. I agree that the battle for open is one worth fighting, and while WebM proved a point the long tail of h.264 content has proven near-insurmountable to overcome.
One question about your notes at the end: I thought that ad-encumbered content on YouTube required the Flash player instead of the one with h.264 content, as the requirement was to be able to support the un-DOM-hackable “you must watch this ad for at least 15s” aspect. Is that not the case? Are they serving otherwise Flash-ad-encumbered video as raw h.264 to Chrome and Safari?
So what about MeeGo users? Only android?
Re: Mike and YouTube — https://www.youtube.com/html5 says that “Videos with ads are not supported (they will play in the Flash player)”
I still think we should have a “purity” switch (whatever you call it) that allows people to disable proprietary codec support. I agree that we need to ship it by default, though, where the hardware supports it.
mfgdfgdfgfdg, That is a good question. I don’t imagine this just being Android. We have put a lot of effort into both Maemo and now Meego. We revived the Qt port and now Nokia is shipping Firefox on the N9 (MeeGo 1.2). But it is true, most of our focus has been on Android and B2g.
I am pretty sure we can support MPAPI. People are already playing h.264 content on MeeGo through gstreamer and ffmpeg (with media plugins).
Doug
Ask the User if they would like the allow the non-free H264 plug-in on their Firefox installation/update… If yes, then provide a dialogue box explaining they will now be downloading libavcodec_plugin from VideoLAN.. No licence fee, easily customised like any plugin..
Problem solved?
Re: “Google has at best transcoded only about half the videos into WebM. E.g., this YouTube search for “cat” gives ~1.8M results, while the same one for WebM videos gives 704K results.”
I don’t think that’s indicative of the number of WebM transcodes available, at least not directly. I’d suggest it’s indicative of the number of videos playable in the HTML5 player which also have WebM transcodes available. You can argue that it doesn’t make a difference in practical terms, but knowing where YouTube’s implementation problems lie is, I feel, an important distinction to make.
Earlier this year I noticed that all the Vevo music videos on YouTube (or all the ones I tried anyway) were playable with the HTML5 player using WebM transcodes. Some time later that changed back and the Vevo videos were no longer playable, but I suspect they still have WebM transcodes lurking behind the scenes.
On YouTube’s HTML5 page (https://www.youtube.com/html5) they note that videos with ads are not supported by the HTML5 player. My guess is that this went “wrong” due to some bug and videos with ads started working in the HTML5 player or, alternatively, Google made that change deliberately but subsequently decided it wasn’t working the way they wanted it to so they turned it off again.
I used to be able to play Beyonce’s “Dance For You” (https://www.youtube.com/watch?v=PGc9n6BiWXA) in Firefox with HTML5 and WebM. Today I can’t play it in either Firefox or Internet Explorer 9 with HTML5 video. Even if Firefox adds H.264 support tomorrow I still won’t be able to listen to Beyonce. And this outcome is fundamentally unacceptable to me.
I’ve written some comments on this, in “Video, fragmentation, and Firefox”, which you can find on my blog: https://ossguy.com/?p=1137 I hope they are useful in the decision-making process. I’d be happy to engage in further discussion about this, either here or on my blog.
@Denver: your post seems to ignore the mobile problem, except by arguing both for battery-burning ARM CPU-based decoding, and citing possible WebM hardware support not in most devices this year or even next.
Also, you assume we won’t make H.264 work on Windows XP, but I said we would (Linux too) — we just hadn’t worked out all the details. It’s possible we will use Flash, which is on almost all PCs.
Another thing: you completely ignore how Mozilla relies on Flash fallback (on desktop; no Flash on mobile) for almost all users to see H.264 in practice. How is that principled of us, while using OS H.264 decoder plugins instead of Flash is unprincipled? There’s no difference provided all OSes are supported.
Finally, you seem to think Mozilla can survive and be a shining example to shame companies such as Apple and Google, fierce competitors whose business leaders would shed at most half a tear if Mozilla died. This is naive, to say the least.
I realize that you care passionately about unencumbered video, but most users and too many developers do not. Mozilla is not the 600 pound gorilla able to move markets away from de-facto standards as Apple did in rejecting Flash, and Google didn’t do in keeping H.264 from video working on Android and in Chrome. Even our long-time ally Opera on some OSes uses OS/HW-integrated H.264 decoders.
As lone hold-out, Mozilla would be like RMS’s laptop (https://wiki.debian.org/DebianYeeloong). Sorry, feel-good impostures of no practical value to most of our users do not advance Mozilla’s mission.
/be
“We may have to continue to fall back on Flash on some desktop OSes. I’ll write more when I know more about desktop H.264, specifically on Windows XP.”
Is there a newsgroup thread where this topic is being discussed?
[replying to Brendan]
It’s unfortunate that I included those two together as they’re very different points. The point about WebM hardware support is intended to say that WebM is ready for implementation into standalone digital cameras and video cameras.
The point about decoding 1080p WebM on current ARM chips was my more important point and I didn’t expand on it enough. It’s possible that TI implemented a battery-draining mostly-software decoder for WebM. And it’s unfortunate that I didn’t choose a better example. In any case, my point here is that WebM decoding can be as good as H.264 decoding on the same hardware.
The part I glossed over is that “hardware decoding” usually means “we do some stuff in hardware”, not “we pass this video stream to a chip and it passes back a series of bitmaps”. For example, H.264 and WebM/VP8 both use discrete cosine transforms, which are implemented in hardware by many (most?) ARM chips, such as in the NEON extensions (see https://www.fftw.org/ for some details). So when you have “hardware support for H.264”, you usually have “hardware support for WebM” as well. The place where WebM lags is in the software that translates decoding instructions into the appropriate assembly that will utilize the hardware that’s already there. This is primarily a lack of software implementations, not a lack of hardware support.
I think it’s premature to say that “we will make H.264 work on Windows XP and Linux” when the details aren’t worked out yet. In this case especially, the devil is in the details; I think it will be quite hard to get even Flash to do what you want (ie. pass it H.264 and have it decode without some kludgy SWF wrapper). As we’ve seen, Adobe doesn’t move quickly to support things they don’t care about.
Part of what bothers me here are the assumptions about what types of software are ok to ask people to use. When the only concrete solution being mentioned is Flash, the implicit assumption is that asking people to use proprietary software is ok. Now I’m not saying that there are a lot of people who use absolutely no proprietary software, but I am saying that it’s important to reduce our dependence on proprietary software whenever possible and that it’s improper to ask people to use proprietary software as a result.
I don’t think any non-Flash solutions to playing H.264 in Firefox will be better from a free software perspective than using Flash. Usually to get a patent license, it’s necessary to make the patented software non-free to prevent redistribution.
The implicit assumption here is that by not supporting H.264, Mozilla is forcing people to use Flash. I think some people will use Flash, but some people will think “I wonder why some videos don’t work” and then discover how patented technologies are holding back the open web and choose to stay away from H.264 as a result (yes, this will be a small number). At least in this case they will not be lulled into a false sense of security, assuming that there are no restrictions on playing H.264 (since they long ago agreed to the OS EULA that allows them to play H.264 so they won’t think of its restrictions again, while without H.264 support, users will be confronted with a new EULA (Flash or otherwise) if that choose to use H.264 that may cause them to think about what they’re getting into).
Right now Firefox is not forcing me or others to use Flash. We can choose not to and still use the web just fine (see https://ossguy.com/?p=1037 ). But if Firefox supported H.264, many people who were not previously forced to use Flash to view Firefox-supported content would now be forced to install it (or similarly-restrictive software) if they want the same experience as other Firefox users.
Firefox still has a higher usage share than Chrome (by most accounts), though it is admittedly falling. But I think that user loyalty should not be underestimated. Even if usage share falls below Chrome, the overall loyalty of users to Firefox will be greater, exactly because Mozilla is a non-profit that cares about the open web. I would not play down Mozilla’s influence in driving codec trends.
As I said above, this is not about making sure that people like RMS can run Firefox. This is about keeping Firefox away from dependence on non-free and patented software so that it can remain nimble in the future and won’t be tempted to make even worse compromises that negatively impact the open web in the long-term. I hope Mozilla will choose to not compromise here.
@PD: linked from the post:
https://groups.google.com/group/mozilla.dev.platform/browse_frm/thread/fb14de8b9ad84e15?hl=en&scoring=d&
/be
@Denver: Firefox has over 400m users. Most don’t have a clue about why video might not work. Most have Flash. For you to model most users on yourself is unrealistic.
Most users just want “it” to work. Most users won’t use a phone or tablet that can’t play the same video leading phones and tablets play. This is so basic it should go without saying. Committed free-software supporters and other users who don’t care or want only free decoders are the all-too-rare exception.
Again (over and over), on mobile we don’t even get users without working video. B2G does not get onto phones.
My words about Windows XP and Linux are a commitment as well as a prediction. We are working with Flash folks and we may even be able to wrap H.264 in a SWF and hand it off to unaltered Flash players — but we may well get some help from Adobe.
Worst case, we would rely on authored fallback on dying WinXP. I doubt it will come to that, but I assert that is better than nothing, and much better than Mozilla dying on the mobile hill without H.264.
On Linux we will use OS-installed decoders. If you decline to install them, more power to you. You are *not* representative of most users, even on Linux.
Your final paragraph gives Mozilla too much presumptive blame by imputing too much market power to us. Remember, our share of mobile is very small right now.
You are effectively asking us to stay on desktop, where as you note we are under withering competitive attack from well-funded big companies. Low ten digits (US$) of bundling and advertising attack. As I wrote, sticking to desktop only is too likely to mean decline and irrelevance.
Be clear to yourself and in what you write about the likely effects of what you propose. Talking about hardware decoding issues (I did a simulated software MPEG-2 decoder in my MicroUnity past life, I know the issues) does nothing to respond to the mobile predicament we face trying to get users this year and next.
Device turnover may help in two or more years. The best case then would be WebM and H.264 equally hardware optimized, and growing WebM content — but H.264 still “table stakes”. We will continuously evaluate.
/be
Just putting this out there: if we’re going to talk about “most” users, then “most” users will not install Firefox on their phone, because they use a phone that comes with a very good browser already.
@Stephen: let me get this straight: you think Firefox on Android cannot get much user share, so we should get even fewer users by hamstringing ourselves?
I agree mobile is tougher for alternative browser vendors. Browsers are less-used than apps (mostly for search or finding things — apps are for “doing task X”). And unlike the situation with IE in 2004, the mobile OS vendors do a better job on the default browser (yet not as good as they should — iOS Safari continues to disappoint, and even Chrome for ICS has issues for me that Firefox can solve better, e.g. the AwesomeBar).
In spite of the challenges for alternative browser vendors, Dolphin has >10M users and venture funding. I bet Firefox can go far once we finish the native front end and high-quality rendering work we’re preparing to ship. We’ll see.
But whatever happens on Android, let’s talk about our other mobile initiative. You dodged the B2G issue completely. We have much bigger upside there. Care to comment?
/be
@Brendan I said nothing about being hamstrung. I see that this discussion is emotional for you, and I apologise for wading in.
What is B2G, exactly? https://www.mozilla.org/en-US/b2g/ reads like so much marketing + buzzword spin. Is it an alternate OS for OEMs to use? Either way, it doesn’t seem directly relevant to Firefox.
@Stephen: my use of “hamstringing” is not emotional, it’s descriptive and rational. You assert we won’t get much Firefox usage on mobile (could be, I allowed, while citing some competitve wins by another alternative browser). Why do you believe rejecting H.264 on mobile Firefox will help or not hurt? I’ve plainly argued that it will hurt.
https://wiki.mozilla.org/B2G is a better link for technical info. But you can always use your favorite search engine, news search too, to find out more about B2G. That you don’t know, and don’t seem to care to search, makes it hard for me to give much of a hoot about educating you :-|. Playing the “oh, sorry you are emotional” line also doesn’t advance the debate. Let’s argue vigorously, not assert once and then back off out of misconstrued courtesy :-).
Mozilla is building for the mobile Bell’s-Law shift where browsers become OS wallpaper, so it ain’t all about Firefox (which you think won’t do well on mobile anyway — so now I may see your point: we are going down anyway, may as well die nobly — is that right? If so, we must part company).
BTW, here’s a relatively new group/list where media codec discussion is ongoing:
https://groups.google.com/group/mozilla.dev.media/browse_frm/thread/37abeb8d6044c8c8/5d977d7d4a23bd55#5d977d7d4a23bd55
/be
@Brendan (in response to your reply to me):
It sounds like Mozilla’s mission is not what I thought it was, which is fine, but good to know. When https://www.mozilla.org/ says “we’re dedicated to keeping [the web] free, open and accessible to all”, I suppose for Firefox this means “free” is as in “free beer”, “open” means one can view the code or docs that it implements, and “accessible to all” means “to all people who are willing to run non-free software”. I’m not trying to be snide here, I’m just trying to understand what it means. If you think the statement implies more freedom, openness, or accessibility than I’m giving it, I’d be happy to hear it.
My views on Mozilla’s mission echo those that Zizzle expresses in the comments on https://hacks.mozilla.org/2012/03/video-mobile-and-the-open-web/ : Mozilla’s position is unique in that it doesn’t need to worry about the bottom line and it should use that to its advantage, by not trying to “compete” in the traditional sense, but to be a leader in freedom and openness for the web.
If the main push for H.264 is Boot2Gecko, then I’m a bit confused. In the case of B2G, Mozilla would have some degree of control over the hardware (as the OS would have to be designed for particular devices) so it could choose to use devices that support certain instructions that most optimally help to decode WebM. I’m not sure if the goal is to provide ROMs for existing devices or to work with manufacturers to produce a ships-with-B2G device, but this comment applies in both cases. In the latter case, it would be even easier since Mozilla could work with the manufacturer select chips that contain the instructions that are most useful for WebM decoding, perhaps even burning a chip of their own with the WebM hardware decoding reference implementation.
Since you worked with MicroUnity, I won’t belabor the above point too much as you probably have more experience with video decoding than I do. It just seems to me that with all the hardware out there and the hardware instructions available that decoding WebM quickly on mobile devices should not be insurmountable in the near-term. But if you feel it is currently insurmountable and that Mozilla must move to mobile before mobile WebM decoding improves, then I suppose that’s the direction you will take.
I am implicitly assuming in this discussion that by not including H.264 in Firefox, sites will eventually switch to WebM. This might be the biggest difference in the approaches that you and I take to the H.264 issue (if you believe otherwise). I can understand the fear that if Firefox doesn’t support H.264, then eventually no one will use Firefox anymore and the whole web will just use H.264 (I don’t think this will happen, though it might). But I hope you will agree that by supporting H.264, Firefox will be discouraging any transitions to WebM that are currently underway, which effectively brings us to the same end result of everyone just publishing H.264 (except with perhaps a higher Firefox usage share). And discouraging these transitions now will make it harder to fight against people moving to HEVC and other patented codecs that will undoubtedly follow. It is better to nip this problem in the bud now, even if it results in lower Firefox usage share for a little while.
@Brendan I never said anyhing about rejecting H.264 in general, or on mobile in particular.
Given that I made comments about Firefox and you accused me of “dodging the B2G issue” I was trying to figure out how it’s relevant, and why I should have addressed it. It does not seem to be an app people install at all, and certainly not Firefox, so of course my comment that “most” mobile users are unlikely to switch to Firefox does not apply to B2G in any way.
You cite Dolphin’s 10M users and funding as evidence that Firefox can make headway on at least Android. I’m curious: do you consider 10M “most”? Does having a funding source change the way you perceive what should be done independant of “most”?
(Note, my use of “most” is in reference to your earlier uses of the word, and not necessarily to actual majorities. I intend it to mean whantever you intended to mean by your use, which I think was “non-technical users I’ve encountered and heard report of”, but could certainly mean something else 🙂 )
@Stephen: my apologies, I misread your original comment to mean “we can’t get most/many users with an alternative mobile browser, so why not stick with unencumbered-only HTML5 video”.
If I understand your point now, it’s that “most users” doesn’t matter because we can’t get many with Firefox on mobile.
If I’m reading you right, two responses:
1. We’re optimistic that we can do better than 10M users, and we’re giving it our best shot. It seems worth the attempt right now, even if we fail.
2. Funding for sustainability is important, but like most modern web outfits, we go for satisfied users and hope to receive revenue without losing them after. We don’t have a revenue plan yet for Firefox on Android (search pays little as people do fewer/different searches on mobile).
Hope this helps. Sorry again for over-interpreting your initial comment.
/be
Re: “We don’t have a revenue plan yet for Firefox on Android (search pays little as people do fewer/different searches on mobile).”
Speaking of revenue, does Mozilla have any future ambitions to release a Mozilla branded and unlocked B2G mobile phone?
@Ray: I do, and Mozilla obviously is not trying to be zero-revenue in our new mobile initiatives! No secret there. We have partners as revealed at MWC. We are working on business models.
/be
@Denver: Mozilla has always supported plugins that handle encumbered formats, and we view that as consistent with our mission. Why do *you* think relying on Flash fallback for H.264 from HTML5 video is ok, but relying on OS decoders is not ok?
Not having to worry about the bottom line is a myth. Not-for-profits too must worry but not in the same way as for-profits. The worry shifts to sustaining a mix of donations and user-activity-based revenue, mostly search.
Nothing in this life is free, and pretending something is will get you in big trouble quickly.
You are missing something simple about B2G: it does not gain market share by magically bidding into existence lots of WebM-encoded video. B2G must work on the same video content Apple and Android play.
I’m glad you wrote “I am implicitly assuming in this discussion that by not including H.264 in Firefox, sites will eventually switch to WebM.” This is a flawed assumption. Here’s why:
* On desktop, Flash fallback is authored and we embed Flash. It handles H.264 and it all “just works” — and there’s no pressure on authors to dual-encode.
* On mobile, we have too little share to make authors dual-encode. Trying to grow our share fails — chicken and egg problem. We are on the wrong side of Metcalfe’s Law.
Your mistake is treating desktop and mobile the same, and assuming that our not handling H.264 from HTML5 video somehow causes authors to dual-encode. Flash fallback on desktop relieves authors of having to dual-encode, and (with no Flash on mobile) puts us in a bind on mobile.
Do you see the predicament now?
/be
Brendan: just playing devils advocate, but removing flash fallback on desktop is also an option. There you might still have enough pull.
@Matthew: no, that would just give Chrome more advantage. The game theory on desktop is quite simple. Mobile is not much more complicated. Failing to distinguish the two is a problem, but in either game Prisoners who “defect” without enough market share just lose share. Cost of switching browsers is much lower than aggregate cost of dual-encoding.
/be
Via @AsaDotzler, the one and only Robert O’Callahan, aka roc (unhatched @rocallahan on twitter, https://robert.ocallahan.org/) in one sentence, 56 words, ~313 characters:
https://yro.slashdot.org/comments.pl?sid=2736863&cid=39420461
/be
@Brendan:
Totally irrelevant nerd correction, but:
I’d suggest that if browser codec choice was modelled as the prisoners dilemma then “defecting” would be shipping patented codecs. Mozilla is trying to co-operate, but the other browsers (rationally) see this as a chance to stick one over Mozilla (which is being “idealistic”), which in turn leads to Mozilla having to “defect” as well and now everyone is equal again and has no advantage but all players are down over $5Million a year compared with the unattainable best solution of all shipping the same free codecs.
[replying to Brendan]
This goes back to the assumption I outlined earlier (not supporting H.264 will cause developers to switch to WebM instead of causing more users to install Flash), which you discuss below and I’ll reply to in more detail (also below).
That’s fair. I guess our projections of how Firefox usage share will change given the two scenarios differs. Perhaps mine is more idealistic (less realistic)? I’m not sure. I do understand that a non-profit serving no one is not a useful non-profit.
What video content that Apple and Android play would B2G be missing out on? Both play DRMed and un-DRMed H.264, but I’ve seen very few sites (other than YouTube) that provide un-DRMed H.264. I’m guessing that B2G will not support DRMed H.264 so the amount of H.264 that B2G could actually play seems to be small.
When I say “DRMed”, I also kind of mean unobfuscated H.264 video provided only via Flash, since Android/iOS can’t usually play these. But perhaps I see a lot more of these than someone browsing on mobile, since I do most web browsing from a desktop browser without Flash and only a little bit of browsing on my phone, where I seldom, if ever, watch videos.
I’m guessing that a lot of sites just assume Flash for desktop browsers and deliver “real” H.264 when a mobile browser is detected. This is a terrible idea in my opinion, but might be more prevalent than I think (leading to a possibly-skewed view of how much non-Flash H.264 there is out there).
If Firefox supported H.264, I don’t see things getting better (using less Flash) on the desktop. Lots of sites already assume all desktops have Flash so they don’t even bother using <video>. I guess this will eventually improve if all browsers do H.264, but for the next few years, I think most users will be saddled with both a Flash dependency _and_ a patented/proprietary codec dependency, whereas without H.264 support in Firefox, they would only have the former (which we could then wean them off of by getting sites to use WebM).
Again, I don’t know where all this “H.264 in <video>” is coming from. I see a lot of H.264 in use, but mostly via Flash, and usually DRMed. Are a lot of sites serving up H.264 in <video> (and not also in WebM) that I just don’t know about?
@BOD: In the general Prisoner’s Dilemma browser game, “defect” means “not cooperate”, and “cooperate” means do cross-browser open standards whether de-facto or de-jure — and per SJ, and I hate this but he set this standard: “open can be encumbered”.
@Denver: please don’t “guess”. Apple killed Flash on mobile, the MeFeedia chart shows how this has rapidly moved H.264 video to HTML5.
Indeed mobile browsers are detected differently and fed “real” H.264 — and your opinion that this is “terrible” doesn’t amount to a hill of beans. It’s a fact Mozilla and all other mobile browser vendors must face.
“If Firefox supported H.264, I don’t see things getting better (using less Flash) on the desktop.”
You changed the predicament. Whatever happens on desktop, we must support H.264 HTML5 video on mobile.
Please address this instead of changing the subject to (incorrect) speculation about Flash on the desktop. Again the MeFeedia data show <video> is rapidly being adopted for H.264 and Flash is relegated to fallback content, as my blog post showed.
“Are a lot of sites serving up H.264 in <video> (and not also in WebM) that I just don’t know about?”
Yes. You don’t surf the same page reference string as users all around the world. Also you surf with a desktop user agent. And authors are not double-encoding (YouTube aside, and even not all of YT).
Try using a mobile browser, Apple or Android (top two, ~80%), or just change your user-agent: header with a Firefox add-on. You’ll see dramatically different content, including H.264 in HTML5 video.
/be
@Brendan:
I apologize for arguing based on conclusions I made from guessing. Sometimes I get so fed up with the way people have messed up the web that I don’t want to confirm that it’s as broken as it is. But it is broken and we have to deal with it that way.
I confirmed that more sites than I expected to work did play some video (at least previews) with H.264 <video> (or similar) on Android, while delivering Flash on a desktop browser. In particular, ABC and CBS seemed to do this.
It’s sad that we’re in the state you described for H.264 <video> on mobile. I think pushing back by supporting only royalty-free formats is worthwhile, as I believe there is time to build usage share while people still don’t care avoid video on mobile that much. But I can see why Mozilla might not want to keep pushing back right now.
I remain worried about the way in which this compromise will affect Mozilla’s decision-making in the future. I think it will be hard to say no to other royalty-bearing codecs (like HEVC) in the future (your article argued one reason H.264 was ok is that patents expire). And I’m perhaps most concerned that Mozilla will feel it necessary (and no longer problematic, since it supports H.264 already) to support particularly pernicious forms of DRM that the media industry might dictate. In particular, I notice that Netflix and Hulu are unlikely to provide H.264 <video> in the near future (or ever) and I fear that Mozilla will feel it necessary to boost usage share by saying “we can view that, just like Android and iOS”.
@denver
“I notice that Netflix and Hulu are unlikely to provide H.264 in the near future (or ever) and I fear that Mozilla will feel it necessary to boost usage share by saying “we can view that, just like Android and iOS”.”
Not true, Netflix is one of the companies behind the HTML5 DRM standard that is in it’s draft stages. This will stop users from right clicking and saving the video to their pc. Once DRM support is added to the HTML5 spec then bbc iplayer, itv player, netflix, lovefilm and countless other popular VOD companies will add support for HTML5 in conjunction with flash/silverlight.
@Denver: I meant what I wrote about us fighting the long war. We are working with Xiph on new formats. We need to double down there, which will require not only big partners (we’re talking to some) who stick to their guns, but (my personal opinion) royalty-free patent pooling or other means of IPR defense.
We’re here for the long haul, we hack law as well as code, we fight on all winnable fronts.
On the HEVC front we may well be too late — in many ways it rides on H.264 and that die is cast. But it is not the last word either, and too asymmetric for all applications (e.g., real time two-way).
You and @Sam are right to bring up DRM. This came up at the SXSW browser wars panel this year. But our take on it is not that DRM is inevitable as conceived today.
Mozilla’s vision is that you, the user, own your media (and other ownables: apps, and data in general). You can take it with you, it hangs off your portable identity (browserid, now PersonaID, a decentralized system), it does not belong to the iTunes or Android store, or die in some data coffin.
We’ll be getting into this more this year, in order to fight attempts to standardize the bogus walled-garden DRM model (region-locking on steroids: you don’t own your media, Apple does).
/be
but we need DRM though so that we can rent movies/tv shows otherwise we will only be able to buy them at a higher cost otherwise they won’t be able to stop their customers from choosing to stream the movie but actually save it. DRM in this sense (solely the removal of the ability to save the video to your pc) is a good thing, it could work on any html5 capable browser that supports that spec. DRM of the actual video itself so that you can’t play the downloaded video on another device is obviously a bad thing but the standard put forward by google, netflix and others doesn’t do anything nasty like that.
Music videos won’t go html5 as they don’t want customers to be able to download a 1080p copy to save for free and rightly so hence drm would be needed for those to stop people from stealing them for free.
I hope mozilla implements the new drm proposal by google/netflix and others from the 3mins i spent reading it sounded very good to me. We just need to get youtube to promise not to add drm for all videos, just music videos and partners that wish to protect their videos, that way we can easily save our viral videos to our pc’s but can’t steal a music video, tv show, movie etc.
@Sam: yes, renting as well as buying — time-based leases, or other models (I hate having only one day to watch, can imagine one day with two pauses permitted within a week, or variations).
/be
@Sam: “but we need DRM though so that we can rent movies/tv shows otherwise we will only be able to buy them at a higher cost otherwise they won’t be able to stop their customers from choosing to stream the movie but actually save it.”
If I’m part of the We, I say We don’t need DRM. DRM does nothing at all for Us. The They you mention seem to need DRM. I don’t know why They need it because DRM just doesn’t work. It never has and it never will:
https://www.streamingmedia.com/Articles/Editorial/Featured-Articles/DRM-Is-Dead-79353.aspx
But DRM is apparently a comforting fiction for Them, so They continue to insist on it.
“Music videos won’t go html5 as they don’t want customers to be able to download a 1080p copy to save for free and rightly so hence drm would be needed for those to stop people from stealing them for free.”
I don’t get it. Music videos have gone HTML5. Look, here are just three of many:
https://www.youtube.com/watch?v=1oQeWUxM4s8
https://www.youtube.com/watch?v=Ekdq1jbZLFU
https://www.youtube.com/watch?v=JZ1Mi77nogQ
It’s even the case that all of them have WebM transcodes. Additionally, in my part of the world at least, music videos are routinely broadcast at 720p and 1080i on free to air digital television, a medium that is trivial to copy. Yes, you too can record a copy of Rebecca Black’s “Friday” off the TV and become overwhelmed with guilt (though overwhelmed, perhaps, but rather more a musical turpitude than a moral one). If music videos don’t need DRM now, why will they need it in future?
some music videos are available in html5 yes, but not all. as for ripping drm yes that is possible but the average user doesn’t know how to do that, they do know how to right click and choose “save as” though. If we don’t get HTML5 DRM then we won’t have the streaming tv/movie services moving to HTML5, they will stick with flash. Flash sucks, i want to use html5 therefore i need DRM to be added to the HTML5 spec.
@Brendan:
It’s sad to see this will likely continue for HEVC. Hopefully that’s the end of the road for royalty-bearing codecs in Mozilla software. I, too, hope that fixing the law will resolve these issues, but I see that as being several decades away. Lessig and others’ work in fixing Congress will take some time.
I thought I understood your position on DRM (Mozilla will never endorse or implement it), but then you replied to Sam’s post about using DRM with “yes, renting as well as buying”. What did you mean there?
@Denver: you saw my reply at 3:23pm yesterday, about Mozilla’s portable, persistent, federated-identity user-based vision for rights to owned/rented data.
Yes, DRM won’t die. Say you want to watch Iron Man, or the forthcoming sequel to Dr. Horrible’s Sing-along Blog. You might torrent it but if the price is right and one can buy it, most users will. I personally don’t mind supporting some of the current glorious and mostly deleterious mess that is H’wood. YMMV.
But DRM in device- or vertical-silo-exclusive form sucks and abuses users who have devices that die, or who wish to switch silos. Mozilla’s vision accommodates the rights-owners (I know, going back to Muddy Waters, these rights-owners are too often swindlers and thieves who stole the rights from the creators; but not always) without abusing the users.
Or you could ‘torrent at will. That has its own costs and right now, too few users. If “bad DRM” takes over, torrenting will rise — a social feedback loop but also with risks (police state growth and action against torrenters, e.g.). Mozilla is not out to escalate. We seek wins for users, even if the users pay some rights-owners who may not deserve it. Users also may pay some who do.
/be
@Brendan:
Ok, that does make me feel more at ease. I think the most direct line that addresses my concern is “Mozilla’s vision accommodates the rights-owners…without abusing the users.” I’m glad “without abusing the users” is in there, though I worry that “abuse” might come to mean something different than I think it does, similar to how the meaning of Mozilla’s mission was not what I thought it meant. I’m interested to see how Mozilla implements this “user-based vision for rights to owned/rented data”.
I appreciate how involved you’ve been in the general discussion here. There are not many organizations/people that make the same effort to explain their decisions.
@Denver: you wrote:
“It’s sad that we’re in the state you described for H.264 on mobile. I think pushing back by supporting only royalty-free formats is worthwhile, as I believe there is time to build usage share while people still don’t care avoid video on mobile that much. But I can see why Mozilla might not want to keep pushing back right now.”
i wanted to close the loop on this point too. Thanks for your comments and thoughtful, evidence-based approach — much appreciated.
The mobile player to try rejecting H.264 HTML5 video is one with mobile browser or OS market power. Mozilla does not yet have such power. Android might get there but it manifestly can’t yet take the chance of dropping H.264, as I argued. It certainly couldn’t in past years while coming up against Apple.
You may be suggesting that we rely on our brand and values to attract new users on mobile, but too few who want working video will put up with the lack of it for a point of principle lost on most users. The mass market is harsh this way.
All this reinforces the need to research and develop decisively better unencumbered formats, and even to mount affirmative patent defenses for them. When we have both market power and better formats (with hardware support), then we can consider deprecating H.264.
/be
@Brendan:
I realize that I mis-typed a word in my previous post: ‘avoid’ should be ‘about’ in “while people still don’t care avoid video on mobile that much”. But it sounds like you still got the gist of it.
My premise was that people “who want working [H.264] video” would be relatively few, since there isn’t that much mobile video out there yet. And so perhaps Mozilla could push users toward the right solution in the same way Apple did for Flash by rejecting it from iOS (mostly for usability, in my opinion). I agree that this situation is different, as it is hard for most users to see a difference in usability or other factors that are directly beneficial to them, but I feel it is an avenue worth pursuing.
I’m not optimistic that a royalty-free codec rivaling the current proprietary-codec-of-the-year will be developed within this decade or even the next. This is partly why I am pushing for WebM now, rather than taking the position that we should accept proprietary codecs now and wait for royalty-free codecs to catch up. I think it will be very hard to match the progress that WebM has made with the millions of dollars Google has poured into it. And there will be far fewer people who care to work on the next royalty-free codec once Firefox supports H.264. Furthermore, the key players in the existing desktop and mobile realms (I mean Microsoft and Apple specifically) have a vested interest in keeping the latest codec proprietary (at least to deter the little guys from competing) and both embody the worldview that people should pay for “IP”.
I don’t want to wait decades for royalty-free video and I feel that we still have time to avoid that outcome. But I do see that this will cause less usage share for Mozilla than supporting H.264 would in the short-term. Whether this reduction in usage share would extinguish Mozilla altogether or at least make it ineffective is unclear; I don’t think it would, but I think you feel differently. Time will tell how closely our respective projections match reality.
@Denver: I figured out already that you meant “about”.
What, pray tell, is going to *increase* WebM usage?
You previously conceded there is in fact H.264 in HTML5 sent to mobile devices, e.g. on CBS and ABC sites. Your last comment seems to walk back from that to “isn’t that much”. That is not so as far as I can tell — there’s a lot of H.264 in HTML5 video.
/be
@Brendan:
What I meant is that there is not very much video being viewed on mobile devices relative to the time people spend doing other activities on mobile devices. And subsequently there is relatively little video being published for mobile (yes, CBS and ABC have some, but I think most people are spending a lot more time with Twitter, etc. than those sites). Even the video people do watch on mobile is primarily from YouTube (52% in H1 2011, according to https://gigaom.com/broadband/youtube-global-mobile-bandwidth/ ), which provides WebM for over 99% of views these days. So I think there is time for other devices to get popular and cause video publishers to add WebM to the formats they publish (as the percentage of time people spend watching video on mobile increases). In the meantime, people will still find a device that doesn’t play H.264 to be useful for 90+% of what they do on a mobile device.
I feel it’s likely that my above comments will continue to be unpersuading, which is fine. My main point is that Mozilla should stay true to what I believe its mission says and if that means trying really hard to get free codecs supported, then after a few years having to tell the world, “sorry, there is no open, unencumbered web”, I think that’s what should be done. This is not a great outcome, but it’s better than giving up on the open, unencumbered web for now and hoping that it will exist at some point in the future (a promise from Mozilla that it won’t support any known-encumbered codecs after HEVC would allay some of the fears that an open, unencumbered web is just a hope, not a plan). As I said, I think an open, unencumbered web will take decades to achieve if Mozilla chooses to support H.264 now.
brendan, the microsoft html5 video extension… v1.05 doesn’t work for me on firefox 12 beta 2, on youtube.com/html5 it shows h264 as unavailable, i’ve tried it on a html5 site and indeed it doesn’t work for h264 videos. Is this a bug in firefox or microsoft’s plugin or? Will you release a test plugin for firefox to enable h264 soon so that users who want to test the plugin can try h264?
@Denver: “I think that’s what should be done.” We already did that.
@Sam: we haven’t yet added H.264 HTML5 video yet. Watch the bugs.
/be
i’m not sure what you mean brendan. I know you don’t have the ability to watch h264 by default by microsoft’s h264 plugin doesn’t work for me with firefox 12, i’m not sure it works on earlier versions or not. What do you mean by watch the bugs?
I hope you release an add-on that adds h264 support as a beta plugin for users to test out before you add it into firefox as i’d like to use h264 right away.
thanks.
@SAM
sorry the code example didnt go thru, check this page
to get Microsoft H264 MP Plugin for firefox to work the source has to add type in the source eg:
https://blogs.msdn.com/b/b8/archive/2012/03/28/touch-hardware-and-windows-8.aspx
@Sam, sorry, I misread your comment. If @Kise’s advice does not help, could you please file a bug? Could be our issue, Microsoft’s, or both, but bugzilla.mozilla.org is the place, not my blog comments.
/be