This week, a number of Mozillians attended the annual Game Developers Conference in San Francisco to demonstrate how the Web is a competitive platform for gaming and game development.
We’ve worked very hard over the past couple of months on technologies used to speed up the Web for game development, including asm.js, a JavaScript optimization technology and pure-JS-subset target language for compilers; Emscripten, a C++ to JavaScript compiler; and OdinMonkey, the SpiderMonkey JavaScript engine’s compiler for asm.js source.
On Wednesday, we were happy to announce that we’ve teamed up with Epic Games to combine these technologies to bring Epic’s Unreal Engine 3 to the Web.
This is significant because it demonstrates to everyone that game developers and publishers can now take advantage of the reach and scale of the Web without the additional user acquisition friction and even higher costs (infinite on iOS :-P) associated with third-party plugins. These technology advancements also mean improved performance and the ability to port lots of games to the Web on desktops, laptops, and mobile devices. (Note well that asm.js code runs pretty fast on modern browsers, and super-fast when optimized as by OdinMonkey.)
I was personally thrilled to meet Tim Sweeney, CEO and founder of Epic Games, for breakfast before GDC Thursday. I’d been aware of Tim since his invited talk at POPL 2006. Tim and I were on very much the same page on many things, including JS’s ascendance as a safe target assembly-like language for C++ via asm.js and beyond.
P.P.S. Hot rumor: WebGL in IE11? Source, judge for yourself. I have hopes.
UPDATE: withinwindows.com says WebGL is coming, disabled by default so far, and (look for “UPDATE3” run together at the original source) with a different shader language, IESL based on HLSL, from GLSL ES.
2nd UPDATE: From “UPDATE4” at François Remy’s blog post, GLSL is the default and you set a registry flag to get HLSL. Cool!
Pictures (the first shows the room less than half-full, pre-press-conference):
Founders and executives:
The Mozilla crew who helped at the press conference (others were in Barcelona doing equally important pre-MWC work):
Here is our stunning, always-crowded booth in Hall 8 (“App Planet”):
Two days later, the excitement ramped even higher with news of a fifth device maker, Sony.
Our success in attracting support from these partners is due in part to our ability to innovate and standardize the heretofore-missing APIs needed to build fully-capable smartphones purely from web standards. Therefore I will give an update below to my post from last MWC with details on API standardization progress over the last year.
I want not only to identify the good work being done, but also to reinforce that:
We are collaboratively building these APIs to fill real gaps in the standards.
We actively work to get them standardized across all operating systems and browsers, as part of Mozilla’s mission.
We have made terrific progress in the last year thanks to Firefox OS, and many of the APIs work in Firefox for Android too.
We will continue to update API specs and code as new devices and sensors come to market.
The missing APIs must be added to the web platform in order to enable the billions of new mobile users who will be coming online in the next few years to have affordable web-based phones, tablets, and apps. Emerging market consumers and developers generally cannot afford increasingly higher-end, native-app-advantaged smartphones from the two bigs.
The main reason that many developers historically have turned to native OS-specific app platforms and tools is because that was the only way to work with the different hardware elements and proprietary features (e.g., battery status, dialer, SMS, payments system) present on mobile devices. Those developers who bravely soldiered on, trying to use HTML5, were forced to envelop their code in compiled native app wrappers such as Phonegap generates.
There was no technical reason for OS proprietors to disadvantage their web stacks, lock developers into their native stacks, or make their store an exclusive trust anchor. Yet the incumbents did hold the web back by withholding APIs.
Just as the first browsers’ creators exposed file pickers and alert dialogs to work with the underlying OS, we are finally filling in the web platform to include the missing smartphone APIs. This levels the web vs. native playing field. And by following our usual best practices of working in the open and submitting all of our work to the standardization process, we can be confident that the missing functionality will become available on all platforms eventually.
When these APIs are available on all web rendering engines — even desktops and laptops, where the APIs make sense — then developers can be sure that their code will run interoperably, no matter what the underlying operating system. Fragmentation, if any, could arise at the OS level, e.g., due to not updating the OS’s web engine (as happened in Android 2.3).
Barring OS-specific update failures, developers can be sure because they have consolidated cross-OS, cross-browser market power. In today’s competitive browser markets, web developers rule the roost; browser and web engine purveyors seek to win their favor. Developers will not bother with up-and-coming operating systems that deviate more than epsilon from the standards.
The example set by our good friends at Samsung in patching WebKit based on the new specifications validates our approach. It means that if Tizen devices become numerous, it is highly likely that mobile web applications that work on Firefox OS will work on Tizen.
Likwise, we are working with Microsoft on Pointer Events, since touch-enabled web apps must today unify mouse and touch events, and the previous W3C multi-touch work foundered on patent assertions.
The orders-of-magnitude increase in Javascript performance across all engines in the last few years has been stunning. This boost helps to address another web vs. native “gap”. And just around the corner is asm.js — more on that in a future post!
Other performance enhancements include continued work on off main thread compositing and GL layers. Yes, it’s true: the modern mobile-focused web rendering model includes implicit threading in the form of the Compositor thread, and the GPU’s parallel hardware.
As always, standards in progress are subject to change, but they require prototype implementations and user-testing (“user” meaning both developer and consumer). Mozilla remains committed to playing fairly by not forging de-facto standards out of prototypes, rather proposing before disposing and in the end tracking whatever is standardized.
(This is why we are vendor-prefixing the less certain among our new APIs — but I think we should aim to unprefix quickly once we have emerging consensus among implementors.)
Here’s what we’ve achieved in the last year and what we’re working on:
Battery Status is now a W3C Candidate Recommendation (CR), and there was discussion recently about moving it to Proposed Recommendation (the last step before a REC, the status of an officially endorsed W3C recommended standard, indicating readiness for deployment within its problem domain).
WebTelephony. Mozillians: Ben Turner, Jonas Sicking, Philipp von Weitershausen. This is being worked on by Intel and Telefónica folks as the Telephony API in the SysApps Working Group.
WebSMS. Mozillians: Mounir Lamouri, Jonas Sicking. Being worked on as the Messaging API in the SysApps WG.
Pointer Lock: this API is largely stable and implemented in multiple browsers. Mozilla is participating in the W3C Pointer Events working group along with Microsoft, Google, Opera, jQuery, and others. Mozillian: Matt Brubeck, together with Microsoft’s Jacob Rossi.
Open WebApps has been proposed to W3C. We seem to have reached consensus to use the Mozilla proposal as a “First Public Working Draft” (FPWD) in the SysApps WG with Samsung as co-editor.
Alarm API is now a W3C working draft. Intel is helping edit, a great example of collaboration.
Web Activities, a simplified version of Web Intents. We focused on a simpler subset of Web Intents, which has proven to be a good idea. The feature set of Web Activities is working well, while Web Intents is currently going through a restart.
Push Notifications. This API is still in an early state. We have some prototypes but not yet shipping code. Getting this right is hard given that we want to create something secure, scaleable, and highly available.
WebFM API. We have an initial draft, but the W3C SysApps WG has decided to focus on a smaller set of specs first to build common ground. This API was deemed to be lower priority and so will have to wait a bit before it goes through standardization.
WebPayment, implemented by Fernando Jiménez Moreno of Telefónica. There is a W3C Community Group entirely focused on payments, and now a W3C Task Force. We will work in these groups to make a flexible API for payments part of the web standards.
UPDATE: Contacts Manager API, Eduardo Fullea and Jose M. Cantera of Telefónica are editing.
Developers interested in getting to know the new APIs better should look at this Mozilla Hacks post by Robert Nyman, which provides guidance on how to use them along with code examples.
So, Mozilla is leading the effort to complete the web platform on mobile devices. Our work is bearing fruit, as shown by the industry support for Firefox OS and the W3C’s efforts to standardize the new APIs. Even Andy Rubin had some kind words for us (thanks, Andy!).
The main prize for me remains the picture of smiling “mobile web developers” (read: “web developers”) that I can see just around the corner. Based on our MWC experience this year, the Web is in good shape to win that prize.
[This is an extended essay on the news out of Norway yesterday. See the closing for encouragement toward Opera and its fans, whatever the open source projects they choose to join, from me on behalf of Mozilla. /be]
Founder Flashback
I wrote about the founding of HTML5 in June, 2004, without dropping that acronym, mentioning only “Opera and others” as partners, because Apple was shy. Fragments of memory:
@t was then working for Microsoft (he’s at Mozilla now, it’s great to have him) and sitting in front of me during the second day of the workshop.
Vested interests touted XForms as inevitable (the UK Insurance Industry had standardized on it!).
Browser vendors were criticized as everything from uncooperative to backward in resisting the XML Utopia.
JavaScript took its lumps, including from Bert (whom I met years later at SXSW and got on fine with).
Hixie (pre-Google, already working on Web Forms 2) and Håkon of Opera joined David Hyatt of Apple, Mozilla’s @davidbaron and me at a San Jose pub afterward. There I uncorked and said something like “screw it, let’s do HTML5!”
We intended to do the work in the WHAT-WG. Håkon reminded us that the only likely path to Royalty-Free patent covenants remained the W3C, so we should aim to land specs there. He also correctly foresaw that Microsoft would not join our legal non-entity, rather would prefer the W3C.
In early January, I heard from an old friend, who wrote “Some major shift is happening at Opera, senior developers are being laid off, there’s a structural change to the company….” Other sources soon confirmed what became news yesterday: Opera was droppingPresto for WebKit (the chromium flavor).
One of my sources wrote about Mozilla and why it matters to him:
Smart people, and people I know and like, at that. Interesting work going on in [programming languages and operating systems]. Probably the last relatively independent web technology player (not something I cared about historically, but after leaving the Apple iOS fiefdom for the Google Android fiefdom, I find I’m just as locked in and feel just as monitored).
I hear this kind of comment more often lately. I think it’s signal, not noise.
Why Mozilla Matters
The January signal reminded me of mail I wrote to a colleague about “How Mozilla is different”:
We innovate early, often, and in the open: both standards and source, draft spec and prototype implementation, playing fair and revising both to achieve consensus.
Our global community cuts across search, device, social network businesses and agendas, and wants the end-to-end and intertwingled mix-and-match properties of the Web, even if people can’t always describe these technical properties precisely, or at all. What they want most is to not be “owned”. They seek user sovereignty.
We restored browser competition by innovating ahead of standards in user-winning ways. We’re doing it again, and with allies pulling other platforms up: Samsung is patching WebKit to have the B2G-pioneered device APIs.
We advance the vision of you owning your own experience and data, attached to your identity, with whatever services you choose integrated. Examples include the Social API and the new Firefox-sync project that outsources storage service via DropBox, Box, etc.
The last point is hugely important. It’s why we have had such good bizdev and partnering with Firefox OS. But it is also why we have a strongly trusted brand and community engagement: people expect us to be an independent voice, to fight the hard but necessary fight, to be willing to stand apart some times.
I see these as critical distinguishing factors. No competing outfit has all of these qualities.
Thoughts on WebKit
I read webkit-dev; some very smart people post there. Yet the full WebKit story is more complex and interesting than shallow “one WebKit” triumphalist commentary suggests.
First, there’s not just one WebKit. Web Developers dealing with Android 2.3 have learned this the hard way. But let’s generously assume that time (years of it) heals all fragments or allows them to become equivalent.
WebKit has eight build systems, which cause perverse behavior. Peel back the covers and you’ll find tension about past code forks: V8 vs. Apple’s JavaScriptCore (“Nitro”), iOS (in a secret repository at apple.com for years, finally landing), various graphics back ends, several network stacks, two multi-process models (Chrome’s, stealth-developed for two years; and then Apple’s “WebKit2” framework). Behind the technical clashes lie deep business conflicts.
When tensions run high the code sharing is not great. Kudos to keyleaders from the competing companies for remaining diplomatic and trying to collaborate better.
Don’t get me wrong. I am not here to criticize WebKit. It’s not my community, and it has much to offer the world on top of its benefits to several big companies — starting with providing a high-quality open-source implementation of web standards alongside Mozilla’s. But it is not the promised land (neither is Mozilla).
If you are interested in more, see Eric Seidel’s WebKit wishes post for a heartfelt essay from someone who used to work at Apple and now works (long time on Chrome) at Google.
My point is that there’s no “One WebKit”. Distribution power matters (more below near the end of this essay). Conflicting business agendas hurt. And sharing is always hard. There are already many cooks.
In software, it’s easier in the short run to copy design than code, and easier to copy code than share it. In this spirit the open-source JS engines (two for WebKit) have learned from one another. Mozilla uses JSC’s YARR RegExp JIT, and we aim to do more such copying or sharing over time.
Web Engine Trade-offs
Some of you may still be asking: beyond copying design or sharing pieces of code, why won’t Mozilla switch to WebKit?
Answering this at the level of execution economics seems to entail listing excuses for not doing something “right”. I reject that implied premise. More below, but at all three levels of vision, strategy, and certainly execution, Mozilla has good reasons to keep evolving Gecko, and even to research a mostly-new engine.
But to answer the question “why not switch” directly: the switching costs for us, in terms of pure code work (never mind loss of standards body and community leverage), are way too high for “switching to WebKit” on any feasible, keep-your-users, current-product timeline.
XUL is one part of it, and a big part, but not the only large technical cost. And losing XUL means our users lose the benefits of the rich, broad and deep Firefox Add-ons ecosystem.
In other words, desktop Firefox cannot be a quickly rebadged chromium and still be Firefox. It needs XUL add-ons, the Awesome Bar, our own privacy and security UI and infrastructure, and many deep-platform pieces not in chromium code.
Opera, as a pure business without all of XUL, the Mozilla mission, and our community, has far lower technical switching costs. Especially with Opera desktop share so low, and Opera Mini as a transcoding proxy that “lowers” full web content and so tends to isolate content authors from “site is broken, blame the browser” user-retention problems that afflict Opera’s Presto-on-the-desktop engine and browser.
Nevertheless, my sources testify that the technical switching costs for Opera are non-trivial, in spite of being lower relative to Mozilla’s exorbitant (multi-year, product-breaking) costs. This shows that the pure “business case” prevailed: Opera will save engineering headcount and be even more of a follower (at least at first) in the standards bodies.
The Big Picture
At the Mozilla mission level, monoculture remains a problem that we must fight. The web needs multiple implementations of its evolving standards to keep them interoperable.
Hyatt said pretty much the last sentence (“the web needs more implementations of its evolving standards to keep them interoperable “) to me in 2008 when he and I talked about the costs of switching to WebKit, technical and non-technical. That statement remains true, especially as WebKit’s bones grow old while the two or three biggest companies sharing it struggle to evolve it quickly.
True, some already say “bring on the monoculture”, imagining a single dominant power will distribute and evergreen one ideal WebKit. Such people may not have lived under monopoly rule in the past. Or they don’t see far enough ahead, and figure “après moi, le déluge” and “in the long run, we are all dead.” These folks should read John Lilly’s cautionary tumblr.
@andreasgal notes that the W3C tries not to make a standard without two or more interoperating prototype implementations, and that this now favors independent Gecko and WebKit, since the only other major engine for WebKit to pair with is Microsoft’s Trident. We shall see, but of course sometimes everyone cooperates, and politics makes strange bedfellows.
I expect more web engines in the next ten years, not fewer, given hardware trends and the power wall problem. In this light, Mozilla is investing not only in Gecko now, we are also researching Servo, which focuses on the high-degree parallel (both multicore CPU, and the massively parallel GPU) hardware architectures that are coming fast.
If we at Mozilla ever were to lose the standards body leverage needed to uphold our mission, then I would wonder how many people would choose to work for or with us. If Servo also lacked research partners and good prospects due to changing technology trends, we would have more such retention troubles. In such a situation, I would be left questioning why I’m at Mozilla and whether Mozilla matters.
But I don’t question those things. Mozilla is not Opera. If we were a more conventional business, without enough desktop browser-market share, we would probably have to do what Opera has done. But we’re not just a business, and our desktop share seems to be holding or possibly rising — due in part to the short-term wins we have been able to build on Gecko.
Future Web Engines
So realistically, to switch to WebKit while not dropping out of the game for years, we would have to start a parallel stealth effort porting to WebKit. But we do not have the hacker community (paid and volunteer) for that. Nowhere near close, given XUL and the other dependencies I mentioned, including the FFOS ones. (Plus, we don’t do “stealth”.)
The truth is that Gecko has been good for us, when we invest in it appropriately. (There’s no free lunch with any engine, given the continuously evolving Web.) We could not have done Firefox OS or the modern Firefox for Android without Gecko. These projects continue to drive Gecko evolution.
And again, don’t forget Servo. The multicore/GPU future is not going to favor either WebKit or Gecko especially. The various companies investing in these engines, including us but of course Apple, Google, and others, will need to multi-thread as well as process-isolate their engines to scale better on “sea of processors” future hardware.
There’s more to it than threads: due to Amdahl’s Law both threads and so-called “Data Parallelism”, aka SIMD, are needed at fine grain in all the stages of the engine, not just in image and audio/video decoding. But threads in C++ mean more hard bugs and security exploits than otherwise.
I learned at SGI, which dived into the deep end of the memory-unsafe SMP kernel pool in the late ’80s, to never say never. Apple and Google can and probably will multi-thread and even SIMD-parallelize more of their code, but it will take them a while, and there will be productivity and safety hits. Servo looks like a good bet to me technically because it is safer by design, as the main implementation language, Rust, focuses on safety as well as concurrency.
Other approaches to massively parallel hardware are conceivable, and the future hardware is not fully designed yet, so I think we should encourage web engine research.
Strategic Imperatives
Beyond these considerations, we perceive a new strategic imperative, both from where we sit and from many of our partners: the world needs a new, cleanish-slate, safer/parallel, open-source web engine project. Not just for multicore devices, but also — for motivating reasons that vary among us and our partners — to avoid depending on an engine that an incredibly well-funded and lock-in-prone competitor dominates, namely WebKit.
Such a safer/parallel engine or engines should then be distributed and feed back on web standards, pushing the standards in more declarative and implicitly parallelizable directions.
Indeed more is at stake than just switching costs, standards progress, and our mission or values.
One pure business angle not mentioned above is that using an engine like WebKit (or what is analogous in context, e.g., Trident wrappers in China) reduces your product to competing on distribution alone. Front end innovations are not generally sticky; they’re in any event easy to clone. Consider Dolphin on Android. Or observe how Maxthon dropped from 20+% market share to below 1% in a year or so in China. Observe also the Qihoo/360 browser, with not much front end innovation, zooming from 0 to 20% in the same time frame.
Deep platform innovations can be sticky, and in our experience, they move the web forward for everyone. JS performance is one example. Extending the web to include device APIs held back for privileged native app stacks is another. So long as these platform innovations are standards-based and standardized, with multiple and open source implementations interoperating, and the user owns vendor-independent assets, then “quality of implementation and innovation” stickiness is not objectionable in my view.
Back to Mozilla. We don’t have the distribution power on desktop of our competitors, yet we are doing well and poised to do even better. This is an amazing testimony to our users and the value to them of our mission and the products we build to uphold it. In addition to Firefox on the desktop, I believe that Firefox OS and Firefox for Android (especially as a web app runtime) will gain significant distribution on mobile over the next few years.
Take Heart
So take heart and persevere. It is sad to lose one of the few remaining independent web platforms, Presto, created by our co-founder of HTML5 and the unencumbered <video> element. I hope that Opera will keep fighting its good fight within WebKit. Opera fans are always welcome in Mozilla’s community, at all levels of contribution (standards, hacking, engagement).
We don’t know the future, but as Sarah Connor carved, “No fate but what we make.” Whatever happens, Mozilla endures.
A short blog post to let everyone in the Mozilla community know the latest news.
As the “About Brendan” sidebar says, I’m a founder of Mozilla, and since 2005 I have had the title of CTO. That job has entailed work on technical strategy, Web standards, influencing/partnering, Mozilla Research, public speaking, and talent-scouting. (Oh, and I still code.)
I’m happy to announce that as CTO I will now also be managing our product and platform engineering teams. This is a privilege and an obligation I take seriously. I look forward to working even more with community members including Mozilla’s engineering staff to advance the Open Web on all platforms, especially the mobile ones rising to overshadow desktop computers.
Mozilla is many things: mission-based organization, community of hackers and supporters, social movement representing the people using the Web above all other agendas. To fulfill our mission, we must serve users with excellent products. Successful products require great design and engineering, and also (increasingly) partnering.
We can’t fulfill our mission without winning products. We can’t have winning products without focus on the technology trends, including the rise of mobile, and also of multicore and GPU processing, and yet more laser-like focus on serving the people who use our products.
The watch-word for me is therefore “focus”. Competition and partnering have focused Mozilla in the last few years, and all to the good in my view. We are on the right track with Firefox OS, Firefox for Android, Firefox on the desktop, and our other initiatives (two to name here: Persona and Web Maker). We need to continue, with even sharper focus on what’s in front of us.
In particular, on mobile this means not just great user interfaces and fast, smooth performance. It also means, as I said to @stshank, expanding up the stack to fight proprietary lock-in that diminishes developer and user experience. We did it with Firefox in 2004, we can do it again with Firefox OS, Firefox for Android, Persona and beyond.
I remain the buck-stopper for Mozilla module ownership and other technical delegated decision-making disputes. Mitchell remains governance conflict buck-stopper and Chief Lizard-Wrangler.
This is great news: four out of the five reformers won.
Back-story: in late 2010, TBL invited me to join the TAG. I declined with thanks for two reasons:
I was (at the time, especially) over-committed to standards (JS, mostly) and Mozilla work (e.g., setting up Mozilla Research).
The TAG was not engaged “toothfully” (my word) on Web problems faced by developers, rather it seemed focused on XML and Semantic Web matters, and therefore I would be odd-TAG-member-out.
Having both (a) more time and (b, the point of this story) three or four kindred-spirit candidates would have changed my mind. I hope my decline-with-thanks message helped in some small way to bring about today’s good news.
Kudos to the reformers for running, to the W3C Advisory Committee representatives who voted in the reformers, and to the W3C and TBL for being open to reform.
Tracking bug 799318 for the above two and the missing Mac OS X bug, plus the Windows XP solution described next.
The idea for Windows XP is to use Flash. According to roc, “we believe it may be possible to use Flash unmodified. Modern Flash has APIs to let us inject compressed data to the media engine without going through their networking layer, and we can recover rendered video frames.”
So, hard work still to-do ahead of us, but nothing that we can’t overcome (knock on wood).
We are taking the fight for unencumbered formats to the next battlefront, WebRTC, also as promised. More on that front later.
As always, the dev-media list (mailman “subscribe” protocol; also a Google Group) is a fine place to discuss any of this.
I blogged early in 2011 about Harmony of My Dreams, to try to fit in one page some dream-sketches (if not sketchy dreams — the #-syntax ideas were sketchy) of what I thought were the crucial elements of ECMAScript Harmony, the name I coined for the standardized future of JavaScript.
Now this dream is coming true, not just in ES6 draft specs but in prototype implementations in top browsers. Here I’ll tout Firefox 15, which released almost six weeks ago (yes, this means Firefox 16 is tomorrow, and Firefox 17 beta and 18 aurora too — these all have yet more new goodies in them — isn’t Rapid Release fun?). Per the MDN docs, the SpiderMonkey JS engine shipped in Firefox 15 sports the following new prototype-implemented draft ES6 features:
Default parameters
This extension (AKA “parameter default values”) is too sweet, and it will help put the arguments object out to pasture:
Implementation credit goes to Benjamin Peterson for his work implementing default parameters, and to Jason Orendorff for his always-excellent code reviews. See this bug for followup work to track the latest ES6 agreement on how passing undefined (and only undefined) should trigger defaulting.
We have a few details to iron out still about scope, I suspect (based on this es-discuss message and its thread).
Rest parameters
Even sweeter than default parameters are rest parameters, and I bet they are ahead of default parameters in making arguments a bad memory some fine day:
Again credit goes to Benjamin and Jason for their work.
Spread in array literals
The dual of rest is called “spread”, and it should work in call expressions as well as array literals. The latter is implemented in Firefox 16 (now in the beta channel):
js> a = [3, 4, 5]
[3, 4, 5]
js> b = [1, 2, ...a]
[1, 2, 3, 4, 5]
Thanks once again to Benjamin (a star Mozilla intern this summer) and Jason.
Spread in call expressions is not yet implemented:
js> function f(...r) { return r; }
js> function g(a) { return f(...a); }
typein:20:0 SyntaxError: syntax error:
typein:20:0 function g(a) { return f(...a); }
typein:20:0 .........................^
But I believe it is coming soon — bug 762363 is the one to watch, patch, and test.
for-of iteration
I blogged and spoke about for-of at TXJS 2011. The of contextual keyword, also in CoffeeScript, goes where in goes in for-in loops, in order to trigger the new iteration protocol (which is based on Python’s).
js> for (var v of [1, 2, 3]) print(v)
1
2
3
Arrays are iterable out of the box in ES6. This is a huge usability win! Unwary newcomers hoping for Pythonic value iteration can now avoid the trap of for-in on arrays iterating string-coerced keys rather than values.
Objects are not iterable without the programmer opting in explicitly:
js> for (var [k, v] of {p: 3, q: 4, r: 5}) print(k, v)
typein:24:0 TypeError: ({p:3, q:4, r:5}) is not iterable
To opt in, call an iterator factory, that is, a function that returns a fresh iterator for its parameter. Or simply give your objects or their common prototype an iterator property whose value is an iterator factory method: a function that returns the desired fresh iterator given its this parameter.
We require opt-in to avoid future-hostility against custom iterators for collection objects. Such objects probably do not want any kind of general property iterator default, which if left on Object.prototype, might be object-detected and prevent installation of the correct custom iterator factory.
The easiest way to create such an iterator factory is to write a generator function:
js> function items(o) { for (var k in o) yield [k, o[k]]; }
js> for (var [k, v] of items({p: 3, q: 4, r: 5})) print(k, v)
p 3
q 4
r 5
Note that SpiderMonkey has not yet implemented the ES6 generator function* syntax. We also haven’t added the ES6 features of delegating to a sub-generator via yield* and of returning a value from a generator (as in PEP 380). We’ll get to these soon.
Map
Have you ever wanted to map from arbitrary keys to values, without having the keys be implicitly converted to strings and therefore possibly colliding? ES6 Map is for you:
js> var objkey1 = {toString: function(){return "objkey1"}}
js> var objkey2 = {toString: function(){return "objkey2"}}
js> var map = Map([[objkey1, 42], [objkey2, true]])
js> map.get(objkey1)
42
js> map.get(objkey2)
true
The Map constructor takes any iterable, not just an array, and iterates over its key/value array-pairs.
Of course you can update a Map entry’s value:
js> map.set(objkey1, 43)
js> map.get(objkey1)
43
And you can add new entries with arbitrary key and value types:
js> map.set(objkey2, objkey1)
js> map.set(objkey1, objkey2)
js> for (var [k, v] of map) print(k, v)
objkey1 objkey2
objkey2 objkey1
hi 44
stringkey 44!
but now there’s a cycle between the objkey1 and objkey2 entries. This will tie up space in the table that must be manually released by breaking the cycle (or by dropping all references to the map):
js> map.delete(objkey1)
true
js> map.delete(objkey2)
true
js> for (var [k, v] of map) print(k, v)
hi 44
stringkey 44!
Setting the objkey1 and objkey2 variables to null is not enough to free the space in map tied up by the cycle. You must map.delete.
If your map is not exposed via an API by which arbitrary values could be passed as key and value to map.set, you won’t have to worry about cycles. And if the map itself becomes garbage soon (for sure), no worries. But for leak-proofing with arbitrary key/value cycles, see WeakMap, below.
Set
When you just want a set of arbitrary values, it’s a pain to have to use a map and burn code and memory on useless true values for the keys. So ES6 also offers Set:
js> var set = Set([1, true, "three"])
js> set.has(1)
true
js> set.has(2)
false
js> for (var e of set) print(e)
1
true
three
js> set.size()
3
As with Map, with a Set you can delete as well as add:
js> set.delete("three")
true
js> for (var e of set) print(e)
1
true
js> set.size()
2
js> set.add("three")
js> set.size()
3
An object element keyed by its identity works just as well as any other type of element.
js> var four = {toString: function(){return '4!'}}
js> set.add(four)
js> set.has(four)
true
js> for (var e of set) print(e)
1
true
three
4!
Unlike Map there is no cyclic leak hazard with arbitrary elements, although a WeakSet taking only object elements would still be helpful for automatic element removal when no other references to an element object remain. This idea has come up in connection with proxies and symbols, but I’ll save that for another post.
WeakMap
As noted above, with Map, making a cycle among map keys and values can tie up space in the table, and in the heap in all objects linked along the cycle or reachable from those objects, even when no references outside of the table to the key objects still live. Non-object keys, which can be recreated (forged) by writing literal string-equated expressions, have no such hazard.
So far so good, wm has a cycle but the objkey1 and objkey2 variables still keep the objects alive. Let’s cut the external references and force garbage collection:
At this point wm is empty. We can’t tell, however: there’s no way to enumerate a WeakMap, as doing so could expose the GC schedule (in browsers, you can’t call gc() to force a collection). Nor can we use wm.has to probe for entries, since we have nulled our objkey references!
A WeakMap is therefore close friends with the JS garbage collector. The GC knows when no references to a key object survive, and can collect the entry for that key — and for any cyclic entries in the table tied in a knot by their values being keys of other entries.
This special GC handling adds overhead, which ordinary Map users should not have to suffer.
What’s more, WeakMap accepts only object keys to enforce the no-forged-key rule necessary for the GC to be able to collect entries whose keys no longer survive — otherwise when could you ever GC an entry for key "if", which is typically interned along with the other JS reserved identifiers forever?
An entry with a key such as 42 or "42!" might be GC’ed if no copies of the key’s primitive value exist, even though the value could be recreated at any time (primitive types have value identity, not reference identity).
Of course, the GC cannot keep count of live instances of 42 very efficiently — or at all — depending on the JS engine’s implementation details. And strings are not observably shared via references and therefore counted, either (small ones could be copied, and are in many engines).
This is all a bit of a brain bender, and probably more than the average Map user needs to know, but the need for WeakMap compared to separate weak reference (on the ES7 radar!) and Map facilities is real. Smalltalkers discovered it decades ago, and called the weak key/value pair an Ephemeron (note: @awbjs, who witnessed the discovery, testified to me that the wikipedia page’s credits are incomplete).
Proxy
The draft ES6 spec has evolved since Proxies were first prototyped, but the good news is that the new Proxy spec can be implemented on the old one (which was prototyped in SpiderMonkey and V8) via Tom Van Cutsem‘s harmony-reflect library. The even better news is that the built-in direct proxies implementation has just landed in SpiderMonkey.
Tom’s __noSuchMethod__ implementation using direct proxies:
With this approach, you have to insert MethodSink just before the end of the prototype chain of an object that wants __noSuchMethod__‘s magic, using the __proto__de facto standard that will be a de jure standard in ES6. The Object.prototype.__noSuchMethod__ backstop throws to catch bugs where the MethodSink was not on a receiver’s prototype chain.
This implementation does not just call the __noSuchMethod__ hook when a missing method is invoked, as shown after the obj.bar() line above. It also creates a thunk for any get of a property not in the target object and not in Object.prototype:
I think this is an improvement on my original__noSuchMethod__ creation all those years ago in SpiderMonkey.
(Avid SpiderMonkey fans will cheer the switch to source recovery from decompilation evident in the result from Function.prototype.toString when evaluating obj.bar, thanks to Benjamin Peterson’s fine work in bug 761723.)
RegExp sticky (y) flag
This flag causes its regular expression to match in the target string starting from the index held in the lastIndex property of the regexp. Thus ^ can match at other than the first character in the target string. This avoids O(n2) complexity when lexing a string using a regexp, where without y one would have to take successive tail slices of the string and match at index 0.
String startsWith, endsWith, contains
These explain themselves by their names and they’re pretty simple, but also handier and more readable than the equivalent indexOf and lastIndexOfexpansions.
Number isNaN, isFinite, toInteger, isInteger
The first two are not super-exciting, but worthwhile to avoid implicit conversion mistakes in specifying the isNaN and isFinite global functions which date from ES1 days:
True fact: isNaN(" ") returns false because a string containing spaces converts (I was influenced by Perl; hey, it was the ’90s!) to the number 0, which sure enough is not a NaN. Dave Herman used this to good effect in the fun bonus segment of his BrazilJS talk.
The Integer static methods also avoid implicitly converting non-numeric arguments (e.g., "foo" to NaN). Their main purpose is to provide built-in IEEE-754 integral-double handling:
Notice how once you exceed the bits in IEEE double’s mantissa, Number.isInteger may return true for what you might wish were precise floating point results. Better luck in ES7 with value objects, which would enable new numeric types including IEEE-754r decimal.
Older prototype implementations
SpiderMonkey of course supports many Harmony proposals implemented based on ES4 or older drafts, e.g., const, let, generators, and destructuring. These will be brought up to spec as we hammer out ES6 on the anvil of Ecma TC39, heated in the forge of es-discuss, and user-tested in Firefox, Chrome, and other leading browsers. I hope to blog soon about other progress on the ES6 and ES7 “Harmony” fronts. A few clues can be found near the end of my Strange Loop slides.
/be
PS: I colorized the code examples in this post using the fun Prism.js library adapted lightly to ES6. Many thanks to Lea Verou, et al., for Prism.js!
I gave two talks recently, first at O’Reilly Media’s go-big-with-JavaScript FluentConf, and then at my favorite regional JS conference, the delightful TXJS (gorgeous site design), curated and stage-managed by Alex Sexton, Rebecca Murphey, and other usual suspects.
My Fluent video was up in record time, one achievement that the O’Reilly folks can brag about:
There I played the JS clip from Gary Bernhardt‘s hilarious Wat lightning talk, and gave bleeding-edge demos that my TXJS talk updated (video link for TXJS to be posted as soon as it’s available).
At TXJS I mixed new metaphors like mixed drinks, harkened back to my last year’s TXJS talk, and gave the latest demos.
The big-picture story is renewed humility in the face of community feedback. My goal is to help the JS standards body be the best it can be as one of several symbiotic critters in an evolving system.
The spiral lives, and it works — although newcomers to es-discuss sometimes think consensus has been achieved within one 16-message thread. Not so fast! But consensus on ES6 is being achieved.
Lots of learning and re-learning here:
Versioning is an anti-pattern on the web, wherefore 1JS.
Syntax as programming language user-interface should evolve, but unlike standard library additions, developers can’t fix it or abstract over it (no macros yet, sigh). I made JS’s object model mutable at the start so people could polyfill, and they continue to do so for good reason. New syntax has to be minimal, righteous, and user-tested to get into Harmony, and that’s the way it should be.
My old “dunder-proto” (LOL, @littlecalculist was inspired in this pronunciation of __proto__ by Dunder-Mifflin) vanquished triangle (ugly typography didn’t help triangle either).
People still rant about JS’s privileged client-side status, but it’s a curse more than a blessing (Python, Lua, Ruby all would have been frozen badly if wedged into Netscape 2 in 1995).
The only hope is mutualism in an evolutionary struggle toward something better than either TC39 or JS developers could achieve by themselves. I tend to believe that this struggle will end well, better than master-planned would-be alternatives that don’t have a prayer of catching on as native-VM-implemented-yet-cross-browser on the Web as currently constituted.
JSFixed represents both a cry from the heart by some in the JS developer community, and an attractor for bikeshedding and tons of noise. But with excellent curation by @angustweets, @valueof, @rwaldron, and @KitCambridge, it has produced a reasonable set of recommendations. Now it’s up to TC39 to engulf the recommendations that it can stomach, as if we were a hungry cell and they a primitive bacterium, and then for both sides to find mutual wins in the resulting ensemble.
These are straightforward, but I expanded on maximin classes in the next slide.
Only => (fat arrow) made it into ES6; thin arrow may be an arrow too far, but I will try again at the July TC39 meeting.
The existential operator, ?., is relatively straightforward. I will put it on the agenda for July. It could fit in ES6, IMHO, without breaking any budgets or agreements.
I intend to write up Object.prototype.forEach as a strawman, based on JSFixed’s recommendation. It relies on Object.keys order, which in turn is for-in order, but no big deal. It’s a useful object method, shadowed by Array.prototype.forEach.
JS in 2gyr? More like at birth + 35yr. CoffeeScript forks off and rejoins with Ruby bacterial DNA engulfed. Is that Dart near the bottom? 😛
Evolution does sometimes paint a clade into a dead-end corner, or leave the evolved system with harsh dilemmas and “pick any two” trilemmas.
The quest for shorter function syntax runs afoul of this binding. As I said in my talk, if we are counting fairly, then => and -> are not a single short function syntax, they are two syntaxes grouped by being “arrows” or having two chars ending in >.
This slide refers to the default operator proposal for Harmony, which I’ve recently edited based on several rounds of es-discuss and twitter feedback. It is looking good, IMHO, with the only remaining issue (also open for parameter default values) of whether null as well as undefined should trigger defaulting.
Somehow, @rmurphey knew I would be speaking about Unicorns.
ES6 is already partly implemented in top browsers, and it is coming to more browsers soon. Time to start experimenting with it and giving feedback.
So yeah: 3D games, Flash, and C/C++ in JS. Can you dig it?
I close by praising Wesley Snipes again (this time with a jwz memory), and taunting those who doubt either JS or Passenger 57.
I do not taunt for the sake of JS in itself, which started life as good-not-great where it wasn’t just silly. Rather, for JS as an over-maligned, unique evolving system that somehow still unites a standards group (made of competing browser vendors) with faithful JS developers, working together for a better symbiosis.
No other language has the curse or the blessing of this fate. Let’s do our best. Thanks to the JSFixed crew for doing theirs.
relies on Automatic Semicolon Insertion (ASI) and so cannot be minified except by parsing fully (including ASI), observing the significance of the newline after clearMenus(), and inserting a semicolon when stripping that newline.
FWIW, I agree with Doug’s canonically grumpy tone if not his substance; more below on the substance.
I also agree with @cramforce and @jedschmidt that the && line is an abusage, allowed due to JS’s C heritage by way of Java, but frowned upon by most JS hackers; and that an if statement would be much better style (and, I take it, help JSMin do right). But this particular criticism is too ad hoc to help resolve the general “Let me have my ASI freedom and still minify, dammit!” debate.
TC39 is considering the use of ! as an infix operator. This code will break in the future. Fix it now. Learn to use semicolons properly. ! is not intended to be a statement separator. ; is.
The !-as-infix-operator idea is proposed as syntactic sugar for promises, which may or may not make it into Harmony with that exact syntax, or with any syntactic sugar at all.
Doug’s right that ! is not a statement terminator or “initiator”. And (my point here), neither is newline.
But search for [nlth] in the proposed promises grammar and you’ll see something surprising about ASI and infix operators: we can add new infix operators in the future, whether new contextual keyword-operators (e.g., is and isnt — BTW these are in doubt) or retasked, existing unary-prefix operators, provided that we insist on [no LineTerminator here] immediately to the left of any such infix operator.
(In ECMA-262, [no LineTerminator here] is used in so-called “restricted productions” to make contextually-significant newlines, e.g., after return without any expression of the return value on the same line.)
This future-friendliness to new infix operators comes directly from ASI as a newline-sensitive error correction procedure, as the example at top demonstrates. Try other examples using a leading identifier on a well-formed second line and you’ll see the same effect. Removing the newline introduces an early error, which creates homesteading space for new infix operators in a later edition of ECMA-262. Examples:
let flag = x is y; // no n before 'is'!
x ! p = v; // Q(x).put(’p’, v)
An aside on coding style: if we add new infix operators used in restricted productions, this gives weight to the JS coding style that puts infix operators in multiline expressions at the end of continued lines, rather than at the beginning of continuation lines.
So while I agree with Doug on those two lines of code from Bootstrap (an excellent JS library, BTW) exhibiting poor style, it is not the case that such code as written could break in the future, even if we were to adopt the !-as-infix-operator strawman. The first line terminator in that example is indeed significant.
The moral of this story: ASI is (formally speaking) a syntactic error correction procedure. If you start to code as if it were a universal significant-newline rule, you will get into trouble. A classic example from ECMA-262:
a = b + c
(d + e).print()
Similar hazards arise with [, /, and unary + and -. Remember, if there wasn’t an error, ASI does not apply.
This problem may seem minor, but JS file concatenation ups the ante. For this reason some style guides (Dojo, IIRC) advocate starting your reusable JS file with ;, but people don’t know and it’s easy to forget.
I wish I had made newlines more significant in JS back in those ten days in May, 1995. Then instead of ASI, we would be cursing the need to use infix operators at the ends of continued lines, or perhaps \ or brute-force parentheses, to force continuation onto a successive line. But that ship sailed almost 17 years ago.
The way systematic newline significance could come to JS is via an evolution of paren-free that makes it to Harmony status. I intend to work on this in the strawman, but not for ES6.
Some of the github issue comments are naive or idealistic to the point of being silly. Since when does any programming language not have syntax arguments? All living, practical languages that I know of, even those with indentation-based block structure and similar restrictions, have degrees of freedom of expression that allow abusage as well as good usage. Language designers can try to reduce degrees of freedom, but not eliminate them completely.
My two cents: be careful not to use ASI as if it gave JS significant newlines. And please don’t abuse && and || where the mighty if statement serves better.
I’ll also say that if it were up to me, in view of JS’s subtle and long history, I’d fix JSMin. But I would still log a grumpy comment or two first!
If we start to try to make “Mozilla” mean “those people who share not only the Mozilla mission but also my general political / social / religious / environmental view” we will fail. If we focus Mozilla on our shared consensus regarding the Mozilla mission and manifesto then the opportunities before us are enormous.
Mozilla’s diversity is a success condition. Our mission and our goal is truly global. Our mission taps into a shared desire for respect and control and user sovereignty that runs across cultures and across many other worldviews. We may even offend each other in some of our other views. Despite this, we share a commitment to the Mozilla mission. This is a remarkable achievement and important to our continued success.
I agree with every word of this, and I believe it applies to other communities of which I’m a member. If not, these communities will tend to splinter, and that is likely to be a net loss for everyone.
Background
A donation that I made in support of California Proposition 8 four years ago became publicknowledge and sparked a firestorm of comments in the last few days, mostly on Twitter.
People in other countries or other U.S. states do not know why “Mozilla” was listed in the donation data. Donors above a certain amount are required by the State of California to disclose their employer. Mozilla had nothing to do with the donation.
I’m not going to discuss Prop 8 here or on Twitter. There is no point in talking with the people who are baiting, ranting, and hurling four-letter abuse. Personal hatred conveyed through curse words is neither rational nor charitable, and strong feelings on any side of an issue do not justify it.
In contrast, people expressing non-abusive anger, sadness, or disagreement, I understand, grieve, and humbly accept.
No Hate
Ignoring the abusive comments, I’m left with charges that I hate and I’m a bigot, based solely on the donation. Now “hate” and “bigot” are well-defined words. I say these charges are false and unjust.
First, I have been online for almost 30 years. I’ve led an open source project for 14 years. I speak regularly at conferences around the world, and socialize with members of the Mozilla, JavaScript, and other web developer communities. I challenge anyone to cite an incident where I displayed hatred, or ever treated someone less than respectfully because of group affinity or individual identity.
Second, the donation does not in itself constitute evidence of animosity. Those asserting this are not providing a reasoned argument, rather they are labeling dissenters to cast them out of polite society. To such assertions, I can only respond: “no”.
If we are acquainted, have good-faith assumptions, and circumstances allow it, we can discuss 1:1 in person. Online communication doesn’t seem to work very well for potentially divisive issues. Getting to know each other works better in my experience.
The Larger Point
There’s a larger point here, the one Mitchell made: people in any group or project of significant size and diversity will not agree on many crucial issues unrelated to the group or project.
I know people doing a startup who testify that even at fewer than 20 employees, they have to face this fact. It’s obviously true for much larger communities such as JS and Mozilla. Not only is insisting on ideological uniformity impractical, it is counter-productive.
So I do not insist that anyone agree with me on a great many things, including political issues, and I refrain from putting my personal beliefs in others’ way in all matters Mozilla, JS, and Web. I hope for the same in return.
