8 October 2012

Harmony of Dreams Come True

This blog focuses on portions of the new-in-ES6 stuff I presented in my Strange Loop 2012 closing keynote, which was well-received (reveal.js-based HTML slides, some from my Fluent 2012 keynote, many of those originally from Dave Herman‘s Web Rebels 2012 talk [thanks!], can be viewed here; notes courtesy Jason Rudolph).

UPDATE: the Strange Loop keynote video is up.


I blogged early in 2011 about Harmony of My Dreams, to try to fit in one page some dream-sketches (if not sketchy dreams — the #-syntax ideas were sketchy) of what I thought were the crucial elements of ECMAScript Harmony, the name I coined for the standardized future of JavaScript.

Now this dream is coming true, not just in ES6 draft specs but in prototype implementations in top browsers. Here I’ll tout Firefox 15, which released almost six weeks ago (yes, this means Firefox 16 is tomorrow, and Firefox 17 beta and 18 aurora too — these all have yet more new goodies in them — isn’t Rapid Release fun?). Per the MDN docs, the SpiderMonkey JS engine shipped in Firefox 15 sports the following new prototype-implemented draft ES6 features:

Default parameters

This extension (AKA “parameter default values”) is too sweet, and it will help put the arguments object out to pasture:


js> function f(a = 0, b = a*a, c = b*a) { return [a, b, c]; }
js> f()
[0, 0, 0]
js> f(2)
[2, 4, 8]
js> f(2, 3)
[2, 3, 6]
js> f(2, 3, 4)
[2, 3, 4]

Implementation credit goes to Benjamin Peterson for his work implementing default parameters, and to Jason Orendorff for his always-excellent code reviews. See this bug for followup work to track the latest ES6 agreement on how passing undefined (and only undefined) should trigger defaulting.

We have a few details to iron out still about scope, I suspect (based on this es-discuss message and its thread).

Rest parameters

Even sweeter than default parameters are rest parameters, and I bet they are ahead of default parameters in making arguments a bad memory some fine day:


js> function f(a, b, ...r) { print(Array.isArray(r)); return r.concat(a, b); }
js> f(1, 2)
true
[1, 2]
js> f(1, 2, 3)
true
[3, 1, 2]
js> f(1, 2, 3, 4, 5)
true
[3, 4, 5, 1, 2]

Again credit goes to Benjamin and Jason for their work.

Spread in array literals

The dual of rest is called “spread”, and it should work in call expressions as well as array literals. The latter is implemented in Firefox 16 (now in the beta channel):


js> a = [3, 4, 5]
[3, 4, 5]
js> b = [1, 2, ...a]
[1, 2, 3, 4, 5]

Thanks once again to Benjamin (a star Mozilla intern this summer) and Jason.

Spread in call expressions is not yet implemented:


js> function f(...r) { return r; }
js> function g(a) { return f(...a); }
typein:20:0 SyntaxError: syntax error:
typein:20:0 function g(a) { return f(...a); }
typein:20:0 .........................^

But I believe it is coming soon — bug 762363 is the one to watch, patch, and test.

for-of iteration

I blogged and spoke about for-of at TXJS 2011. The of contextual keyword, also in CoffeeScript, goes where in goes in for-in loops, in order to trigger the new iteration protocol (which is based on Python’s).


js> for (var v of [1, 2, 3]) print(v)
1
2
3

Arrays are iterable out of the box in ES6. This is a huge usability win! Unwary newcomers hoping for Pythonic value iteration can now avoid the trap of for-in on arrays iterating string-coerced keys rather than values.

Objects are not iterable without the programmer opting in explicitly:


js> for (var [k, v] of {p: 3, q: 4, r: 5}) print(k, v)
typein:24:0 TypeError: ({p:3, q:4, r:5}) is not iterable

To opt in, call an iterator factory, that is, a function that returns a fresh iterator for its parameter. Or simply give your objects or their common prototype an iterator property whose value is an iterator factory method: a function that returns the desired fresh iterator given its this parameter.

We require opt-in to avoid future-hostility against custom iterators for collection objects. Such objects probably do not want any kind of general property iterator default, which if left on Object.prototype, might be object-detected and prevent installation of the correct custom iterator factory.

The easiest way to create such an iterator factory is to write a generator function:


js> function items(o) { for (var k in o) yield [k, o[k]]; }
js> for (var [k, v] of items({p: 3, q: 4, r: 5})) print(k, v)
p 3
q 4
r 5

(This example uses destructuring, too.)

Note that SpiderMonkey has not yet implemented the ES6 generator function* syntax. We also haven’t added the ES6 features of delegating to a sub-generator via yield* and of returning a value from a generator (as in PEP 380). We’ll get to these soon.

Map

Have you ever wanted to map from arbitrary keys to values, without having the keys be implicitly converted to strings and therefore possibly colliding? ES6 Map is for you:


js> var objkey1 = {toString: function(){return "objkey1"}}
js> var objkey2 = {toString: function(){return "objkey2"}}
js> var map = Map([[objkey1, 42], [objkey2, true]])
js> map.get(objkey1)
42
js> map.get(objkey2)
true

The Map constructor takes any iterable, not just an array, and iterates over its key/value array-pairs.

Of course you can update a Map entry’s value:


js> map.set(objkey1, 43)
js> map.get(objkey1)
43

And you can add new entries with arbitrary key and value types:


js> map.set("stringkey", "44!")
js> for (var [k, v] of map) print(k, v)
objkey1 43
objkey2 true
stringkey 44!
js> map.size()
3

You can even use a key as a value:


js> map.set(objkey2, objkey1)
js> map.set(objkey1, objkey2)
js> for (var [k, v] of map) print(k, v)
objkey1 objkey2
objkey2 objkey1
hi 44
stringkey 44!

but now there’s a cycle between the objkey1 and objkey2 entries. This will tie up space in the table that must be manually released by breaking the cycle (or by dropping all references to the map):


js> map.delete(objkey1)
true
js> map.delete(objkey2)
true
js> for (var [k, v] of map) print(k, v)
hi 44
stringkey 44!

Setting the objkey1 and objkey2 variables to null is not enough to free the space in map tied up by the cycle. You must map.delete.

If your map is not exposed via an API by which arbitrary values could be passed as key and value to map.set, you won’t have to worry about cycles. And if the map itself becomes garbage soon (for sure), no worries. But for leak-proofing with arbitrary key/value cycles, see WeakMap, below.

Set

When you just want a set of arbitrary values, it’s a pain to have to use a map and burn code and memory on useless true values for the keys. So ES6 also offers Set:


js> var set = Set([1, true, "three"])
js> set.has(1)
true
js> set.has(2)
false
js> for (var e of set) print(e)
1
true
three
js> set.size()
3

As with Map, with a Set you can delete as well as add:


js> set.delete("three")
true
js> for (var e of set) print(e)
1
true
js> set.size()
2
js> set.add("three")
js> set.size()
3

An object element keyed by its identity works just as well as any other type of element.


js> var four = {toString: function(){return '4!'}}
js> set.add(four)
js> set.has(four)
true
js> for (var e of set) print(e)
1
true
three
4!

Unlike Map there is no cyclic leak hazard with arbitrary elements, although a WeakSet taking only object elements would still be helpful for automatic element removal when no other references to an element object remain. This idea has come up in connection with proxies and symbols, but I’ll save that for another post.

WeakMap

As noted above, with Map, making a cycle among map keys and values can tie up space in the table, and in the heap in all objects linked along the cycle or reachable from those objects, even when no references outside of the table to the key objects still live. Non-object keys, which can be recreated (forged) by writing literal string-equated expressions, have no such hazard.

ES6 WeakMap rides to the rescue:


js> var wm = WeakMap()
js> wm.set(objkey1, objkey2)
js> wm.set(objkey2, objkey1)
js> wm.has(objkey1)
true
js> wm.get(objkey1)
({toString:(function (){return "objkey2"})})
js> wm.has(objkey2)
true
js> wm.get(objkey2)
({toString:(function () {return 'objkey1'})})

So far so good, wm has a cycle but the objkey1 and objkey2 variables still keep the objects alive. Let’s cut the external references and force garbage collection:


js> objkey1 = null
null
js> gc()
"before 286720, after 282720n"
js> wm.get(objkey2)
({toString:(function () {return 'objkey1'})})
js> objkey2 = null
null
js> gc()
"before 286720, after 282624n"

At this point wm is empty. We can’t tell, however: there’s no way to enumerate a WeakMap, as doing so could expose the GC schedule (in browsers, you can’t call gc() to force a collection). Nor can we use wm.has to probe for entries, since we have nulled our objkey references!

A WeakMap is therefore close friends with the JS garbage collector. The GC knows when no references to a key object survive, and can collect the entry for that key — and for any cyclic entries in the table tied in a knot by their values being keys of other entries.

This special GC handling adds overhead, which ordinary Map users should not have to suffer.

What’s more, WeakMap accepts only object keys to enforce the no-forged-key rule necessary for the GC to be able to collect entries whose keys no longer survive — otherwise when could you ever GC an entry for key "if", which is typically interned along with the other JS reserved identifiers forever?

An entry with a key such as 42 or "42!" might be GC’ed if no copies of the key’s primitive value exist, even though the value could be recreated at any time (primitive types have value identity, not reference identity).

Of course, the GC cannot keep count of live instances of 42 very efficiently — or at all — depending on the JS engine’s implementation details. And strings are not observably shared via references and therefore counted, either (small ones could be copied, and are in many engines).

This is all a bit of a brain bender, and probably more than the average Map user needs to know, but the need for WeakMap compared to separate weak reference (on the ES7 radar!) and Map facilities is real. Smalltalkers discovered it decades ago, and called the weak key/value pair an Ephemeron (note: @awbjs, who witnessed the discovery, testified to me that the wikipedia page’s credits are incomplete).

Proxy

The draft ES6 spec has evolved since Proxies were first prototyped, but the good news is that the new Proxy spec can be implemented on the old one (which was prototyped in SpiderMonkey and V8) via Tom Van Cutsem‘s harmony-reflect library. The even better news is that the built-in direct proxies implementation has just landed in SpiderMonkey.

Tom’s __noSuchMethod__ implementation using direct proxies:


js> var MethodSink = Proxy({}, {
  has: function(target, name) { return true; },
  get: function(target, name, receiver) {
    if (name in Object.prototype) {
      return Object.prototype[name];
    }
    return function(...args) {
      return receiver.__noSuchMethod__(name, args);
    }
  }
});
js> void Object.defineProperty(Object.prototype,
  '__noSuchMethod__',
  {configurable: true, writable: true, value: function(name, args) {
    throw new TypeError(name + " is not a function");
  }});
js> var obj = { foo: 1 };
js> obj.__proto__ = MethodSink;
({})
js> obj.__noSuchMethod__ = function(name, args) { return name; };
(function (name, args) { return name; })
js> obj.foo
1
js> obj.bar()
"bar"
js> obj.toString
function toString() {
    [native code]
}

With this approach, you have to insert MethodSink just before the end of the prototype chain of an object that wants __noSuchMethod__‘s magic, using the __proto__ de facto standard that will be a de jure standard in ES6. The Object.prototype.__noSuchMethod__ backstop throws to catch bugs where the MethodSink was not on a receiver’s prototype chain.

This implementation does not just call the __noSuchMethod__ hook when a missing method is invoked, as shown after the obj.bar() line above. It also creates a thunk for any get of a property not in the target object and not in Object.prototype:


js> obj.bar
(function (...args) {
      return receiver.__noSuchMethod__(name, args);
    })
js> var thunk = obj.bar
js> thunk()
"bar"

I think this is an improvement on my original __noSuchMethod__ creation all those years ago in SpiderMonkey.

(Avid SpiderMonkey fans will cheer the switch to source recovery from decompilation evident in the result from Function.prototype.toString when evaluating obj.bar, thanks to Benjamin Peterson’s fine work in bug 761723.)

RegExp sticky (y) flag

This flag causes its regular expression to match in the target string starting from the index held in the lastIndex property of the regexp. Thus ^ can match at other than the first character in the target string. This avoids O(n2) complexity when lexing a string using a regexp, where without y one would have to take successive tail slices of the string and match at index 0.

String startsWith, endsWith, contains

These explain themselves by their names and they’re pretty simple, but also handier and more readable than the equivalent indexOf and lastIndexOf expansions.

Number isNaN, isFinite, toInteger, isInteger

The first two are not super-exciting, but worthwhile to avoid implicit conversion mistakes in specifying the isNaN and isFinite global functions which date from ES1 days:


js> Number.isNaN("foo")
false
js> isNaN("foo")
true

True fact: isNaN(" ") returns false because a string containing spaces converts (I was influenced by Perl; hey, it was the ’90s!) to the number 0, which sure enough is not a NaN. Dave Herman used this to good effect in the fun bonus segment of his BrazilJS talk.

The Integer static methods also avoid implicitly converting non-numeric arguments (e.g., "foo" to NaN). Their main purpose is to provide built-in IEEE-754 integral-double handling:


js> Math.pow(2,53)/3
3002399751580330.5
js> Number.isInteger(Math.pow(2,53)/3)
false
js> Math.pow(2,54)/3
6004799503160661
js> Number.isInteger(Math.pow(2,54)/3)
true

Notice how once you exceed the bits in IEEE double’s mantissa, Number.isInteger may return true for what you might wish were precise floating point results. Better luck in ES7 with value objects, which would enable new numeric types including IEEE-754r decimal.

Older prototype implementations

SpiderMonkey of course supports many Harmony proposals implemented based on ES4 or older drafts, e.g., const, let, generators, and destructuring. These will be brought up to spec as we hammer out ES6 on the anvil of Ecma TC39, heated in the forge of es-discuss, and user-tested in Firefox, Chrome, and other leading browsers. I hope to blog soon about other progress on the ES6 and ES7 “Harmony” fronts. A few clues can be found near the end of my Strange Loop slides.

/be

PS: I colorized the code examples in this post using the fun Prism.js library adapted lightly to ES6. Many thanks to Lea Verou, et al., for Prism.js!

24 Responses to “Harmony of Dreams Come True”

  1. kc says:

    ES6 looks great. Some of the ES5 stuff now looks like premature standardisation. definePropertyBlah..

    Fat arrow functions with lexical |this|?

    Are JSC on board?

    Be useful to have a Google doc spreadsheet showing the feature and implementation status for the various engines which are implementing/prototyping before final standardisation.

  2. Brendan Eich says:

    @KC: apart from some default reversed-boolean issues, ES5′s meta-object API is working out well and Proxies build on it in ES6. Before ES5 there was no way to define non-enumerable properties, for example. ES5 filled that gap and modulo usability glitches did it well enough.

    Fat arrow with lexical |this| is in ES6. I left it out to focus on APIs and compilation, but x => x (example identity function) is “in”.

    Apple participates and is on board. Some implementation effort there, even: http://trac.webkit.org/search?q=es6.

    @kangax did a nice ES5 “Spreadsheet” (just a big HTML table) tracking implementation across the browsers. It would be nice to have that for ES6. Maybe he’ll do it, but it’s fair game for anyone following es-discuss and able to test.

    /be

  3. > but x => x (example identity function) is “in”.

    BTW, did anyone ever finish the push for no-parens in the no-args case? E.g. `=> 5`

  4. Brendan Eich says:

    @kangax already had an ES6 page, and just updated it:

    http://kangax.github.com/es5-compat-table/es6/

    Huzzah!

    /be

  5. Brendan Eich says:

    @Domenic: some TC39ers were against CoffeeScript’s paren-free zero args shorthand, and to reach consensus I cut it.

    /be

  6. Tom Hunt says:

    I love the new Set type but it would be nice if it contained some commonly used set operations like union, intersection, difference, etc. It seems like I’ll have to define this on my own to make this type useful and would prefer a native implementation.

  7. Brendan Eich says:

    @Tom: good point, and worth a post to es-discuss. You or me? You have my endorsement, so feel free to go first.

    /be

  8. Kevin C says:

    Declarative syntax for map and set would be nice. Maybe with macros post ES6.

    // map & set are macros. expand to api calls.
    let mymap = map {‘one’:1, ‘two’: 2};
    let myset = set [4,5,6];

  9. Brendan Eich says:

    @Kevin: Map allows any value as key, so an object literal is not expressive enough.

    But you’re right about macros — they can specialize any balanced delimited form. The list of pairs approach the Map constructor takes could be sugared with macrology, possibly even to allow arbitrary key type in a “map literal”.

    /be

  10. A Lee Wade says:

    You mentioned no paren-free for Coffeescript’s zero arg shorthand, but what about your if/for/while paren-free proposal from last year? Any chance that will make it in ES7 ?

  11. Kevin C says:

    ‘specialize any balanced delimited form’

    Very useful.

    Any value as key – new and/or Call parens enough to distinguish?:

    let mymap = map {new Point(4,4): 4, new Point(5,5):5 };

    // and to ref variables in the lexical env
    let x,y = 1,2;
    let mymap = map {(x):1, (y):2 }

  12. Kevin C says:

    Generators/Comprehensions – maybe lead with the ‘for’. Deviates from Python but i find it more readable. Easier to parse?

    let comp = [for (i of lst) i*i];
    let gen = (for (i of lst) i*i);

    // without parens
    let gen = (for i of lst => i * i);

  13. Has there been any discussion of how the new types would work with JSON? Say I create a Set and want to store the Set in localStorage, so I use JSON.stringify(), I assume it would look like a standard Array. But when I use JSON.parse(), how would I know to convert the data back to a Set?

  14. As a side note, currently Firefox will turn a Set into an empty object if I use JSON.stringify(). Which means I would need to manually convert it to an Array beforehand. e.g.:

    JSON.parse(JSON.stringify({ha:Set([1,true,'hi']})) === {ha:{}};

  15. Brendan Eich says:

    @Trevor: see the thread starting at Nicholas C. Zakas’s post:

    https://mail.mozilla.org/pipermail/es-discuss/2012-October/025478.html

    JSON does not by itself express sets or maps, but you can add your own replacer (on stringify) and reviver (on parse).

    /be

  16. Ric Johnson says:

    Brendan,
    I am glad to see these features finally start to come to fruition after so many years! I do have a question- how can we make sure that older browsers do not have errors? With HTML5, we can do progressive enhancement to detect browser features, but that will not be possible when we change the language itself. I saw some mention on the Wiki that the script tag may get changed with a version attribute but I did not know if that was decided. Projects like Traceur or Typescript may let me write ES6 code today and translate it to valid javascript for the current clients, but how do we do inside the browser?
    {name: Ric}

  17. [...] For Dart to fulfill its wider ambitions, it would have to be adopted by all of the major browser vendors. Here are three reasons why this is not going to happen. 1) Apple and Google are bitter rivals at this point. 2) Microsoft has come out with its own solution to what it perceives as JavaScript's shortcomings — TypeScript — and delivered Dart a dismissive backhand when introducing it. 3) Firefox is well down the road towards implementing the standards-track ECMAScript 6. [...]

  18. Brendan Eich says:

    @Ric: the way to develop in ES6 and support older browsers in a year will involve compiling to ES5 or ES3, via tools such as

    http://code.google.com/p/traceur-compiler/
    https://github.com/matthewrobb/six

    and the like (you mentioned TypeScript, it adds value on top of ES6 and may have to change to track ES6, but it’s another option).

    There is no <script> version attribute value for ES6, although RFC 4329 did mandate that a version attribute must not be ignored, yet without defining its values. The “1JS” breakthrough early this year, see Dave Herman’s post:

    https://mail.mozilla.org/pipermail/es-discuss/2011-December/019112.html

    and followups, means you use ES6 just like you used ES5 and ES3 when they were new: new APIs can be detected and even polyfilled some of the time (people are forward-polyfilling already). New syntax must be avoided unless you down-compile.

    /be

  19. Yoshihisa Kimura says:

    Seems to be wrong in one place for me.

    s/map.delete(objkey1)/map.delete(objkey2)/

  20. Brendan Eich says:

    @Yoshihisa: thanks, fixed.

    /be

  21. MJ Kim says:

    Hi, Brendan Eich.

    HTML5 developers working in South Korea, MJ, Kim.

    I would like to put your blog post on my blog translated into Korean.

    You registered Post:https://brendaneich.com/2012/10/harmony-of-dreams-come-true/

    Translated into English (URL): http://www.html5dev.kr/401

    This translation can be published on my blog, by allowing Would you like to share?

    Thank you.

  22. I see one potential problem with the “rest” parameters. From here:
    http://wiki.ecmascript.org/doku.php?id=harmony:rest_parameters
    Rest parameters “Capture trailing arguments after one or more fixed formal parameters”

    Why require the first fixed parameter?

    I use the arguments object for logging. I have code that automatically iterates through all of my functions, and replaces them with a wrapper method. The wrapper method looks at the arguments object, and logs all the arguments to the method, it then calls the original method using the arguments object.

    In this way the arguments object is really useful for meta-programming, and doing things you would need macros for in LISP. For meta-programming, requiring the first fixed parameter though is actually a step back from the relatively clean interface of arguments.

    As a side note, another thing that has made meta-programming javascript more difficult is the addition of proxies, and the ability to “freeze” objects. This makes it more difficult to traverse a generic object graph (for things like serialization, logging), since you can’t count on the ability to flag the object as traversed. It would be really nice of there were an API to traverse an object graph to get around this limitation, i.e.
    function BFSTraversal(object, callback);
    function DFSTraversal(object, callback);

    Presumably a javascript implementations already have to implement something like this for the sake of JSON.stringify (which is specified to detect graph cycles), so it would be nice to get at the underlying, efficiently implemented browser graph traversal algorithm.

  23. Jean Bonbeur says:

    I think that Javascript is a burning platform.
    And sadly, ES6 Harmony is not what people are waiting and expecting to help them build large web applications.

    Look at Dart, TypeScript, CoffeeScript… what we want is classes, optional types, sane & cleaned up semantics, ide help, tooling, etc…

    I wonder how, after 10years of failure, the ECMAScript committee still continues to deny reality. But the world has changed and Javascript’s future looks grim now.

  24. Brendan Eich says:

    Jean: JS is the most popular language on github.com, with booming indie-developer, big-company-recruits-the-talent, JS conference, and JS book businesses. If that’s a burning platform, then all I can say is: burn, baby, burn! :-P

    TypeScript is quite a good piece of work for Visual Studio users, and smartly aligned with ES6. I’ll have more to say about it, but you shouldn’t carelessly mix it up in a list with Dart.

    CoffeeScript is pure JS runtime semantics, just syntactic sugar. And much of that sugar is going into ES6 (some was based on ES4, as a matter of fact): destructuring, comprehensions, => functions, classes.

    You seem not to be paying attention to ES6, BTW. Classes (a la CoffeeScript) are *in*.

    Finally, your “10 years of failure” is obnoxious in light of Microsoft sitting on the web with the IE monopoly, which shut down the Ecma TC39 (TG1) group in 2003. We had to revive browser competition with Firefox starting in 2004 to get the Ecma process restarted. While ES4 failed, ES5 is a success story, and ES6 is already wanted by developers, prefigured by compilers such as https://github.com/matthewrobb/six and http://traceur-compiler.googlecode.com/, and being prototype-implemented in top browsers.

    So do try to pay attention and make your negative comments a bit more accurate next time.

    /be