Brendan Eich

Chris,

You seem to be repeating falsehoods in blogs since the Proposed ECMAScript 4th Edition Language Overview was published, claiming dissenters including Microsoft were ignored by me, or “shouted down” by the majority, in the ECMAScript standardization group. Assuming you didn’t know better, and someone was misinforming you, you (along with everyone reading this letter) know better now. So I’ll expect to see no more of these lies spread by you.

Everything that has been done to advance JavaScript to the proposed ECMAScript 4th Edition (ES4) in Ecma TC39-TG1 has been in the open for well over a year, first via the es4-discuss mailing list (complete archive) and exports of our group’s wiki, along with public talks I’ve given. Then by ecmascript.org hosting the reference implementation for ES4 as proposed, along with our public trac database. And finally by the whole wiki being fully opened up and hosted live — including Microsoft’s “ES3.1” proposals.

These ES3.1 proposals came suddenly in March after years of passive participation by Microsoft, while the rest of the group worked on ES4. They have not been touched since April. They weren’t published via wiki export at that time, by agreement among all parties in TG1 including Microsoft and Doug Crockford. For one thing, they were (and still are) extremely rough and incomplete, with unanswered comments.

Almost all of the non-deprecation ES3.1 proposals were lifted directly from ES4 proposals (see my comment here).

One document reaffirms an agreement made in March: any ES3.1 candidate specification would be developed as a subset of the ES4 reference implementation, so it could be tested mechanically for forward and backward compatibility. This agreement has not been kept.

In February, an early warning was given by Microsoft, and met with vigorous debate (no shouting). Readers should take the time to follow this link, because I think it gets to the heart of the dispute, which is superficially over whether ECMAScript should remain small and mostly the same as it is in Edition 3. The pattern of general assertions about small being beautiful and sufficient for web application developers’ needs, met by specific arguments listing use-cases where JS does not scale in time or space, or lacks basic data integrity, type safety, and programming in the large support, in turn met by absolutely zero specific counter-arguments — this is a pattern we have seen over and over in TG1 this year.

The small-is-beautiful generalization alternates with don’t-break-the-web, again without specifics in reply to specific demonstrations of compatibility. Oh, and sometimes fix-security-first is put forth, without sound and sufficient solutions. At best, we have a fundamental conflict of visions and technical values between the majority and the minority. However, the obvious conflict of interest between the standards-based web and proprietary platforms advanced by Microsoft, and the rationales for keeping the web’s client-side programming language small while the proprietary platforms rapidly evolve support for large languages, does not help maintain the fiction that only clashing high-level philosophies are involved here.
Readers may not know that Ecma has no provision for “minor releases” of its standards, so any ES3.1 that was approved by TG1 would inevitably be given a whole edition number, presumably becoming the 4th Edition of ECMAScript. This is obviously contentious given all the years that the majority of TG1, sometimes even apparently including Microsoft representatives, has worked on ES4, and the developer expectations set by this long-standing effort.

A history of Microsoft’s post-ES3 involvement in the ECMAScript standard group, leading up to the overt split in TG1 in March, is summarized here.
The history of ECMAScript since its beginnings in November 1996 shows that when Microsoft was behind in the market (against Netscape in 1996-1997), it moved aggressively in the standards body to evolve standards starting with ES1 through ES3. Once Microsoft dominated the market, the last edition of the standard was left to rot — ES3 was finished in 1999 — and even easy-to-fix standards conformance bugs in IE JScript went unfixed for eight years (so three years to go from Edition 1 to 3, then over eight to approach Edition 4). Now that the proposed 4th edition looks like a competitive threat, the world suddenly hears in detail about all those bugs, spun as differences afflicting “JavaScript” that should inform a new standard.

Sorry, but most of those JScript deviations are not candidate de facto standards — they are just your bugs to fix. They should not stall ES4 for one second. I’m testifying as an eyewitness, and I have other witnesses in TG1 who can confirm what I write here. Everyone in the majority of TG1 gave time to the late-breaking dissenters from Microsoft and Yahoo!. We went out of our way to read, cross-reference, and comment on the “3.1” proposals. Activity on those proposals stopped in April.

Anyone who wants to check my claims against the TG1 meeting notes can go to the meetings page on the wiki, and follow the links. The face-to-face notes starting in March are easy to scan for political content (including my dissenting notes), and I invite anyone who is interested to have a look.

The meeting notes were edited by many hands, including Microsoft folks to their evident satisfaction, at the time of the meetings. The change logs are public in the wiki. There was no ignoring of minority positions. There was no failure to listen on the part of the majority. On the other hand I did not stop all progress on ES4 until unanimous agreement was reached on how to proceed (since Ecma and ISO do not require unanimity to make standards).

There certainly was no shouting down of the dissenters — that’s a bold lie in view of the well-attended and friendly dinners sponsored by the face-to-face meeting hosts. As the links above demonstrate, there has been no secrecy on the part of the majority of TG1, other than the secrecy requested by Microsoft and required by Ecma. Rather the opposite: Doug Crockford and Microsoft’s reps would depart for private meetings, in the middle of the March, April, and especially the May TG1 face-to-face meetings.

My TG1 Convenor’s Report, dated 4 May 2007, to Ecma TC39 (chaired by Jim Miller of Microsoft) is available to Ecma members at the Ecma members website. In it, I informed our parent TC39 committee that “[a] sub-group of TG1 is investigating possibly standardizing a smaller, compatible language, but this proposal has not yet been adopted.” Since then, the sub-group has failed to present a coherent proposal — JScript informative documents are not proposals — and my next report to Ecma will have to say so plainly.

You may have been misled, but now that everyone can see almost all of the primary sources (the remaining documents are hidden behind Microsoft, Yahoo!, and Ecma firewalls), please stop repeating falsehoods about either my conduct as TG1 Convenor, or the treatment of dissent by the majority favoring ES4 in TG1. If you have any documents to publish that shed light on Microsoft’s actions and intentions in Ecma TC39-TG1, I welcome them.

/be

From the inimitable Chris Double of Mozilla’s New Zealand brain trust: moving, rotating, scaling <video> in SVG (so who needs yet another non-standard plugin?). If you make a build from Chris’s code, don’t forget to view-source the demo.

/be

My Ajax Experience West keynote covers a lot of ground, with slant-wise truth telling the over-arching theme. Mozilla believes in fairly radical open source action, including open strategy. In that spirit, three new projects, the first already known via jresig:

ActionMonkey, the project to join forces between SpiderMonkey and Tamarin, being driven by Jason Orendorff with Edward Lee and some old-timers and newcomers lending a hand. The first stage of development integrates MMgc into SpiderMonkey as a replacement for SpiderMonkey’s GC. (NB: the wannabe mascot has a day job.)

IronMonkey, the project to map IronPython and IronRuby to Tamarin, so that Tamarin becomes multi-lingual, over time delivering high performance for all languages. This effort is being spearheaded by Seo Sanghyeon (whose last name I apologize for misspelling — or for using a different spelling of, at any rate — in my slides). The idea is to make use of the MsPL-licensed open source that Microsoft has kindly given the world, and tend necessary changes or additional code in downstream truly-open repositories that can accept patches from non-MS employees, such as FePy.

ScreamingMonkey, the project to add browser-specific script-engine glue to Tamarin, enabling it as a <script> tag handler in other browsers, starting with IE. Mark Hammond is leading this charge. Where it will end, one can only speculate. At a minimum, if you have Tamarin installed on your system in the future, you may need only a small download to plug it into other browsers. Best case, it comes with all the relevant glue in its main distribution. We’ll see.

I’m excited to announce these projects. They address various concerns voiced by bloggers of the form “what about other languages?” and “what about IE?”. Comments welcome.

/be

As Dave Herman just posted at Lambda-the-Ultimate, we ECMA members who have been working on the successor to the JavaScript standard have a new www.ecmascript-lang.org website, which now hosts the “milestone 0” release of the ES4 Reference Implementation, written in Standard ML of New Jersey.

As Dave notes, this is a “pre-release” in the Open Source, “release early and often” sense. We are not done yet, but the Reference Implementation passes over 90% of the ES3 tests that we host in cvs.mozilla.org. I should note here that the Reference Implementation already handles a great deal of ES4 in addition to most of ES3, so for example the self-hosted built-in classes are mostly there.

We know of some bugs, and we welcome well-written reports of more to fix. If you observe something that might be a bug, but you aren’t ready to file a Trac ticket, feel free to ask for advice on the es4-discuss list.

Extravagant praise to Dave for setting up the site and working on all the details of hosting the source files and binaries. Also to Graydon for tremendous work creating and developing much of the SML code. SML has been a good choice for us, although the performance of our (intentionally not very optimized) reference code can be poky. However, Dave says he is working on a MLton port, which should give a nice performance boost (MLton is an optimizing whole-program SML compiler).

This is just the beginning. Our plan is to finish the Reference Implementation over the summer and then work on specification language to surround pretty-printed excerpts of the SML and self-hosted ES4 code. At the same time, Mozilla, Adobe, and anyone who wants to help will bring up the new language on Tamarin in the new Mercurial repository. I’ll have more to say about that in a bit.

/be

Adobe open-sources Flex under the MPL — a good second step. Congratulations to all the people who made it happen.

My browser-war battle-heightened senses tell me you’ll see more open-sourcing, and no small amount of “openness” hype, from several big players in the coming days. As we mozilla.organs learned the hard way, getting the license right and hosting the code in a public repository are not enough (no pixie dust).

Successful open source projects combine meritocratic leadership, “doing” more than “talking”, and breadth through well-scoped extension mechanisms. It’s not enough to do great work by oneself: each committer who has the stamina and remains engaged must spend time listening to users and developers, grooming helpers and successors, and refactoring or even redesigning to support what becomes, module by module, a platform.

Platforms are for applications to stand upon, for building higher tiers of platform, and ultimately (in my view) for enabling user innovation. Mozilla’s platform has at least three tiers: Web, XUL, Gecko-embedded. Firefox has addons as a tier distinct from those.

Of course, closed-source platforms exist, but the successful ones hold up so much code that they end up de-facto standards, and often go open-source just to reduce costs born by the proprietor (there are better reasons than that, don’t get me wrong).

One way of evaluating Adobe’s success in open-sourcing Flex will be to look for more than just added components and widgets, and discern higher-level as well as different platform tiers. For instance, would patches to add a web-targeted compiler back end, in addition to the Flash Player SWF-targeted back end that’s there now, be accepted if they were developed?

Or how about harmonizing XUL and MXML, making changes to underlying layout and widget models on both sides in order to converge the two, so that (with trivial XML vocabulary translation) MXML could be interpreted by browsers that support XUL, as well as compiled into a SWF bound for the Flash Player?

For many observers and fans, an open-source Flash Player is the holy grail. If that is coming, it will probably take a while. But there’s a small and (I think) rising chance that “openness” competition among big players will speed up the trend at Adobe.

Then what? Obviously, forked and fragmented Player variants are not helpful, and the main distribution will be the contested ground. A better Flash Player via open-source development would be fine: good not great.

Blending Flash code into multiple browser rendering pipelines, with seamless text input (including focus) and output, an integrated DOM, full and fast scripting, and unified CSS processing would be even better. Add high-quality, unencumbered <video>, and web-friendly 3D (including an LOD-capable scene-graph model, rather than a bulky, eagerly created DOM), and you’re close to my idea of the promised land.

Adobe is moving in the right direction. That’s good enough for me today.

A small presentation I gave at SXSW asks what it means to be “Open” in the sense of open standards, open source, and a web whose major content formats are not controlled by a single vendor (I credited Nat Friedman’s amusing keynote from last year’s Free Software & Open Source Symposium for inspiration of the OED historical “Open” example).

Ted Leung (see followups) blogged about this topic already, specifically in connection with Adobe.

Ted refers to Anne Zelenka’s Why Open is Good and How Open Could Be Good for Flash post, which is worth reading, including the comments. In one comment, John Dowdell of Adobe asks whether the essence of “Open” is the ability to fork. Anne notes the loaded connotation of “fork” but agrees that John’s analysis is “not incorrect.”

I would go further. Forking is an extreme point in a continuum of options that exist with open source. The option to fork must exist as a feedback mechanism, but it need not be used in order for users to gain benefits not available with closed source and proprietary standards. Forking can be the right thing, or it can be a kind of mutually-assured-destruction option that keeps everyone acting in the interest of not forking.

Prior to even approaching a fork, open standards and open source both empower user-driven innovation. This is old news to the Mozilla user community, who have been building and feeding back innovations for the life of the project, increasing over time to include Firefox add-ons and GreaseMonkey user scripts. (BTW, I am pushing to make add-on installation not require a restart in Firefox 3, and I intend to help improve and promote GreaseMonkey security in the Firefox 3 timeframe too.) Without forking, even to make private-label Firefoxes or FlashPlayers, users can innovate ahead of the vendor’s ability to understand, codify, and ship the needed innovations.

Consider just the open standards that make up the major web content languages: HTML, CSS, DOM, JS. These mix in powerful ways that do not have correspondences in something like a Flash SWF. There is no DOM built inside the FlashPlayer for a SWF; there’s just a
display list. There’s no eval in ActionScript, and ActionScript features a strict mode that implements a static type checker (with a few big loopholes for explicit dynamic typing). You can’t override default methods or mutate state as freely as you can in the browser content model. Making a SWF is more like making an ASIC — it’s “hardware”, as Steve Yegge argues.

This is not necessarily a bad thing; it’s certainly different from the Open Web.

I assert that there is something wrong with web-like “rich” formats that aren’t hyperlink-able or indexable by search-engines. You could argue that these bugs could be fixed, and Flash is wisely becoming more URI-addressable and view-source-able over time. But it still ain’t the Web. It is not hand-authored, easily tweaked incrementally, copy-and-paste-able. It’s
hardware.

Given the stagnation of the web under IE and the failure of Java on the client, there’s a place for Flash, and I’m sure Microsoft would argue that means there’s a place for WPF/E (but I don’t see the point: either bet on the Web, or bet against it in a transcendently different way from trying to overtake Flash). If I were VP of Engineering in a 10,000 person company, I would want the security blanket of the C-like syntax and a static type system for a well-known, big-bell-curve language like AS3, C#, or Java.

And indeed AS3, C#, and Java all are marketed to big companies with hordes of “hardware” hackers creating fixed-point UIs. Dead UIs, in other words, not enlivened by user innovation. And sorry to say (to Yahoo! and Google, who may be exceptions to the rule), the big innovations don’t come from the big dumb companies.

Meanwhile, the Web is alive precisely because of the distributed extensibility of its content languages, both on the server side and in the client JS/DOM world, not to mention add-ons and GreaseMonkey.

Dare Obasanjo argues that developers crave single-vendor control because it yields interoperation and compatibility, even forced single-version support. Yet this is obviously not the case for anyone who has wasted time getting a moderately complex .ppt or .doc file working on both Mac and Windows. It’s true for some Adobe and Microsoft products, but not all, so something else is going on. And HTML, CSS, DOM and JS interoperation is better over time, not worse. TCP/IP, NFS, and SMB interoperation is great by now. The assertion fails, and the question becomes: why are some single-vendor solutions more attractive to some developers? The answers are particular, not general and implied simply by the single-vendor condition.

Implicit in my writing is the assumption (conclusion, really) that browsers can adopt the necessary advanced rendering and faster virtual-machine programming language support that the “rich client platforms” boast (or promise in version 2.0). For instance, we will support an OpenGL-ES binding for canvas in the near term, in Firefox 3 if vlad and I can help it. There’s no technical reason this can’t be interoperably supported by other browsers in the near term.

Can the Open Web keep improving enough to stave off the Closed Web? Of course, Mozilla is here to help, and we want every browser to support the nascent standards we support. But at some point the answer does depend on Microsoft upgrading IE without forking web standards from what Firefox, Safari, and Opera agree on now and in their upcoming releases, which are not yet implemented in IE7.

This will mean incompatibility for old IE content, or more painful version checking in the IE engine, or a mix. Note how Flash 9 includes both the AS2 and AS3 (Tamarin) engines, since Adobe could not keep compatibility in one engine. Gecko has had to put up with more de-facto quirks implementation than we might prefer; we sucked it up to get market share. IE can do likewise, for a change to avoid losing more market share.

Such a brute-force versioning approach could work well for Microsoft, as it did for Adobe in Flash 9. Microsoft can most afford to put more megabytes on disk via Windows Update than can browser vendors such as Mozilla and Opera, who rely on
voluntary downloads.

Anyway, I’m committed to improving the Open Web, even at the expense of Firefox market share if it comes to it (but we are still growing, and Firefox still has the second-place leverage to improve the Web quickly). If we should fail and just make a fancier browser whose nascent standards are not adopted by IE, at least we tried. The alternative is to renounce innovation and let the proprietary rich clients move the Closed Web forward. That might be convenient for some (big company) developers. It’s not in the interest of the larger developer community, or of the public, in my book.

Let me know what you think.

This is not an original thought, but I write with some authority here.

I hacked Unix kernel code out of grad school at SGI, in SGI’s “good old days” (1985-1992). Among other things, we took single-threaded (ST) kernel code and multi-threaded (MT) it on SGI’s SMP boxes. I won a free trip to New Zealand and Australia in 1990 along with Bent Hagemark, with kernel source on magtape in our hot little hands, on account of others’ bugs in adapting some single-threaded (ignoring interrupts) AT&T “Streams” code to SGI’s SMP kernel, and made fixes in the field (thanks to the SGI sales guys in Brisbane, we even got two nights on the Gold Coast in compensation — not long enough!).

You must be this tall to hack on threaded systems, and that means most programmers should run away crying. But they don’t. Instead, as with most other sharp tools, the temptation is to show how big one is by picking up the nearest ST code and jamming it into a MT embedding, or tempting race-condition fate otherwise. Occasionally the results are infamous, but too often, with only virtual fingers and limbs lost, no one learns.

Threads violate abstractions six ways to Sunday. Mainly by creating race conditions, deadlock hazards, and pessimistic locking overhead. And still they don’t scale up to handle the megacore teraflop future.

We can hope for better static analyses to find all races. In the real world, the code is C or C++ and there’s no hope for static salvation. Sure, some languages try to put deadlocks in a syntactic cage, but that walks right into the overhead problem, in spite of heroic VM-based optimizations. Unexpected costs, even if constant or linear, can sink any abstraction. For example (still piquant to Mozilla hackers busy deCOMtaminating), virtual method calls cost; they should be avoided where you’re writing hardware. The same goes for locks: not all abstractions must be MT-safe; some must be ST and fast.

So my default answer to questions such as the one I got at last May’s Ajax Experience, “When will you add threads to JavaScript?” is: “over your dead body!”

There are better ways. Clueful hackers keep rediscovering Erlang. Then there is STM. One retro stylist I know points to an old language-based solution, Hermes.

A requirement for JS3 (along with hygienic macros) is to do something along these more implicit lines of concurrency support. In all the fast yet maintainable MT systems I’ve built or worked on, the key idea (which Will Clinger stated clearly to me over lunch last fall) is to separate the mutable unshared data from the immutable shared data. Do that well, with language and VM support, and threads become what they should be: not an abstraction violator from hell, but a scaling device that can be composed with existing abstractions.

So here’s a promise about threads, one that I will keep or else buy someone a trip to New Zealand and Australia (or should I say, a trip here for those over there): JS3 will be ready for the multicore desktop workload.

Does this mean we won’t be hand-coding MT Gecko? I think Graydon said it best: “I’d rather eat glass.” Most high-level concurrent Mozilla programming will be in JS3 on the back of an evolved Tamarin.

For Mozilla 2, I proposed that we use Oink, or really Elsa, to build a tool that can help automate deCOMtamination, switching to C++ exceptions instead of nsresults, and similar tasks beyond the reach of sed or perl. The idea is to specify pattern and replacement abstract syntax trees (ASTs), and have the tool match the pattern against sub-tree instances, producing a version of the source with the replacement AST pretty-printed instead of the matching sub-tree.

Graydon came up with the insight of using source coordinates, which the Oink framework tracks accurately, to tell which lines in a given primary source (.cpp file, not .i file that results from the C pre-processor) correspond to a particular AST. The tool would read the source, counting lines until the range to delete is reached, then copy each such line matching the pattern to replace with a “- ” prepended. Then write lines pretty-printed from the replacement AST with “+ ” prepended. Add a bit more boilerplate and context lines, and you have a patch.

Graydon didn’t just make this breakthrough, he prototyped it, with a hardwired pattern to replace struct foo pointers with struct bar pointers in a test file:

$ cat test.cc
struct foo {};
struct holdsAFoo
{
char x;
static const foo * const y;
int *z;
};
$ ./staticprint -fs-print-rewrite test.cc
--- test.cc
+++ test.cc
@@ -5,1 +5,1 @@
-  static const foo * const y;
+static struct bar const * const y;
$ ./staticprint -fs-print-rewrite test.cc | patch -p0
patching file test.cc
$ cat test.cc
struct foo {};
struct holdsAFoo
{
char x;
static struct bar const * const y;
int *z;
};

All this wants is an Oink tool of its own (instead of a hack to staticprint), a little more indentation fidelity (bonus points for respecting the source file’s Emacs and vim modelines), C++ purity (no struct before bar needed), patch context and hunk coalescing, and generalized AST pattern/replacement options, and we’ll be able to make major systematic changes by generating patches. Neat!

/be

I just committed the initial revision of mozilla/js/tamarin, the open-source ActionScript Virtual Machine contributed by Adobe. But I had nothing to do with this fine work. It’s the product of some talented folks at Adobe (originally at Macromedia; at least one of whom has moved on and is doing a startup).

Tamarin, the open-source AVM, is the JIT-oriented JavaScript VM that I mentioned in my last post. We have already developed, via a two-day marathon hacking session, a set of proof-of-concept patches to integrate it with SpiderMonkey. The resulting Franken-VM was compiling and linking when I last laid hands on it, but not quite limping. This exercise produced many insights to guide more durable integration. And it was great fun interacting with the new open-source contributors from Adobe, whom I would like to welcome to Mozilla:

• Jeff Dyer, self-hosting compiler architect
• Steven Johnson, Tamarin developer
• Tom Reilly, garbage collector developer
• Rick Reitmaier, JIT developer
• Dan Smith, Tamarin module owner
• Edwin Smith, Tamarin creator and VM architect
• Erik Tierney, Tamarin developer

My apologies to those whose names I haven’t yet learned. I’ve had the pleasure of getting to know most of the above-named folks through the ECMA TG1 (JavaScript standards) group. They are good people and skillful hackers, and I’m grateful that they are adding their strength to the Mozilla project.

I should point out that, although Adobe did not release open source for its existing, free AS3 compiler, it did contribute a self-hosting (AS3-in-AS3) compiler prototype written by Jeff Dyer. We intend to develop this jointly into an ES4/JS2 self-hosting compiler, implementing the full and final ECMA-262 Edition 4 spec when it is done. In the mean time, if you want to generate ABC for Tamarin, you have open source options.

I’ll be on IRC in a little over nine hours to chat about this great news and answer questions, and then I’m off to Web 2.0. Later this week I will blog again about some early progress on another Mozilla 2 front.

/be

Update: Frank Hecker of the Mozilla Foundation clearly and concisely answers many basic questions about What This All Means, questions to which I tend to give short shrift in this too-technical blog of mine. Recommended.