Can you detect WebDriver sessions from inside a web page?

When you install a fresh copy of Firefox and run it for the first time, it opens a few tabs. For example, since August 2017, new installs of Firefox open the Firefox Privacy Notice in one tab on first run. The actual content can vary between versions and across platforms.

Last year we noticed some unusual traffic patterns on these pages: Lots of visitors came from countries like Finland that generally make up a tiny fraction of our traffic; many were on linux; and many came from ISPs named after the world’s largest data center operators.

If you’ve done much web engineering, you’re probably running some variation of this heuristic in your mind:

1. Do I know anyone who uses a remote linux box in a data center for normal web browsing? (Maybe.)
2. Do I know enough people who do that to make any kind of blip in the large volume of traffic that visits www.mozilla.org? (Nope.)
3. What else might explain lots of linux boxes installing Firefox and running it? (Testing automation, perhaps?)
4. What testing automation tools drive browsers? (Selenium WebDriver!)

At this point we’re back to the title of this piece: Can you detect WebDriver sessions from inside a web page? Because that sure would be interesting, right? If your website gets a lot of testing bot traffic, and you could detect it, then you would know more about your site’s traffic, for example. Or maybe you could do something more interesting with bot traffic than just show it whatever content it requested. Etc.

Lucky for us, we’re not the first to ask this question. Lots of people have asked it before. The best answers I found in my own searches are over on StackOverflow in response to “Can a website detect when you are using selenium with chromedriver?” and “Selenium WebDriver is detectable“. Answers to those questions suggest an entire suite of client-side tests we could run to discover if WebDriver is driving a browser. For example, one respondent says that the html tag of the document will include an attribute added by WebDriver (or by geckodriver, the Firefox proxy for WebDriver). Another suggests testing the presence of navigator.webdriver. And so on.

Other responses to the question say it’s impossible. Their line of reasoning goes like this: WebDriver is running a browser for the purposes of simulating real browsing activity, and a good simulation should be indistinguishable from the authentic thing, so of course WebDriver should be undetectable.

One interesting feature of these conversations is they appear to include lots of people who want to be sure their bot activity won’t be detected by whatever website they point it at. For example, one of the most detailed responses explains how the respondent circumvented detection on a site that sells concert tickets. “It worked,” they say. It would appear that my own use case is rather tame, in comparison.

Well, there was enough controversy around the topic to inspire a bit of coding on my part. I built a trivial little tool that runs all the tests I could find; if any of the tests can discern WebDriver at the controls, my WebDriver test would pass.

Digression: While the code itself is trivial, getting it to that state was definitely not. I code so rarely these days that I have to reestablish my development environment each time. That means I end up with the latest versions of e.g. node.js and the node bindings for WebDriver and the Jasmine testing framework. And that means I run into all manner of incompatibility among libraries and between documented/example usage and actual usage. It feels like exploring a new continent full of undiscovered affordances.

Nevertheless, I eventually banged out some code that appears to run. And then I ran it.

So, can you detect WebDriver sessions from inside a web page?

Before I get to that I want to point out that I have the great luck to work at an amazing company. I get to rub shoulders (or whatever the IRC equivalent of that is) with people who help shape the future of our most critical technologies (by way of specifications and implementations) and help determine technology’s role in society (by way of advocacy, education and straight-up lawsuits against the federal government, when necessary), and that makes me proud.

Some of the people I work with contribute a lot to automated testing tools like WebDriver. They are very patient when someone like me bungles into their IRC channel to ask this sort of thing. So when my tests completely failed to ascertain whether a bot was at the wheel of my browser, I reached out to those folks and asked: Should I be able to detect WebDriver sessions?

And they said, “Probably not.” Which was my experience, too.

A patch is in the works to expose an attribute on navigator, such that navigator.webdriver !== false, when geckodriver is driving, and when the person who launched geckodriver has not made any code changes to obscure their ticket-scalping side business.

Also, several companies market tools designed to “fingerprint” a browser, because doing so creates all sorts of opportunity to trade the identity of the browser (and the person running it) for profit. (It’s like silver mining, except the miners ride Uber instead of mules and the silver is extracted from your soul!) Of course such technology might also be used to thwart networks of bots, since bots can theoretically be identified in the same manner.

Meanwhile, if you just want to add some simple JavaScript to your page so you can innocently ask a browser whether its driver is a robot, you’re out of luck. For now.

 

 

Debugging WebExtension Popups

Note: In the time since I last posted here I have been doing a bit more hands-on web development [for example, on the View Source website]. Naturally this has led me to learn new things. I have learned a few things that may be new to others, too. I’ll drop those here when I run across them.

---

I have been looking for a practical way to learn about WebExtensions, the new browser add-on API in Firefox. This API is powerful for a couple reasons: It allows add-on developers to build add-ons that work across browsers, and it’s nicer to work with than the prior Firefox add-on API (for example, it watches code and reloads changes without restarting the browser).

So I found a WebExtensions add-on to hack on, which I’ll probably talk about in a later post. The add-on has a chrome component, which is to say it includes changes to the browser UI. Firefox browser chrome is just HTML/CSS/JavaScript, which is great. But it took me a little while to figure out how to debug it.

The tools for doing this are all fairly recent. The WebExtension documentation on MDN is fresh from the oven, and the capabilities shown below were missing just a few months ago.

Here’s how to get started debugging WebExtensions in the browser:

First, enable the Browser Toolbox. This is a special instance of Firefox developer tools that can inspect and debug the browser’s chrome. Cool, eh? Here’s how to make it even cooler:

• Set up a custom Firefox profile with the Toolbox enabled, so you don’t have to enable it every time you fire up your development environment. Consider just using the DevPrefs add-on, which toggles a variety of preferences (including Toolbox) to optimize the browser for add-on development.
• Once you have a profile with DevPrefs installed, you can launch it with your WebExtension like so: ./node_modules/.bin/web-ext run --source-dir=src --firefox-binary {path to firefox binary} --firefox-profile {name of custom profile} (See the WebExtensions command reference for more information.

Next, with the instance of Firefox that appears when you run the above command, go to the Tools -> Web Developer -> Browser Toolbox menu. A window should appear that looks just like a standard Firefox developer tools window. But this window is imbued with the amazing ability to debug the browser itself. Try it: Use the inspector to look at the back button!

In that window you’ll see a couple small icons near the top right. One looks like a waffle. This button makes the popup sticky — just like a good waffle. This is quite helpful, since otherwise the popup will disappear the minute you try to inspect, debug, or modify it using the Browser Toolbox.

Next to the waffle is a button with a downward arrow on it. This button lets you select which content to debug — so, for example, you could select the HTML of your popup. When you have a sticky popup selected, you can inspect and hack on its HTML and CSS just like you would any other web content.

This information is now documented in great detail on MDN. Check it out!

Experiments: Services

The vision for our Services products is to bring the power of MDN directly into professional web developers’ daily coding environments. Experiments in this area will take the form of web services built iteratively. Each iteration should either attract enthusiastic users or provide market insight that helps guide the next iteration.

In addition to exploring the market for developer services, these experiments will also explore new architectures, form factors and contribution pathways for MDN’s information products.

Four services have been identified for exploration so far.

1. Compatibility Data service

The compatibility data service (a.k.a. Browsercompat.org) is a read/write API intended to replace the tables of compatibility data that currently accompany many features in MDN’s reference documentation. The project is justified for maintenance reasons alone: Unstructured compatibility data on MDN is very difficult to keep current because it requires editors to maintain every page (in every language) where identical data might appear. It offers a fantastic opportunity to answer several questions likely to recur in MDN’s future evolution:

• Can we maintain so-called “micro-services” without creating significant engineering overhead?
• Can we build reasonable contribution pathways and monitoring processes around structured data residing in a separate data store?
• Is the effort involved in “decomposing” MDN’s wiki content into structured APIs justified by the improvement in reusability such services provide?

These questions are essential to understand as we move toward a future where data are increasingly structured and delivered contextually.

Status of this experiment: Underway and progressing. Must achieve several major milestones before it can be assessed.

2. Security scanning service

In surveys run in Q4 2014 and Q1 2015, a large number of MDN visitors said they currently do not use a security scanning service but are motivated to do so. This experiment will give many more web developers access to security scanning tools. It will answer these questions:

• Can we disrupt the security scanning space with tools aimed at individual web developers?
• Can we help more web developers make their web sites more secure by providing services in a more familiar form factor?
• Is there value in releasing services for web developers under the MDN brand?

Status of this experiment: Underway and progressing toward an MVP release. Must achieve several major milestones before it can be assessed.

3. Compatibility scanning service

In the surveys mentioned above, a large number of MDN visitors said they currently do not use a compatibility scanning tool but are motivated to do so. This experiment will build such a tool using a variety of existing libraries. It will answer these questions:

• Are web developers enthusiastic about using a tool that promises to make their web sites more compatible across devices?
• What form factor is most effective?
• Can we successfully create automation from MDN’s information products and contribution workflows?

Status of this experiment: MVP planned for Q2/Q3 2015.

4. Accessibility scanning service

Also in the surveys mentioned above, a large number of MDN visitors said they currently do not scan for accessibility but are motivated to do so. This experiment will build an accessibility scanning service that helps answer the questions above, as well as:

• If the tool fits into their workflow, will more developers make their web sites more accessible?

Status of this experiment: MVP planned for Q2/Q3 2015.

The market success of any of the latter three services would make possible an additional experiment:

5. Revenue

Professional web developers are accustomed to paying for services that increase their capacity to deliver high-quality professional results. The success of such services as Github, Heroku, NewRelic and many others is evidence of this.

MDN services that bring the high quality of MDN into professional web developers’ workflows may be valuable enough to generate revenue for Mozilla. This possibility depends on a number of important milestones before it is feasible, such as…

• Market demand for services built
• Community discussion about paid services under the MDN banner
• Analysis of appropriate pricing and terms
• Integration with payment systems

In other words, this cannot happen until services prove themselves valuable. Meanwhile, simply discussing it is an experiment: Is the possibility of MDN generating revenue with valuable developer-facing services conceivable?

MDN Product Talk: The Series

1. Introduction
2. Business Context
3. The Case for Experiments
4. Product Vision
5. Reference Experiments
6. Learning Experiments
7. Services Experiments

Experiments: Learning

The vision for our Learning products is to use the power of MDN to teach web development to future web developers. Experiments with new content (the Learning Area), new formats (Content Kits) and new contribution paths (the MDN Fellowship) are underway. Each of these experiments is the culmination of months of planning and development.

Furthermore, experiments in the Reference product such as Helpfulness Ratings, the Metrics Dashboard and Interactive Code Samples promise new opportunities to improve MDN’s Learning products.

In other words, MDN’s Learning products enjoy plenty of enthusiasm and forward momentum. I propose just two additional experiments to supplement those already underway.

1. Discussion Areas

Someone recently created a page in MDN’s new Learning Area asking for help:

“Good day, I have a problem with nsIWebProgressListener for extension firefox. I need a status is target blank, mean: when user on click to tag a (<a href=”www.mysite.com” target=”blank”>) then we have a this status. 

Please, help me.
Thanks all!”

Learning Area wiki pages are not well suited to this kind of discussion. The Learning Area does include a link to the Webmaker forums which are definitely more suitable. But they are not focused on the same audience of aspiring web developers.

I have spoken several times with learners who chose to invest in development training at a coding bootcamp such as The Turing School or Code Fellows. For them, the peer group afforded by such opportunities was very compelling.

Learners need a way to discuss their subject with others who are learning the same thing at the same time. We should identify a forum/forums focused on the same audience and subjects as MDN’s Learning Area. A dedicated area of the Webmaker forums is probably the most lightweight place to explore the usage and implications of this feature. We should be able to answer these questions through observation:

• Do people use the forum?
• Do they help each other?
• What management does the forum require?

With those questions answered we can decide whether a more involved solution is justified.

Status of this experiment: Product stakeholders recognize the gap, but nobody is currently working to address it. The proposal needs more discussion and commitment.

2. Exercises in Github

Analysis and surveys of the online learning space suggest that online learners are constrained by sandboxes. They learn from online tools how to code; they do not learn from online tools how to build and launch web sites.

The Turing School maintains its syllabi, lessons and activities in a Github Repository. This creates additional opportunities for students to interact with git and Github, both critical elements of the modern web developer’s toolkit.

MDN can adapt this approach to work for thousands of learners. Specifically, MDN can create activities in Github that require users…

1. to fork and/or clone a repository
2. to deploy their work to Github pages

This approach offers numerous benefits:

• Learners use the same source code management tools that real web developers use
• Learners create measurable artifacts of their progress, which can help identify opportunities to improve lessons or activities
• Learners build a website in their own Github repository that may help demonstrate their skills to future employers

The open-source Nodeschool.io offers training in this format. We could create a Nodeschool.io activity to accompany one of MDN’s lessons and track usage that way, thereby learning answers to these questions:

• Do people start the activities?
• Do they complete them?
• Does their work demonstrate fluency in the lesson the activity accompanies?

Status of this experiment: Proposal to Learning Area stakeholders met with enthusiasm. Needs specification and an engineering commitment.

MDN Product Talk: The Series

1. Introduction
2. Business Context
3. The Case for Experiments
4. Product Vision
5. Reference Experiments
6. Learning Experiments
7. Services Experiments

Experiments: Reference

The vision of MDN’s Reference product is to use the power of MDN to build the most accessible, authoritative source of information about web standard technologies for web developers. Accomplishing this vision means optimizing and improving on the product’s present success.

Optimization requires measurement, and MDN’s current measurements need improvement. Below I describe two measurement improvements underway, plus a few optimization experiments:

1. Helpfulness Ratings

Information quality is the essential feature of any reference, but MDN currently does not implement direct quality measures. Bug 1032455 hypothesizes that MDN’s audience would provide qualitative feedback that will help measure and improve MDN’s content quality. But qualitative feedback is a new feature on MDN that we need to explore. Comment 37 on that bug suggests that we use a 3rd-party “micro-survey” widget to help us understand how to get the most from this mechanism before we implement it in our own codebase. The widget will help us answer these critical questions:

• How can we convince readers to rate content? (We can experiment with different calls to action in the widget.)
• How do we make sense of ratings? (We can tune the questions in the widget until their responses give us actionable information.)
• How can we use those ratings to improve content? (We can design a process that turns good information gleaned from the widget into a set of content improvement opportunities; we can solicit contributor help with those opportunities.)
• How will we know it is working? (We can review revisions before and after the widget’s introduction; our own qualitative assessment should be enough to validate whether a qualitative feedback mechanism is worth more investment.)

If the 3rd-party widget and lightweight processes we build around it make measurable improvements, we may wish to invest more heavily into…

• a proprietary micro-survey tool
• dashboards for content improvement opportunities
• integration with MDN analytics tools

Status of this experiment: MDN’s product council has agreed with the proposal and vendor review bugs for the 3rd party tool are filed.

2. Metrics Dashboard
In an earlier post I depicted the state of MDN’s metrics with this illustration:

The short summary of this is, MDN has not implemented sufficient measures to make good data-driven decisions. MDN doesn’t have any location to house most of those measurements. Bug 1133071 hypothesizes that creating a place to visualize metrics will help us identify new opportunities for improvement. With a metrics dashboard we can answer these questions:

• What metrics should be on a metrics dashboard?
• Who should have access to it?
• What metrics are most valuable for measuring the success of our products?
• How can we directly affect the metrics we care about?

Status of this experiment: At the 2015 Hack on MDN meetup, this idea was pitched and undertaken. A pull request attached to bug 973612 includes code to extract data from the MDN platform and add it to Elasticsearch. Upcoming bugs will create periodic jobs to populate the Elasticsearch index, create a Kibana dashboard for the data and add it (via iframe) to a page on MDN.

3. Social Sharing
For user-generated content sites like MDN, social media is an essential driver of traffic. People visiting a page may be likely to share the page with their social networks and those shares will drive more traffic to MDN. But MDN lacks a social sharing widget (among other things common to user-generated content sites):

Bug 875062 hypothesizes that adding a social sharing widget to MDN’s reference pages could create 20 times more social sharing than MDN’s current average. Since that bug was filed MDN saw some validation of this via the Fellowship page. That page included a social sharing link at the bottom that generated 10 times as many shares as MDN’s average. This experiment will test social sharing and answer questions such as…

• What placement/design is the most powerful?
• What pages get the most shares and which shares get the most interaction?
• Can we derive anything meaningful from the things people say when they share MDN links?

Status of this experiment: The code for social sharing has been integrated into the MDN platform behind a feature flag. Bug 1145630 proposes to split-test placement and design to determine the optimal location before final implementation.

4. Interactive Code Samples

Popular online code sandboxes like Codepen.io and JSFiddle let users quickly experiment with code and see its effects. Some of MDN’s competitors also implement such a feature. Surveys indicate that MDN’s audience considers this a gap in MDN’s features. Anecdotes indicate that learners consider this feature essential to learning. Contributors also might benefit from using a code sandbox for composing examples since such tools provide validation and testing opportunities.

These factors suggest that MDN should implement interactive code samples, but they imply a multitude of use cases that do not completely overlap. Bug 1148743 proposes to start with a lightweight implementation serving one use case and expand to more as we learn more. It will create a way for viewers of a code sample in MDN to open the sample in JSFiddle. This experiment will answer these questions:

• Do people use the feature?
• Who uses it?
• How long do they spend tinkering with code in the sandbox?
• Was it helpful to them?

The 3rd party widget required for the Helpfulness Ratings experiment can power the qualitative assessment necessary to know how this feature performs with MDN’s various audiences. If it is successful, future investment in this specific approach (or another similar approach) could…

• Allow editors of a page to open samples in JSFiddle from the editing interface
• Allow editors of a sample to save it to an MDN page
• Create learning exercises that implement the sandbox

Status of this experiment: A pull request attached to Bug 1148743 will make this available for testing by administrators.

5. Akismet spam integration

Since late 2014 MDN has been the victim of a persistent spam attack. Triaging this spam is a constant vigil for MDN contributors and staff. Most of the spam is blatant: It seems likely that a heuristic spam detection application could spare the human triage team some work. Bug 1124358 hypothesizes that Akismet, a popular spam prevention tool, might be up to the task. Implementing this bug will answer just one question:

• Can Akismet accurately flag spam posts like the ones MDN’s triage team handles, without improperly flagging valid content?

Status of this experiment: Proposed. MDN fans and contributors with API development experience are encouraged to reach out!

MDN Product Talk: The Series

1. Introduction
2. Business Context
3. The Case for Experiments
4. Product Vision
5. Reference Experiments
6. Learning Experiments
7. Services Experiments

MDN Product Talk: Vision

As I wrote this post, MDN’s documentation wiki hit a remarkable milestone: For the first time ever MDN saw more than 4 million unique visitors in a single month.

I always tell people, if we have a good quarter it’s because of the work we did three, four, and five years ago. It’s not because we did a good job this quarter.

– Jeff Bezos


Mozilla’s MDN project envisions a world where software development begins with web development — a world where developers build for the web by default. That is our great ambition, an audacious summary of MDN’s raison d’être. I discussed MDN’s business strategy at length in an earlier post. In this post I will talk about MDN’s product strategy.

Several posts ago I described MDN as a product ecosystem — “…a suite of physical products that fits into a bigger ecosystem that may entail services and digital content and support”, to use one designer’s words. The components of MDN’s product ecosystem — audience, contributors, platform, products, brand, governance, campaigns, and so forth — are united by a common purpose: to help people worldwide understand and build things with standard web technologies.

The Future

The efforts we undertake now must help people years hence. But projecting even a few years into the future of technology is … challenging, to say the least. It is also exactly what MDN’s product vision must do. So what does the future look like?

Looking at the future, we see futuristic shapes emerging from fog. We can’t tell yet whether they are riding hoverboards or wearing virtual reality headsets in self-driving cars. Maybe they are piloting webcam-equipped moth cyborgs. Will hoverboards implement web-standard APIs? Will MDN contributors need watch-optimized contribution pathways? We cannot know now.

We can be confident about a few things:

1. Future information tools will take a marvelous variety of forms, from fashion accessories to appliances to autonomous vehicles. There is no replacement for the web; it will appear in some shape on all of these devices.
2. Future information tools will deliver information when and where it is needed. Digging for information in search results will be less common than it is today, even among web developers.
3. The future will have even more demand for capable web developers than the present has. Many of them will read documentation in their own language.

The three MDN products under heavy development now — the mature Reference product (MDN’s documentation wiki) and the new Services and Learning products — will evolve to meet this future:

Reference

In the future, web developers will still need a source of truth about open web technology development. MDN’s last 10 years of success have established it as that source of truth. Millions of web developers choose MDN over other online sources because, as a reference, MDN is more authoritative, more comprehensive, and more global. The vision of our Reference product is to use the power of MDN to build the most accessible, authoritative source of information about standard and emerging web technologies for web developers.

Services

In the future, MDN will still be an information resource, but its information will take different shapes depending on how and where it is accessed. MDN may look like the present-day reference product when it is rendered in a web browser; but sometimes it may render in a browser’s developer tools, in a pluggable code editor, or elsewhere. In those instances the information presented may be more focused on the things developers commonly need while coding. Some developers may still access MDN via search results; others will get something from MDN the moment they need it, in the context where it is most helpful. MDN’s articles will be used for learning and understanding; but subsets of MDN’s information may also power automation that enhances productivity and quality. These new uses all share one characteristic: They bring MDN’s information closer to developers through a service architecture. The vision for our Services products is to bring the power of MDN directly into professional web developers’ daily coding environments.

Learning

The future’s web developers are learning web development right now. MDN’s present-day material is already essential to many of them even though it is aimed at a more advanced audience. MDN’s new learning content and features will deliver beginner-focused content that authoritatively covers the subjects essential to becoming a great web developer. Unlike many other online learning tools, MDN needn’t keep learners inside the platform: We can integrate with any 3rd-party tool that helps learners become web developers, and we can create opportunities for web developers to learn from each other in person. The vision for our Learning products is to use the power of MDN to teach web development to future web developers.

The power of MDN

The success of all three products depends on something I above call “the power of MDN” — something that sets MDN apart from other sources of information about web development.

I have previously described information about web development as an “oral tradition”. Web development is a young, complex and constantly changing field. It is imperfectly documented in innumerable blogs, forums, Q&A sites and more. MDN’s unique power is its ability to aggregate the shared experience of web developers worldwide into an authoritative catalog of truth about web technologies.

This aspect of MDN is constant: We carry it with us into the future, come what may. For any MDN product to succeed at scale it must implement a contribution pathway that allows web developers to directly contribute their knowledge about web development. MDN’s products advance the field of web development at a global scale by sharing essential information discovered through the collective experience of the world’s web developers.

Together we are advancing the field. In 10 years web development will be concerned with new questions and new challenges, thanks to the state of the art that MDN aggregates and promulgates. We build on what we know; we share what we know on MDN. Here’s to another 10 years!

MDN Product Talk: The Series

1. Introduction
2. Business Context
3. The Case for Experiments
4. Product Vision
5. Reference Experiments
6. Learning Experiments
7. Services Experiments

MDN Product Talk: The Case for Experiments

This is the third post in a series that is leading up to a roadmap of sorts — a set of experiments I will recommend to help MDN both deepen and widen its service to web developers.

The first post in the series introduced MDN as a product ecosystem. The second post in the series explained the strategic context for MDN’s product work. In this post I will explain how an experimental approach can help us going forward, and I’ll talk about the two very different kinds of experiments we’re undertaking in 2015.

In the context of product development, “experiments” is shorthand for market-validated learning — a product discovery and optimization activity advocated by practitioners of lean product development and many others. Product development teams run experiments by testing hypotheses against reality. The primary benefit of this methodology is eliminating wasted effort by building things that people demonstrably want.

Now my method, though hard to practise, is easy to explain; and it is this. I propose to establish progressive stages of certainty.

–Francis Bacon (introducing the scientific method in Novum Organum)

Product experiments all…

• have a hypothesis
• test the hypothesis by exposing it to reality (in other words, introducing it to the market)
• deliver some insight to drive further product development

Here is a concrete example of how MDN will benefit from an experimental approach. We have an outstanding bug, “Kuma: Optional IRC Widget“, opened 3 years ago and discussed at great length on numerous occasions. This bug, like so many enhancement requests, is really a hypothesis in disguise: It asserts that MDN would attract and retain more contributors if they had an easier way to discuss contribution in realtime with other contributors.

That hypothesis is untested. We don’t have an easy way to discuss contribution in realtime now. In order to test the bug’s hypothesis we propose to integrate a 3rd-party IRC widget into a specific subset of MDN pages and measure the result. We will undoubtedly learn something from this experiment: We will learn something about the specific solution or something about the problem itself, or both.

Understanding the actual problem to be solved (and for who) is a critical element of product experimentation. In this case, we do not assert that anonymous MDN visitors need a realtime chat feature, and we do not assert that MDN contributors specifically want to use IRC. We assert that contributors need a way to discuss and ask questions about contribution, and by giving them such a facility we’ll increase the quality of contribution. Providing this facility via IRC widget is an implementation detail.

This experiment is an example of optimization. We already know that contribution is a critical factor in the quality of documentation in the documentation wiki. This is because we already understand the business model and key metrics of the documentation wiki. The MDN documentation wiki is a very successful product, and our focus going forward should be on improving and optimizing it. We can do that with experiments like the one above.

In order to optimize anything, though, we need better measurements than we have now. Here’s an illustration of the key components of MDN’s documentation wiki:

Visitors come to the wiki from search, by way of events, or through links in online social activity. If they create an account they become users and we notify them that they can contribute. If they follow up on that notification they become returners. If they contribute they become contributors. If they stop visiting they become disengaged users. Users can request content (in Bugzilla and elsewhere). Users can share content (manually).

All the red and orange shapes in the picture above represent things we’re measuring imperfectly or not at all. So we track the number of visitors and the number of users, but we don’t measure the rate by which visitors become users (or any other conversion rate). We measure the rates of content production and content consumption, but we don’t measure the helpfulness of content. And so forth.

If we wanted to add a feature to the wiki that might impact one of these numbers, how would we measure the before and after states? We couldn’t. If we wanted to choose between features that might affect these numbers, how would we decide which metric needed the most attention? We couldn’t. So in 2015 we must prioritize building enough measurements into the MDN platform that we can see what needs optimization and which optimizations make a difference. In particular, considering the size of content’s role in our ecosystem, we must prioritize features that help us better understand the impact of our content.

Once we have proper measurements, we have a huge backlog of optimization opportunities to consider for the documentation wiki. Experiments will help us prioritize them and implement them.

As we do so, we are also simultaneously engaged in a completely different kind of experimentation. Steve Blank describes the difference in his recent post, “Fear of Failure and Lack of Speed In a Large Corporation”. To paraphrase him: A successful product organization that has already found market fit (i.e. MDN’s documentation wiki) properly seeks to maximize the value of its existing fit — it optimizes. But a fledgling product organization has no idea what the fit is, and so properly seeks to discover it.

This second kind of experimentation is not for optimization, it is for discovery. MDN’s documentation wiki clearly solves a problem and there is clearly value in solving that problem, but MDN’s audience has plenty of problems to solve (and more on the way), and new audiences similar to MDN’s current audience have similar problems to solve, too. We can see far enough ahead to advance some broad hypotheses about solving these problems, and we now need to learn how accurate those hypotheses are.

Here is an illustration of the different kinds of product experiments we’re running in the context of the overall MDN ecosystem:

The left bubble represents the existing documentation wiki: It’s about content and contribution; we have a lot of great optimizations to build there. The right bubble represents our new product/market areas: We’re exploring new products for web developers (so far, in services) and we’re serving a new audience of web learners (so far, by adding some new areas to our existing product).

The right bubble is far less knowable than the left. We need to conduct experiments as quickly as possible to learn whether any particular service or teaching material resonates with its audience. Our experiments with new products and audiences will be more wide-ranging than our experiments to improve the wiki; they will also be measured in smaller numbers. These new initiatives have the possibility to grow into products as successful as the documentation wiki, but our focus in 2015 is to validate that these experiments can solve real problems for any part of MDN’s audience.

As Marty Cagan says, “Good [product] teams are skilled in the many techniques to rapidly try out product ideas to determine which ones are truly worth building.  Bad teams hold meetings to generate prioritized roadmaps.” On MDN we have an incredible opportunity to develop our product team by taking a more experimental approach to our work. Developing our product team will improve the quality of our products and help us serve more web developers better.

In an upcoming post I will talk about how our 2015 focus areas will help us meet the future. And of course I will talk about specific experiments soon, too.

MDN Product Talk: The Series

1. Introduction
2. Business Context
3. The Case for Experiments
4. Product Vision
5. Reference Experiments
6. Learning Experiments
7. Services Experiments

MDN Product Talk: Business Context

This post is intended to frame the business context that MDN operates in. This is not a product strategy. It is the second in a series of posts.

One of the questions I asked in my introductory post was, “What does solving problems for MDN’s audience accomplish for Mozilla?”. In other words, what opportunities make this investment worthwhile?

It’s not always comfortable to ask such questions, but they’re important. We operate within constraints; everything we do has an opportunity cost. Why MDN? I’m glad to say it is easy to answer.

MDN is a public service sponsored by Mozilla, a mission-driven nonprofit. Mozilla does not measure success by revenue, we measure success by our global impact. Products at Mozilla are expected to deliver impact on Mozilla’s mission and support at least one of Mozilla’s main product lines.

Mission Impact

Mozilla’s mission is to build the internet as a global public resource open and accessible to all. Web developers are critical to the mission. They choose the platforms and technologies they use to build products:

• Deploying to the web is a developer choice.
• Using open web standards is a developer choice.
• Building content that works across a multitude of devices is a developer choice.
• Making content accessible to a broad, global audience is a developer choice.

The success of Mozilla’s mission depends on these developer choices. Developers make choices pragmatically — they choose the possible. MDN makes building open and accessible on the web more possible. MDN empowers all developers to make choices aligned with Mozilla’s mission and thereby create the web the world needs.

How do we measure this impact? Look to future posts for more discussion of this.

Top-line Organizational & Product Support

MDN supports Mozilla’s top-line goal to create meaningful relationships with Mozilla product users worldwide. MDN’s current audience contributes approximately 1-2% toward Mozilla’s overall relationship goals; this number is naturally limited by the size of MDN’s market.

MDN is naturally aligned with Mozilla Learning. Mozilla’s learning products build contributors to the web. MDN empowers expert and professional web makers and is uniquely positioned to address those who aspire to be expert or professional web developers. MDN’s alignment with Mozilla’s learning initiatives is obvious, but only recently intentional; in 2015 we will experiment to learn more about how to directly support Mozilla Learning.

Market

In 2015 MDN will focus on three segments — two of them new.

MDN historically has served web developers who want to build standards-based web sites. After a decade of providing high-quality information about web technologies on its documentation wiki, MDN is considered an authority among these developers. (For instance: Instructors at several “web developer bootcamps” in my town tell me they actively guide students toward MDN, away from MDN’s competitors, because they only trust MDN.) MDN serves more of these developers every month and now serves 25-50% of the global market. These developers are MDN’s core users and the choices they make directly impact Mozilla’s mission.

MDN’s authority, expertise and capabilities position it uniquely to serve web learners who aspire to be web developers. The job market for developers is strong and growing. The number of jobs in both web development and software development is expected to grow by 20% or more in the U.S. by 2020. Global growth is expected to follow a similar trend. MDN can match these new web developers with product features that will help them become web developers building standards-based web sites.

MDN’s authority, scale and unique content position it to serve professional web developers who want to build standards-based web sites professionally. Recent surveys indicate that more than half of MDN’s visitors meet that description. A large number of these professionals say they are motivated to build web products that embody Mozilla values like accessibility, compatibility, and security, but are under-served by tools to make doing so feasible. MDN can build new products and features to fill the gap, increasing these professionals’ ability to build standards-based web sites on deadline.

2015 Goals

Considering those opportunities, in 2015 the MDN team has committed to…

1. “Relationships”: Support Mozilla’s overall goals by continuing the (double digit per year) organic growth of our original segment, web developers who want to build standards-based web sites.
2. “Teaching”: Support Mozilla’s learning initiatives by exploring and implementing new capabilities that will serve a new market and segment, web learners who aspire to be web developers.
3. “Services”: Add a new product, “Services” to serve a new segment, professional web developers who want to build standards-based web sites professionally.

KPIs for these live on the Mozilla wiki with other information about the MDN team’s overall vision, mission and objectives.

Summary

Can you put a TL;DR at the bottom of a post? Let’s find out. After this post and yesterday’s, we can now say…

MDN is a product ecosystem that serves an audience whose choices are critical to the success of Mozilla’s mission. MDN serves them at a scale unique within Mozilla and in the market. In 2015 MDN will continue serving web developers who want to build standards-based web sites while building prototypes and experiments to solve problems for web learners and professional web developers.

MDN Product Talk: The Series

1. Introduction
2. Business Context
3. The Case for Experiments
4. Product Vision
5. Reference Experiments
6. Learning Experiments
7. Services Experiments

MDN Product Talk: Introduction

This is the first of a series of blog posts about MDN, the product I am product manager for. The series discusses MDN’s business case, product strategy, and a series of experiments we can run on MDN in 2015 (and beyond) to help it continue to serve its audience and the web. Many people familiar with MDN will consider some of the following posts obvious; to them I say, “stay with me.” Not all of this will be obvious to everyone. Some may be novel to everyone. Any of it may need clarification or improvement, which I will learn from gentle comments.

As a new member of the MDN team, I taxed my colleagues in pursuit of answers to such questions as…

• What is MDN?
• Whom is MDN for?
• What are their problems?
• How does MDN solve those problems?
• What does solving those problems accomplish for Mozilla?

I posed such questions to almost everyone I could corner, from members of Mozilla’s steering committee to random web developers I met at kids’ birthday parties. I interrogated historic mailing list threads. I scanned through hundreds of enhancement requests in our bug backlog. I read books about product management. I doodled architectural diagrams and flowcharts. And finally I think I came to an understanding. I hope sharing it is helpful. So what is MDN? Is it a documentation site? A developer program? A wiki? A network? A suite of products? A single product? A brand? A railway station? It depends on whom you ask and when. Few of the people I cornered could say exactly what MDN is or ought to be. Most had an idea of what MDN ought to accomplish, but their ideas varied wildly. One thing is clear: MDN is successfully serving a large portion of its potential market and has been doing so in its present form for nearly 10 years. Which is great. But… what is MDN? For most of MDN’s primary audience — web developers — MDN is a documentation wiki (i.e. a user-generated content site or UGC) serving trustworthy information about open web technologies (primarily HTML, JavaScript and CSS). And of course the audience is correct. But MDN is also a brand that resonates with its audience and fans. And MDN is also a community of contributors who care about MDN. And MDN is also a platform where all these things mingle and mix. All told, MDN includes (at least) 6 product offerings in operation or development and (at least) 8 significant enhancement projects underway, as well as numerous activities and projects that sustain the documentation wiki. The interplay between these activities is complex. OK, great; but what is MDN?!? Considering its scope — the variety of its critical stakeholders, the complexity of its value chain, the number of supporting activities underway — I think MDN can only be considered a product ecosystem.

• MDN is a strong brand supported a number of products and activities.
• Chief among these is a platform, the documentation wiki, which is also called MDN.
• Within the platform content is huge: MDN’s primary audience visits solely for the content, because the content is valuable. The MDN brand’s authority and the MDN platform’s scale both depend on the MDN content’s quality.
• As an open-source web application serving user-generated content about open standards, contribution overlaps almost every aspect MDN — especially content creation. The MDN content’s quality depends in large part on contribution.
• Various marketing efforts support MDN, strengthening its brand, attracting new visitors, and activating contributors. MDN marketing efforts among developers also support Mozilla’s brand.
• In response to a clear market need, the MDN team is experimenting with some developer-facing services that will be partly supported by the content and platform.

That’s a lot of moving pieces — and we haven’t even begun talking about whom MDN serves and what problems MDN solves for them! Look for answers to those questions and much more in coming posts. As we go I’ll use the above diagram to help contextualize things. More to come! :wq

MDN Product Talk: The Series

1. Introduction
2. Business Context
3. The Case for Experiments
4. Product Vision
5. Reference Experiments
6. Learning Experiments
7. Services Experiments

Report: Web Compatibility Summit

Last week I spoke at Mozilla’s Compatibility Summit. I spoke about compatibility data. Here are my slides.

Compatibility data…

• is information curated by web developers about the cross-browser compatibility of the web’s features.
• documents browser implementation of standards, browser anomalies (both bugs and features), and workarounds for common cross-browser bugs. It theoretically covers hundreds of browser/version combinations, but naturally tends to focus on browsers/versions with the most use.
• is partly structured (numbers and booleans can answer the question, “does Internet Explorer 10 support input type=email?”) and partly unstructured (numbers and booleans cannot say, “Safari Mobile for iOS applies a default style of opacity: 0.4 to disabled textual <input> elements.”).

Compatibility data changes all the time. As a colleague pointed out last week, “Every six weeks we have all new browsers” — each with an imperfect implementation of web standards, each introducing new incompatibilities into the web.

Web developers are often the first to discover and document cross-browser incompatibilities since their work product is immediately impacted. And they do a pretty good job of sharing compatibility data. But as I said in my talk: Compatibility data is an oral tradition. Web developers gather at the town pump and share secrets for making boxes with borders float in Browser X. Web developers sit at the feet of venerated elders and hear how to make cross-browser compatible CSS transitions. We post our discoveries and solutions in blogs; in answers on StackOverflow; in GitHub repositories. We find answers on old blogs; in countless abandoned PHP forums; on the third page of search results.

There are a small number of truly canonical sources for compatibility data. We surveyed MDN visitors in January and learned that they refer to two sources far more than any others: MDN and caniuse.com. MDN has more comprehensive information — more detailed, for more browser versions, accompanied by encyclopedic reference materials. caniuse has a much better interface and integrates with browser market share data. Both have good communities of contributors. Together, they represent the canon of compatibility data.

Respondents to our recent survey said they use the two sites differently: They use caniuse for planning new sites and debugging existing issues, and use MDN for exploring new features and when answering questions that come up when writing code.

On MDN, we’re working on a new database of compatibility data with a read/write API. This project solves some maintenance issues for us, and promises to create more opportunities for web developers to build automation around web compatibility data (for an example of this, see doiuse.com, which scans your CSS for features covered by caniuse and returns a browser coverage report).

Of course the information in MDN and caniuse is only one tool for improving web compatibility, which is why the different perspectives at the Compat Summit were so valuable.

If we think of web compatibility as a set of concentric circles, MDN and caniuse (and the entire, sprawling web compatibility data corpus) occupy a middle ring. In the illustration above, the rings get more diffuse as they get larger, representing the increasing challenge of finding and solving incompatibility as it moves from vendor to web developer to site.

• By the time most developers encounter cross-browser compatibility issues, those issues have been deployed in browser versions. So browser vendors have a lot of responsibility to make web standards possible; to deploy as few standards-breaking features and bugs as possible. Jacob Rossi from Microsoft invited Compat Summit participants to collaborate on a framework that would allow browser vendors to innovate and push the web forward without creating durable incompatibility issues in deployed websites.
• When incompatibilities land in browser releases, web developers find them, blog about them, and build their websites around them. At the Compat Summit, Alex McPherson from Quickleft presented his clever work quantifying some of these issues, and I invited all present to start treating compatibility data like an important public resource (as described above).
• Once cross-browser incompatibilities are discussed in blog posts and deployed on the web, the only way to address incompatibilities is to politely ask web developers to fix them. Mike Taylor and Colleen Williams talked about Mozilla’s and Microsoft’s developer outreach activities — efforts like Webcompat.com, “bug reporting for the internet”.

At the end of the Compat Summit, Mike Taylor asked the participants whether we should have another. My answer takes the form of two questions:

1. Should someone work to keep the importance of cross-browser compatibility visible among browser vendors and web developers?
2. If we do not, who will?

I think the answer is clear.

Special thanks to Jérémie Patonnier for helping me get up to speed on web compatibility data.