has an undiagnosed addiction to steamed pork buns
and works on webby stuff at Mozilla. more »

Sandboxing the Sandbox

With almost 2 billion downloads, add-ons have proven to be a huge part of Firefox’s growth and popularity over the last 5 years. As Firefox continues to be adopted by non-technical, mainstream users, the security and consumer experience of installing third party add-ons becomes increasingly more important. It’s with these users in mind that I propose some major changes to the way add-ons are submitted and distributed through Mozilla’s official add-ons gallery.

First, some background

This month marks the three year anniversary of the “sandbox” review model being introduced on addons.mozilla.org. Veteran Mozilla contributors and add-on fans may remember what the process was like prior to the sandbox: add-ons were inaccessible until they were reviewed, new add-ons and updates were listed together in the same queue, and the quality of some of the reviewed add-ons was questionable.

The current add-on submission and review process was designed to surface unreviewed add-ons in a “sandbox” to testers who wanted to try them out and write reviews, while still keeping them far away from casual Firefox users just looking to customize their browser. We hoped this would alleviate developer frustration over long review times and raise the quality bar, as not every add-on would have to be “public” and reviewed in order to be distributed on the site. In its original incarnation, the sandbox excelled at keeping untested add-ons from everyday users, but was found lacking in usability for advanced users: no one could figure out the process of signing up for an account and then opting in to sandbox access in order to see the unreviewed add-ons.

In the years since there have been two iterations affecting the discoverability of the sandbox: the first removed the opt-in requirement and allowed anyone logged in with an account to see the sandbox. This made the sandbox a bit more usable and easier to understand; however, feedback from developers indicated that more needed to be done. The second iteration, currently in use today, went even further by letting logged-out users browse and install unreviewed add-ons by simply checking a box.

We get a lot of community feedback on our ideas and site features, but almost all of that comes from developers who, of course, want to make their add-ons as easy to get as possible. We rarely hear from the non-technical users installing these add-ons, which can sometimes lead to decisions that are too developer-focused. I think the current state of unreviewed add-ons is an example of one of those decisions, and that we’ve gone too far in making them widely available. Unreviewed add-ons, which are potentially harmful to your computer and your data, are trivial to find and too easy to install without understanding the risk.

Since the inception of the sandbox, we’ve had very few incidents involving unreviewed add-ons, and have been quick to investigate and respond to any reports we’ve received from our users concerning these add-ons. But as we strive to make Firefox add-ons and our website more consumer friendly and encourage the hundred million users already enjoying add-ons to tell their friends about customization, “very few” incidents is a few too many.

What are the issues?

I helped design the original sandbox and had a part in all of its subsequent iterations, and I’ve heard from developers, users, and security experts on it. I’ve agreed with many of the issues brought forward, and we think it’s time to make some big changes. In my opinion, the biggest issues with the current system are:

  • We use misleading terminology. Untested, unapproved add-ons are called “experimental”, which hints that the add-on is not ready for prime time, but doesn’t evoke enough (or any) levels of alarm or warning for a casual user. In computer vernacular, when something is in a sandbox, it is trapped in a secure container and can do no harm. This is not the case at all with the addons.mozilla.org sandbox.
  • Screenshot of Experimental buttonsThe interface doesn’t say what it means. For logged-in users, unreviewed add-ons have a green install button, the same as reviewed add-ons. People like to click on green, shiny buttons. For logged-out users, the only thing standing in the way is a checkbox that says “Let me install this experimental add-on.” We don’t explain what this actually means (“not reviewed”) or let the user make an informed decision unless they click on “What’s this?”.
  • Untested add-ons are too popular. Yesterday there were 40,000 downloads of add-ons in the sandbox. While this makes up less than 3% of all downloads yesterday, I have serious doubts that all 40,000 downloads came from advanced testers who fully understand the risks of installing unreviewed add-ons.

We need to limit the exposure of unreviewed add-ons so that if an incident does occur, only the few people who have have made an educated decision and accepted the risk will be affected.

What can we do to fix this?

Some have proposed removing the sandbox and unreviewed add-ons entirely. It would certainly solve all of the above issues, but would leave us back where we were in 2006 when developers had to wait a week or two before anyone could try out their add-on. As an add-on developer myself, I know that after I finish working on something, I want to let my friends try it out immediately.

Others have proposed spinning off a separate website for add-ons that have not been reviewed, or encouraging a third party not affiliated with Mozilla to create a website for such creations to live. This may address most of the issues listed above, but I don’t think it would be in the best interest of users. If a user arrives at the only add-on they can find that has a certain feature, they’ll probably install it regardless of any warnings the site may have (keeping in mind a third party would not have to have any warnings at all). I think it’s important that Mozilla provide some form of home for these add-ons until they can be reviewed so that we can take the appropriate cautions for users that do wish to try out the add-ons.

I think there’s a middle ground, and that’s what I’m proposing today. There are three main components:

1. Unapproved add-ons will not be shown in the add-ons gallery.
Users can be sure that every add-on they come across browsing around the website has been tested and approved by Mozilla’s editors. This would mainly affect browsing through categories, search results, and collections.
2. Users who are linked directly to an unapproved add-on by its developer can install the add-on.
After submission, developers can send the link to their unapproved add-on to anyone they’d like to try it out. This can be on their blog, website, email, instant message, etc. However, these direct links will be disjointed from the normal add-ons gallery and make it clear that this add-on has not been reviewed, as well as offer the user the option to leave and go to the gallery with only reviewed add-ons.

Screenshot of new button

3. There will be a maximum amount of time an unapproved add-on can be hosted on the site.
There are 7,500 add-ons currently in the sandbox that have never been reviewed by a human because the developers have never nominated them to show up in the “public” site. Some of these add-ons have been there for years, and are usually the source of the problems we do have.

To address this, we think a critical part of fixing the review process is setting the expectation that all add-ons submitted to the site must undergo review and using a time limit to enforce that.

Under this proposal, an add-on’s submission to addons.mozilla.org will begin its 30-day incubation period. As noted above, the add-on will not appear in the gallery but can be accessed through a direct link from the developer during this period.

The add-on must apply to the gallery sometime during this incubation period, whether on day 1 or day 30. Reminder emails will be sent at certain intervals to remind the developer of this timeframe, but if the add-on has not applied to the gallery during the 30-day window, its listing will expire and no longer be available on the website, even with a direct link.

If the developer has allowed an add-on to expire, he or she may still submit a gallery application after the 30-day window, but the add-on will not be re-activated for download until it has been approved.

The add-ons team tried to come up with details that we think are reasonable and not too complicated, yet still address many of the edge-cases we’re sure to encounter. I haven’t included all the details of this part of the propsal here, but those interested should read the full proposal and give feedback in the discussion thread.

Some additional important details that are worth mentioning in this summary are:

  • This policy will apply to add-ons currently in the sandbox, and those add-ons will expire 30 days after this new process is implemented on the website, if it is adopted. Those authors will be given months of advanced notice in order to nominate their add-ons for review before the deadline rush.
  • Currently, users who install “experimental” add-ons do not automatically update to new versions in order to limit the exposure of unreviewed code. Many users and developers do not realize or expect this, and it’s not adequately explained. Under this proposal, unapproved add-ons already have a limited exposure window, so automatic updates will be turned on for all add-ons, even if unapproved.
  • New versions of add-ons will not be available for download until they have been approved. Currently, 95% of add-on updates are reviewed in under 5 days.

If implemented, this new system will make some big changes to the add-ons website, but I think they’re needed. We’re asking for community feedback on all parts of the proposal, although if you’d like to comment, please take the time to read the full proposal and then add your thoughts in the discussion thread.

19 comments

Antonin Hildebrand

Hi Justin. Thanks for sharing this story about upcoming changes.

I’m a creator of some free extensions, mainly development tools extending Firebug: https://addons.mozilla.org/en-US/firefox/user/50466. And I want to share two stories with you.

Here is my first story. I left all of my extensions as “experimental” because I didn’t see much value in going through approval process every time I release a new version. I tried it once in the beginning with my FireRainbow extension and was rejected because it is keeping global variables which may conflict with other extensions. That is true, but I’m using 3rd party library for syntax coloring and I was unable to convince its authors to adapt their codebase to AMO guidelines. And I didn’t want to patch it and maintain my own fork. It is not that big deal, it could be done, but it just does not pay off. I have simply more important things to do. The fail is in first place in Firefox architecture that it allows shared namespace for all extensions.

My second story is about AMO rules being sometimes counter productive. My other extension FireQuery modifies Firebug’s HTML panel. Many people installed version for Firebug 1.4 and when Firebug 1.5 came out (and was updated via autoupdater) it broke its HTML panel. I released a new fixed version way before Firebug 1.5 was out, but AMO didn’t provide updates for experimental addons at that time so people had no clue there is a fixed version available. You can find many people on the twitter and over the internets blaming Firebug 1.5 to be broken and showing it in bad light. Sad!

Maybe I’m special case because my extensions are for developers, but I want to urge you to make the rules simple and developer-friendly. Honestly I’m lazy to read full proposal doc, even your article is quite brain teaser and you can see i’m quite involved. What about other developers? Anyway, “30-day incubation with expiration” rule seems to me like show stopper right now.

thanks for listening

Justin Scott (fligtar)

Hi Antonin,

Thanks for your stories. My take on the first is that yes, I agree, it’s unfortunate that a tradeoff of being able to do anything with a Firefox add-on also means things like global namespacing problems exist. Jetpack will fix that, but until then, we really do have to enforce that policy because many add-ons do break each other (and sometimes Firefox itself) because of namespace collisions.

As for your second story about unreviewed add-ons not getting updated, that is one of the things the new proposal would fix, as all add-ons submitted to the site would have updates enabled.

martimus8

I’ve been using Mozilla products over the last several years and I have seen many changes in the sandbox. Some add-ons, that were always safe, I’ve re/downloaded… they eventually end back up in the sandbox eternally for no apparent reason. It’s great to have some additional clarification and security precautions but I would really like to continue to evaluate the new additions as well. Team evaluations even from outside sources can be a good thing for Mozilla addons. Most project authors are learning the responsibilities of having a third party project management tool such as github or sourceforge. Many sites also offer “dynamic links” that automatically create a tar ball or a zip ball based off the latest and greatest trunk. Cutting people like myself out of the “testing loop” seems to be a bad idea.

I can only assume this was related to the video downloader addon that some virus scanners identified as containing malware. I only heard of one person that actually got the “recall notice” in real life. Heuristics on signature identification aren’t exactly accurate imho.

Anyhow *steps off the podium for now*… these are my thoughts.

Dave Hulbert

Slightly off topic, but is there a way to see in the Extensions manager which addons are experimental? If I’ve installed some experimental addons, these are more likely to cause a browser crash and, as you explained above, won’t be getting automatic updates. I think it’s great what is proposed for the AMO site, so people understand more about untested addons, but the Extensions manager is an important place to find information about your extensions too. Do I have to go through all my 30+ installed extensions’ AMO pages to see if any are experimental and need a manual update? (I’m sending this from my phone and don’t have Firefox available to see if I’ve just missed some of the UI; sorry if it already exists)

Dave Garrett

No, there is no way to discern sandbox/experimental/unreviewed/whathaveyou status in the Add-ons Manager. This status is a function of AMO, not Firefox itself. It would be nice if information like this was available there but it isn’t.

Dave Garrett

How does this new system affect individual file statues for an otherwise public addon? Will pending updates on the “all versions” page simply get the “unreviewed” style button with a prompt of some kind? What about older files that either weren’t reviewed or were sandboxed by an admin or the developer?

Jonathan

Hi,

While I appreciate the effort to fix some obvious flaws in the current AMO design, I think you are going in the wrong direction. As some other people did, let me share a few facts with you.

All of my extensions started as “experimental”; I’m developing three. What I uploaded for “GMail Conversation View” in the first place was merely a proof-of-concept. However, I needed exposure so that I could get feedback, ideas from users. What happened is that the users kept coming back for new versions, and left constructive and useful remarks by email. This allowed me to do a quick feedback-improve development cycle and the addon saw many versions in a few month’s time. Now I’ve settled for a Public add-on. The whole user base has been upgraded, that’s great. However, the development cycles are slower. It’s been one month today since I submitted a new version for review, and during this time, new users don’t see the latest version, only the reviewed version on the addon’s page. I don’t get feedback on new versions anymore, and only a few users install the development builds I provide. For another addon, I tried posting in Mozillazine’s development forums, but no one took the time to provide feedback. As soon as I uploaded it to AMO, even experimental, I started getting comments, and ideas for improvement.

Another point is that some experimental addon’s quality might be acceptable. One of my extensions is still experimental. That’s because it has a low user base, and because the review pointed out some nitty-gritty details that I don’t have the time currently to fix. I’m not saying that the review is bad : actually, it pointed out important things that MUST be fixed if I am to provide a high-quality addon. However, the issues mostly have to do with the configuration dialog, and can be circumvented. And I still believe the addon is useful as is, as do the users who left a comment.

Reflecting on the current state of AMO is good, and I think that “kicking out” old addons that have been there for years and have not had a single update is an excellent idea. This will remove noise from AMO and globally improve the contents. However, the community might suffer from the changes you propose, as I tried to outline with the examples above.

Jonathan

Matthew Wilson

What would be the story for binary add-ons? For example, I have an image viewer (https://addons.mozilla.org/en-US/firefox/addon/8345, not updated for FF 3.6) and last I heard, it wouldn’t be possible for it to get out of the sandbox, because the binary itself can’t be audited.

tomh

The first 2 I totally agree with, the time limiting one I’m not so sure about.

I know the idea is to make sure that there is a limit to the time security issues can be around, however the first 2 steps could have a large enough effect to stop that.

One idea is that experimental add-on could undergo a “security only” review, eg if an add-on is somehow unfinished or for debug purposes or requires someone to edit a database ect. Then it would be appropriate to let people download the add-on and not appropriate for it to be included in the main site. If it went through a safety review, it could stay in the incubator indefinitely.

Dave Dash

It would be nice for end-users to assign their level of trust to add-on developers. For example, I could say that I implicitly trust any new versions of the Password Exporter Addon whether or not their updates are in the review queue.

Whereas I might not grant that same benefit to the Delicious Addon since their unreviewed addons aren’t very stable sometimes.

Pavel Cvrček (JasnaPaka)

“1. Unapproved add-ons will not be shown in the add-ons gallery.”
I don’t think this is good idea. Many developers don’t have highly visible sites where users can find newly created add-on. With this rule we are going back to the days when sandboxed add-ons weren’t visible. As you said developers want to have his work visible immediately.

“2. Users who are linked directly to an unapproved add-on by its developer can install the add-on.”
Good idea. I think that special notification what “unapproved add-on” means is necessary.

“3. There will be a maximum amount of time an unapproved add-on can be hosted on the site.”
Good idea. I just want to say that one month is maybe too short period. I had one of my add-on in sandbox for longer time because I wasn’t sure that has enough quality for being public.

One idea:
Hide “experimental add-ons” in search by default.

Robert Kaiser

IMHO, there’s current three kinds of add-ons that are counted as “experimental”:
1) Newly submitted add-ons that were never reviewed/approved.
2) New versions of already reviewed/approved add-ons.
3) Add-ons not hosted on AMO.

I don’t think the same policies should apply to all of them. While we surely can’t guarantee that a new version doesn’t introduce bad things, it’s much less likely to be bad than a new add-on, and the same is true for add-ons that are e.g. hosted on mozdev and added to the AMO list by the new feature allowing that.

I tend to get reports from users who fail to find add-ons on AMO because the new SeaMonkey-2.0-supporting versions haven’t been reviewed yet or they are hosted externally. And new versions get few reviews as people don’t get informed by update notifications – there’s not even a possibility to opt in to such updates. I think we should try to ease things for those users in some way – and yes, I know it’s far from trivial to do that.

ff3.6ftw

I think the Problematic Extensions page which helps users solve conflicting add-ons should be more visible and known to more users, especially those who are testing experimental add-ons and are unaware of this page. Maybe include a link on the pages of the add-ons which have issues. I know this will probably be another column in a database, but it should help users resolve their issues and in the long term be more satisfied with what AMO has to offer.
http://kb.mozillazine.org/Problematic_extensions

And for developers who aren’t aware:
https://wiki.mozilla.org/Extension_Manager:Addon_Conflict_Resolution

Hope that helps anyone, I’d like to know if this idea would not be hard to implement/confusing for users.

To Robert Kaiser:
I think the issue of lack of user reviews for experimental add-ons will be fixed. Justin proposed that they will be updated automatically, and since this is limited to who will most likely give reviews, I think this is the perfect balance. I think to a large extent this will be going the right way.

Mithgol the Webmaster

If an add-on developer is a busy man and thus he cannot develop his add-on from the scratch in 30 days, then your policy effectively means that his development should start elsewhere (not in AMO), otherwise the beta-testers won’t be allowed to download the month-old add-on.

For example, if I start my add-on in AMO this week-end (March 27+28) and if after a couple of fortnights (about April 25) I feel it like ready for beta-testing, bad luck: the add-on is no longer available for download.

Justin Scott (fligtar)

That’s very true. AMO is for add-on distribution, not project hosting, even today.

ietab

One consideration is that this will cost you the ability to gauge the consumer demand for unreviewed add-ons, which might be used to influence the review queue priority. An example is an update we created to the popular IE Tab product, which quit working as of Fx 3.6. Under the current system they can find and use this update while it is waiting to be reviewed (and indeed many thousand per day do), but under the new system it would be much harder for users to find this add-on (they would need to be directed right to it).

A related thought: If all add-ons and updates were being reviewed in 1 day, then I don’t think you would need to provide unreviewed add-on hosting at all. I know you are doing your best, I just want to point out what I think the source of the problem is.

Posicionamiento web

This are some great suggestion, they should totally apply this many times we don’t know from who or where is the add-on if it has malicious code or whatever, good input.

Mariscos

To be honest i never read the privayc policy, now i will be more carefull iwht htis osrt of thing, thanks firefox

Armeria Avarez

I think for the non technical final user Firefox confuse and disturb a litle in offering so frecuently actualizations of addons at the startup. It is noted that the soft is for more advanced users.

Regards,
Armeria Alvarez

(this post is no longer accepting new comments)

show 8 pings

8 pings

Mozilla está considerando ajustar el sistema de publicación de nuevas extensiones - Zona Firefox

[...] Justin Scott publicó ayer una propuesta que sin eliminar el área experimental podría significar una mejora significativa en el proceso de admisión de nuevas extensiones. Scott se enfoca en que el actual sandbox tiene tres problemas mayores. El primero es que llamar “experimental” a una extensión que nunca ha sido probada por un ser humano no conlleva el suficiente nivel de alarma para un usuario sin conocimientos informáticos profundos. El segundo, que la interfaz es confusa, pues cuando el usuario está logueado en AMO se encuentra con botones verdes de “Instalar” tanto en las extensiones aprobadas como en las “experimentales”. El tercero: el área experimental está adquiriendo una desmedida popularidad, con alrededor de 40.000 descargas al día, lo que significa un riesgo potencial si consideramos que muchas de estas extensiones no han sido convenientemente probadas. [...]