Florian Quèze

To content | To menu | To search

Sunday, July 27 2014

Firefox is awesome

"Hey, you know what? Firefox is awesome!" someone exclaimed at the other end of the coworking space a few weeks ago. This piqued my curiosity, and I moved closer to see what she was so excited about.

When I saw the feature she was delighted of finding, it reminded me a similar situation several years ago. In high school, I was trying to convince a friend to switch from IE to Mozilla. The arguments about respecting web standards didn't convince him. He tried Mozilla anyway to please me, and found one feature that excited him.
He had been trying to save some images from webpages, and for some reason it was difficult to do (possibly because of context menu hijacking, which was common at the time, or maybe because the images were displayed as a background, …). He had even written some Visual Basic code to parse the saved HTML source code and find the image urls, and then downloaded them, but the results weren't entirely satisfying.
Now with Mozilla, he could just right click, select "View Page Info", click on the "Media" tab, and find a list of all the images of the page. I remember how excited he looked for one second, until he clicked a background image in the list and the preview stayed blank; he then clicked the "Save as" button anyway and… nothing happened. Turns out that "Save as" button was just producing an error in the Error Console. He then looked at me, very disappointed, and said that my Mozilla isn't ready yet.
After that disappointment, I didn't insist much on him using Mozilla instead of IE (I think he did switch anyway a few weeks or months later).

A few months later, as I had time during summer vacations, I tried to create an add-on for the last thing I could do with IE but not Firefox: list the hostnames that the browser connects to when loading a page (the add-on, View Dependencies, is on AMO). I used this to maintain a hosts file that was blocking ads on the network's gateway.
Working on this add-on project caused me to look at the existing Page Info code to find ideas about how to look through the resources loaded by the page. While doing this, I stumbled on the problem that was causing background image previews to not be displayed. Exactly 10 years ago, I created a patch, created a bugzilla account (I had been lurking on bugzilla for a while already, but without creating an account as I didn't feel I should have one until I had something to contribute), and attached the patch to the existing bug about this background preview issue.
Two days later, the patch was reviewed (thanks db48x!), I addressed the review comment, attached a new patch, and it was checked-in.
I remember how excited I was to verify the next day that the bug was gone in the next nightly, and how I checked that the code in the new nightly was actually using my patch.

A couple months later, I fixed the "Save as" button too in time for Firefox 1.0.

Back to 2014. The reason why someone in my coworking space was finding Firefox so awesome is that "You can click "View Page Info", and then view all the images of the page and save them." Wow. I hadn't heard anybody talking to me about Page Info in years. I did use it a lot several years ago, but don't use it that much these days. I do agree with her that Firefox is awesome, not really because it can save images (although that's a great feature other browsers don't have), but because anybody can make it better for his/her own use, and by doing so making it awesome for millions of other people now but also in the future. Like I did, ten years ago.

Friday, July 25 2014

Converting old Mac minis into CentOS Instantbird build slaves

A while ago, I received a few retired Mozilla minis. Today 2 of them started their new life as CentOS6 build slaves for Instantbird, which means we now have Linux nightlies again! Our previous Linux build slave, running CentOS5, was no longer able to build nightlies based on the current mozilla-central code, and this is the reason why we haven't had Linux nightlies since March. We know it's been a long wait, but to help our dear Linux testers forgive us, we started offering 64bit nightly builds!

For the curious, and for future reference, here are the steps I followed to install these two new build slaves:

Partition table

The Mac minis came with a GPT partition table and an hfs+ partition that we don't want. While the CentOS installer was able to detect them, the grub it installed there didn't work. The solution was to convert the GPT partition table to the MBR older format. To do this, boot into a modern linux distribution (I used an Ubuntu 13.10 live dvd that I had around), install gdisk (sudo apt-get update && sudo apt-get gdisk) and use it to edit the disk's partition table:

sudo gdisk /dev/sda
Press 'r' to start recovery/transformation, 'g' to convert from GPT to MBR, 'p' to see the resulting partition table, and finally 'w' to write the changes to disk (instructions initially from here).
Exit gdisk.
Now you can check the current partition table using gparted. At this point I deleted the hfs+ partition.

Installing CentOS

The version of CentOS needed to use the current Mozilla build tools is CentOS 6.2. We tried before using another (slightly newer) version, and we never got it to work.

Reboot on a CentOS 6.2 livecd (press the 'c' key at startup to force the mac mini to look for a bootable CD).
Follow the instructions to install CentOS on the hard disk.
I customized the partition table a bit (50000MB for /, 2048MB of swap space, and the rest of the disk for /home).

The only non-obvious part of the CentOS install is that the boot loaded needs to be installed on the MBR rather than on the partition where the system is installed. When the installer asks where grub should be installed, set it to /dev/sda (the default is /dev/sha2, and that won't boot). Of course I got this wrong in my first attempts.

Installing Mozilla build dependencies

First, install an editor that is usable to you. I typically use emacs, so: sudo yum install emacs

The Mozilla Linux build slaves use a specifically tweaked version of gcc so that the produced binaries have low runtime dependencies, but the compiler still has the build time feature set of gcc 4.7. If you want to use something as old as CentOS6.2 to build, you need this specific compiler.

The good thing is, there's a yum repository publicly available where all the customized mozilla packages are available. To install it, create a file named /etc/yum.repos.d/mozilla.repo and make it contain this:

[mozilla]
name=Mozilla
baseurl=http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/
enabled=1
gpgcheck=0

Adapt the baseurl to finish with i386 or x86_64 depending on whether you are making a 32 bit or 64 bit slave.

After saving this file, you can check that it had the intended effect by running this comment to list the packages from the mozilla repository: repoquery -q --repoid=mozilla -a

You want to install the version of gcc473 and the version of mozilla-python27 that appear in that list.

You also need several other build dependencies. MDN has a page listing them:

yum groupinstall 'Development Tools' 'Development Libraries' 'GNOME Software Development'
yum install mercurial autoconf213 glibc-static libstdc++-static yasm wireless-tools-devel mesa-libGL-devel alsa-lib-devel libXt-devel gstreamer-devel gstreamer-plugins-base-devel pulseaudio-libs-devel

Unfortunately, two dependencies were missing on that list (I've now fixed the page):
yum install gtk2-devel dbus-glib-devel

At this point, the machine should be ready to build Firefox.

Instantbird, because of libpurple, depends on a few more packages:
yum install avahi-glib-devel krb5-devel

And it will be useful to have ccache:
yum install ccache

Installing the buildbot slave

First, install the buildslave command, which unfortunately doesn't come as a yum package, so you need to install easy_install first:

yum install python-setuptools python-devel mpfr
easy_install buildbot-slave

python-devel and mpfr here are build time dependencies of the buildbot-slave package, and not having them installed will cause compiling errors while attempting to install buildbot-slave.

We are now ready to actually install the buildbot slave. First let's create a new user for buildbot:

adduser buildbot
su buildbot
cd /home/buildbot

Then the command to create the local slave is:

buildslave create-slave --umask=022 /home/buildbot/buildslave buildbot.instantbird.org:9989 linux-sN password

The buildbot slave will be significantly more useful if it starts automatically when the OS starts, so let's edit the crontab (crontab -e) to add this entry:
@reboot PATH=/usr/local/bin:/usr/bin:/bin /usr/bin/buildslave start /home/buildbot/buildslave

The reason why the PATH environment variable has to be set here is that the default path doesn't contain /usr/local/bin, but that's where the mozilla-python27 packages installs python2.7 (which is required by mach during builds).

One step in the Instantbird builds configured on our buildbot use hg clean --all and this requires the purge mercurial extension to be enabled, so let's edit ~buidlbot/.hgrc to look like this:
$ cat ~/.hgrc
[extensions]
purge =

Finally, ssh needs to be configured so that successful builds can be uploaded automatically. Copy and adapt ~buildbot/.ssh from an existing working build slave. The files that are needed are id_dsa (the ssh private key) and known_hosts (so that ssh doesn't prompt about the server's fingerprint the first time we upload something).

Here we go, working Instantbird linux build slaves! Figuring out all these details for our first CentOS6 slave took me a few evenings, but doing it again on the second slave was really easy.

Tuesday, May 20 2014

Summer of Code 2014 projects

The coding period is starting for Google Summer of Code 2014. I am pleased to list the 28 projects (7 more than last year!) that are mentored this year under the Mozilla banner.

The title of each project is linked to the location where the student will be posting weekly updates on their progress, if you want to follow along with a project you are interested in. I’m sure they would appreciate any help or advice you have :-) Please help them write useful code during the summer!

Project Title Student Mentor
MDN – Bug Fixing and Polishing Akshay Aurora David Walsh
Firefox OS Games Resources Andre Alves Garzia Soledad Penades
Development of both online as offline speech recognition to B2G and Firefox Andre Natal Guilherme Gonçalves
Porting key Meemoo modules to NoFlo Vilson Vieira Forrest Oliphant
Functional Test Suite and Features for QA Taskboard - One and Done Pankaj Malhotra Bob Silverberg
Implement Zest recorder and runner Sunny Gogoi Simon Bennetts
Improve font handling and text shaping in Servo Patryk Obara Patrick Walton
Add learning capability in the Gaia Keyboard prediction Sukant Garg Jan Jongboom
Browsercast Gabriel Ivanica Greg Wilson
Migrate from YUI2 to YUI3 . Bug 453268 Ishitva Goel Byron Jones
Implement Speech Synthesis on Desktop Firefox Yashasvi Girdhar Eitan
Add markdown support to Bugzilla Koosha Khajehmoogahi David Lawrence
Improved messaging functionality and interaction in the SMS app Kumar Rishav Gabriele Svelto
Update Lightning Invitations to Latest Specification Malintha Fernando Mohit Kanwal
Implement XMLHttpRequest in Servo Manish Goregaokar Josh Matthews
Instantbird: WebRTC Support Mayank Kumar BenediktP
Instantbird: Indexed Chat History and Infinite Conversation Scrollback Nihanth Subramanya aleth
Peer instruction on the Web Piotr Banaszkiewicz mhoye
Implementation of SVG Backend for PDF.js Pramodh KP yury
A Desktop Environment based on B2G Towfique Anam Fabrice Desré
Math virtual keyboard Raniere Silva Salva
Improve Calendar Backends Reid Anderson Philipp Kewisch
Kitherder, web application for the Security Mentorships program Renee Cheung Yvan Boily
FileLinks in IMs / File transfer Saurabh Anand clokep
Thunderbird - Make the unit test framework work with maildir mailbox format Suyash Agarwal R Kent James
Mozilla Intellego -- Terminology-driven automatic translation of web sites. Tharshan Muthulingam Jeff
Implement Electrolysis Support for Addon SDK Tomislav Jovanovic gozala
Mochitest Failure Investigator Vaibhav Agrawal joelmaher

Tuesday, April 22 2014

Thoughts about mentoring Summer of Code students

I've mentored Google Summer of Code students 3 years in a row, and I would like to share with new GSoC mentors what I've learned, in the hope that it can help them get more out of the summer. I'm going to assume that most mentors share these 3 goals:
  • get usable code at the end of the summer
  • train long term contributors who will stay active after the end of the summer
  • avoid spending disproportionate amounts of time mentoring.
I can't overstate how important it is to give the student good working habits from the very beginning. If you setup a good working relationship from the beginning, you are on for an awesome summer. If you let a student do things that frustrate you without sending immediate feedback about it, you'll both suffer during the whole summer.
Here are some frequent problems that are likely to happen if you don't pay enough attention in the beginning:
  • Because of the way school works, students are used to hiding their work until it's ready to be evaluated. Training them to show work-in-progress code so that you can provide early feedback and effectively help them takes time. You'll likely need to ask "show me the code, please" several times, encourage them, and comfort them that it's OK to show unfinished non-working code.
  • Students are used to receiving feedback from one person, and are likely to send their work only to their mentor when requesting feedback or expecting evaluation. Encourage the student to discuss the project with all the relevant people and more generally to interact with the community; avoid private discussion of technical matters. Doing this has 2 interesting consequences: the time you'll spend mentoring the student will be dramatically reduced because the mentoring load will be spread across all your team (for me this often meant that when my student asked a question on IRC, he received a good and timely answer before I had even read the question!), and the student will feel like he's part of your team, which increases the likelihood of staying around after the summer.
  • Unless given specific directions, students tend to work on a branch or on a repository somewhere, and think about merging their code or getting a review only at the end of the summer. It's common to see student applications where the schedule says that they will do all the coding work, and then get a review during the last week of the summer to land the code. Experienced Mozilla developers knows there's no way sending a several-months-of-work-worth patch to someone's review queue and expecting a timely review will work. But students usually dramatically underestimate both the time it takes to get a review AND the time it takes to address the feedback (probably because at school they hand their work and only expect a grade, or a pass/fail in return). As a mentor it's your job to ensure they get their code through review. Urge students to write small patches, and get them reviewed as soon as possible. Students (and other developers ;)) will be tempted to 'just add one more little feature before attaching the patch'. Resist! Filing a new bug is easy. If the schedule of a student involves writing code without submitting it for review and inclusion, this raises a red flag for me: is there really no way to break this down to smaller pieces that could land separately? Don't hesitate to rediscuss the schedule with the student if you aren't fully happy with it.
  • Given that the students are being paid money to participate in GSoC, it makes sense that lots of students will consider summer of code as "work", and have money be the primary motivator; which means that once they get the final payment, they disappear. This feeling can't be avoided in all cases, but if we can get the student to instead feel that the money is an opportunity to not do any other work during that time to focus full-time on volunteering for a project they care about, the student is likely to stay around for much longer than the summer. This can be done by treating the student more like a volunteer than like someone paid to be on your team. Get the student to have code landing early, and often. Get the student addicted to restarting his nightly build and seeing his code is in use there. Encourage the student to not only fix things directly related to his GSoC project, but to also spend a few hours here and there fixing his own itches and looking into other bugs that make the product better for his own use cases.
Here is one suggestion to get students started: during the community bounding period, assign them a 'small' project to warm up. The ideal characteristics of such a project are:
  • will get the student to go through all the processes of your team (using bugzilla, attaching patches, requesting review, handling review comments, …)
  • will get the student to touch code areas that will be relevant during the summer.
  • can be completed by the student before the ends of the community bounding period (think of it as a 2-weeks project, but size it like something you would do in a rainy week-end. Remember that most of the time spent by the student will not be actual coding, but learning to interact with your team, and this is the very reason why this project is done during the community bounding period).
  • once this small project lands, it will be highly visible, and will cause the student to receive appreciation from other people in your team/community.
It's not always possible to find a project fitting perfectly here, but it's usually possible to find something close enough to fit the purpose of getting the student started. The Instantbird team will be using this 'small community bounding project' approach to starting the mentoring relationship for the third time this year.
I tend to think that getting the student started on the right track is the most important thing a mentor needs to do, and if it's done correctly the rest of the mentoring is easy and pleasurable.

Thursday, March 20 2014

Summer of Code Student application deadline approaching

This is just a reminder to students interested in applying for Google Summer of Code 2014: the application deadline is "21 March 19:00 UTC" and no late application will be accepted.

If you intend to apply to Google Summer of Code this year and haven't submitted your application yet, don't wait: apply now!

Friday, February 28 2014

Google Summer of Code Student who went on to become a Mentor/Org Admin

Since 2006, I've probably tried every possible role as a Google Summer of Code participant: accepted student, rejected student, submitter of a rejected organization application, mentor of successful students, mentor of a failing student, co-administrator, and now administrator. Here's my story:

  • In 2006, I had the pleasure of being selected to participate as a student for the Mozilla organization. The work I did on the Page Info dialog eventually shipped as part of Firefox 3.0.
  • In 2007 I applied as a student again, to a different organization, but unfortunately wasn't selected. However, I received an email from someone from the organization who told me the application was good and they would have liked to take me as a student if they had received more slots. He offered to mentor me if I decided to move forward with the project anyway, which I did! I completed the project and in October 2007, Instantbird 0.1 was released. Instantbird is a cross platform, easy to use, instant messaging client based on Mozilla technologies.
  • In 2007, I was selected by Mozilla for an internship in the Firefox team, and spent the end of the year in their headquarters in Mountain View, California. I think this was in large part because people were happy about the work I did in 2006 as a Summer of Code student.
  • In 2008, 2009 and 2010, I focused on Instantbird, which had a growing community of volunteers improving it every day. I sent Summer of Code organization applications on behalf of the Instantbird community, but they weren't accepted. When I asked for feedback, I was told that we should try to work with Mozilla.
  • And this is what we did in 2011! Gerv agreed to let us add Instantbird project ideas to the Mozilla idea list. In 2011 I mentored a student, who completed successfully his work on implementing XMPP in JavaScript for Instantbird. In October I attended the mentor summit in Mountain View.
  • In 2012, I mentored another student, who did some excellent work on improving the user experience for new users during the first run of Instantbird.
  • In 2013, Gerv, who had been handling Summer of Code for Mozilla since the beginning in 2005, asked me if I would be interested in eventually replacing him as an Administrator. After some discussion, I accepted the offer, and we agreed to have a transition period. In 2013, I was backup administrator. I also mentored for the third year in a row. Unfortunately I had to fail my student. This was frustrating, but also a learning experience.
  • In 2014, I submitted the Summer of Code organization application on behalf of Mozilla, it was accepted and we are looking forward to another great summer of code!
  • On a more personal note, after being a Mozilla volunteer for years (since 2004), working on the Thunderbird team (from 2011 to 2012), and then on WebRTC apps (since October 2012), I'm starting in March 2014 as a full time engineer in the Firefox team. Which is the team for which I was a Summer of Code student, 8 years ago.

Thursday, February 27 2014

What I remember of flying British Airways…

I'll remember British Airways as a an airline where the service is as poor as the ticket is expensive. Here is how my journey from San Francisco to France happened on February 14, 15 AND 16.

BA286

It started going awry at the end of the BA286 flight (which by the way was late: departed at 20:38 instead of 19:45 from SFO). When the aircraft started its descent to London Heathrow, cold water started dripping from the ceiling of the cabin, onto my shoulder and the armrest of my seat (45G). I pressed the button to call a flying attendant. Nobody came. When leaving the aircraft, I casually mentioned the problem to a flying attendant who was standing by so that this could get fixed and avoid a shower for the new passenger. His reply was "oh, it's normal, it's condensation, it happens sometimes. There's nothing we can do about it. It's just that you are our unlucky passenger today."
Well, maybe there's nothing British Airways can do about it, but when I had the exact same incident a few years ago on an Air France flight, the hostess rushed to stop the rain by absorbing all the water that was on the ceiling with a pile of paper towels, and apologized for the inconvenience.
This small incident was actually just the start of a continued shower of disappointments during this journey.

Heathrow

Heathrow was not my final destination, but just a (dis)connection. A long connection (2+ hours) which became a very long connection, because the BA322 flight to Paris was delayed. Originally scheduled to depart at 17:50 from gate B33 (or at least I think that's the gate the person at the passport check told me), it was announced as "gate opens at 19:59"!
It was quite obvious I would miss my train connection in CDG (21:06) and have to change my ticket. I estimated the new arrival time in Paris: given that the flight was now expected to depart around 20:30, there was no way it could reach Paris before 22:07, the time of the last train toward Lille. Taking the BA322 flight meant being stuck in CDG for the night.
After noticing this, I researched the other possible options, and reached out to the British Airways staff to request their assistance.
Another possible option was taking the BA336 flight for Paris Orly (it was late too, and wasn't closed yet. It would have taken me to Paris at a time when finding a train connection from Paris would have still been possible). The staff told me that flight was already full and there was no way to get a seat there. Maybe that's true, but I have doubts, given that I heard numerous calls more than half an hour after that indicating that the gate would close soon and passengers should proceed immediately.
The last possible option was to take the metro to London St. Pancras, and take a Eurostar to Lille. That was actually the safest and fastest possible option, but the BA staff was very reluctant, saying that BA would probably not pay for that ticket, and that getting back my checked in suitcase would be quite a hassle.
They advised that I stick with the BA322 flight, however late it would be, and figure out something in Paris with the BA staff there, although BA would possibly not pay the hotel because the contract was to transport me to Paris, and the train wasn't part of the contract, blah blah blah. They said that I should keep receipts of any incurred cost and file a complaint to British Airways, because even though they had no obligation to pay for the hotel, they do study cases on a case by case basis, and may decide to do something.
They gave me a £5 voucher for a refreshment and instructed me to return to the main gate.
With that voucher I got a small (but £4.70, because, you know, everything is super expensive in airports…) tuna sandwich that kept me from starving, but not from being hungry.

BA322

The flight finally departed at 21:21 (according to FlightAware), and landed at 23:04. It seems the flight was shorter than usual (maybe the captain was trying to catchup a little bit on the schedule?), and it seemed to take the crew by surprise when they hear that we were starting our descent to Paris and people should return to their seats. Only about two third of the passengers had been offered snacks and hot drinks. It was too late for the others. I was hoping to get some food and a coffee during the flight to stay awake and be ready to affront the inevitable complications in Roissy; no luck… All I got was a coke.

Roissy

Of course, at the Roissy airport, the British Airways staff did a really good job of being invisible and impossible to find. The only person I could talk to was the lady at the baggage claim desk, who was as nice as she was unable to do anything useful for me. The best she could do was call some British Airways people, who said they wouldn't deal with the passengers in Paris, and that anyway, the delay was not their fault, but only due to the bad weather. No, it was not! Most flights in Heathrow were delayed, yes, but not by more than 3 hours. (I actually have a photo showing that at 19:42, all the flights with original departure times before 18:30 but BA322 had their gate already open, or had already departed.)
The person at the baggage claim desk told me to follow the signage to the information desk where they could help me find an hotel. I did. And finally found a closed desk, with nobody around. At midnight, Roissy is deserted, and the remaining people are mostly homeless people. The only man I could find near the information desk was busy collecting and storing away luggage trolleys. He indicated me where I should go to find the hotel shuttle. I waited there in the cold for what seemed like half an hour, then hopped into the shuttle. I had no idea of which hotel to go to. I asked the bus driver for advice about which one would be the least expensive. He dropped me at the "Première Classe", which indeed seemed to be a cheap hotel with minimal service. The guy at the front desk scolded me for arriving at his hotel at 1am without a reservation, told me how lucky (really?!?) I was that a room was available, and urged me to at least call from the airport next time (to be honest, I don't see how I could have done that without help from the British Airways staff, especially at a time when the airport's info desk was closed). 1am, I reached my room and collapsed on the bed. The night cost me 65.47euros, that I think should have been paid by British Airways.
Next morning, I took a quick breakfast, a the shuttle back to the airport, and finally took a train to Lille.

Arrival

I reached home around noon. The same evening, I started not feeling very well… and I spent the next 2 days in bed with fever. I have no doubt that this was at least in large part (if not fully) caused by British Airways' incapability of taking care of passengers blocked because of delays. I had almost no food on Saturday (a small breakfast in the first flight, and a tiny sandwich in Heathrow. I had no food in the flight to Paris, despite the tickets stating there would be snacks. I was dropped in the middle of the night in a cold and deserted airport, with no connection departing from it until the next day, and with nobody taking care of setting up accommodation for the night. Is this the level of service you would expect for a ticket that was paid more than $2,000? I'm disappointed. Very disappointed.

Thursday, February 6 2014

Mozilla and Google Summer of Code, FOSDEM talk

Last Saturday at FOSDEM, I had the pleasure of giving a talk with Gerv about Mozilla's participation in Google Summer of Code. If you missed us, you can see the slides.

This post is also a good opportunity to remind you to add your ideas to our brainstorm wiki page now if you are interested in mentoring a student this summer.

Friday, January 24 2014

Project ideas for Summer of Code 2014

Google will be running this year the 10th edition of Summer of Code. Mozilla has had the pleasure of participating every year so far, and we are hoping to participate again this year. In the next 3 weeks, we need to prepare a list of suitable projects to support our application.

Can you think of a 3-month coding project you would love to guide a student through? This is your chance to get a student focusing on it for 3 months! Summer of Code is a great opportunity to introduce new people to your team and have them work on projects you care about but that aren't on the critical path to shipping your next release.

Here are the conditions for the projects:

  • completing the project should take roughly 3 months of effort;
  • any part of the Mozilla project (Firefox, Firefox OS, Thunderbird, Instantbird, SeaMonkey, Bugzilla, L10n, NSS, IT, and many more) can submit ideas, as long as they require coding work;
  • there is a clearly identified mentor who can guide the student through the project.


If you have an idea, please put it on the Brainstorming page, which is our idea development scratchpad. Please read the instructions at the top – following them vastly increases your chances of your idea getting added to the formal Ideas page.

The deadline to submit project ideas and help us be selected by Google is February 14th.

Please feel free to discuss with me any Summer of Code question you may have.

Tuesday, January 21 2014

Instantbird bugs should now go to bugzilla.mozilla.org

From 2007 to late 2013, Instantbird bugs were tracked on bugzilla.instantbird.org. This was fine as long as Instantbird was completely separate from all other Mozilla products, but when Instantbird's chat backend was re-used to add instant messaging to Thunderbird, we started having bugs that were relevant for both Instantbird and Thunderbird. Having them in two separate bug trackers was messy and caused lots of duplicates.

We started discussing this at MozCampEU2012 in Warsaw. It quickly became apparent that getting a product created for Instantbird and another one for the chat core wasn't a problem, but that importing our existing bugs would require more work. The plan at this time was to import the existing BIO (bugzilla.instantbird.org) database into BMO (bugzilla.mozilla.org). Unfortunately, while it seems bugzilla already supports importing bugs from plenty of other bug trackers, it doesn't support importing bugs from itself. Some people from the BMO team thought writing such an importer would be a nice and not-very-difficult project, so they agreed to work on it. We filed a bug to figure out the details. And nothing happened for months, because there were (understandably) higher priorities for the BMO team. Ten months later, the BMO team concluded that they were unlikely to ever get time for this project, and suggested we work on another solution instead: using BzAPI to do the import ourselves. This was more work than we could get done immediately, so we kept that in mind for later. Patrick Cloke started experimenting with BzAPI but couldn't get enough of my and others' attention to complete the project.

In December 2013, we decided to move forward with the import, and requested the creation of the products just before the holidays. We drafted a plan in which we broke down the work that needed to happen in several steps that could each be performed by a (relatively) simple and independent script:

  1. first, download all the data from bugzilla.mozilla.org in JSON, create a file for the data of each bug
  2. slice the content of each file from 1. into a stream of events (each event is an action performed by the bugzilla users).
  3. transform the sliced events into events that would apply on BMO (eg. change product names when they don't match exactly on BIO and BMO)
  4. merge the events of all bugs into a single file; without duplicates (eg. when marking a bug as a duplicate of another bug, this creates events on both bugs, but we only need to import one, the other happens automatically).
  5. replay events on BMO: this needs to create a map of new bug and attachment numbers, and replace them automatically in comments.

The plan looked simple on paper (well, on an etherpad). Reality got in the way, with plenty of not-really-expected complications:

  • BzAPI has lots of documented (eg. can't create an attachment at the same time as filing a bug; can't set the component without also setting the product, ...) and undocumented (eg. UTF8 is supported when filing a bug or adding comments, but not when commenting on an attachment) limitations.
  • The format of data downloaded from BzAPI is significantly different from the format of data we need to send it to do changes. This is especially true for attachment flags.
  • Sometimes the values in bug fields don't match what's in the history of events that happened to the bug. This problem occurs when users changed their bugzilla email address, when products got renamed. The worst case we had to deal with was a target milestone that was renamed, then a new target milestone was created with the original name, and some bug's milestone was changed from the renamed milestone to the new milestone with the original name. This caused an interesting complication because we ended up with changes where the mapped value was the same before and after the change.
  • We initially started using the js shell for our scripts, but to do HTTP requests, we needed XPCOM, so we switched to xpcshell instead. Unfortunately, xpcshell's readline() function truncates input data after 255 characters. Later I discovered that xpcshell doesn't handle UTF8 input/output by default, and had to use nsIScriptableUnicodeConverter for that.

Our scripts are all in the bio-merge hg repository. The result of step 4. is a 120MB text file with each line being the JSON representation of an event that needs to be replayed. We tested replaying events on bugzilla-dev.allizom.org first for a single bug, then for a few set of bugs that had dependencies. Finally we replayed our whole set of bugs (2294 bugs, 20984 events), and after fixing the issues that surfaced during that test run, we were ready to do the final import.

The final import took 848 minutes to complete (14 hours). 240 events failed to replay because some components didn't have on BMO the exact names the script expected. I fixed the script and replayed these again.

We finished by adding comments in all existing BIO bugs giving the new URL, and closed BIO (it's now read-only).  As of 2013-12-30, new Instantbird bugs should be filed on bugzilla.mozilla.org, in the Instantbird product for UI bugs, in the Chat Core product for backend issue (this part of the code is shared with Thunderbird), and in the Instantbird Servers product of issues with our servers.

- page 1 of 13