Tag Archives: retrospective

Fedora 15 QA retrospective

It’s that time again.  Time to review, discuss and learn from any hiccups encountered while testing the previous release.  No matter how successful a release is, there is always something to improve on.  Fedora 15 was no exception.

First, some ground rules … we’re not trying to solve the world problems with this retrospective.  It’s focused on Fedora QA, our interactions with other teams, and what we can improve on to better accomplish our goals.  For a more general release retrospective, Jared already has an app for that.  Next, this isn’t the place to point fingers and taunt other teams/people.  Use your own blog for that :).  Let’s keep the retrospective as objective and actionable as possible.  For example, when recording issues, it’s important to know the bug(s) involved and what factors contributed to the problem(s).  I’m not interested in what Steve thinks Mary said that Jillian heard after passing out at 31 flavors last night.

Next, a bit of history.  The process starts early each release when I draft a retrospective wiki page to track issues as they happen.  I break the page down into three groups; 1) what worked (sometimes we get things right!), 2) what didn’t work (learn from our mistakes etc…), 3) wishlist (aka pony).  It’s just my nature to want to know what went wrong, and try to fix it.  Accordingly, most of the content focuses on problems we encounter during the release.  Also, I wanted to have some concept of scope … which is where the wishlist/pony group comes in.  I still don’t know if it really fits, but I find it interesting to see what comments people have without constraints.

The retrospective concept doesn’t really work if only one person provides feedback.  So, I try to get the word out so anyone can add issues.  This is an area where I could certainly do better, probably by blogging more about it.  As the release progresses, I beg/plead/nag people closest to the issues to add some notes to the wiki.  I’m sure most people get tired of my nagging for content, but I’ve seen too often where we forget the causes of a critical problem, or don’t properly record the conditions leading up to a failure.  Without accurate data on the cause of problems, any corrective measures are just busy work.  So, the release chugs along, and hopefully the list of good/bad/wishlist issues matures.

mental note … I need to document this in a SOP

Okay … so all that preamble just to let folks know I completed a draft of recommendations based on the Fedora 15 QA retrospective feedback.  I’ll be sending this out to the test@ list for review shortly.  Check it out, and tell me what you think.  See something missing, don’t agree with something … just let me know.

Image by Camilo via flickr used under a Creative Commons license.



Fedora 14 QA retrospective

For several releases now, the Fedora QA team hosted retrospectives aimed at identifying process gaps and recommending corrective action.  For those unfamiliar with the term retrospective, these events are sometimes referred to as lessons learned or, the morbidly named, post-mortem.  We treat the retrospectives as living documents throughout a release.  Noting the good, bad and ugly in an objective manner as issues surface.  I’ve found recording issues as they happen paints a much more accurate picture, rather than trying to remember everything all at once after the product has been released declared GOLD.  At the end of a release, we discuss, debate, propose recommendations and prioritize the work for the upcoming release.

Rear view mirror taken from http://www.lessonsfromthecockpit.com/blog/2009/4/5/dont-look-back.html

We followed this process for several Fedora releases and most recently, with Fedora 13, created TRAC tickets to manage progress towards recommendations.  There are still a few tickets remaining, but that’s okay.  This is the first time we used TRAC to manage recommendations.  I wouldn’t be surprised if we over estimated how much we could accomplish in the short time between releases.  Of course, another way to look at this list is … if you’re interested in getting more involved with the Fedora QA team, review the tickets, grab one that looks interesting to you.

For Fedora 14, the retrospective process will continue.  I started the wiki page a bit later than I prefer, but the page is up and ready for your ideas and suggestions … https://fedoraproject.org/wiki/Fedora_14_QA_Retrospective.  To help the discussion, I’ll list my two most recent entries …

  1. Virtualization Test Day – Participation in the virtualization test day was low and not what we had hoped for.  Why? What factors contributed to low attendance?  What can we do to improve virtualization test day attendance in the future?
  2. Root cause analysis of RHBZ#638091 – Kernel panic during boot after updating to image:Package-x-generic-16.pngglibc-2.12.90-13 (see bodhi).  The system would be fine until the prelink command ran, and then it would never boot and all commands would segfault on a running system. The issue was initially given positive proventester feedback as the system survived after a reboot.  The feedback was later reverted after the testers system was unusable+unbootable after prelink had run.  The good news, the problem was discovered in updates-testing before it landed in updates. However, it would be good to have some documented test procedures for a subset of packages to guide proventesters

What other issues/ideas would you like to list on the Fedora 14 QA Retrospective?  While you’re in the mood for feedback, please note that the QA team isn’t the only group interested in keeping track of issues.  Check out John Poelstra’s thoughts on the Fedora 14 Schedule Retrospective, as well as Kevin Fenzi’s growing list of Updates lessons.