Tag Archives: qa

Fedora 15 QA retrospective

It’s that time again.  Time to review, discuss and learn from any hiccups encountered while testing the previous release.  No matter how successful a release is, there is always something to improve on.  Fedora 15 was no exception.

First, some ground rules … we’re not trying to solve the world problems with this retrospective.  It’s focused on Fedora QA, our interactions with other teams, and what we can improve on to better accomplish our goals.  For a more general release retrospective, Jared already has an app for that.  Next, this isn’t the place to point fingers and taunt other teams/people.  Use your own blog for that :).  Let’s keep the retrospective as objective and actionable as possible.  For example, when recording issues, it’s important to know the bug(s) involved and what factors contributed to the problem(s).  I’m not interested in what Steve thinks Mary said that Jillian heard after passing out at 31 flavors last night.

Next, a bit of history.  The process starts early each release when I draft a retrospective wiki page to track issues as they happen.  I break the page down into three groups; 1) what worked (sometimes we get things right!), 2) what didn’t work (learn from our mistakes etc…), 3) wishlist (aka pony).  It’s just my nature to want to know what went wrong, and try to fix it.  Accordingly, most of the content focuses on problems we encounter during the release.  Also, I wanted to have some concept of scope … which is where the wishlist/pony group comes in.  I still don’t know if it really fits, but I find it interesting to see what comments people have without constraints.

The retrospective concept doesn’t really work if only one person provides feedback.  So, I try to get the word out so anyone can add issues.  This is an area where I could certainly do better, probably by blogging more about it.  As the release progresses, I beg/plead/nag people closest to the issues to add some notes to the wiki.  I’m sure most people get tired of my nagging for content, but I’ve seen too often where we forget the causes of a critical problem, or don’t properly record the conditions leading up to a failure.  Without accurate data on the cause of problems, any corrective measures are just busy work.  So, the release chugs along, and hopefully the list of good/bad/wishlist issues matures.

mental note … I need to document this in a SOP

Okay … so all that preamble just to let folks know I completed a draft of recommendations based on the Fedora 15 QA retrospective feedback.  I’ll be sending this out to the test@ list for review shortly.  Check it out, and tell me what you think.  See something missing, don’t agree with something … just let me know.

Image by Camilo via flickr used under a Creative Commons license.

 

{{Key_press}}

If you haven’t heard me say it before … “I really like media wiki.”  It’s tremendously flexible, clearly very well-used and maintained, and as there is an abundance of useful content on mediawiki.org. If you are trying to do something on the wiki, chances are someone has already done it.

For example, displaying test results on the wiki (refer to {{Result}}).  While this was started by Will Woods, it later adapted based on experiences with templating on OLPC’s wiki.  Also adapted from OLPC’s wiki is our use of {{QA/Test_Case}}.  Finally, as Athmane recently pointed out, I borrowed {{!}} and {{=}} from mediawiki so you can make use of those characters when using a template.  Who cares?  Well, if you’ve ever attempted to write an admonition (e.g. {{admon/warning}}), you may have noticed it doesn’t format properly when you include certain characters (notably = and | which have special meaning in mediawiki template syntax).

Without “escaping” the special characters, your template will render improperly as seen below …

{{admon/warning|Warning!|Sometimes a=b and b=c. 
In that case, a=c. :-| }}
{{admon/warning|Warning!|Sometimes a=b and b=c.  In that case, a=c. :-| }}

This doesn't look correct.

This can be corrected by using the {{!}} and {{=}} templates, as shown below.

{{admon/warning|Warning!|Sometimes a{{=}}b and b{{=}}c. 
In that case, a{{=}}c. :-{{!}} }}
{{admon/warning|Warning!|Sometimes a{{=}}b and b{{=}}c. In that case, a{{=}}c. :-{{!}} }}

Much better!

Today I found another really helpful series of templates for something I’ve wanted for a long time.  In Fedora QA, we write and maintain a lot of test cases on the wiki.  Often in a test case, one describes a series of steps, or procedure, to perform a particular function in a specific manner.  We extensively use <pre>, lists (both forms # and <ol>) and helper templates such as {{filename}}, {{package}} and {{command}}.

It’s always bugged me that we didn’t have a consistent, and visually striking manner for representing keyboard events.  For example, telling a tester to … “Hit <code>Ctrl-Alt-Backspace to stop Xorg” ([1]) just doesn’t stand-out to me enough.  What can I say … I’m a visual person.  I stumbled upon an awesome series of templates from mediawiki.org for displaying key combinations.  Enter … {{Key_press}}.  With the help of many smaller templates, I’ve added this support, along with the rock solid “upstream” documentation, to the fedoraproject.org Wiki.  I’m not sure who to thank for the template, as there are a significant number of editors.

So instead of something like  …

# Activate the overview by pressing the Windows key,
 pressing <code>Alt-F1</code>, or by moving the mouse
 to the top-left corner of the screen.

One can write …

# Activate the overview by pressing the 
 {{key press|Win}} key, pressing {{key press|Alt|F1}},
 or by moving the mouse to the top-left corner of the
 screen

Which results in something my brain responds to much better…

Much better!

From my docbook XML days, I appreciate structured ways to write content and separation between content and the look’n’feel.  However, I dislike editing content in XML.  Templates like {{command}}, {{filename}} and now {{Key_press}} offer a ways to accomplish the same thing with mediawiki.  I plan to use this more when documenting keyboard events in future test cases.  Hopefully, someone else finds this useful as well.

PreUpgrade Test Day this Thursday (2011-03-17)

Upgrade your system...

Have you seen this dialog?  It’s from a pretty handy utility called preupgrade.  Preupgrade takes some of the complexity out of upgrading your Fedora system.  But like any software, to keep running smoothly, it needs testing and maintenance.

For the past several releases, Fedora QA has hosted several test days (F14 and F13) to shake out bugs related to system upgrades using the preupgrade tool.  Those same tests are also included in our regular release validation matrix.  I’d like to think that regular, organized testing of preupgrade (and installer upgrades) contributes to a more reliable upgrade experience once Fedora GA’s.  Thanks to Hurry for helping to organize these events. 🙂

Testing involves performing system upgrades using the latest preupgrade package.  There are test instructions (including test cases) on the wiki pageFedora QA and Richard Hughes (preupgrade maintainer) will be in the IRC channel #fedora-test-day on Freenode all day to help with testing and answer any questions you might have.  If you’re not familiar with IRC, a web-based (WebIRC) client is available.

To participate, you’ll need a Fedora 14, or Fedora 13 system ready for upgrade.  Test systems can be bare metal or virtualized guests, it doesn’t matter.  If you don’t have a previous release handy, don’t worry … live images for previous releases are a great way to quickly install a working desktop.

Unfortunately, preupgrade doesn’t test itself, so we need your help.  This is a good chance to get a head start on Fedora 15 Beta testing, and help identify preupgrade-related bugs early.    See you on Thursday!

Fedora 15 Alpha release candidate in sight

If you’ve been following the Fedora 15 schedule, you probably are aware that the Alpha is looming.  Getting the Alpha to line-up with the release criteria has always been a challenge.  There is only so much you can cram into a six month release schedule.  The QA team starts testing candidate builds before F15 has been branched from rawhide.  In addition, due to the gcc-4.6 feature, all packages must be rebuilt.  Testing a moving target is … difficult.

In addition to daily rawhide/branched updates testing, we attempt to reduce the risk with several small early test runs (called rawhide acceptance test), a simulated release candidate ISO drop (called the test compose), and we pepper the weeks leading up to a release candidate testing (F-15-Alpha.TC1), and blocker bug reviews.  The first review was held last Friday, and another is planned for this Friday.

F15Alpha blocker bugs

Proposing a blocker bug is easy.  Simply add the text F15Alpha to the Blocks field of any bugzilla.redhat.com bug. It’s also helpful to provide some impact statement as to what users or use cases are hindered by the bug. Often, we cut and paste from the release criteria to keep things simple.  You can find the full details on the wiki.  If you aren’t sure … send a message to test@lists.fedoraproject.org with details.

The F15Alpha blocker list is very manageable at the moment.  However, we have another test compose (TC2), and release candidate (RC) yet to test.  Fedora contributors perform miracles on a regular basis, so I wouldn’t be surprised if we manage to line up all the moving targets in time. If you’d like to help get Fedora 15 Alpha released on time (QA sign-off is February 23), please contribute test feedback or keep approved blocker bugs updated and accurate.

The current approved, but unresolved, blocker bugs are listed below for convenience …

  • 676485 (distribution) – repoclosure fails on 15-Alpha.TC1
  • 674978 (gdm) – GDM in autologon error loop
  • 676904 (livecd-tools) – osmin.img is sometimes not a valid squashfs on livecd images
  • 672265 (livecd-tools) – “Install to harddrive” fails with “not a live image”
  • 670379 (firstboot) – Unknown lvalue ‘ValidNoProcess’ in section ‘Service’.
  • 579838 (binutils) – glibc not compatible with AMD Geode LX

Image by tanakawho via flickr used under a Creative Commons license

FUDCon Tempe plan

Busy week!  Fedora 15 Test Days started with a fairly visible change to network device names.  A fair amount of autoqa testing and review in preparation for the upcoming release.  And of course … FUDCon Tempe kicks off tomorrow morning.  Leading up to each event, I always experience some anxiety trying to figure out how best to make use of my time, and Red Hat’s dollars.  Like clockwork, that anxiety returned on the weeks leading up to FUDCon Tempe.  I thought I’d share my goals for the event before the fun gets started.  No sense in delaying further, the fun starts in the morning …

  • AutoQA talk – I’m assuming most of the users+developers in attendance have heard about the autoqa project before.  It has occupied many cycles, especially on the lead up to autoqa-0.4.4.  I’d like to pitch a talk to discuss where we are, roadblocks/challenges, what’s coming next, and the shameless solicitation for help
  • Autotest packaging – After an in depth search for reasonable flights, Lucas Rodrigues, upstream autotest maintainer, arrived in Phoenix on Thursday.  With Lucas in town, we’d like to host a hackfest session to help package autotest so it is ready for package review.  The web front-ends in autotest are built using gwt, which has some java build requirements.  I’m no java expert, so one idea is package autotest without the web front-end.  A FUDCon hackfest session seems like a perfect place to work through the issues.
  • Test case management – A few weeks back, Hurry began identifying Fedora QA workflows and use cases in an effort to determine if the nitrate project would be a good fit for our test events.  Samuel Greenfeld (OLPC) provided some feedback on the list and has been experimenting with nitrate as well as other test case management systems.  I’m anxious to talk with Samuel, document his experiences and use cases in the hope we can better understand our own needs.
  • QA – Fedora 15 is heating up.  This week it was the Network Device renaming test day, next week, the first of several GNOME3 test days.  I’d like to sync up with Adam Williamson and offer any help preparing tests for the upcoming test day.  GNOME3 is a huge change from what you might be used to.  It’s going to frighten some folks, it’s already angered others, but hopefully some people will welcome the change.  Either way it’s just that … a big change, and it’s going to need a lot of help filing bugs and identifying any gaps.

Hope to see you at FUDCon Tempe!

Fedora QA special thanks …

During the Report and Summary section of the installation validation SOP, we outline a procedure for wrapping up a community test run. We have similar wrap-ups for test days and desktop validation. Typically, this involves highlighting the bugs encountered and calling out any issues worth noting (e.g. outstanding efforts or troublesome areas).

We use the fedoraproject.org/wiki to plan and track test events. Using the wiki has a lot of benefits for us, primarily, it’s dead-simple to use and it’s integrated with FAS. When in doubt on any particular wiki syntax, there is a huge mediawiki community available to provide guidance (irc, forums and docs). One drawback with using the wiki is gathering metrics. For example, it’s bugged me that we haven’t been able to call out special thanks to test event contributors during an event recap. The good news is that the raw data is there, we just need some duck-tape to pull it together. To demonstrate, see the command-line below (mind the pipes please) …

# for PLAN in Install Desktop; do  \
   echo -e "\n== F-14-Final-RC1 Testers - $PLAN ==" ;  \
   curl --stderr /dev/null "https://fedoraproject.org/w/index.php?title=Test_Results:Fedora_14_Final_RC1_$PLAN&action=raw" \
   | grep -A999999 "^=.*Test Matrix.*=$" \
   | sed -e "s|{{\(result[^}]*\)}}|\n\1 |g" | grep "^result" \
   | gawk 'BEGIN{FS="[| ]"} $3{print $3} !$3{print "UNTESTED"}' \
   | sort | uniq -c | sort -bgr ; done

== F-14-Final-RC1 Testers - Install ==
     53 robatino
     53 kparal
     37 rhe
     37 jlaska
     23 jkeating
      8 cyberpear
      6 mkrizek
      2 rspanton
      2 newgle1
      2 dramsey
      1 UNTESTED
      1 bcl

== F-14-Final-RC1 Testers - Desktop ==
     16 UNTESTED
     14 mkrizek
     12 Masami
      7 kparal
      7 jskladan
      7 jreznik
      4 dramsey
      2 jsmith
      2 jlaska

Certainly not the prettiest command-line, and as you can imagine, bash is notoriously easy to mess-up. I wonder if it would be better to scrub the history of a wiki page instead? Or perhaps, use python-mwlib for a more programmatic approach?

Ignoring the data collection method, I’d like to focus on what the data tells me…

  1. First, along with Adam Williamson, I’d like to send a special thanks to our QA contributors listed above. As our test efforts mature, one thing is certain … we can’t do much of anything without our community. My impression has been that substantive discussion around quality topics on test@lists.fedoraproject.org has been on the increase in recent releases. Additionally, with release validation and proventesters, my sense has been that participation and engagement has been on the rise. While this is just one data set, I hope that the metrics collection method discussed here can help explore my theory.
  2. The data above is for only one test event (Fedora 14 Final Release Candidate #1), so there are many more contributors from previous events that I can’t list in a single blog. It certainly isn’t my intent to exclude them, but going forward I’d like to include the above data in each of our test events (test days and release validation) so we can recognize contributions.
  3. UNTESTED – Geez, we have quite a bit of UNTESTED results in the Desktop validation matrix. That’s not entirely a surprise since we focus primarily on ensuring the default desktop (GNOME) and KDE are tested. For remaining desktop spins, we reach out to the appropriate spin to provide test feedback using the provided framework. There are already some ideas on the F-14 QA Retrospective page on improving this coverage.

I’m still thinking about the data, so I may have more ideas in the next few days. In the meantime, feel free to contribute your comments.

Fedora 14 QA retrospective

For several releases now, the Fedora QA team hosted retrospectives aimed at identifying process gaps and recommending corrective action.  For those unfamiliar with the term retrospective, these events are sometimes referred to as lessons learned or, the morbidly named, post-mortem.  We treat the retrospectives as living documents throughout a release.  Noting the good, bad and ugly in an objective manner as issues surface.  I’ve found recording issues as they happen paints a much more accurate picture, rather than trying to remember everything all at once after the product has been released declared GOLD.  At the end of a release, we discuss, debate, propose recommendations and prioritize the work for the upcoming release.

Rear view mirror taken from http://www.lessonsfromthecockpit.com/blog/2009/4/5/dont-look-back.html

We followed this process for several Fedora releases and most recently, with Fedora 13, created TRAC tickets to manage progress towards recommendations.  There are still a few tickets remaining, but that’s okay.  This is the first time we used TRAC to manage recommendations.  I wouldn’t be surprised if we over estimated how much we could accomplish in the short time between releases.  Of course, another way to look at this list is … if you’re interested in getting more involved with the Fedora QA team, review the tickets, grab one that looks interesting to you.

For Fedora 14, the retrospective process will continue.  I started the wiki page a bit later than I prefer, but the page is up and ready for your ideas and suggestions … https://fedoraproject.org/wiki/Fedora_14_QA_Retrospective.  To help the discussion, I’ll list my two most recent entries …

  1. Virtualization Test Day – Participation in the virtualization test day was low and not what we had hoped for.  Why? What factors contributed to low attendance?  What can we do to improve virtualization test day attendance in the future?
  2. Root cause analysis of RHBZ#638091 – Kernel panic during boot after updating to image:Package-x-generic-16.pngglibc-2.12.90-13 (see bodhi).  The system would be fine until the prelink command ran, and then it would never boot and all commands would segfault on a running system. The issue was initially given positive proventester feedback as the system survived after a reboot.  The feedback was later reverted after the testers system was unusable+unbootable after prelink had run.  The good news, the problem was discovered in updates-testing before it landed in updates. However, it would be good to have some documented test procedures for a subset of packages to guide proventesters

What other issues/ideas would you like to list on the Fedora 14 QA Retrospective?  While you’re in the mood for feedback, please note that the QA team isn’t the only group interested in keeping track of issues.  Check out John Poelstra’s thoughts on the Fedora 14 Schedule Retrospective, as well as Kevin Fenzi’s growing list of Updates lessons.