Why bother? Well, when writing test cases, or related pages, I frequently provide command-line samples or output. I like to ensure that the output remains relevant over time. Even more, I dislike having to frequently update a page release after release to ensure the commands or output are still relevant. For some common QA-related <pre> tag examples, checkout the following links …

• How to update AutoQA repoinfo.conf – provide copy and paste-able configuration files used whenever a Fedora release branches or reaches end-of-life.
• QA:SOP Installation Test – Provides a sample command-line script to count test day participants.
• QA:Testcase Anaconda save traceback to bugzilla – Specify boot arguments for various installer traceback tests

Okay, you get the idea. The fun part is hidden in the first example. What if you want to take advantage of some awesome wiki templates, such as {{FedoraVersion}}? If you try that … the <pre> tags will show you just that … pre-formatted text as shown below.

mock -r fedora-{{FedoraVersion||previous}}-x86_64 --init

If you like puzzles, and are hell-bent on having templates expansion inside <pre> tags, let me save you a tremendous waste of time searching for the correct solution. … To expand mediawiki templates inside <pre> tags … you can use a miscellaneous parser function. The function is a bit strange, but works as follows …

{{#tag:pre|
mock -r fedora-{{FedoraVersion||previous}}-x86_64 --init
}}

The above syntax will produce the following wiki output …

mock -r fedora-14-x86_64 --init

All that to emit two characters! I know, but I’m lazy and this solution helps me to never update that particular wiki page ever again. It will always point to the proper release thanks to Template:FedoraVersion. Also, thanks to a post from Das Gurke on the Customizing-MediaWiki mwusers forum for pointing out the right answer.

If you have other creative uses for the #tag:tagname parser function, please share!

First, some ground rules … we’re not trying to solve the world problems with this retrospective.  It’s focused on Fedora QA, our interactions with other teams, and what we can improve on to better accomplish our goals.  For a more general release retrospective, Jared already has an app for that.  Next, this isn’t the place to point fingers and taunt other teams/people.  Use your own blog for that .  Let’s keep the retrospective as objective and actionable as possible.  For example, when recording issues, it’s important to know the bug(s) involved and what factors contributed to the problem(s).  I’m not interested in what Steve thinks Mary said that Jillian heard after passing out at 31 flavors last night.

Next, a bit of history.  The process starts early each release when I draft a retrospective wiki page to track issues as they happen.  I break the page down into three groups; 1) what worked (sometimes we get things right!), 2) what didn’t work (learn from our mistakes etc…), 3) wishlist (aka pony).  It’s just my nature to want to know what went wrong, and try to fix it.  Accordingly, most of the content focuses on problems we encounter during the release.  Also, I wanted to have some concept of scope … which is where the wishlist/pony group comes in.  I still don’t know if it really fits, but I find it interesting to see what comments people have without constraints.

The retrospective concept doesn’t really work if only one person provides feedback.  So, I try to get the word out so anyone can add issues.  This is an area where I could certainly do better, probably by blogging more about it.  As the release progresses, I beg/plead/nag people closest to the issues to add some notes to the wiki.  I’m sure most people get tired of my nagging for content, but I’ve seen too often where we forget the causes of a critical problem, or don’t properly record the conditions leading up to a failure.  Without accurate data on the cause of problems, any corrective measures are just busy work.  So, the release chugs along, and hopefully the list of good/bad/wishlist issues matures.

mental note … I need to document this in a SOP

Okay … so all that preamble just to let folks know I completed a draft of recommendations based on the Fedora 15 QA retrospective feedback.  I’ll be sending this out to the test@ list for review shortly.  Check it out, and tell me what you think.  See something missing, don’t agree with something … just let me know.

Image by Camilo via flickr used under a Creative Commons license.

For example, displaying test results on the wiki (refer to {{Result}}).  While this was started by Will Woods, it later adapted based on experiences with templating on OLPC’s wiki.  Also adapted from OLPC’s wiki is our use of {{QA/Test_Case}}.  Finally, as Athmane recently pointed out, I borrowed {{!}} and {{=}} from mediawiki so you can make use of those characters when using a template.  Who cares?  Well, if you’ve ever attempted to write an admonition (e.g. {{admon/warning}}), you may have noticed it doesn’t format properly when you include certain characters (notably = and | which have special meaning in mediawiki template syntax).

Without “escaping” the special characters, your template will render improperly as seen below …

{{admon/warning|Warning!|Sometimes a=b and b=c.
In that case, a=c. :-| }}

This doesn't look correct.

This can be corrected by using the {{!}} and {{=}} templates, as shown below.

{{admon/warning|Warning!|Sometimes a{{=}}b and b{{=}}c.
In that case, a{{=}}c. :-{{!}} }}

Much better!

Today I found another really helpful series of templates for something I’ve wanted for a long time.  In Fedora QA, we write and maintain a lot of test cases on the wiki.  Often in a test case, one describes a series of steps, or procedure, to perform a particular function in a specific manner.  We extensively use <pre>, lists (both forms # and <ol>) and helper templates such as {{filename}}, {{package}} and {{command}}.

It’s always bugged me that we didn’t have a consistent, and visually striking manner for representing keyboard events.  For example, telling a tester to … “Hit <code>Ctrl-Alt-Backspace to stop Xorg” ([1]) just doesn’t stand-out to me enough.  What can I say … I’m a visual person.  I stumbled upon an awesome series of templates from mediawiki.org for displaying key combinations.  Enter … {{Key_press}}.  With the help of many smaller templates, I’ve added this support, along with the rock solid “upstream” documentation, to the fedoraproject.org Wiki.  I’m not sure who to thank for the template, as there are a significant number of editors.

So instead of something like  …

# Activate the overview by pressing the Windows key,
 pressing <code>Alt-F1</code>, or by moving the mouse
 to the top-left corner of the screen.

One can write …

# Activate the overview by pressing the
 {{key press|Win}} key, pressing {{key press|Alt|F1}},
 or by moving the mouse to the top-left corner of the
 screen

Which results in something my brain responds to much better…

From my docbook XML days, I appreciate structured ways to write content and separation between content and the look’n'feel.  However, I dislike editing content in XML.  Templates like {{command}}, {{filename}} and now {{Key_press}} offer a ways to accomplish the same thing with mediawiki.  I plan to use this more when documenting keyboard events in future test cases.  Hopefully, someone else finds this useful as well.

Have you seen this dialog?  It’s from a pretty handy utility called preupgrade.  Preupgrade takes some of the complexity out of upgrading your Fedora system.  But like any software, to keep running smoothly, it needs testing and maintenance.

For the past several releases, Fedora QA has hosted several test days (F14 and F13) to shake out bugs related to system upgrades using the preupgrade tool.  Those same tests are also included in our regular release validation matrix.  I’d like to think that regular, organized testing of preupgrade (and installer upgrades) contributes to a more reliable upgrade experience once Fedora GA’s.  Thanks to Hurry for helping to organize these events.

Testing involves performing system upgrades using the latest preupgrade package.  There are test instructions (including test cases) on the wiki page.  Fedora QA and Richard Hughes (preupgrade maintainer) will be in the IRC channel #fedora-test-day on Freenode all day to help with testing and answer any questions you might have.  If you’re not familiar with IRC, a web-based (WebIRC) client is available.

To participate, you’ll need a Fedora 14, or Fedora 13 system ready for upgrade.  Test systems can be bare metal or virtualized guests, it doesn’t matter.  If you don’t have a previous release handy, don’t worry … live images for previous releases are a great way to quickly install a working desktop.

Unfortunately, preupgrade doesn’t test itself, so we need your help.  This is a good chance to get a head start on Fedora 15 Beta testing, and help identify preupgrade-related bugs early.    See you on Thursday!

Day#1 – Barcamp

Catching up with fenzi …

During the first time slot, I spent some time catching up with familiar nicks/faces.  I spoke with Kevin Fenzi, who is always extremely helpful+approachable on irc and in-person, on the current state of the AutoQA project.  We discussed some of the challenges we’ve had soliciting contributors.  The consensus was that once we have tests+results that directly impact maintainers, we can expect more participation.  It’s been a long time since the last release of autoqa, but once depcheck and upgradepath start posting results to bodhi, I’m sure maintainers will get excited (or at least agitated).

What gives mockbuild?

One test idea that Kevin suggested was to write a new rpmlint test to ensure that no files in a package are owned by the user: mockbuild.  You’ve probably seen such errors when installing a package.

warning: user mockbuild does not exist - using root
warning: user mockbuild does not exist - using root
warning: user mockbuild does not exist - using root

It looks sloppy, and it’s something we should be able to detect and prevent.

AutoQA BarCamp talk

Next, I hosted an AutoQA talk.  I prepared some slides to provide talking points, however I was hoping for a more interactive discussion.  I think the talk went well.  Both Chris Lumens and Will Woods were in attendance and were able to provide more in-depth details on the installer unittests and depcheck test that will be landing in the next release (autoqa-0.4.4).

Several good test ideas surfaced, including Dave Malcolm’s ideas on artificial ignorance.  Dan Walsh asked if we could detect whenever dependencies might unintentionally change, and pull in undesired packages.  For example, accidentally including xorg-x11-server-Xorg on a Server SIG image, or increasing the size of Live images beyond the desired 700 Mib (or 1Gib).  Andrew Overholt discussed an eclipse test suite that he’d like run for him after koji builds.  It’s a fairly comprehensive test suite, but early discussions seem like it shouldn’t be too much trouble to wrap and run using AutoQA. .  Finally, some of the installer folks were asking about resultsdb, which is planned for a future autoqa release.  Hopefully more on that later.

Other sessions…

After the AutoQA talk, I attended the Cloud: The future of computing talk by Mike McGrath.  Good talk, with an active demonstration and screenshots.  We have some preliminary support in a branch of AutoQA for provisioning virtual test systems on the fly, so it’s neat to review other ways of solving this problem.  We’re always interested in any off-the-shelf solutions we can integrate with to dynamically allocate and connect to virtual test systems.  Obviously, for tests that need bare metal hardware … this wouldn’t work.  However, a large portion of our tests don’t have this requirement and work just as nice on virtual systems.

Next, I attended the Meet the Anaconda team talk by David Cantrell.  In Fedora QA, we work with them pretty closely during installation validation, so it was really nice to get to know them a bit better.  Throughout the talk, each member of the team presented their current projects.  There is a lot of exciting work planned for the next 3-4 Fedora releases, and the good news is that the team documented their ideas on the wiki.

After a few diversions, I attended Using the SELinux Sandbox by Dan Walsh.  SELinux sandbox allows administrators to take untrusted content, run it through one or more SELinux filters, and have confidence that the content won’t damage to their system/files.  Since Dan’s initial blog post on the subject, a lot has changed.  Dan demonstrated running an entire desktop session in a sandbox, including firefox, openoffice, evince and other desktop applications.  Pretty cool stuff.

Day#2 – More Barcamp, and some hackfest

UI Mockups…

I’ve always been slow when it comes to doing mock-ups.  The process usually involves the gimp (not the one that’s sleeping) and copy+pasting from screenshots to assemble something that looks somewhat decent.  It takes me a long time to make anything reasonable, which means … I don’t do it often.  Of course, this means it takes even longer the next time I try.  Designing UI mockups in Inkscape by Máirín Duffy was really helpful to demonstrate an easier way to rapidly generate UI mock-ups.  I don’t think this was the first time she gave this talk, but it was the first time I attended.  Great demos and tons of examples.  I’ll start with inkscape next time.  And remember, d saves time!

I skipped the next session, and instead spent time catching up with fellow home brewer Chris Lumens.  His keg has a leak, and this must be stopped.

What’s next for Fedora Engineering?

The last talk of the day was The Next Big Fedora Engineering Project by spot.  Similar in structure to the anaconda team talk, members of Tom’s group presented ideas for future engineering projects.  The good news, was that all the ideas were interesting, and they all seemed to have wide support (by a show of hands in the room).  The bad news, was that all the ideas were interesting and seemed to have wide support.  Two that stood out to me were the Fedora message bus and several discussions on how to identify and present contributor metrics.  The most notable idea, was the Fedora RPG.  I’m sure many different groups have this challenge, but in QA, we have no shortage of activities that generate data.  It’s impossible to manual monitor all the data points (bugs, TRAC tickets, bodhi feedback, mailing list, IRC and wiki contributions) so you can  give kudos where deserved.  The RPG concept offers a new twist on this metrics challenge.  I’d love to take advantage of this for QA so we have better insight on community contributions.

Let the hacking begin …

Lucas and I cornered spot for some ideas on how we can resolve the packaging challenges in autotest.  The current autotest package installs all content into the directory /usr/share/autotest.  This really works well for the autotest scheduler since it rsync’s /usr/share/autotest/client to all test client systems.  The directory has all it needs to run jobs and report results and works well across all distributions.  Unfortunately, it’s not likely we’ll get an exception for this format during Fedora package review.  Lucas began the difficult work of adjusting autotest python and configuration files so it would function with content in FHS friendly locations (e.g. /etc, /var,  site-packages etc…).  I worked on the easier task of updating our existing autotest.spec to accommodate the new layout.  We made a good bit of progress, but there is still a lot of work remaining to ensure that autotest will continue operating.

Day#3 – Hackfest

More autotest please…

After settling in, Lucas and I picked up right where we left off.  Autotest relies on an older version of Django that is no longer in Fedora.  Upstream plans to update to the latest Django release, but that’s not on the short-term horizon.  In the meantime, I package Django-1.0.4 privately.  However, if we want autotest accepted into Fedora, I need to formally package this older Django in a manner where it can co-exist with the current Django.  Toshio and Luke shared ideas they’ve used for packaging compat versions of CherryPy.  Those tricks were just what the doctor ordered.

I learned about using setuptools (instead of distutils) to build and install the egg.  The egg installation doesn’t interfere with the regular Django package, and allows callers to use the compat version by specifying:

__requires__ = 'Django==1.0.4'
import pkg_resources

Compat-Django-1.0.4 is packaged and about ready for review, my remaining task is to review the current Django security advisories and patch as necessary.  Thanks to Steve Milner for the security guidance.

GNOME Shell discussion

I had a quick talk with Chris Aillon, J5 and Adam Williamson regarding test ideas for the upcoming GNOME Shell Test Days.  This was a really good conversation, and one that can only take place in person.  Always amazing how much you can cover in person, as opposed to IRC or email.  For more details, read Adam’s blog post on the subject.

Back to autotest…

Throughout the day, Lucas continued hacking on autotest.  Like I said, this is a large change to the autotest code-base and will take some time to get right.  I helped where I could with small patches and brainstorming, but Lucas ran the show.  His changes are available for testing in github.  I plan to revisit and test his branch once F15Alpha is behind us.  Once again, this would be nearly impossible to do over IRC/email.  Or at least, we wouldn’t have made this much progress in such a short time.  Kudos to Lucas for his work!

Nitrate

I posted earlier about my plans for FUDCon Tempe.  I planned to talk with Samuel Greenfeld to understand his test case management system (TCMS) needs and use cases.  Time was running short on Monday, so we only managed to talk briefly.  Samuel has a good grasp on what other open-source TCMS offerings are available, and the benefits and drawbacks to each.  We talked about the current status of nitrate, and Hurry’s plans to host a demo instance so we can further explore modelling test plans and gathering data.  If anyone is interested in helping, please ask questions on the nitrate-devel list and take a look at the TCMS requirements Hurry is tracking.

I had a fun and productive time at FUDCon Tempe.  Too often, I under-estimate the value of face-to-face interaction with people I interact with on a regular basis.  You know, the whole networking thing.  Of course, when you call it that, I don’t want to do it.  Anyway, with FUDCon Tempe behind me, it’s time to get Fedora 15 Alpha out the door.

In addition to daily rawhide/branched updates testing, we attempt to reduce the risk with several small early test runs (called rawhide acceptance test), a simulated release candidate ISO drop (called the test compose), and we pepper the weeks leading up to a release candidate testing (F-15-Alpha.TC1), and blocker bug reviews.  The first review was held last Friday, and another is planned for this Friday.

F15Alpha blocker bugs

Proposing a blocker bug is easy.  Simply add the text F15Alpha to the Blocks field of any bugzilla.redhat.com bug. It’s also helpful to provide some impact statement as to what users or use cases are hindered by the bug. Often, we cut and paste from the release criteria to keep things simple.  You can find the full details on the wiki.  If you aren’t sure … send a message to test@lists.fedoraproject.org with details.

The F15Alpha blocker list is very manageable at the moment.  However, we have another test compose (TC2), and release candidate (RC) yet to test.  Fedora contributors perform miracles on a regular basis, so I wouldn’t be surprised if we manage to line up all the moving targets in time. If you’d like to help get Fedora 15 Alpha released on time (QA sign-off is February 23), please contribute test feedback or keep approved blocker bugs updated and accurate.

The current approved, but unresolved, blocker bugs are listed below for convenience …

• 676485 (distribution) – repoclosure fails on 15-Alpha.TC1
• 674978 (gdm) – GDM in autologon error loop
• 676904 (livecd-tools) – osmin.img is sometimes not a valid squashfs on livecd images
• 672265 (livecd-tools) – “Install to harddrive” fails with “not a live image”
• 670379 (firstboot) – Unknown lvalue ‘ValidNoProcess’ in section ‘Service’.
• 579838 (binutils) – glibc not compatible with AMD Geode LX

Image by tanakawho via flickr used under a Creative Commons license

• AutoQA talk – I’m assuming most of the users+developers in attendance have heard about the autoqa project before.  It has occupied many cycles, especially on the lead up to autoqa-0.4.4.  I’d like to pitch a talk to discuss where we are, roadblocks/challenges, what’s coming next, and the shameless solicitation for help
• Autotest packaging – After an in depth search for reasonable flights, Lucas Rodrigues, upstream autotest maintainer, arrived in Phoenix on Thursday.  With Lucas in town, we’d like to host a hackfest session to help package autotest so it is ready for package review.  The web front-ends in autotest are built using gwt, which has some java build requirements.  I’m no java expert, so one idea is package autotest without the web front-end.  A FUDCon hackfest session seems like a perfect place to work through the issues.
• Test case management – A few weeks back, Hurry began identifying Fedora QA workflows and use cases in an effort to determine if the nitrate project would be a good fit for our test events.  Samuel Greenfeld (OLPC) provided some feedback on the list and has been experimenting with nitrate as well as other test case management systems.  I’m anxious to talk with Samuel, document his experiences and use cases in the hope we can better understand our own needs.
• QA – Fedora 15 is heating up.  This week it was the Network Device renaming test day, next week, the first of several GNOME3 test days.  I’d like to sync up with Adam Williamson and offer any help preparing tests for the upcoming test day.  GNOME3 is a huge change from what you might be used to.  It’s going to frighten some folks, it’s already angered others, but hopefully some people will welcome the change.  Either way it’s just that … a big change, and it’s going to need a lot of help filing bugs and identifying any gaps.

Hope to see you at FUDCon Tempe!

We use the fedoraproject.org/wiki to plan and track test events. Using the wiki has a lot of benefits for us, primarily, it’s dead-simple to use and it’s integrated with FAS. When in doubt on any particular wiki syntax, there is a huge mediawiki community available to provide guidance (irc, forums and docs). One drawback with using the wiki is gathering metrics. For example, it’s bugged me that we haven’t been able to call out special thanks to test event contributors during an event recap. The good news is that the raw data is there, we just need some duck-tape to pull it together. To demonstrate, see the command-line below (mind the pipes please) …

# for PLAN in Install Desktop; do  \
   echo -e "\n== F-14-Final-RC1 Testers - $PLAN ==" ;  \
   curl --stderr /dev/null "https://fedoraproject.org/w/index.php?title=Test_Results:Fedora_14_Final_RC1_$PLAN&action=raw" \
   | grep -A999999 "^=.*Test Matrix.*=$" \
   | sed -e "s|{{\(result[^}]*\)}}|\n\1 |g" | grep "^result" \
   | gawk 'BEGIN{FS="[| ]"} $3{print $3} !$3{print "UNTESTED"}' \
   | sort | uniq -c | sort -bgr ; done

== F-14-Final-RC1 Testers - Install ==
     53 robatino
     53 kparal
     37 rhe
     37 jlaska
     23 jkeating
      8 cyberpear
      6 mkrizek
      2 rspanton
      2 newgle1
      2 dramsey
      1 UNTESTED
      1 bcl

== F-14-Final-RC1 Testers - Desktop ==
     16 UNTESTED
     14 mkrizek
     12 Masami
      7 kparal
      7 jskladan
      7 jreznik
      4 dramsey
      2 jsmith
      2 jlaska

Certainly not the prettiest command-line, and as you can imagine, bash is notoriously easy to mess-up. I wonder if it would be better to scrub the history of a wiki page instead? Or perhaps, use python-mwlib for a more programmatic approach?

Ignoring the data collection method, I’d like to focus on what the data tells me…

1. First, along with Adam Williamson, I’d like to send a special thanks to our QA contributors listed above. As our test efforts mature, one thing is certain … we can’t do much of anything without our community. My impression has been that substantive discussion around quality topics on test@lists.fedoraproject.org has been on the increase in recent releases. Additionally, with release validation and proventesters, my sense has been that participation and engagement has been on the rise. While this is just one data set, I hope that the metrics collection method discussed here can help explore my theory.
2. The data above is for only one test event (Fedora 14 Final Release Candidate #1), so there are many more contributors from previous events that I can’t list in a single blog. It certainly isn’t my intent to exclude them, but going forward I’d like to include the above data in each of our test events (test days and release validation) so we can recognize contributions.
3. UNTESTED – Geez, we have quite a bit of UNTESTED results in the Desktop validation matrix. That’s not entirely a surprise since we focus primarily on ensuring the default desktop (GNOME) and KDE are tested. For remaining desktop spins, we reach out to the appropriate spin to provide test feedback using the provided framework. There are already some ideas on the F-14 QA Retrospective page on improving this coverage.

I’m still thinking about the data, so I may have more ideas in the next few days. In the meantime, feel free to contribute your comments.

<pre>
This is <strong>bold <span style="color: red;">red</span></strong> text
<a href="#link">This is a link</a>
<span style="color:green;
                font-size:xx-large;
                font-family:'Times New Roman', Times, serif;
                font-weight:bolder;">big and green times</span>
</pre>

This will display in a browser as …

This is bold red text
This is a link
big and green times

Introducing mediawiki into the mix complicates things slightly. By design (as security precaution I’d guess), mediawiki does not honor all HTML tags on a wiki page. However, it does recognize and permit some tags, <pre> being one of them. As noted previously, we use the <pre> quite a bit to call out command execution or file contents.

Where the wrinkle comes is trying to use mediawiki templates inside a <pre> block. Well why would you want to do that? Often, our command samples or file contents include Fedora release version specific information. Since Fedora releases come out about every 6 months, for every wiki page that doesn’t need to reference an explicit Fedora release, someone would need to modify every wiki page that references a Fedora version. At a quick glance, that comes to about ~179 wiki pages. As I mentioned, not all of those pages would need an update when a new release comes out, but it at least gives a rough estimate.

After some searching, I came across a forum post which recommends a solution that works. To demonstrate using a mediawiki template inside a <pre> block, I’ll use the wiki page Creating_a_Test_Day_Live_Image. On that page, we recommend checking out a git branch of spin-kickstarts whose name matches the upcoming Fedora release. I’d like to use the FedoraVersion template so we don’t need to constantly update this page each time a new version of Fedora is released.

To make use of the FedoraVersion mediawiki template inside a <pre> block, you would enter the following …

<pre&lt;noinclude&gt;&lt;/noinclude&gt;>
git clone 'git://git.fedorahosted.org/spin-kickstarts.git' -b F-{{FedoraVersion|number|next}}
</pre&lt;noinclude&gt;&lt;/noinclude&gt;>

This instructs mediawiki to format the page as follows:

git clone 'git://git.fedorahosted.org/spin-kickstarts.git' -b F-14

I have no idea why the magic combination of noinclude mediawiki tags works. Bonus points to anyone who can answer: why does this work?