Feeds:
Posts
Comments

Estimate, Testimate, Guesstimate

Estimating the test effort is often one of the first tasks one has to face when joining a new project. The project manager wants to know how much time and effort of the test periods (sometimes all from Unit to Acceptance Test), and your job is to find these numbers for her/him.

The most common estimating method is the Sinatra method, also known as “I did it my way”. I prefer to use some known methologies for estimation. It’s important to use more than one method, so that you can compare and see if you get (more or less) the same results. Also, remember that you estimate the number of hours, not the duration. There are lot of dependencies in a project, and if you have to wait one week for a programmer who has other priorities, you may allready be too late if you estimated the duration.

It is important to include estimating assumptions. If you write “testing product A will take 500 hours”, you may have promised more than you wanted to… Product A may change during development time, or perhaps someone forgot to give you the entire architecture, so there was more to test than you thought? Thus, when describing your estimate, write down which parts are included.

Prediction is very difficult, especially about the future 

Niels Bohr

 

The first two methods are probably the most common for estimating. The important thing about these two, is to use both!

 

Bottom-up and Top-down

In Bottom-up estimation, you “split as much as possible”. Separate the software into as tiny bits as possible. The smaller the bits, the easier it is to see how much time it will take to test them. You should also take into consideration how complex each part is, which may add to testing effort. Integration points increase the amount of testing needed. When estimation has been done for each bit, these are summed to see the overall estimate. This method provides a good start for a burndown chart, by the way…

In Top-down estimation, experience/comparison is often used. Comparing to other, similar projects  Using Work Breakdown Structure (WBS), you consider the various parts of the system which is due for testing, and then estimate each part. Remember to include documentation, demo, defect handling, retesting, regression test etc.

“One of the most robust findings in forecasting, human judgment, and software estimation studies is that “combination works.”5 Apparently it doesn’t matter whether the combination involves a simple average of estimates from different methods or a sophisticated weighting algorithm. A simple average offers a robust combination method unless one estimation method or expert is obviously more reliable than another. As shown elsewhere,6 though, an expert’s technical skill level can be a poor indicator of accuracy, and it’s rarely obvious, in advance, which expert will be the better estimator. This is one reason a simple average of outputs from different estimation experts and methods frequently offers the most robust and accurate combination method.” (Jørgensen, M., Simula Research Laboratory, “Practical Guidelines for Expert-Judgment-Based Software Effort Estimation”, 2005).

 

Parametric sizing

With Parametric estimating, you use the relationship and similarities which exists for each part subject to the estimation. First, come to an agreement regarding how many units it will take to complete a particular task. Then, compare similar parts using the technique, to find number of units needed. One exampler of parametric sizing, is the Planning Poker used in Scrum.

 

Expert Judgment (Delphi)

  • Used to assess input to the estimating process Subjective opinion.
  • Expertise is provided by any group or individual with specialised knowledge or training.
  • Watch out for consensus and arguments.

 

Three point estimation

The expected completion times are found by using the three time estimates: Optimistic/minimum, pessimistic/maximum and most likely.

First, Make three estimates: Optimistic + Pessimistic + Expected. Then use the formula, to find the expected completion time: M = (min + 3x exp + max) / 5

 This method can be used with common sense, but it is nicely combined with Expert Judgment (Delphi).

 

References / Further reading

http://www.stepinforum.org/stepinsummit2007/docs/Samgram%20-%20TestEstimationTechniques.pdf

 http://www.pmicmass.com/mtg_archive/2011/Planning%20Poker.pdf

http://en.wikipedia.org/wiki/Three-point_estimation

Sql for the Tester

Some general SQL knowledge is a huge advantage for a tester. Controling the expected result in a test can sometimes be a troublesome, e.g. if the results are spread around various screens (or front-end isn’t available yet). Using SQL to gather data directly from the database also gives you an oppurtunity to control sums, number of rows etc, which are invaluable for e.g. migration tests.

Different projects use different variants of SQL. Oracle, msSql, mySQL and IBM DB2 are some of the most commonly used. They have some minor differences, but fthe basics are mostly the same. I will not go into details on syntax here, because there are plenty of sites around which does that better. W3Schools and SQL Course are nice entering points for a beginner, and if you’re stuck on something later, there’s allways Google… :)

 

Describe myTable

<function> Oracle MsSql mySql IBM DB2
Describing a table DESCRIBE myTable; exec sp_columns myTable; DESCRIBE myTable; describe SELECT * FROM myTable;

Describing a table lists all the fields (with details about each field). This is useful when creating your select statements, as you can specify which fields you want to select (instead of “select *”), and you can see which fields to use in your WHERE-criteria. Also, listing the details for each fields like this, can reveal problems which occur because fields have been defined differently across the database (a value with two decimals one place and no decimals somewhere else may cause trouble, or a post-number definded as text contra number in two tables will cause unneccesary typecasting (read trouble) for the programmers).

So, after describing the table, you can easily map the values shown in a screen to the table fields, and then just select each field for the given customer:

SELECT field_1, field_2, field_3
FROM myTable
WHERE customer_ID = 1;

 

Select sum / count;

myTable cotains something like this:

ID Cust_id Account_type Account_balance
1 00789789789 Holding 1000,00
2 00123123123 Holding 250,00
3 00789789789 Credit 500,00

 

<function> Oracle MsSql mySql IBM DB2
Find sum for a given column SELECT SUM(Account_balance)
FROM myTable
GROUP BY Cust_id;
SELECT SUM(Account_balance)
FROM myTable
GROUP BY Cust_id;
SELECT Cust_id, Sum(Account_balance)
FROM myTable
GROUP BY Cust_id;
SELECT SUM(Account_balance)
FROM myTable
GROUP BY Cust_id;
Find number of rows
for given criteria.
SELECT COUNT(*)
FROM myTable
WHERE Account_type LIKE “Holding”
SELECT COUNT(*)
FROM myTable
WHERE Account_type LIKE “Holding”
SELECT COUNT(*)
FROM myTable
WHERE Account_type LIKE “Holding”
SELECT COUNT(*)
FROM myTable
WHERE Account_type LIKE “Holding”

SUM and COUNT are useful in several situations. You can use it to see the number/sum of transactions (pr. customer, pr. date, pr. product etc). You can use it to see if number of customers is the same in the new system as in the one you migrated from. You can use it to see if a screen lists all the users attached to a customer, etc. etc.

 

This stuff is pretty basic, but still very handy when testing. In my next post, I will write more on how to connect using command-line tools (rather than various heavy GUI tools), and how you can use these to have generic SQL-scripts which you can use with parameters. This comes in handy e.g. when integrating with HP Quality Center

Here is a trick to see the defects which are not linked to a run (more precisely to a test set) :

1. Go to the Defects module

2. Define a filter with a cross filter on the test sets where the test set open date is <= [To-Day].

3. Click OK : you should see only the defects that are linked to a test set (either directly or indirectly, through a test instance or a run) ; all the listed defects should display a red paper-clip in the second column of the defects grid.

4. Click on define a filter again.

5. Copy the filter to the clipboard (second icon at the top of the dialog box).

6. Open a new document in any text editor (Notepad is OK).

7. Edit / Paste the filter definition. You should see a line with the following text :

EXISTS_IN_IDS:Y,

8. Replace Y with N.

9. Select the whole text and Edit / Copy.

10. Go back to the Define Filter dialog box and paste the new filter definition (third icon at the top of the dialog box). There will be no change in the way the filter definition is displayed in that dialog box, but…

11. Click OK to apply the new filter : the defects grid will show all the defects that are NOT linked to any test set (either directly or indirectly..). None of the displayed defect should have a red paper-clip (some of them may display it : it means they are linked to a test – in the Test Plan module – or to a requirement).

This trick is applicable to all cross-filter definitions.

Solution to my question, by Christian Grzelka, SQAForums.

This Excel Report list defects with Status changed to Reopen.
Useful to see how many defects was not fixed properly and had to be reopened.
SELECT

BUG.BG_BUG_ID as “Defect Id”,

BG_SUMMARY as “Defect Summary”,

AP_OLD_VALUE as “Old value”, — Status prior to Reopen

AP_NEW_VALUE as “New value”, — Reopen

au_time as “Time changed”, — When it was reopened

au_user as “Changed by” — Who reopened it

 
FROM audit_log, audit_properties, bug
WHERE AUDIT_PROPERTIES.AP_NEW_VALUE = ‘Reopen’

and AU_Action_ID = AP_Action_ID

AND AU_ENTITY_ID = BG_BUG_ID

AND AU_ENTITY_TYPE = ‘BUG’

AND AP_FIELD_NAME = ‘BG_STATUS’

order by bg_bug_id 

The following code can be used to see how long it takes from a defect is submitted to defect closure.
Our defects goes from Submitted (via Assigned, Open, Fixed, Verified) to Closed.
 
This simple query is all it takes to see how long it took from detection to closure: 

SELECT

BUG.BG_BUG_ID as “Defect Id”,

left(BG_SUMMARY, 6) as “Screen”, — We add screen unique ID as first 6 bytes of defect.

BG_PRIORITY as “Priority”,

BG_SEVERITY as “Severity”,

datediff(“d”, BG_CLOSING_DATE, AU_TIME) as “Fix Time”, — Calculate LifeSpan

AU_TIME as “Submitted”, — When defect was submitted

BG_CLOSING_DATE as “Closing Time” — When defect was closed (date only unfortunatly)

FROM audit_log, audit_properties, bug

WHERE

AP_NEW_VALUE = ‘Submitted’ — (initial status; change to “New” or appropriate) 

and AU_Action_ID = AP_Action_ID — table connect

AND AU_ENTITY_ID = BG_BUG_ID — table connect

AND AU_ENTITY_TYPE = ‘BUG’

AND AP_FIELD_NAME = ‘BG_STATUS’

AND BG_STATUS = ‘Closed’ — can only see entire LifeSpan for closed defects

order by “Priority”, “Severity”, “Fix Time” — Want to see how long it took for high priority defects first.

 
In addition, Post-processing can be done like this to format the output in the Excel Spreadsheet:
Sub QC_PostProcessing()

Dim MainWorksheet As Worksheet

‘ Make sure your worksheet name matches!

Set MainWorksheet = ActiveWorkbook.Worksheets(“Sheet1″)

Dim DataRange As Range

Set DataRange = MainWorksheet.UsedRange

 
    ‘ Set header row to bold
    Rows(“1:1″).Select

    Selection.Font.Bold = True

    ‘ Set colum “Fix Time” to bold

    Selection.Find(What:=”Fix Time”, After:=ActiveCell, LookIn:=xlFormulas, _

        LookAt:=xlPart, SearchOrder:=xlByRows, SearchDirection:=xlNext, _

        MatchCase:=False, SearchFormat:=False).Activate

    activecell.entireColumn.select

    Selection.Font.Bold = True

 
    Cells.Select

    Cells.EntireColumn.AutoFit

    ‘ Borders
    DataRange.Select

    Selection.Borders(xlDiagonalDown).LineStyle = xlNone

    Selection.Borders(xlDiagonalUp).LineStyle = xlNone

    With Selection.Borders(xlEdgeLeft)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlEdgeTop)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlEdgeBottom)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlEdgeRight)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlInsideVertical)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlInsideHorizontal)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    DataRange.Select

End Sub 

 
 
This gives output like this: 
Defect Id Screen Priority Severity Fix Time Submitted Closed
54 abc123 1 – High 1. Critical defect -21 21.12.2010 16:01 11.01.2011 00:00
90 abc321 1 – High 1. Critical defect -8 04.01.2011 10:19 12.01.2011 00:00
91 abc213 1 – High 1. Critical defect -8 04.01.2011 11:48 12.01.2011 00:00

As a consultant moving from project to project, it’s sometimes hard to keep track of all tools, document templates and experience gained over the years. PCs are renewed, email addresses change, USB-drives gets corrupted and notebooks disappears… Thus, when you’re stuck on a problem you know you’ve solved before, it can be very frustrating when trying to recollect the required tools/documents you once wrote down to solve it.

Behold; Evernote.

bilde  

Evernote is not a new app. It’s been on the market for quite some time, and I’m finding it ever more usefull. I’m using it to keep a home inventory (with images of receipts attached to photos of items in my home), I’m storing receipes and various stuff, and most important (for this blog’s angle): I’ve created a category labeled “Test”, in which I store documents, templates, link to tools (or a copy of them), etc etc etc.

By using Evernote like this, I can access what I need from any computer around the world. Evernote has a desktop version, but is fully web-functional from a computer or i.e. an iPhone app.

I can store anything I want there, including notes from my Project notebook (hand written); just take a snapshot of it with your cellphone, add a tag (and description and more if you want), and it’s in Evernote in seconds. You can also store entire webpages, or just the selected text, using a bookmarklet in your browser. You can also take snapshots of your iPhone screen:

bilde

You can also make notes public, which means you can share your valuable files with others who might need them.

This has been usefull to my several times, and the more I use, the more I understand the advantages of it. Now I’m completing my collection of manuals for my hi-fi, I’m creating a base of code snippets, and I’m keeping all details for the upcoming summer holiday there as well (tickets, car rental details, receipts from inflight orders etc).

Smplst

Simplicity is the highest form of sophistication
   – Leonardo DaVinchi

Make things as simple as possible, but not simpler.
   – Albert Einstein

Beauty lies in the abscence of the superfluous.
   – unknown

Receive with simplicity everything that happens to you.
   – Rashi

Everything can and should be improved.
Not a single day should go by without some kind of improvement being made.
Don’t just criticize, suggest an improvement.
   – Kaizen

Follow

Get every new post delivered to your Inbox.