Estimate, Testimate, Guesstimate

Estimating the test effort is often one of the first tasks one has to face when joining a new project. The project manager wants to know how much time and effort of the test periods (sometimes all from Unit to Acceptance Test), and your job is to find these numbers for her/him.

The most common estimating method is the Sinatra method, also known as “I did it my way”. I prefer to use some known methologies for estimation. It’s important to use more than one method, so that you can compare and see if you get (more or less) the same results. Also, remember that you estimate the number of hours, not the duration. There are lot of dependencies in a project, and if you have to wait one week for a programmer who has other priorities, you may allready be too late if you estimated the duration.

It is important to include estimating assumptions. If you write “testing product A will take 500 hours”, you may have promised more than you wanted to… Product A may change during development time, or perhaps someone forgot to give you the entire architecture, so there was more to test than you thought? Thus, when describing your estimate, write down which parts are included.

Prediction is very difficult, especially about the future 

Niels Bohr

 

The first two methods are probably the most common for estimating. The important thing about these two, is to use both!

 

Bottom-up and Top-down

In Bottom-up estimation, you “split as much as possible”. Separate the software into as tiny bits as possible. The smaller the bits, the easier it is to see how much time it will take to test them. You should also take into consideration how complex each part is, which may add to testing effort. Integration points increase the amount of testing needed. When estimation has been done for each bit, these are summed to see the overall estimate. This method provides a good start for a burndown chart, by the way…

In Top-down estimation, experience/comparison is often used. Comparing to other, similar projects  Using Work Breakdown Structure (WBS), you consider the various parts of the system which is due for testing, and then estimate each part. Remember to include documentation, demo, defect handling, retesting, regression test etc.

“One of the most robust findings in forecasting, human judgment, and software estimation studies is that “combination works.”5 Apparently it doesn’t matter whether the combination involves a simple average of estimates from different methods or a sophisticated weighting algorithm. A simple average offers a robust combination method unless one estimation method or expert is obviously more reliable than another. As shown elsewhere,6 though, an expert’s technical skill level can be a poor indicator of accuracy, and it’s rarely obvious, in advance, which expert will be the better estimator. This is one reason a simple average of outputs from different estimation experts and methods frequently offers the most robust and accurate combination method.” (Jørgensen, M., Simula Research Laboratory, “Practical Guidelines for Expert-Judgment-Based Software Effort Estimation”, 2005).

 

Parametric sizing

With Parametric estimating, you use the relationship and similarities which exists for each part subject to the estimation. First, come to an agreement regarding how many units it will take to complete a particular task. Then, compare similar parts using the technique, to find number of units needed. One exampler of parametric sizing, is the Planning Poker used in Scrum.

 

Expert Judgment (Delphi)

• Used to assess input to the estimating process Subjective opinion.
• Expertise is provided by any group or individual with specialised knowledge or training.
• Watch out for consensus and arguments.

 

Three point estimation

The expected completion times are found by using the three time estimates: Optimistic/minimum, pessimistic/maximum and most likely.

First, Make three estimates: Optimistic + Pessimistic + Expected. Then use the formula, to find the expected completion time: M = (min + 3x exp + max) / 5

 This method can be used with common sense, but it is nicely combined with Expert Judgment (Delphi).

 

References / Further reading

http://www.stepinforum.org/stepinsummit2007/docs/Samgram%20-%20TestEstimationTechniques.pdf

 http://www.pmicmass.com/mtg_archive/2011/Planning%20Poker.pdf

http://en.wikipedia.org/wiki/Three-point_estimation

Sql for the Tester

Some general SQL knowledge is a huge advantage for a tester. Controling the expected result in a test can sometimes be a troublesome, e.g. if the results are spread around various screens (or front-end isn’t available yet). Using SQL to gather data directly from the database also gives you an oppurtunity to control sums, number of rows etc, which are invaluable for e.g. migration tests.

Different projects use different variants of SQL. Oracle, msSql, mySQL and IBM DB2 are some of the most commonly used. They have some minor differences, but fthe basics are mostly the same. I will not go into details on syntax here, because there are plenty of sites around which does that better. W3Schools and SQL Course are nice entering points for a beginner, and if you’re stuck on something later, there’s allways Google…

 

Describe myTable

<function>	Oracle	MsSql	mySql	IBM DB2
Describing a table	DESCRIBE myTable;	exec sp_columns myTable;	DESCRIBE myTable;	describe SELECT * FROM myTable;

Describing a table lists all the fields (with details about each field). This is useful when creating your select statements, as you can specify which fields you want to select (instead of “select *”), and you can see which fields to use in your WHERE-criteria. Also, listing the details for each fields like this, can reveal problems which occur because fields have been defined differently across the database (a value with two decimals one place and no decimals somewhere else may cause trouble, or a post-number definded as text contra number in two tables will cause unneccesary typecasting (read trouble) for the programmers).

So, after describing the table, you can easily map the values shown in a screen to the table fields, and then just select each field for the given customer:

SELECT field_1, field_2, field_3
FROM myTable
WHERE customer_ID = 1;

 

Select sum / count;

myTable cotains something like this:

ID	Cust_id	Account_type	Account_balance
1	00789789789	Holding	1000,00
2	00123123123	Holding	250,00
3	00789789789	Credit	500,00

 

<function>	Oracle	MsSql	mySql	IBM DB2
Find sum for a given column	SELECT SUM(Account_balance) FROM myTable GROUP BY Cust_id;	SELECT SUM(Account_balance) FROM myTable GROUP BY Cust_id;	SELECT Cust_id, Sum(Account_balance) FROM myTable GROUP BY Cust_id;	SELECT SUM(Account_balance) FROM myTable GROUP BY Cust_id;
Find number of rows for given criteria.	SELECT COUNT(*) FROM myTable WHERE Account_type LIKE “Holding”	SELECT COUNT(*) FROM myTable WHERE Account_type LIKE “Holding”	SELECT COUNT(*) FROM myTable WHERE Account_type LIKE “Holding”	SELECT COUNT(*) FROM myTable WHERE Account_type LIKE “Holding”

SUM and COUNT are useful in several situations. You can use it to see the number/sum of transactions (pr. customer, pr. date, pr. product etc). You can use it to see if number of customers is the same in the new system as in the one you migrated from. You can use it to see if a screen lists all the users attached to a customer, etc. etc.

 

This stuff is pretty basic, but still very handy when testing. In my next post, I will write more on how to connect using command-line tools (rather than various heavy GUI tools), and how you can use these to have generic SQL-scripts which you can use with parameters. This comes in handy e.g. when integrating with HP Quality Center

HPQC: Showing defects with no links to Test Set

Here is a trick to see the defects which are not linked to a run (more precisely to a test set) :

1. Go to the Defects module

2. Define a filter with a cross filter on the test sets where the test set open date is <= [To-Day].

3. Click OK : you should see only the defects that are linked to a test set (either directly or indirectly, through a test instance or a run) ; all the listed defects should display a red paper-clip in the second column of the defects grid.

4. Click on define a filter again.

5. Copy the filter to the clipboard (second icon at the top of the dialog box).

6. Open a new document in any text editor (Notepad is OK).

7. Edit / Paste the filter definition. You should see a line with the following text :

EXISTS_IN_IDS:Y,

8. Replace Y with N.

9. Select the whole text and Edit / Copy.

10. Go back to the Define Filter dialog box and paste the new filter definition (third icon at the top of the dialog box). There will be no change in the way the filter definition is displayed in that dialog box, but…

11. Click OK to apply the new filter : the defects grid will show all the defects that are NOT linked to any test set (either directly or indirectly..). None of the displayed defect should have a red paper-clip (some of them may display it : it means they are linked to a test – in the Test Plan module – or to a requirement).

This trick is applicable to all cross-filter definitions.

Solution to my question, by Christian Grzelka, SQAForums.

HPQC Excel Report: Defects reopened

This Excel Report list defects with Status changed to Reopen.

Useful to see how many defects was not fixed properly and had to be reopened.

SELECT

BUG.BG_BUG_ID as “Defect Id”,

BG_SUMMARY as “Defect Summary”,

AP_OLD_VALUE as “Old value”, — Status prior to Reopen

AP_NEW_VALUE as “New value”, — Reopen

au_time as “Time changed”, — When it was reopened

au_user as “Changed by” — Who reopened it

 

FROM audit_log, audit_properties, bug

WHERE AUDIT_PROPERTIES.AP_NEW_VALUE = ‘Reopen’

and AU_Action_ID = AP_Action_ID

AND AU_ENTITY_ID = BG_BUG_ID

AND AU_ENTITY_TYPE = ‘BUG’

AND AP_FIELD_NAME = ‘BG_STATUS’

order by bg_bug_id 

HPQC Excel Report: Defect LifeCycle TimeSpan

The following code can be used to see how long it takes from a defect is submitted to defect closure.

Our defects goes from Submitted (via Assigned, Open, Fixed, Verified) to Closed.

 

This simple query is all it takes to see how long it took from detection to closure: 

SELECT

BUG.BG_BUG_ID as “Defect Id”,

left(BG_SUMMARY, 6) as “Screen”, — We add screen unique ID as first 6 bytes of defect.

BG_PRIORITY as “Priority”,

BG_SEVERITY as “Severity”,

datediff(“d”, BG_CLOSING_DATE, AU_TIME) as “Fix Time”, — Calculate LifeSpan

AU_TIME as “Submitted”, — When defect was submitted

BG_CLOSING_DATE as “Closing Time” — When defect was closed (date only unfortunatly)

FROM audit_log, audit_properties, bug

WHERE

AP_NEW_VALUE = ‘Submitted’ — (initial status; change to “New” or appropriate) 

and AU_Action_ID = AP_Action_ID — table connect

AND AU_ENTITY_ID = BG_BUG_ID — table connect

AND AU_ENTITY_TYPE = ‘BUG’

AND AP_FIELD_NAME = ‘BG_STATUS’

AND BG_STATUS = ‘Closed’ — can only see entire LifeSpan for closed defects

order by “Priority”, “Severity”, “Fix Time” — Want to see how long it took for high priority defects first.

 

In addition, Post-processing can be done like this to format the output in the Excel Spreadsheet:

Sub QC_PostProcessing()

Dim MainWorksheet As Worksheet

‘ Make sure your worksheet name matches!

Set MainWorksheet = ActiveWorkbook.Worksheets(“Sheet1″)

Dim DataRange As Range

Set DataRange = MainWorksheet.UsedRange

 

    ‘ Set header row to bold

    Rows(“1:1″).Select

    Selection.Font.Bold = True

    ‘ Set colum “Fix Time” to bold

    Selection.Find(What:=”Fix Time”, After:=ActiveCell, LookIn:=xlFormulas, _

        LookAt:=xlPart, SearchOrder:=xlByRows, SearchDirection:=xlNext, _

        MatchCase:=False, SearchFormat:=False).Activate

    activecell.entireColumn.select

    Selection.Font.Bold = True

 

    Cells.Select

    Cells.EntireColumn.AutoFit

    ‘ Borders

    DataRange.Select

    Selection.Borders(xlDiagonalDown).LineStyle = xlNone

    Selection.Borders(xlDiagonalUp).LineStyle = xlNone

    With Selection.Borders(xlEdgeLeft)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlEdgeTop)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlEdgeBottom)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlEdgeRight)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlInsideVertical)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    With Selection.Borders(xlInsideHorizontal)

        .LineStyle = xlContinuous

        .Weight = xlThin

        .ColorIndex = xlAutomatic

    End With

    DataRange.Select

End Sub 

 

 

This gives output like this: 

Defect Id	Screen	Priority	Severity	Fix Time	Submitted	Closed
54	abc123	1 – High	1. Critical defect	-21	21.12.2010 16:01	11.01.2011 00:00
90	abc321	1 – High	1. Critical defect	-8	04.01.2011 10:19	12.01.2011 00:00
91	abc213	1 – High	1. Critical defect	-8	04.01.2011 11:48	12.01.2011 00:00

EverNote – remember everything

As a consultant moving from project to project, it’s sometimes hard to keep track of all tools, document templates and experience gained over the years. PCs are renewed, email addresses change, USB-drives gets corrupted and notebooks disappears… Thus, when you’re stuck on a problem you know you’ve solved before, it can be very frustrating when trying to recollect the required tools/documents you once wrote down to solve it.

Behold; Evernote.

  

Evernote is not a new app. It’s been on the market for quite some time, and I’m finding it ever more usefull. I’m using it to keep a home inventory (with images of receipts attached to photos of items in my home), I’m storing receipes and various stuff, and most important (for this blog’s angle): I’ve created a category labeled “Test”, in which I store documents, templates, link to tools (or a copy of them), etc etc etc.

By using Evernote like this, I can access what I need from any computer around the world. Evernote has a desktop version, but is fully web-functional from a computer or i.e. an iPhone app.

I can store anything I want there, including notes from my Project notebook (hand written); just take a snapshot of it with your cellphone, add a tag (and description and more if you want), and it’s in Evernote in seconds. You can also store entire webpages, or just the selected text, using a bookmarklet in your browser. You can also take snapshots of your iPhone screen:

You can also make notes public, which means you can share your valuable files with others who might need them.

This has been usefull to my several times, and the more I use, the more I understand the advantages of it. Now I’m completing my collection of manuals for my hi-fi, I’m creating a base of code snippets, and I’m keeping all details for the upcoming summer holiday there as well (tickets, car rental details, receipts from inflight orders etc).

Smplst

Simplicity is the highest form of sophistication
   – Leonardo DaVinchi

Make things as simple as possible, but not simpler.
   – Albert Einstein

Beauty lies in the abscence of the superfluous.
   – unknown

Receive with simplicity everything that happens to you.
   – Rashi

Everything can and should be improved.
Not a single day should go by without some kind of improvement being made.
Don’t just criticize, suggest an improvement.
   – Kaizen