.Net软件测试自动化之道

pcndxx

贡献于2013-10-26

字数:0 关键词: .NET开发

James D. McCaffrey .NET Test Automation Recipes A Problem-Solution Approach 6633FM.qxd 4/3/06 1:54 PM Page i .NET Test Automation Recipes: A Problem-Solution Approach Copyright © 2006 by James D. McCaffrey All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13: 978-1-59059-663-0 ISBN-10: 1-59059-663-3 Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1 Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. Lead Editor: Jonathan Hassell Technical Reviewer: Josh Kelling Editorial Board: Steve Anglin, Ewan Buckingham, Gary Cornell, Jason Gilmore, Jonathan Gennick, Jonathan Hassell, James Huddleston, Chris Mills, Matthew Moodie, Dominic Shakeshaft, Jim Sumser, Keir Thomas, Matt Wade Project Manager: Elizabeth Seymour Copy Edit Manager: Nicole LeClerc Copy Editor: Julie McNamee Assistant Production Director: Kari Brooks-Copony Production Editor: Katie Stence Compositor: Lynn L’Heureux Proofreader: Elizabeth Berry Indexer: Becky Hornak Cover Designer: Kurt Krames Manufacturing Director: Tom Debolski Distributed to the book trade worldwide by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax 201-348-4505, e-mail orders-ny@springer-sbm.com, or visit http://www.springeronline.com. For information on translations, please contact Apress directly at 2560 Ninth Street, Suite 219, Berkeley, CA 94710. Phone 510-549-5930, fax 510-549-5939, e-mail info@apress.com, or visit http://www.apress.com. The information in this book is distributed on an “as is” basis, without warranty. Although every precau- tion has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work. The source code for this book is available to readers at http://www.apress.com in the Source Code section. 6633FM.qxd 4/3/06 1:54 PM Page ii 6633FM.qxd 4/3/06 1:54 PM Page iii 6633FM.qxd 4/3/06 1:54 PM Page iv Contents at a Glance About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii About the Technical Reviewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix ■CHAPTER 1 API Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 ■CHAPTER 2 Reflection-Based UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 ■CHAPTER 3 Windows-Based UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 ■CHAPTER 4 Test Harness Design Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 ■CHAPTER 5 Request-Response Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 ■CHAPTER 6 Script-Based Web UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 ■CHAPTER 7 Low-Level Web UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 ■CHAPTER 8 Web Services Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 ■CHAPTER 9 SQL Stored Procedure Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 ■CHAPTER 10 Combinations and Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 ■CHAPTER 11 ADO.NET Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 ■CHAPTER 12 XML Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 ■INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 v 6633FM.qxd 4/3/06 1:54 PM Page v 6633FM.qxd 4/3/06 1:54 PM Page vi Contents About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii About the Technical Reviewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix PART 1 ■ ■ ■ Windows Application Testing ■CHAPTER 1 API Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Storing Test Case Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2 Reading Test Case Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Parsing a Test Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.4 Converting Data to an Appropriate Data Type . . . . . . . . . . . . . . . . . . . . . 9 1.5 Determining a Test Case Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.6 Logging Test Case Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.7 Time-Stamping Test Case Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.8 Calculating Summary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.9 Determining a Test Run Total Elapsed Time . . . . . . . . . . . . . . . . . . . . . 19 1.10 Dealing with null Input/null Expected Results . . . . . . . . . . . . . . . . . . 20 1.11 Dealing with Methods that Throw Exceptions . . . . . . . . . . . . . . . . . . 22 1.12 Dealing with Empty String Input Arguments . . . . . . . . . . . . . . . . . . . . 24 1.13 Programmatically Sending E-mail Alerts on Test Case Failures . . . 26 1.14 Launching a Test Harness Automatically . . . . . . . . . . . . . . . . . . . . . . . 28 1.15 Example Program: ApiTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 vii 6633FM.qxd 4/3/06 1:54 PM Page vii ■CHAPTER 2 Reflection-Based UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.1 Launching an Application Under Test . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.2 Manipulating Form Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.3 Accessing Form Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.4 Manipulating Control Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.5 Accessing Control Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.6 Invoking Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 2.7 Example Program: ReflectionUITest . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 ■CHAPTER 3 Windows-Based UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.1 Launching the AUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.2 Obtaining a Handle to the Main Window of the AUT . . . . . . . . . . . . . . 68 3.3 Obtaining a Handle to a Named Control . . . . . . . . . . . . . . . . . . . . . . . . 73 3.4 Obtaining a Handle to a Non-Named Control . . . . . . . . . . . . . . . . . . . . 75 3.5 Sending Characters to a Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 3.6 Clicking on a Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.7 Dealing with Message Boxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.8 Dealing with Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.9 Checking Application State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.10 Example Program: WindowsUITest . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 ■CHAPTER 4 Test Harness Design Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.1 Creating a Text File Data, Streaming Model Test Harness . . . . . . . . 100 4.2 Creating a Text File Data, Buffered Model Test Harness . . . . . . . . . . 104 4.3 Creating an XML File Data, Streaming Model Test Harness . . . . . . . 108 4.4 Creating an XML File Data, Buffered Model Test Harness . . . . . . . . 113 4.5 Creating a SQL Database for Lightweight Test Automation Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 4.6 Creating a SQL Data, Streaming Model Test Harness . . . . . . . . . . . . 119 4.7 Creating a SQL Data, Buffered Model Test Harness . . . . . . . . . . . . . 123 4.8 Discovering Information About the SUT . . . . . . . . . . . . . . . . . . . . . . . . 126 4.9 Example Program: PokerLibTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 ■CONTENTSviii 6633FM.qxd 4/3/06 1:54 PM Page viii PART 2 ■ ■ ■ Web Application Testing ■CHAPTER 5 Request-Response Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.1 Sending a Simple HTTP GET Request and Retrieving the Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 5.2 Sending an HTTP Request with Authentication and Retrieving the Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5.3 Sending a Complex HTTP GET Request and Retrieving the Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 5.4 Retrieving an HTTP Response Line-by-Line . . . . . . . . . . . . . . . . . . . . 141 5.5 Sending a Simple HTTP POST Request to a Classic ASP Web Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 5.6 Sending an HTTP POST Request to an ASP.NET Web Application . . . 145 5.7 Dealing with Special Input Characters . . . . . . . . . . . . . . . . . . . . . . . . . 150 5.8 Programmatically Determining a ViewState Value and an EventValidation Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.9 Dealing with CheckBox and RadioButtonList Controls . . . . . . . . . . . 156 5.10 Dealing with DropDownList Controls . . . . . . . . . . . . . . . . . . . . . . . . . 157 5.11 Determining a Request-Response Test Result . . . . . . . . . . . . . . . . . 159 5.12 Example Program: RequestResponseTest . . . . . . . . . . . . . . . . . . . . 162 ■CHAPTER 6 Script-Based Web UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 6.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 6.1 Creating a Script-Based UI Test Harness Structure . . . . . . . . . . . . . . 170 6.2 Determining Web Application State . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 6.3 Logging Comments to the Test Harness UI . . . . . . . . . . . . . . . . . . . . . 173 6.4 Verifying the Value of an HTML Element on the Web AUT . . . . . . . . . 174 6.5 Manipulating the Value of an HTML Element on the Web AUT . . . . . 176 6.6 Saving Test Scenario Results to a Text File on the Client . . . . . . . . . 177 6.7 Saving Test Scenario Results to a Database Table on the Server . . 179 6.8 Example Program: ScriptBasedUITest . . . . . . . . . . . . . . . . . . . . . . . . . 181 ■CONTENTS ix 6633FM.qxd 4/3/06 1:54 PM Page ix ■CHAPTER 7 Low-Level Web UI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 7.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 7.1 Launching and Attaching to IE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 7.2 Determining When the Web AUT Is Fully Loaded into the Browser . 190 7.3 Manipulating and Examining the IE Shell . . . . . . . . . . . . . . . . . . . . . . 192 7.4 Manipulating the Value of an HTML Element on the Web AUT . . . . . 194 7.5 Verifying the Value of an HTML Element on the Web AUT . . . . . . . . . 195 7.6 Creating an Excel Workbook to Save Test Scenario Results . . . . . . 198 7.7 Saving Test Scenario Results to an Excel Workbook . . . . . . . . . . . . . 200 7.8 Reading Test Results Stored in an Excel Workbook . . . . . . . . . . . . . . 201 7.9 Example Program: LowLevelUITest . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 ■CHAPTER 8 Web Services Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 8.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 8.1 Testing a Web Method Using the Proxy Mechanism . . . . . . . . . . . . . 212 8.2 Testing a Web Method Using Sockets . . . . . . . . . . . . . . . . . . . . . . . . . 214 8.3 Testing a Web Method Using HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 8.4 Testing a Web Method Using TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 8.5 Using an In-Memory Test Case Data Store . . . . . . . . . . . . . . . . . . . . . 226 8.6 Working with an In-Memory Test Results Data Store . . . . . . . . . . . . 229 8.7 Example Program: WebServiceTest . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 PART 3 ■ ■ ■ Data Testing ■CHAPTER 9 SQL Stored Procedure Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 9.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 9.1 Creating Test Case and Test Result Storage . . . . . . . . . . . . . . . . . . . . 239 9.2 Executing a T-SQL Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 9.3 Importing Test Case Data Using the BCP Utility Program . . . . . . . . . 243 9.4 Creating a T-SQL Test Harness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 9.5 Writing Test Results Directly to a Text File from a T-SQL Test Harness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 9.6 Determining a Pass/Fail Result When the Stored Procedure Under Test Returns a Rowset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 9.7 Determining a Pass/Fail Result When the Stored Procedure Under Test Returns an out Parameter . . . . . . . . . . . . . . . . . . . . . . . . . 254 9.8 Determining a Pass/Fail Result When the Stored Procedure Under Test Does Not Return a Value . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 9.9 Example Program: SQLspTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 ■CONTENTSx 6633FM.qxd 4/3/06 1:54 PM Page x 7e4af1220c26e223bcee6d3ae13e0471 ■CHAPTER 10 Combinations and Permutations . . . . . . . . . . . . . . . . . . . . . . . . . 265 10.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 10.1 Creating a Mathematical Combination Object . . . . . . . . . . . . . . . . . 267 10.2 Calculating the Number of Ways to Select k Items from n Items . . . 269 10.3 Calculating the Successor to a Mathematical Combination Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 10.4 Generating All Mathematical Combination Elements for a Given n and k . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 10.5 Determining the mth Lexicographical Element of a Mathematical Combination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 10.6 Applying a Mathematical Combination to a String Array . . . . . . . . 278 10.7 Creating a Mathematical Permutation Object . . . . . . . . . . . . . . . . . 280 10.8 Calculating the Number of Permutations of Order n . . . . . . . . . . . . 282 10.9 Calculating the Successor to a Mathematical Permutation Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 10.10 Generating All Mathematical Permutation Elements for a Given n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 10.11 Determining the kth Lexicographical Element of a Mathematical Permutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 10.12 Applying a Mathematical Permutation to a String Array . . . . . . . . 291 10.13 Example Program: ComboPerm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 ■CHAPTER 11 ADO.NET Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 11.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 11.1 Determining a Pass/Fail Result When the Expected Value Is a DataSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 11.2 Testing a Stored Procedure That Returns a Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 11.3 Testing a Stored Procedure That Returns a Rowset . . . . . . . . . . . . 309 11.4 Testing a Stored Procedure That Returns a Value into an out Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 11.5 Testing a Stored Procedure That Does Not Return a Value . . . . . . 314 11.6 Testing Systems That Access Data Without Using a Stored Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 11.7 Comparing Two DataSet Objects for Equality . . . . . . . . . . . . . . . . . . 321 11.8 Reading Test Case Data from a Text File into a SQL Table . . . . . . . 324 11.9 Reading Test Case Data from a SQL Table into a Text File . . . . . . . 327 11.10 Example Program: ADOdotNETtest . . . . . . . . . . . . . . . . . . . . . . . . . 329 ■CONTENTS xi 6633FM.qxd 4/3/06 1:54 PM Page xi ■CHAPTER 12 XML Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 12.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 12.1 Parsing XML Using XmlTextReader . . . . . . . . . . . . . . . . . . . . . . . . . . 337 12.2 Parsing XML Using XmlDocument . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 12.3 Parsing XML with XPathDocument . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 12.4 Parsing XML with XmlSerializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 12.5 Parsing XML with a DataSet Object . . . . . . . . . . . . . . . . . . . . . . . . . . 347 12.6 Validating XML with XSD Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 12.7 Modifying XML with XSLT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 12.8 Writing XML Using XmlTextWriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 12.9 Comparing Two XML Files for Exact Equality . . . . . . . . . . . . . . . . . . 356 12.10 Comparing Two XML Files for Exact Equality, Except for Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 12.11 Comparing Two XML Files for Canonical Equivalence . . . . . . . . . 359 12.12 Example Program: XmlTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 ■INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 ■CONTENTSxii 6633FM.qxd 4/3/06 1:54 PM Page xii About the Author ■DR. JAMES MCCAFFREY works for Volt Information Sciences, Inc. He holds a doctorate from the University of Southern California, a master’s in information systems from Hawaii Pacific University, a bachelor’s in mathematics from California State University at Fullerton, and a bachelor’s in psychology from the University of California at Irvine. He was a professor at Hawaii Pacific University, and worked as a lead software engineer at Microsoft on key prod- ucts such as Internet Explorer and MSN Search. xiii 6633FM.qxd 4/3/06 1:54 PM Page xiii 6633FM.qxd 4/3/06 1:54 PM Page xiv About the Technical Reviewer ■JOSH KELLING is a private consultant working in the business software industry. He is formally educated in physics and self-taught as a software developer with nearly 10 years of experience developing business and commercial software using Microsoft technologies. His focus has been primarily on .NET development since it was a beta product. He also enjoys teaching, skiing, hiking, hunting for wild mushrooms, and pool. xv 6633FM.qxd 4/3/06 1:54 PM Page xv 6633FM.qxd 4/3/06 1:54 PM Page xvi Acknowledgments Many people made this book possible. First and foremost, Jonathan Hassell and Elizabeth Seymour of Apress, Inc. drove the concept, writing, editing, and publication of the entire proj- ect. My corporate vice presidents at Volt Information Sciences, Inc., Patrick Walker and Christina Harris, suggested the idea of this book in the first place and supported its develop- ment. The lead technical reviewer, Josh Kelling (Kelling Consulting) did a terrific job at finding and correcting my coding mistakes. I’m also grateful to Doug Walter (Microsoft), who con- tributed significantly to the technical accuracy of this book. Many of the sections of this book are based on a monthly column I write for Microsoft’s MSDN Magazine. My editors at MSDN, Joshua Trupin and Stephen Toub, provided me with a lot of advice about writing, without which this book would never have gotten off the ground. And finally, my staff at Volt—Shirley Lin, Lisa Vo Carlson, and Grace Son—supplied indispensable administrative help. Many Volt software engineers working at Microsoft acted as auxiliary technical and edito- rial reviewers for this book. Primary technical reviewers include: Evan Kaplan, Steven Fusco, Bruce Ritter, Peter Yan, Ron Starr, Gordon Lippa, Kirk Slota, Joanna Tao, Walter Wittel, Jay Gray, Robert Hopkins, Sam Abolrous, Rich Bixby, Max Guernsey, Larry Briones, Kristin Jaeger, Joe Davis, Andrew Lee, Clint Kreider, Craig Green, Daniel Bedassa, Paul Kwiatkowski, Mark Wilcox, David Blais, Mustafa Al-Hasnawi, David Grossberg, Vladimir Abashyn, Mitchell Harter, Michael Svob, Brandon Lake, David Reynolds, Rob Gilmore, Cyrus Jamula, Ravichandhiran Kolandaiswamy, and Rajkumar Ramasamy. Secondary technical reviewers include Jerry Frost, Michael Wansley, Vanarasi Antony Swamy, Ted Keith, Chad Fairbanks, Chris Trevino, David Moy, Fuhan Tian, C.J. Eichholz, Stuart Martin, Justice Chang, Funmi Bolonduro, Alemeshet Alemu, Lori Shih, Eric Mattoon, Luke Burtis, Aaron Rodriguez, Ajay Bhat, Carol Snyder, Qiusheng Gao, Haik Babaian, Jonathan Collins, Dinesh Ravva, Josh Silveria, Brian Miller, Gary Roehl, Kender Talylor, Ahlee Ly, Conan Callen, Kathy Davis, and Florentin Ionescu. Editorial reviewers include Christina Zubelli, Joey Gonzales, Tony Chu, Alan Vandarwarka, Matt Carson, Tim Garner, Michael Klevitsky, Mark Soth, Michael Roshak, Robert Hawkins, Mark McGee, Grace Lou, Reza Sorasi, Abhijeet Shah, April McCready, Creede Lambard, Sean McCallum, Dawn Zhao, Mike Agranov, Victor Araya Cantuarias, Jason Olsan, Igor Bodi, Aldon Schwimmer, Andrea Borning, Norm Warren, Dale Dey, Chad Long, Thom Hokama, Ying Guo, Yong Wang, David Shockley, Allan Lockridge, Prashant Patil, Sunitha Mutnuri, Ping Du, Mark Camp, Abdul Khan, Moss Willow, Madhavi Kandibanda, John Mooney, Filiz Kurban, Jesse Larsen, Jeni Jordan, Chris Rosson, Dean Thomas, Brandon Barela, and Scott Lanphear. xvii 6633FM.qxd 4/3/06 1:54 PM Page xvii 6633FM.qxd 4/3/06 1:54 PM Page xviii Introduction What This Book Is About This book presents practical techniques for writing lightweight software test automation in a .NET environment. If you develop, test, or manage .NET software, you should find this book useful. Before .NET, writing test automation was often as difficult as writing the code for the application under test itself. With .NET, you can write lightweight, custom test automation in a fraction of the time it used to take. By lightweight automation, I mean small, dedicated test harness programs that are typically two pages of source code or less in length and take less than two hours to write. The emphasis of this book is on practical techniques that you can use immediately. Who This Book Is For This book is intended for software developers, testers, and managers who work with .NET technology. This book assumes you have a basic familiarity with .NET programming but does not make any particular assumptions about your skill level. The examples in this book have been successfully used in seminars where the audience background has ranged from begin- ning application programmers to advanced systems programmers. The content in this book has also been used in teaching environments where it has proven highly effective as a plat- form for students who are learning intermediate level .NET programming. Advantages of Lightweight Test Automation The automation techniques in this book are intended to complement, not replace, other test- ing paradigms, such as manual testing, test-driven development, model-based testing, open source test frameworks, commercial test frameworks, and so on. Software test automation, including the techniques in this book, has five advantages over manual testing. We sometimes refer to these automation advantages with the acronym SAPES: test automation has better Speed, Accuracy, Precision, Efficiency, and Skill-Building than manual testing. Additionally, when compared with both open source test frameworks and commercial frameworks, light- weight test automation has the advantage of not requiring you to travel up a rather steep learning curve and perhaps even learning a proprietary scripting language. Compared with commercial test automation frameworks, lightweight test automation is much less expensive and is fully customizable. And compared with open source test frameworks, lightweight automation is more stable in the sense that you have fewer recurring version updates and bug fixes to deal with. But the single most important advantage of lightweight, custom test automa- tion harnesses over commercial and open source test frameworks is subjective—lightweight automation actively encourages and promotes creative testing, whereas commercial and open source frameworks often tend to direct the types of automation you create to the types of tests that are best supported by the framework. The single biggest disadvantage of lightweight test automation is manageability. Because lightweight test harnesses are so easy to write, if you xix 6633FM.qxd 4/3/06 1:54 PM Page xix aren’t careful, your testing effort can become overwhelmed by the sheer number of test har- nesses, test case data, and test case result files you create. Test process management is outside the scope of this book, but it is a challenging topic you should not underestimate when writing lightweight test automation. Coding Issues All the code in this book is written in the C# language. Because of the unifying influence of the underlying .NET Framework, you can refactor the code in this book to Visual Basic .NET with- out too much trouble if necessary. All the code in this book was tested and ran successfully on both Windows XP Professional (SP2) and Windows Server 2003, and with Visual Studio .NET 2003 (with Framework 1.1) and SQL Server 2000. The code was also tested on Visual Studio 2005 (with Framework 2.0) and SQL Server 2005; however, if you are developing in that envi- ronment, you’ll have to make a few minor changes. I’ve coded the examples so that any changes you have to make for VS 2005 and SQL Server 2005 are flagged quickly. I decided that presenting just code for VS 2003 and SQL Server 2000 was a better approach than to sprinkle the book text with many short notes describing the minor development platform differences for VS 2005 and SQL Server 2005. The code in this book is intended strictly for 32-bit systems and has not been tested against 64-bit systems. If you are new to software test automation, you’ll quickly find that coding as a tester is significantly different from coding as a developer. Most of the techniques in this book are coded using a traditional, scripting style, rather than in an object-oriented style. I’ve found that automation code is easier to understand when written in a scripting style but this is a matter of opinion. Also, most of the code examples are not parameterized or packaged as methods. Again, this is for clarity. Most of the normal error-checking code, such as checking the values of input parameters to methods, is omitted. Error-traps are absolutely essential in production test automation code (after all, you are expecting to find errors) but error-checking code is often three or four times the size of the core code being checked. The code in this book is specifically designed for you to modify, which includes wrapping into methods, adding error-checks, incorporating into other test frameworks, and encapsulating into utility classes and libraries. Most of the chapters in this book present dummy applications to test against. By design, these dummy applications are not examples of good coding style, and these applications under test often contain deliberate errors. This keeps the size of the dummy applications small and also simulates the unrefined nature of an application’s state during the development process. For example, I generally use default control names such as textBox1 rather than use descriptive names, I keep local variable names short (such as s for a string variable), I sometimes place multiple statements on the same line, and so forth. I’ve actually left a few minor “severity 4” bugs (typographical errors) in the screenshots in this book; you might enjoy looking for them. In most cases, I’ve tried to be as accurate as possible with my terminology. For example, I use the term method when dealing with a subroutine that is a field/member in a C# class, and I use the term function when referring to a C++ subroutine in a Win32 API library. However, I make exceptions when I feel that a slightly incorrect term is more understandable or readable. For example, I sometimes use the term string variable instead of the more accurate string object when referring to a C# string type item. This book uses a problem-solution structure. This approach has the advantage of organiz- ing various test automation tasks in a convenient way. But to keep the size of the book reasonable, most of the solutions are not complete, standalone blocks of code. This means ■INTRODUCTIONxx 6633FM.qxd 4/3/06 1:54 PM Page xx that I often do not declare variables, explicitly discuss the namespaces and project references used in the solution, and so on. Many of the solutions in a chapter refer to other solutions within the same chapter, so you’ll have to make reasonable assumptions about dependencies and how to turn the solution code into complete test harnesses. To assist you in understand- ing how the sections of a chapter work together, the last section of every chapter presents a complete, standalone program. Contents of This Book In most computer science books, the contents of the book are summarized in the introduction. I will forego that practice and say instead that the best way to get a feel for what is contained in this book is to scan the table of contents; I know that’s what I always do. That said however, let me mention four specific topics in this book that have generated particular interest among my colleagues. Chapter 1, “API Testing,” is in many ways the most fundamental type of all software testing. If you are new to software testing, you will not only learn useful testing techniques, but you’ll also learn many of the basic principles of software testing. Chapter 3, “Windows-Based UI Testing,” presents powerful techniques to manipulate an application through its user inter- face. Even software testers with many years of experience are surprised at how easy UI test automation is using .NET and the techniques in that chapter. Chapter 5, “Request-Response Testing,” demonstrates the basic techniques to test any Web-based application. Web developers and testers are frequently surprised at how powerful these techniques are in a .NET environ- ment. Chapter 10, “Combinations and Permutations,” gives you the tools you need to programmatically generate test cases that take into account all combinations and rearrange- ments of input values. Both new and experienced testers have commented that combinatorics with .NET makes test case generation significantly more efficient than previously. Using the Code in This Book This book is intended to provide practical help for you in developing and testing software. This means that, within reason, you may use the code in this book in your systems and documenta- tion. Obvious exceptions include situations where you are reproducing a significant portion of the code in this book on a Web site or magazine article, or using examples in a conference talk, and so on. Most authors, including me, appreciate citations if you use examples from their book in a paper or article. All code is provided without warranty of any kind. ■INTRODUCTION xxi 6633FM.qxd 4/3/06 1:54 PM Page xxi 6633FM.qxd 4/3/06 1:54 PM Page xxii Windows Application Testing PART 1 ■ ■ ■ 6633c01.qxd 4/3/06 1:57 PM Page 1 6633c01.qxd 4/3/06 1:57 PM Page 2 API Testing 1.0 Introduction The most fundamental type of software test automation is automated API (Application Programming Interface) testing. API testing is essentially verifying the correctness of the individual methods that make up your software system rather than testing the overall system itself. API testing is also called unit testing, module testing, component testing, and element testing. Technically, the terms are very different, but in casual usage, you can think of them as having roughly the same meaning. The idea is that you must make sure the individual build- ing blocks of your system work correctly; otherwise, your system as a whole cannot be correct. API testing is absolutely essential for any significant software system. Consider the Windows- based application in Figure 1-1. This StatCalc application calculates the mean of a set of integers. Behind the scenes, StatCalc references a MathLib.dll library, which contains meth- ods named ArithmeticMean(), GeometricMean(), and HarmonicMean(). Figure 1-1. The system under test (SUT) 3 CHAPTER 1 ■ ■ ■ 6633c01.qxd 4/3/06 1:57 PM Page 3 The goal is to test these three methods, not the whole StatCalc application that uses them. The program being tested is often called the SUT (system under test), AUT (application under test), or IUT (implementation under test) to distinguish it from the test harness system. The techniques in this book use the term AUT. The methods under test are housed in a namespace MathLib with a single class named Methods and have the following signatures: namespace MathLib { public class Methods { public static double ArithmeticMean(params int[] vals) { // calculate and return arithmetic mean } private static double NthRoot(double x, int n) { // calculate and return the nth root; } public double GeometricMean(params int[] vals) { //use NthRoot to calculate and return geometric mean } public static double HarmonicMean(params int[] vals) { // this method not yet implemented } } // class Methods } // ns MathLib Notice that the ArithmeticMean() method is a static method, GeometricMean() is an instance method, and HarmonicMean() is not yet ready for testing. Handling static methods, instance methods, and incomplete methods are the three most common situations you’ll deal with when writing lightweight API test automation. Each of the methods under test accepts a variable number of integer arguments (as indicated by the params keyword) and returns a type double value. In most situations, you do not test private helper methods such as NthRoot(). Any errors in a helper will be exposed when testing the method that uses the helper. But if you have a helper method that has significant complexity, you’ll want to write dedicated test cases for it as well by using the techniques described in this chapter. Manually testing this API would involve creating a small tester program, copying the Methods class into the program, hard-coding some input values to one of the methods under test, running the stub program to get an actual result, visually comparing that actual result CHAPTER 1 ■ API TESTING4 6633c01.qxd 4/3/06 1:57 PM Page 4 with an expected result to determine a pass/fail result, and then recording the result in an Excel spreadsheet or similar data store. You would have to repeat this process hundreds of times to even begin to have confidence that the methods under test work correctly. A much better approach is to write test automation. Figure 1-2 shows a sample run of test automation that uses some of the techniques in this chapter. The complete program that generated the program shown in Figure 1-2 is presented in Section 1.15. Figure 1-2. Sample API test automation run Test automation has five advantages over manual testing: • Speed: You can run thousands of test cases very quickly. • Accuracy: Not as susceptible to human error, such as recording an incorrect result. • Precision: Runs the same way every time it is executed, whereas manual testing often runs slightly differently depending on who performs the tests. • Efficiency: Can run overnight or during the day, which frees you to do other tasks. • Skill-building: Interesting and builds your technical skill set, whereas manual testing is often mind-numbingly boring and provides little skill enhancement. The following sections present techniques for preparing API test automation, running API test automation, and saving the results of API test automation runs. Additionally, you’ll learn techniques to deal with tricky situations, such as methods that can throw exceptions or that can accept empty string arguments. The following sections also show you techniques to man- age API test automation, such as programmatically sending test results via e-mail. CHAPTER 1 ■ API TESTING 5 6633c01.qxd 4/3/06 1:57 PM Page 5 1.1 Storing Test Case Data Problem You want to create and store API test case data in a simple text file. Design Use a colon-delimited text file that includes a unique test case ID, one or more input values, and one or more expected results. Solution 0001:ArithmeticMean:2 4 8:4.6667 0002:ArithmeticMean:1 5:3.0000 0003:ArithmeticMean:1 2 4 8 16 32:10.5000 Comments When writing automated tests, you can store test case data externally to the test harness or you can embed the data inside the harness. In general, external test case data is preferable because multiple harnesses can share the data more easily, and the data can be more easily modified. Each line of the file represents a single test case. Each case has four fields separated by the ‘:’ character—test case ID, method to test, test case inputs separated by a single blank space, and expected result. You will often include additional test case data, such as a test case title, description, and category. The choice of delimiting character is arbitrary for the most part. Just make sure that you don’t use a character that is part of the inputs or expected values. For instance, the colon character works nicely for numeric methods but would not work well when testing methods with URLs as inputs because of the colon that follows “http”. In many lightweight test-automation situations, a text file is the best approach for storage because of simplicity. Alternative approaches include storing test case data in an XML file or SQL table. Weaknesses of using text files include their difficulty at handling inherently hierarchical data and the difficulty of seeing spurious control characters such as extra s. The preceding solution has only three test cases, but in practice you’ll often have thou- sands. You should take into account boundary values (using input values exactly at, just below, and just above the defined limits of an input domain), null values, and garbage (invalid) val- ues. You’ll also create cases with permuted (rearranged) input values like 0002:ArithmeticMean:1 5:3.0000 0003:ArithmeticMean:5 1:3.0000 Determining the expected result for a test case can be difficult. In theory, you’ll have a specification document that precisely describes the behavior of the method under test. Of course, the reality is that specs are often incomplete or nonexistent. One common mistake when determining expected results, and something you should definitely not do, is to feed inputs to the method under test, grab the output, and then use that as the expected value. This approach does not test the method; it just verifies that you get the same (possibly incorrect) output. This is an example of an invalid test system. CHAPTER 1 ■ API TESTING6 6633c01.qxd 4/3/06 1:57 PM Page 6 During the development of your test harness, you should create some test cases that delib- erately generate a fail result. This will help you detect logic errors in your harness. For example: 0004:ArithmeticMean:1 5:6.0000:deliberate failure In general, the term API testing is used when the functions or methods you are testing are stored in a DLL. The term unit testing is most often used when the methods you are testing are in a class (which of course may be realized as a DLL). The terms module testing, component testing, and element testing are more general terms that tend to be used when testing functions and methods not realized as a DLL. 1.2 Reading Test Case Data Problem You want to read each test case in a test case file stored as a simple text file. Design Iterate through each line of the test case file using a while loop with a System.IO.StreamReader object. Solution FileStream fs = new FileStream("..\\..\\TestCases.txt", FileMode.Open); StreamReader sr = new StreamReader(fs); string line; while ((line = sr.ReadLine()) != null) { // parse each test case line // call method under test // determine pass or fail // log test case result } sr.Close(); fs.Close(); Comments In general, console applications, rather than Windows-based applications, are best suited for lightweight test automation harnesses. Console applications easily integrate into legacy test systems and can be easily manipulated in a Windows environment. If you do design a harness as a Windows application, make sure that it can be fully manipulated from the command line. CHAPTER 1 ■ API TESTING 7 6633c01.qxd 4/3/06 1:57 PM Page 7 This solution assumes you have placed a using System.IO; statement in your harness so you can access the FileStream and StreamReader classes without having to fully qualify them. We also assume that the test case data file is named TestCases.txt and is located two directo- ries above the test harness executable. Relative paths to test case data files are generally better than absolute paths like C:\\Here\\There\\TestCases.txt because relative paths allow you to move the test harness root directory and subdirectories as a whole without breaking the har- ness paths. However, relative paths may break your harness if the directory structure of your test system changes. A good alternative is to parameterize the path and name of the test case data file: static void Main(string[] args) { string testCaseFile = args[0]; FileStream fs = new FileStream(testCaseFile, FileMode.Open); // etc. } Then you can call the harness along the lines of C:\Harness\bin\Debug>Run.exe ..\..\TestCases.txt In this solution, FileStream and StreamReader objects are used. Alternatively, you can use static methods in the System.IO.File class such as File.Open(). If you expect that two or more test harnesses may be accessing the test case data file simultaneously, you can use the over- loaded FileStream constructor that includes a FileShare parameter to specify how the file will be shared. 1.3 Parsing a Test Case Problem You want to parse the individual fields of a character-delimited test case. Design Use the String.Split() method, passing as the input argument the delimiting character and storing the return value into a string array. Solution string line, caseID, method; string[] tokens, tempInput; string expected; while ((line = sr.ReadLine()) != null) CHAPTER 1 ■ API TESTING8 6633c01.qxd 4/3/06 1:57 PM Page 8 { tokens = line.Split(':'); caseID = tokens[0]; method = tokens[1]; tempInput = tokens[2].Split(' '); expected = tokens[3]; // etc. } Comments After reading a line of test case data into a string variable line, calling the Split() method with the colon character passed in as an argument will break the line into the parts between the colons. These substrings are assigned to the string array tokens. So, tokens[0] will hold the first field, which is the test case ID (for example “001”), tokens[1] will hold the string identify- ing the method under test (for example “ArithmeticMean”), tokens[2] will hold the input vector as a string (for example “2 4 8”), and tokens[3] will hold the expected value (for exam- ple “4.667”). Next, you call the Split() method using a blank space argument on tokens[2] and assign the result to the string array tempInput. If tokens[2] has “2 4 8”, then tempInput[0] will hold “2”, tempInput[1] will hold “4”, and tempInput[2] will hold “8”. If you need to use more than one separator character, you can create a character array containing the separators and then pass that array to Split(). For example, char[] separators = new char[]{'#',':','!'}; string[] parts = line.Split(separators); will break the string variable line into pieces wherever there is a pound sign, colon, or exclama- tion point character and assign those substrings to the string array parts. The Split() method will satisfy most of your simple text-parsing needs for lightweight test- automation situations. A significant alternative to using Split() is to use regular expressions. One advantage of using regular expressions is that they are more powerful, in the sense that you can get a lot of parsing done in very few lines of code. One disadvantage of regular expressions is that they are harder to understand by those who do not use them often because the syntax is rel- atively unusual compared with most C# programming constructs. 1.4 Converting Data to an Appropriate Data Type Problem You want to convert your test case input data or expected result from type string into some other data type, so you can pass the data to the method under test or compare the expected result with an actual result. Design Perform an explicit type conversion with the appropriate static Parse() method. CHAPTER 1 ■ API TESTING 9 6633c01.qxd 4/3/06 1:57 PM Page 9 Solution int[] input = new int[tempInput.Length]; for (int i = 0; i < input.Length; ++i) input[i] = int.Parse(tempInput[i]); Comments If you store your test case data in a text file and then parse the test case inputs, you will end up with type string. If the method under test accepts any data type other than string you need to convert the inputs. In the preceding solution, if the string array tempInput holds {“2”,”4”,”8”} then you first create an integer array named input with the same size as tempInput. After the loop executes, input[0] will hold 2 (as an integer), input[1] will hold 4, and input[2] will hold 8. Including type string, the C# language has 14 data types that you’ll deal with most often as listed in Table 1-1. Table 1-1. Common C# Data Types and Corresponding .NET Types C# Type Corresponding .NET Type int Int32 short Int16 long Int64 uint Uint32 ushort Uint16 ulong Uint64 byte Byte sbyte Sbyte char Char bool Boolean float Single double Double decimal Decimal Each of these C# data types supports a static Parse() method that accepts a string argument and returns the calling data type. For example, string s1 = "345.67"; double d = double.Parse(s1); string s2 = "true"; bool b = bool.Parse(s2); will assign numeric 345.67 to variable d and logical true to b. An alternative to using Parse() is to use static methods in the System.Convert class. For instance, CHAPTER 1 ■ API TESTING10 6633c01.qxd 4/3/06 1:57 PM Page 10 string s1 = "345.67"; double d = Convert.ToDouble(s1); string s2 = "true"; bool b = Convert.ToBoolean(s2); is equivalent to the preceding Parse() examples. The Convert methods transform to and from .NET data types (such as Int32) rather than directly to their C# counterparts (such as int). One advantage of using Convert is that it is not syntactically C#-centric like Parse() is, so if you ever recast your automation from C# to VB.NET you’ll have less work to do. Advantages of using the Parse() method include the fact that it maps directly to C# data types, which makes your code somewhat easier to read if you are in a 100% C# environment. In addition, Parse() is more specific than the Convert methods, because it accepts only type string as a parameter (which is exactly what you need when dealing with test case data stored in a text file). 1.5 Determining a Test Case Result Problem You want to determine whether an API test case passes or fails. Design Call the method under test with the test case input, fetch the return value, and compare the actual result with the expected result read from the test case. Solution string method, expected; double actual = 0.0; if (method == "ArithmeticMean") { actual = MathLib.Methods.ArithmeticMean(input); if (actual.ToString("F4") == expected) Console.WriteLine("Pass"); else Console.WriteLine("*FAIL*"); } else { Console.WriteLine("Method not recognized"); } Comments After reading data for a test case, parsing that data, and converting the test case input to an appropriate data type if necessary, you can call the method under test. For your harness to be CHAPTER 1 ■ API TESTING 11 6633c01.qxd 4/3/06 1:57 PM Page 11 able to call the method under test, you must add a project reference to the DLL (in this exam- ple, MathLib) to the harness. The preceding code first checks to see which method the data will be applied to. In a .NET environment, methods are either static or instance. ArithmeticMean() is a static method, so it is called directly using its class context, passing in the integer array input as the argument, and storing the return result in the double variable actual. Next, the return value obtained from the method call is compared with the expected return value (sup- plied by the test case data). Because the expected result is type string, but the actual result is type double, you must convert one or the other. Here the actual result is converted to a string with four decimal places to match the format of the expected result. If we had chosen to con- vert the expected result to type double if (actual == double.Parse(expected)) Console.WriteLine("Pass"); else Console.WriteLine("*FAIL*"); we would have ended up comparing two double values for exact equality, which is problematic as types double and float are only approximations. As a general rule of thumb, you should con- vert the expected result from type string except when dealing with type double or float as in this example. GeometricMean()is an instance method, so before calling it, you must instantiate a MathLib.Methods object. Then you call GeometricMean() using its object context. If the actual result equals the expected result, the test case passes, and you print a pass message to console: if (method == "GeometricMean") { MathLib.Methods m = new MathLib.Methods(); actual = m.GeometricMean(input); if (actual.ToString("F4") == expected) Console.WriteLine("Pass"); else Console.WriteLine("*FAIL*"); } You’ll usually want to add additional information such as the test case ID to your output statements, for example: Console.WriteLine(caseID + " Pass"); For test cases that fail, you’ll often want to print the actual and expected values to help diagnose the failure, for example: Console.WriteLine(caseID + " *FAIL* " + method + " actual = " + actual.ToString("F4") + " expected = " + expected); A design question you must answer when writing API tests is how many methods will each lightweight harness test? In many situations, you’ll write a different test harness for every method under test; however, you can also combine testing multiple methods in a single harness. For example, to test both the ArithmeticMean() and GeometricMean() methods, you could combine test case data into a single file: CHAPTER 1 ■ API TESTING12 6633c01.qxd 4/3/06 1:57 PM Page 12 0001:ArithmeticMean:2 4 8:4.6667 0002:ArithmeticMean:1 5:3.0000 0004:GeometricMean :1 2 4 8 16 32:6.6569 0006:GeometricMean :2 4 8:4.0000 (The trailing blank space in “GeometricMean ” is for readability only.) Then you can modify the test harness logic to branch on the value for the method under test: if (method == "ArithmeticMean") { // code to test ArithmeticMean here } else if (method == "GeometricMean ") { // code to test GeometricMean here } else { Console.WriteLine("Unknown method""); } The decision to combine testing multiple methods in one harness usually depends on how close the methods’ signatures are to each other. If the signatures are close as in this example (both methods accept a variable number of integer arguments and return a double), then com- bining their tests may save you time. If your methods’ signatures are very different, then you’ll usually be better off writing separate harnesses. When testing an API method, you must take into account whether the method is stateless or stateful. Most API methods are stateless, which means that each call is independent. Or put another way, each call to a stateless method with a given input set will produce the same result. Sometimes we say that a stateless method has no memory. On the other hand, some methods are stateful, which means that the return result can vary. For example, suppose you have a Fibonacci generator method that returns the sum of its two previous integer results. So the first and second calls return 1, the third call returns 2, the fourth call returns 3, the fifth call returns 5, and so on. When testing a stateful method, you must make sure your test harness logic prepares the method’s state correctly. Your test harness must be able to access the API methods under test. In most cases, you should add a project reference to the DLL that is housing the API methods. However, in some situations, you may want to physically copy the code for the methods under test into your test harness. This approach is necessary when testing a private helper method (assuming you do not want to change the method’s access modifier from public to private). 1.6 Logging Test Case Results Problem You want to save test case results to external storage as a simple text file. CHAPTER 1 ■ API TESTING 13 6633c01.qxd 4/3/06 1:57 PM Page 13 Design Inside the main test case processing loop, use a System.IO.StreamWriter object to write a test case ID and a pass or fail result. Solution // open StreamReader sr here FileStream ofs = new FileStream("..\\..\\TestResults.txt", FileMode.CreateNew); StreamWriter sw = new StreamWriter(ofs); string line, caseID, method, expected; double actual = 0.0; while ((line = sr.ReadLine()) != null) { // parse "line" here if (method == "ArithmeticMean") { actual = MathLib.Methods.ArithmeticMean(input); if (actual.ToString("F4") == expected) sw.WriteLine(caseID + " Pass"); else sw.WriteLine(caseID + " *FAIL*"); } else { sw.WriteLine(caseID + " Unknown method"); } } // while sw.Close(); ofs.Close(); Comments In many situations, you’ll want to write your test case results to external storage instead of, or in addition to, displaying them in the command shell. The simplest form of external storage is a text file. Alternatives include writing to a SQL table or an XML file. You create a FileStream object and a StreamWriter object to write test case results to external storage. In this solution, the FileMode.CreateNew argument creates a new text file named TestResults.txt two directo- ries above the test harness executable. Using a relative file path allows you to move your entire test harness directory structure if necessary. Then you can use the StreamWriter object to write test results to external storage just as you would to the console. CHAPTER 1 ■ API TESTING14 6633c01.qxd 4/3/06 1:57 PM Page 14 When passing in a FileMode.CreateNew “TestResults.txt” argument, if a file with the name TestResults.txt already exists, an exception will be thrown. You can avoid this by using a FileMode.Create argument, but then any existing TestResults.txt file will be overwritten, and you could lose test results. One strategy is to parameterize the test results file name static void Main(string[] args) { string testResultsFile = args[0]; FileStream ofs = new FileStream(testResultsFile, FileMode.CreateNew); StreamWriter sw = new StreamWriter(ofs); // etc. } and pass in a new manually generated test results file name for each run: C:\Harness\bin\Debug>Run.exe Results-12-25-06.txt Alternatives include writing results to a programmatically time-stamped file. Our examples so far have either written test results to the command shell or to a .txt file, but you can write results to both console and external storage: if (actual.ToString("F4") == expected) { Console.WriteLine(caseID + " Pass"); sw.WriteLine(caseID + " Pass"); } else { Console.WriteLine(caseID + " *FAIL*"); sw.WriteLine(caseID + " *FAIL*"); } When the StreamWriter.WriteLine() statement executes, it does not actually write results to your output file. Results are buffered and then flushed out only when the StreamWriter.Close() statement executes. You can force results to be written by explicitly issuing a StreamWriter.Flush() statement. This is usually most important when you have a lot of test cases or when you catch an exception—be sure to close any open streams in either the catch block or the finally block so that buffered results will be written to file and not lost: catch(Exception ex) { Console.WriteLine("Unexpected fatal error: " + ex.Message); sw.Close(); // close other open streams } CHAPTER 1 ■ API TESTING 15 6633c01.qxd 4/3/06 1:57 PM Page 15 1.7 Time-Stamping Test Case Results Problem You want to time-stamp your test case results so you can distinguish the results of different test runs. Design Use the DateTime.Now property passed as an argument to the static CreateDirectory() method to create a time-stamped folder. Alternatively, you can pass DateTime.Now to the FileStream() constructor to create a time-stamped file name. Solution string folder = "Results" + DateTime.Now.ToString("s"); folder = folder.Replace(":","-"); Directory.CreateDirectory("..\\..\\" + folder); string path = "..\\..\\" + folder + "\\TestResults.txt"; FileStream ofs = new FileStream(path, FileMode.Create); StreamWriter sw = new StreamWriter(ofs); Comments You create a folder name using the DateTime.Now property, which grabs the current system date and time. Passing an “s” argument to the ToString() method returns a date-time string in a sortable pattern like “2006-07-30T13:57:00”. You can use many other formatting arguments with ToString(), but a sortable pattern will help you manage test results better than a non- sortable pattern. You must replace the colon character with some other character (here we use a hyphen) because colons are not valid in a path or file name. Next, you create the time-stamped folder using the static CreateDirectory() method, and then you can pass the entire path and file name to the FileStream constructor. After instanti- ating a StreamWriter object using the FileStream object, you can use the StreamWriter object to write into a file named TestResults.txt, which is located inside the time-stamped folder. A slight variation on this idea is to write all results to the same folder but time-stamp their file names: string stamp = DateTime.Now.ToString("s"); stamp = stamp.Replace(":","-"); string path = "..\\..\\TestResults-" + stamp + ".txt"; FileStream ofs = new FileStream(path, FileMode.Create); StreamWriter sw = new StreamWriter(ofs); This variation assumes that an arbitrary result directory is located two directories above the test harness executable directory. If the directory does not exist, an exception is thrown. The test case result file name becomes the time-stamp value appended to the string TestResults- with a .txt extension added, for example, TestResults-2006-12-25T23-59-59.txt. CHAPTER 1 ■ API TESTING16 6633c01.qxd 4/3/06 1:57 PM Page 16 1.8 Calculating Summary Results Problem You want to tally your test case results to track the number of test cases that pass and the number of cases that fail. Design Use simple integer counters initialized to 0 at the beginning of each test run. Solution int numPass = 0, numFail = 0; while ((line = sr.ReadLine()) != null) { // parse "line" here if (method == "ArithmeticMean") { actual = MathLib.Methods.ArithmeticMean(input); if (actual.ToString("F4") == expected) { Console.WriteLine("Pass"); ++numPass; } else { Console.WriteLine("*FAIL*"); ++numFail; } } else { Console.WriteLine("Unknown method"); // no effect on numPass or numFail } } // loop Console.WriteLine("Number cases passed = " + numPass); Console.WriteLine("Number cases failed = " + numFail); Console.WriteLine("Total cases = " + (numPass + numFail)); double percent = ((double)numPass) / (numPass + numFail); Console.WriteLine("Percent passed = " + percent.ToString("P")); CHAPTER 1 ■ API TESTING 17 6633c01.qxd 4/3/06 1:57 PM Page 17 Comments It is often useful to calculate and record summary metrics such as the total number of test cases that pass and the number that fail. If you track these numbers daily, you can gauge the progress of the quality of your software system. You might also want to record and track the percentage of test cases that pass because most product specifications have exit criteria such as, “for mile- stone MM3, a full API test pass will achieve a 99.95% test case pass rate.” You can declare integer variables and initialize them to 0 outside the main test loop. If a test case passes, you increment the pass counter; if the test case fails, you increment the fail counter. After all tests have been run, you can display your summary metrics and/or write them to external storage. In the preceding solution we track the number of test cases that pass and the number that fail and then add them to determine the total number of cases run. You may also want to ini- tialize and insert a counter numCases that increments after every test case so you can verify your test harness logic: if (actual.ToString("F4") == expected) { Console.WriteLine("Pass"); ++numPass; ++numCases; } else { Console.WriteLine("*FAIL*"); ++numFail; ++numCases; } // etc. if ((numPass + numFail) != numCases) Console.WriteLine("Warning: Counter logic failure"); When calculating a percent pass rate, be careful to cast either the numerator or denomi- nator to type double so that the result of the division operation is implicitly converted to type double: double percent = ((double)numPass) / (numPass + numFail); If you don’t cast, you will be performing integer division and always get either 1.0 (100%) or 0.0 (0%) for the result. Instead of using an explicit C# cast to type double, you can perform an implicit cast by multiplying by 1.0: double percent = (numPass * 1.0) / (numPass + numFail); This old technique has the advantage of being language-independent but the disadvantage of doing more work than is necessary. CHAPTER 1 ■ API TESTING18 6633c01.qxd 4/3/06 1:57 PM Page 18 1.9 Determining a Test Run Total Elapsed Time Problem You want to determine the total elapsed run time for a test run. Design Use the DateTime.Now property to record the time when the test run started and when the test run ended. Then use a TimeSpan object to calculate the elapsed time for the test run. Solution DateTime startTime = DateTime.Now; while ((line = sr.ReadLine()) != null) { // run tests } DateTime endTime = DateTime.Now; TimeSpan elapsedTime = endTime - startTime; Console.WriteLine("Elapsed time = " + elapsedTime.ToString()); Comments Calling the DateTime.Now property retrieves the current system time on the test harness machine. You fetch a start time before your tests execute and an end time after the test run concludes. To determine the elapsed time, you find the difference between the start and end times. DateTime objects support an overloaded subtraction operator (“–”) that returns a TimeSpan object. You can think of a DateTime value as being an instant in time and a TimeSpan value as being a time-duration. You have to be somewhat careful about exactly where you place the statements that find the test run start and end times. The guiding principle is to place them as much as possible so that you capture time spent executing your tests, but not so much that you are capturing test harness overhead activities that can vary. The purpose of recording and storing/displaying the total elapsed time of your daily test run is so that you can detect any significant change in the performance characteristics of your API methods. If the total elapsed time of a test run increases greatly one day, then you need to investigate. If you discover that a code change in one of the methods under test produced the performance degradation, you’ll find out immediately and can decide to recast the code or accept the performance penalty. If a code change was not the cause of the performance hit, then you may have a problem with your test harness system (for example some rogue process running and using up CPU time.) Another cause of a change in test run elapsed time would be increasing (or decreasing) the number of tests in the test case data file. CHAPTER 1 ■ API TESTING 19 6633c01.qxd 4/3/06 1:57 PM Page 19 One of the advantages of test automation is that you can execute many thousands of test cases quickly. When you are dealing with a huge number of test case results, you may want to log only summary metrics (the number of cases that passed and the number that failed) and details only about failed test cases. In a situation like this, determining and logging the test run elapsed time is important because it can uncover test harness problems that can be hid- den when you don’t have detailed test results to examine. 1.10 Dealing with null Input/null Expected Results Problem You want to verify the correct handling of null arguments passed to API methods under test. Design Use a special string token to represent null in your test case data file. Add logic to your test harness that converts the null-token to a null input value. Solution Suppose the original ArithmeticMean() method under test is modified to handle null input: public static double ArithmeticMean(params int[] vals) { if (vals == null) return 0.0; // modification double sum = 0.0; foreach (int v in vals) sum += v; return (double)(sum / vals.Length); } // ArithmeticMean You can add a null-token to your test case data like this: 0001:ArithmeticMean:2 4 8:4.6667 0002:ArithmeticMean:NULL:0.0000 Then process the token like this: string line, caseID, method; string[] tokens, tempInput; int[] input = null; double expected, actual; CHAPTER 1 ■ API TESTING20 6633c01.qxd 4/3/06 1:57 PM Page 20 while ((line = sr.ReadLine()) != null) { tokens = line.Split(':'); caseID = tokens[0]; method = tokens[1]; if (tokens[2] == "NULL") // null input input = null; else { tempInput = tokens[2].Split(' '); input = new int[tempInput.Length]; for (int i = 0; i < input.Length; ++i) input[i] = int.Parse(tempInput[i]); } expected = double.Parse(tokens[3]); actual = MathLib.Methods.ArithmeticMean(input); if (actual == expected) Console.WriteLine(caseID + " Pass"); else Console.WriteLine(caseID + " *FAIL*"); } // while Comments Testing API methods for null input arguments is essential. Because we can’t store a null value directly in the test case data, we use the string token “NULL”. Using “NULL” is arbitrary but makes the test case data and code more readable than alternatives like “nil” or “invalid”. When we read and parse a test case, we check for the string “NULL” and then branch to special logic in the test harness. The exact logic you use will depend on the behavior of the method under test. Notice that we assigned null to our input variable at declaration time: int[] input = null; and then reassign a null value if we read “NULL” from the test case file: if (tokens[2] == "NULL") input = null; This is technically unnecessary but makes our code more readable and easier to modify. Dealing with a null expected result uses the same idea as dealing with a null input argument. Suppose a new method named Hypergeometric() is added to the MathLib library under test, and that the Hypergeometric() method returns null if all input arguments are 0. To test, we store a string token such as “NULL” in the test case file: CHAPTER 1 ■ API TESTING 21 6633c01.qxd 4/3/06 1:57 PM Page 21 0001:Hypergeometric:0 0 0 0:NULL: 0002:Hypergeometric:1 3 5 7:2 4: and then add logic to the test harness: object expected = null; while ((line = sr.ReadLine()) != null) { if (tokens[3] == "NULL") expected = null; else // parse tokens[3] into object expected here // etc. } 1.11 Dealing with Methods that Throw Exceptions Problem You want to test a method that throws an exception. Design Embed a special string token in your test case data file to signal that an exception should be thrown, and place the call to the method under test in a try block so you can catch the excep- tion if it is thrown. Solution Add a token “Exception” as the expected value field in your test case data: 0004:GeometricMean :1 2 4 8 16 32:6.6569 0005:GeometricMean :0 0 0 0:Exception 0006:GeometricMean :2 4 8:4.0000 and then process inside the main test loop like this: MathLib.Methods m = new MathLib.Methods(); if (tokens[3] == "Exception") { try { actual = m.GeometricMean(input); } catch(Exception ex) CHAPTER 1 ■ API TESTING22 6633c01.qxd 4/3/06 1:57 PM Page 22 { Console.WriteLine(caseID + " Pass"); continue; } Console.WriteLine(caseID + " *FAIL* no exception thrown"); } else { // use regular test logic } Comments A common situation is that methods will throw an exception for certain input. For example, a method that performs a division operation with one of its input arguments may throw an excep- tion if the value of the argument is 0. In this example, we assume the original GeometricMean() method has been modified so that passing all zero values to the method throws an exception by design. We check for this special input by examining the test case data for an “Exception” string. If we find it, we branch to code that wraps the call to GeometricMean() in a try block. If an excep- tion is thrown as expected, control is transferred to the catch block, and we print a pass result. Then we move to the next test case when the continue statement is executed. If calling GeometricMean() does not throw an exception, control will reach the Console.WriteLine(caseID + " *FAIL* no exception thrown"); statement. Notice you do not want to wrap calls to the method under test that do not throw an exception in a try block because if the method does throw an exception, you’ll get a pass result. Dealing with methods that throw an exception can be messy in terms of integrating that special logic into the “regular” logic of your test harness. Because of this, a good strategy is to create two different lightweight test harnesses—one harness for test cases that do not throw excep- tions and one harness just for cases that do. The preceding solution is designed so that we test only that an exception is thrown, not necessarily a particular exception. In some situations, you may want to check for a specific exception. One way to do this is to embed the exception message in your test case file and check for it in your harness logic. For example, suppose the GeometricMean() method contains this code: if (denominator == 0) throw new Exception("Invalid division"); You could create this test case: 0005:GeometricMean :0 0 0 0:Invalid division and then test inside the main test loop like this: CHAPTER 1 ■ API TESTING 23 6633c01.qxd 4/3/06 1:57 PM Page 23 expected = tokens[3]; // "Invalid division" try { actual = m.GeometricMean(input); } catch(Exception ex) { if (ex.Message == expected) { Console.WriteLine(caseID + " Pass; correct exception"); continue; } else { Console.WriteLine(caseID + " *FAIL*; wrong exception"); } } Console.WriteLine(caseID + " *FAIL*; no exception thrown"); 1.12 Dealing with Empty String Input Arguments Problem You want to test empty string arguments passed to API methods under test. Design Use a special string token to represent an empty string in your test case file and then add branching logic to your test harness that passes a true empty string argument to the API method under test. Solution Create test case data like this: 0001:SubString:put:computer:true 0002:SubString:xyz:computer:false 0003:SubString:emptystring:computer:true CHAPTER 1 ■ API TESTING24 6633c01.qxd 4/3/06 1:57 PM Page 24 and add special logic to the test harness to handle the “emptystring” token like this: tokens = line.Split(':'); if (tokens[2] == "emptystring") // special input arg1 = ""; else arg1 = tokens[2]; bool actual = StringLib.Methods.SubString(arg1, tokens[3]); if (actual == bool.Parse(tokens[4])) Console.WriteLine("Pass"); else Console.WriteLine("*FAIL*"); Comments When testing API methods that accept string arguments, you should always test for empty strings. One way to deal with this is to store a special string token such as “emptystring” in your test case data file and then branch your test case logic when that token is read. Suppose, for example, you are testing a custom StringLib library containing a custom SubString() method that accepts two string arguments and returns true if the first argument is contained within the second argument. By design, the custom SubString() method returns true if an empty string is passed to its first parameter. Unlike null input, it is possible to indirectly store empty string input in a test case data file. For example, the test case data string 0003:SubString::computer:true when parsed by String.Split() into a string array named tokens will store an empty string into tokens[2] because of the two consecutive colon characters. However, in general, it’s much better to store a special string token because it makes your test case data easier to read and validate programmatically. The technique of embedding special string tokens in your test case data file to deal with empty string input can be used to test for other unusual input too. For example, suppose you are testing a method that accepts a character input argument. You will want to test for control characters such as and , and ASCII vs. Unicode characters. You can store strings like “”, “”, and “\u0041” in your test case data and then add special logic to your harness to deal with them: char input; if (tokens[2] == "") // special input input = '\x000d'; else input = char.Parse(tokens[2]); CHAPTER 1 ■ API TESTING 25 6633c01.qxd 4/3/06 1:57 PM Page 25 If you have a lot of special tokens in your test case data file as is often the case, you can keep your harness code cleaner and more scalable by writing a helper method Map(), which converts the input value read from the test case data file into the appropriate value. For example, you could write: private static char Map(string token) { if (token == "") return '\x000d'; else if (token == "") return '\x000a'; // etc. else return char.Parse(token); } and then use it in your harness like this: char input = Map(tokens[2]); 1.13 Programmatically Sending E-mail Alerts on Test Case Failures Problem You want your test harness to programmatically send an e-mail message when a test case fails. Design Use the System.Web.Mail class to create a MailMessage object. Supply properties such as To and Subject, and add details of the test case failure to the Body property. Solution if (method == "ArithmeticMean") { actual = MathLib.Methods.ArithmeticMean(input); if (actual.ToString("F4") == expected) { Console.WriteLine("Pass"); } else CHAPTER 1 ■ API TESTING26 6633c01.qxd 4/3/06 1:57 PM Page 26 { Console.WriteLine("*FAIL*. Sending e-mail"); try { MailMessage m = new MailMessage(); m.From = "Test Automation Harness"; m.To = "you@somewhere.com"; m.Subject = "Test Case Failure"; m.BodyEncoding = System.Text.Encoding.ASCII; m.BodyFormat = MailFormat.Html; m.Priority = MailPriority.High; m.Body = "Test case " + caseID + " failed"; SmtpMail.SmtpServer = "127.0.0.1"; SmtpMail.Send(m); } catch(Exception ex) { Console.WriteLine("Fatal error sending mail: " + ex.Message); } } // test case failed } Comments Because test automation often runs unattended, you may want to send an e-mail message to yourself or one of your team members when a test case fails, so that the test case failure does not lie unnoticed in a log file somewhere. You may also want to send an e-mail message summa- rizing the test run results. There are several ways to programmatically send e-mail in a .NET environment, but using the MailMessage class in the System.Web.Mail namespace is usually the easiest. The code in this solution assumes you have first added a project reference to the System.Web namespace (it’s not accessible by a console application by default) and placed a using System.Web.Mail; statement in your harness code. After instantiating a MailMessage object, you supply values for properties From, To, Subject, BodyEncoding, BodyFormat, and Body. You can also set the optional Priority property to MailPriority.High. You set the BodyEncoding value to one of the encoding representations from System.Text.Encoding. You will usually use Encoding.ASCII, but you can use Encoding.Unicode or Encoding.UTF8 if you want to. The BodyFormat property can be set to MailFormat.Html or MailFormat.Text. Either will work fine as long as there are no quirks in your e-mail system. The Body property is a string that holds the text of the message. At a mini- mum, you’ll want to include the test case ID that failed and the actual and expected return values. CHAPTER 1 ■ API TESTING 27 6633c01.qxd 4/3/06 1:57 PM Page 27 In theory, programmatically sending e-mail is easy, but in practical terms, a lot of things can go wrong. You have to deal with relay servers, proxy servers, network security policies, and firewalls, just to name a few. Because of this, your best strategy is to make sure that whenever feasible, your test harness machine is running as isolated as possible from other machines. This will help prevent unintended side effects as well as make sending e-mail from your test harness much more reliable. 1.14 Launching a Test Harness Automatically Problem You want your test harness program to launch automatically. Design Use the Windows Task Scheduler. Solution You will often want to launch your test automation automatically. For example you might schedule a test automation harness to start at 2:00 A.M. so that it runs overnight, and the test results are ready to review when you come to work. The Windows Task Scheduler makes it easy to schedule tasks in a Windows environment. You specify the test harness executable, the schedule of when you want the harness to run, and the security context under which you want the harness to run. If you have several lightweight test-automation harnesses that you want to run, you can create a .BAT file with commands to launch them. For example, @echo off echo Starting test automation sequence echo. C:\TestHarness1\bin\Debug\Run.exe C:\TestHarness2\bin\Debug\Run.exe C:\TestHarness3\bin\Debug\Run.exe echo. echo Test automation sequence complete If your harnesses explicitly log test case results to external files, then this is all you need. If your harnesses log test case results to the command shell, then you can easily save them to external storage using system redirection such as: C:\TestHarness1\bin\Debug\Run.exe > C:\Results\Harness1Results.txt C:\TestHarness2\bin\Debug\Run.exe > C:\Results\Harness2Results.txt CHAPTER 1 ■ API TESTING28 6633c01.qxd 4/3/06 1:57 PM Page 28 Comments For most lightweight test-automation situations, using .BAT files to manage multiple test har- nesses is simple and effective. An alternative is to write a C# master harness that coordinates and calls the worker automation harnesses. You can write statements that call the worker har- nesses by using the static Start() method in the System.Diagnostics.Process namespace: Console.WriteLine("Starting test automation sequence\n"); Process.Start("C:\\TestHarness1\\bin\\Debug\\Run.exe"); Process.Start("C:\\TestHarness2\\bin\\Debug\\Run.exe"); Process.Start("C:\\TestHarness3\\bin\\Debug\\Run.exe"); Console.WriteLine("\n Test automation sequence complete\n"); and then schedule this one master program using the Task Scheduler. An advantage of using a .BAT file master solution is simplicity. An advantage of using a C# master solution is that you have increased power to do things like catch exceptions and add branching logic to execute a worker harness only if a preceding harness test results meet some condition. Instead of using the Windows Task Scheduler to automatically launch a test harness, you can use the old at command. Scheduling a test harness or master harness program to run using the at command is more difficult than using the Windows Task Scheduler. You should only use at if your test system does not support the Task Scheduler or if you are integrating a new test harness system into an existing system that uses the at command. 1.15 Example Program: ApiTest This program combines many of the techniques we’ve seen in this chapter into a complete lightweight API test harness. The methods under test are ArithmeticMean(), GeometricMean(), and HarmonicMean() as described at the beginning of this chapter. The complete lightweight test harness listing is shown in Listing 1-1. The program reads test case data a line at a time from the file TestCases.txt. Then the harness parses the test case ID, which includes the method to test, input values, and an expected result. Input values are sent to the method under test, and an actual result value is obtained and compared with the test case expected result value. A pass or fail result is sent to the command shell and logged to the file TestResults.txt. Listing 1-1. Program ApiTest using System; using System.IO; using MathLib; // houses methods under test namespace TestAutomation { class Class1 { [STAThread] static void Main(string[] args) CHAPTER 1 ■ API TESTING 29 6633c01.qxd 4/3/06 1:57 PM Page 29 { try { FileStream ifs = new FileStream("..\\..\\TestCases.txt", FileMode.Open); StreamReader sr = new StreamReader(ifs); string stamp = DateTime.Now.ToString("s"); stamp = stamp.Replace(":", "-"); FileStream ofs = new FileStream("..\\..\\TestResults" + stamp + ".txt", FileMode.CreateNew); StreamWriter sw = new StreamWriter(ofs); string line, caseID, method; string[] tokens, tempInput; string expected; double actual = 0.0; int numPass = 0, numFail = 0; Console.WriteLine("\nCaseID Result Method Details"); Console.WriteLine("================================\n"); while ((line = sr.ReadLine()) != null) { tokens = line.Split(':'); caseID = tokens[0]; method = tokens[1]; tempInput = tokens[2].Split(' '); expected = tokens[3]; int[] input = new int[tempInput.Length]; for (int i = 0; i < input.Length; ++i) input[i] = int.Parse(tempInput[i]); if (method == "ArithmeticMean") { actual = MathLib.Methods.ArithmeticMean(input); if (actual.ToString("F4") == expected) { Console.WriteLine(caseID + " Pass " + method + " actual = " + actual.ToString("F4")); sw.WriteLine(caseID + " Pass " + method + " actual = " + actual.ToString("F4")); ++numPass; } else CHAPTER 1 ■ API TESTING30 6633c01.qxd 4/3/06 1:57 PM Page 30 { Console.WriteLine(caseID + " *FAIL* " + method + " actual = " + actual.ToString("F4") + " expected = " + expected); sw.WriteLine(caseID + " *FAIL* " + method + " actual = " + actual.ToString("F4") + " expected = " + expected); ++numFail; } } else if (method == "GeometricMean ") { MathLib.Methods m = new MathLib.Methods(); actual = m.GeometricMean(input); if (actual.ToString("F4") == expected) { Console.WriteLine(caseID + " Pass " + method + " actual = " + actual.ToString("F4")); sw.WriteLine(caseID + " Pass " + method + " actual = " + actual.ToString("F4")); ++numPass; } else { Console.WriteLine(caseID + " *FAIL* " + method + " actual = " + actual.ToString("F4") + " expected = " + expected); sw.WriteLine(caseID + " *FAIL* " + method + " actual = " + actual.ToString("F4") + " expected = " + expected); ++numFail; } } else { Console.WriteLine(caseID + " " + method + " Not yet implemented"); sw.WriteLine(caseID + " " + method + " Not yet implemented"); } } // test case loop CHAPTER 1 ■ API TESTING 31 6633c01.qxd 4/3/06 1:57 PM Page 31 Console.WriteLine("\n========= end test run =========="); Console.WriteLine("\nPass = " + numPass + " Fail = " + numFail); sw.WriteLine(Environment.NewLine + "Pass = " + numPass + " Fail = " + numFail); sr.Close(); ifs.Close(); sw.Close(); ofs.Close(); } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); } Console.ReadLine(); } // Main() } // class Class1 } // ns TestAutomation When run with this test case data file: 0001:ArithmeticMean:2 4 8:4.6667 0002:ArithmeticMean:1 5:3.0000 0003:ArithmeticMean:1 2 4 8 16 32:10.5000 0004:GeometricMean :1 2 4 8 16 32:6.6569 0005:GeometricMean :0:0.0000 0006:GeometricMean :2 4 8:4.0000 0007:HarmonicMean :2 4 8:3.4286 0008:HarmonicMean :2 3 6:3.0000 the output is 0001 Pass ArithmeticMean actual = 4.6667 0002 Pass ArithmeticMean actual = 3.0000 0003 Pass ArithmeticMean actual = 10.5000 0004 *FAIL* GeometricMean actual = 5.6569 expected = 6.6569 0005 Pass GeometricMean actual = 0.0000 0006 Pass GeometricMean actual = 4.0000 0007 HarmonicMean Not yet implemented 0008 HarmonicMean Not yet implemented Pass = 5 Fail = 1 Test case 0004 has a deliberately incorrect expected value to check the validity of the test harness logic. CHAPTER 1 ■ API TESTING32 6633c01.qxd 4/3/06 1:57 PM Page 32 Reflection-Based UI Testing 2.0 Introduction The most fundamental and simplest form of application testing is manual testing through the application’s user interface (UI). Paradoxically, automated testing through a user interface (automated UI testing for short) is challenging. The .NET environment provides you with many classes in the System.Reflection namespace that can access and manipulate an application at run time. Using reflection, you can write lightweight automated UI tests. For example, suppose you had a simple form-based Windows application, as shown in the foreground of Figure 2-1. Figure 2-1. Reflection-based UI testing A user types paper, rock, or scissors into the TextBox control, and a second user selects one of those strings from the ComboBox control. When either user clicks on the Button control, a message with the winner is displayed in the ListBox control. The key code for this dummy application is 33 CHAPTER 2 ■ ■ ■ 6633c02.qxd 4/3/06 1:53 PM Page 33 private void button1_Click(object sender, System.EventArgs e) { string tb = textBox1.Text; string cb = comboBox1.Text; if (tb == cb) listBox1.Items.Add("Result is a tie"); else if (tb == "paper" && cb == "rock" || tb == "rock" && cb == "scissors" || tb == "scissors" && cb == "paper") listBox1.Items.Add("The TextBox wins"); else listBox1.Items.Add("The ComboBox wins"); } Note that this is not an example of good coding, and many deliberate errors are included. For example, the ComboBox player can win by leaving the ComboBox control empty. This simulates the unrefined character of an application while still under development. Using the techniques in this chapter, you can write automated UI tests as shown in the background of Figure 2-1. To write reflection-based lightweight UI test automation, you must be able to perform six tasks programmatically (each test automation task corresponds to a section in this chapter): • Launch the application under test (AUT) from your test-harness program in a way that allows the two programs to communicate. • Manipulate the application form to simulate a user moving and resizing the form. • Examine the application form properties to verify that the resulting state of the applica- tion is correct so you can determine a test scenario pass or fail result. • Manipulate the application control properties to simulate actions such as a user typing into a TextBox control. • Examine the application control properties to verify that the resulting state of the application is correct so you can determine a test scenario pass or fail result. • Invoke the application methods to simulate actions such as a user clicking on a Button control. The techniques in this chapter are very lightweight. The main advantage of using these reflection-based test techniques is that they are very quick and easy to implement. The main disadvantages are that they apply only to pure .NET applications and that they cannot deal with complex test scenarios. The techniques in Chapter 3 provide you with lower-level, more powerful UI test-automation techniques at the expense of increased complexity. CHAPTER 2 ■ REFLECTION-BASED UI TESTING34 6633c02.qxd 4/3/06 1:53 PM Page 34 2.1 Launching an Application Under Test Problem You want to launch the AUT so that you can manipulate it. Design Spin off a separate thread of execution from the test harness by creating a Thread object and then associate that thread with an application state wrapper class. Solution using System; using System.Reflection; using System.Windows.Forms; using System.Threading; class Class1 { [STAThread] static void Main(string[] args) { try { Console.WriteLine("Launching Form"); Form theForm = null; string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; Assembly a = Assembly.LoadFrom(path); Type t1 = a.GetType(formName); theForm = (Form)a.CreateInstance(t1.FullName); AppState aps = new AppState(theForm); ThreadStart ts = new ThreadStart(aps.RunApp); Thread thread = new Thread(ts); thread.ApartmentState = ApartmentState.STA; thread.IsBackground = true; thread.Start(); CHAPTER 2 ■ REFLECTION-BASED UI TESTING 35 6633c02.qxd 4/3/06 1:53 PM Page 35 Console.WriteLine("\nForm launched"); } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); } } // Main() private class AppState { public readonly Form formToRun; public AppState(Form f) { this.formToRun = f; } public void RunApp() { Application.Run(formToRun); } } // class AppState } // class Class1 To test a Windows-based form application through its UI using reflection techniques, you must launch the application on a separate thread of execution within the test-harness process. If, instead, you launch an AUT using the Process.State() method like this: string exePath = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; System.Diagnostics.Process.Start(exePath); the application will launch, but your test harness will not be able to directly communicate with the application because the harness and the application will be running in separate processes. The trick to enable harness-application communication is to spin off a separate thread from the harness. This way, the harness and the application will be running in the same process context and can communicate with each other. Comments If your test harness is a console application, you can add the following using statements so you won’t have to fully qualify classes and objects: using System.Reflection; using System.Windows.Forms; using System.Threading; The System.Reflection namespace houses the primary classes you’ll be using to access the AUT. The System.Windows.Forms namespace is not accessible to a console application by default, so you must add a project reference to the System.Windows.Forms.dll file. The System.Threading namespace allows you to create a separate thread of execution for the AUT. CHAPTER 2 ■ REFLECTION-BASED UI TESTING36 6633c02.qxd 4/3/06 1:53 PM Page 36 Start by getting a reference to the application Form object: Form theForm = null; string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; Assembly a = Assembly.LoadFrom(path); Type t1 = a.GetType(formName); theForm = (Form)a.CreateInstance(t1.FullName); The heart of obtaining a reference to the Form object under test is to use the Assembly. CreateInstance() method. This is slightly tricky because CreateInstance() is called from the context of an Assembly object and accepts an argument for the full name of the instance being created. Furthermore, an Assembly object is created using a factory mechanism instead of the more usual constructor instantiation with the new keyword. Additionally, the full name argu- ment is called from a Type context. In short, you must first create an Assembly object using Assembly.Load(), passing in the path to the assembly. Then you create a Type object using Assembly.GetType(), passing in the full Form class name. And, finally, you create a reference to the Form object under test using Assembly.CreateInstance(), passing in the Type.FullName property. Notice that you must use the full form name (e.g., "AUT.Form1") rather than the shortened form name (e.g., "Form1"). The code to launch the Form under test is best understood by working backwards. The goal is to create a new Thread object and then call its Start() method; however, to create a Thread object, you need to pass a ThreadStart object to the Thread constructor. To create a ThreadStart object, you need to pass a target method to the ThreadStart constructor. This tar- get method must return void, and it is the method to invoke when the thread begins execution. Now in the case of a Form object, you want to call the Application.Run() method. Although it seems a bit awkward, the easiest way to pass Application.Run() to ThreadStart is to create a separate wrapper class: private class AppState { public readonly Form formToRun; public AppState(Form f) { this.formToRun = f; } public void RunApp() { Application.Run(formToRun); } } This AppState class is just a wrapper around a Form object and a call to the Application.Run() method. We do this to pass Application.Run() to ThreadStart in a convenient way. With this class in place, you can instantiate an AppState object and pass Application.Run() indirectly to the ThreadStart constructor: CHAPTER 2 ■ REFLECTION-BASED UI TESTING 37 6633c02.qxd 4/3/06 1:53 PM Page 37 AppState aps = new AppState(theForm); ThreadStart ts = new ThreadStart(aps.RunApp); With the ThreadStart object created, you can create a new Thread, set its properties if nec- essary, and start the thread up: Thread thread = new Thread(ts); thread.ApartmentState = ApartmentState.STA; thread.IsBackground = true; thread.Start(); An alternative to creating a Thread object directly is to call the ThreadPool.QueueUserWorkItem() method. That method creates a thread indirectly and requires a starting method to be passed to a WaitCallBack object. This approach would look like Form theForm = null; string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; Assembly a = Assembly.LoadFrom(path); Type t1 = a.GetType(formName); theForm = (Form)a.CreateInstance(t1.FullName); ThreadPool.QueueUserWorkItem(new WaitCallback(RunApp), theForm); where static void RunApp(object o) { Application.Run(o as Form); } This ThreadPool technique is somewhat simpler than the ThreadStart solution but does not give you as much control over the thread of execution. You can increase the modularity of this technique by refactoring your code as a method: static Form LaunchApp(string path, string formName) { Form result = null; Assembly a = Assembly.LoadFrom(path); Type t = a.GetType(formName); result = (Form)a.CreateInstance(t.FullName); AppState aps = new AppState(result); ThreadStart ts = new ThreadStart(aps.RunApp); Thread thread = new Thread(ts); thread.Start(); return result; } which you can call like this: CHAPTER 2 ■ REFLECTION-BASED UI TESTING38 6633c02.qxd 4/3/06 1:53 PM Page 38 Form theForm = null; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; string formName = "AUT.Form1"; theForm = LaunchApp(path, formName); 2.2 Manipulating Form Properties Problem You want to set the properties of a Windows form-based application. Design Get a reference to the property you want to set using the Type.GetProperty() method. Then use the PropertyInfo.SetValue() method in conjunction with the Form.Invoke() method and a method delegate. Solution string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; Form theForm = LaunchApp(path, formName); // see Section 2.1 Thread.Sleep(1500); Console.WriteLine("\nSetting Form1 Location to x=10, y=20"); System.Drawing.Point pt = new System.Drawing.Point(10,20); object[] o = new object[] { theForm, "Location", pt }; Delegate d = new SetFormPropertyValueHandler(SetFormPropertyValue); if (theForm.InvokeRequired) { theForm.Invoke(d, o); } else { Console.WriteLine("Unexpected logic flow"); } where delegate void SetFormPropertyValueHandler(Form f, string propertyName, object newValue); static void SetFormPropertyValue(Form f, string propertyName, object newValue) CHAPTER 2 ■ REFLECTION-BASED UI TESTING 39 6633c02.qxd 4/3/06 1:53 PM Page 39 { Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); pi.SetValue(f, newValue, null); } Comments To simulate user interaction with a Windows-based form application, you may want to move the form or resize the form. One way to do this using a reflection-based technique is to use the PropertyInfo.SetValue() method. Although the idea is simple in principle, the details are tricky. You can best understand the technique by working backwards. The .NET Framework has a PropertyInfo.SetValue() method that can set the value of a property of an object. But the SetValue() method requires a PropertyInfo object context. However, a PropertyInfo object requires a Type object context. So you start by creating a Type object from the Form object you want to manipulate. Then you get a PropertyInfo object from the Type object, and then you call the SetValue() method. So, if there were no hidden issues you could simply write code like this: theForm = LaunchApp(path, formName); // see Section 2.1 Console.WriteLine("\nSetting Form location to x=10, y=20"); Type t = theForm.GetType(); PropertyInfo pi = t.GetProperty("Location"); Point pt = new Point(10,20); pi.SetValue(theForm, pt, null); Unfortunately, there is a serious hidden issue that you must deal with. Before explaining that hidden issue, let’s examine the SetValue() method. SetValue() accepts three arguments. The PropertyInfo object, whose SetValue() method you call, represents a property, such as a Form object’s Location property. The first argument to SetValue() is the object to manipulate, which in this case is the Form object. The second argument is the new value of the property, which in this example is a new Point object. The third argument is necessary because some properties are indexed. When a property is not indexed, as is usually the case with form controls, you can just pass a null value as the argument. The hidden issue with calling the PropertyInfo.SetValue() method is that you are not calling SetValue() from the main Form thread; you are calling SetValue() from a thread cre- ated by the test-automation harness. In situations like this, you should not call SetValue() directly. A full explanation of this issue is outside the scope of this book, but the conclusion is that you should call SetValue() indirectly by calling the Form.Invoke() method. This is a bit tricky because Form.Invoke() requires a delegate object that calls SetValue() and an object that represents the arguments for SetValue(). So in pseudo-code, you need to do this: if (theForm.InvokeRequired) theForm.Invoke(a method delegate, an object array); else Console.WriteLine("Unexpected logic flow"); The InvokeRequired property in this situation should always be true because the Form object was launched by a different thread (the automation harness). If InvokeRequired is not true, there is a logic error and you may want to print a warning message. CHAPTER 2 ■ REFLECTION-BASED UI TESTING40 6633c02.qxd 4/3/06 1:53 PM Page 40 So, now you need a method delegate. Before you create the delegate, which you can think of as an alias for a real method, you create the real method that will actually do the work: static void SetFormPropertyValue(Form f, string propertyName, object newValue) { Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); pi.SetValue(f, newValue, null); } Notice that this method is almost exactly like the naive code if the whole InvokeRequired hidden issue did not exist. After creating the real method, you create a delegate that matches the real method: delegate void SetFormPropertyValueHandler(Form f, string propertyName, object newValue); In short, if you pass a reference to delegate SetFormPropertyValueHandler(), control is transferred to the associated SetFormPropertyValue() method (assuming you associate the two in the delegate constructor). Now that we’ve dealt with the delegate parameter to the Form.Invoke() method, we have to deal with the object array parameter. This parameter represents arguments that are passed to the delegate and then, in turn, are passed to the associated real method. In this case, the delegate requires a Form object, a property name as a string, and a location as a Point object: System.Drawing.Point pt = new System.Drawing.Point(10,20); object[] o = new object[] { theForm, "Location", pt }; Putting these ideas and code together, you can write delegate void SetFormPropertyValueHandler(Form f, string propertyName, object newValue); static void SetFormPropertyValue(Form f, string propertyName, object newValue) { Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); pi.SetValue(f, newValue, null); } static void Main(string[] args) { Form theForm = null; string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; theForm = LaunchApp(path, formName); // see Section 2.1 Console.WriteLine("\nSetting Form1 Location to 10,20"); CHAPTER 2 ■ REFLECTION-BASED UI TESTING 41 6633c02.qxd 4/3/06 1:53 PM Page 41 System.Drawing.Point pt = new System.Drawing.Point(10,20); object[] o = new object[] { theForm, "Location", pt }; Delegate d = new SetFormPropertyValueHandler(SetFormPropertyValue); if (theForm.InvokeRequired) theForm.Invoke(d, o); else Console.WriteLine("Unexpected logic flow"); //etc. } And now manipulating the properties of the application form is very easy. For example, suppose you want to change the size of the form. Here’s how: Console.WriteLine("\nSetting Form1 Size to 300x400"); System.Drawing.Size size = new System.Drawing.Size(300,400); object[] o = new object[] { theForm, "Size", size }; Delegate d = new SetFormPropertyValueHandler(SetFormPropertyValue); if (theForm.InvokeRequired) { theForm.Invoke(d, o); } else Console.WriteLine("Unexpected logic flow"); Console.WriteLine("\n And now setting Form1 Size to 200x500"); Thread.Sleep(1500); size = new System.Drawing.Size(200,500); o = new object[] { theForm, "Size", size }; d = new SetFormPropertyValueHandler(SetFormPropertyValue); if (theForm.InvokeRequired) { theForm.Invoke(d, o); } else Console.WriteLine("Unexpected logic flow"); You can significantly increase the modularity of this technique by wrapping up the code into a single method combined with a delegate: delegate void SetFormPropertyValueHandler(Form f, string propertyName, object newValue); static void SetFormPropertyValue(Form f, string propertyName, object newValue) CHAPTER 2 ■ REFLECTION-BASED UI TESTING42 6633c02.qxd 4/3/06 1:53 PM Page 42 { if (f.InvokeRequired) { // Console.WriteLine("in invoke required"); Delegate d = new SetFormPropertyValueHandler(SetFormPropertyValue); object[] o = new object[] { f, propertyName, newValue }; f.Invoke(d, o); return; } else { // Console.WriteLine("in the else part"); Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); pi.SetValue(f, newValue, null); } } With this helper method, you can make clean calls in your test harness such as Form theForm = null; string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; theForm = LaunchApp(path, formName); // see Section 2.1 System.Drawing.Point pt = new System.Drawing.Point(10,10); SetFormPropertyValue(theForm, "Location", pt); Thread.Sleep(1500); pt = new System.Drawing.Point(200,300); SetFormPropertyValue(theForm, "Location", pt); Thread.Sleep(1500); This SetFormPropertyValue() wrapper is slightly tricky because it is self-referential. (A recursive method calls itself directly; a self-referential method calls itself indirectly.) When called in the Main() method of your harness, InvokeRequired is initially true because the call- ing automation thread does not own the form. Execution branches to the Form.Invoke() statement, which, in turn, calls the SetFormPropertyValueHandler() delegate that calls back into the associated SetFormPropertyValue() method. But the second time through the wrap- per, InvokeRequired will be false, because the call comes from the originating thread. Control transfers to the else part of the logic, where the PropertyInfo.SetValue() changes the Form property. If you remove the commented lines of code and run, you’ll see how the path of exe- cution works. Because placing Thread.Sleep() delays is so common in UI test automation, you may want to add a delay parameter to all the wrapper methods in this chapter: CHAPTER 2 ■ REFLECTION-BASED UI TESTING 43 6633c02.qxd 4/3/06 1:53 PM Page 43 static void SetFormPropertyValue(Form f, string propertyName, object newValue, int delay) { Thread.Sleep(delay); // other code as before } So, if you wanted to delay 1,500 milliseconds (1.5 seconds), you can call SetFormPropertyValue() like this: Point point = new Point(50,75); SetFormPropertyValue (theForm, "Location", point, 1500); In a lightweight test-automation situation, the most common form properties you will manipulate are Size and Location. However, the techniques in the section allow you to set any form property. For example, suppose you want to manipulate the form title bar. You can do this by passing "Text" as the property name argument and a string for the new title: Form theForm = null; string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; theForm = LaunchApp(path, formName); // see Section 2.1 SetFormPropertyValue(theForm, "Text", "SomeNewTitle"); Thread.Sleep(1500); 2.3 Accessing Form Properties Problem You want to retrieve the properties of an application form object. Design Use the Type.GetProperty() method to get a reference to the property you want to examine. Then use the PropertyInfo.GetValue() method in conjunction with a method delegate to get the value of the property. Solution if (theForm.InvokeRequired) { Delegate d = new GetFormPropertyValueHandler(GetFormPropertyValue); object[] o = new object[] { theForm, "Location" }; Point p = (Point)theForm.Invoke(d, o); Console.WriteLine("Form1 location = " + p.X + " " + p.Y); } CHAPTER 2 ■ REFLECTION-BASED UI TESTING44 6633c02.qxd 4/3/06 1:53 PM Page 44 else { Console.WriteLine("Unexpected logic flow"); } where delegate object GetFormPropertyValueHandler(Form f, string propertyName); static object GetFormPropertyValue(Form f, string propertyName) { Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); object result = pi.GetValue(f, null); return result; } Comments When performing lightweight reflection-based UI test automation, you may want to retrieve properties of the application form such as the Location and Size properties. This allows you to verify the state of the Form object under test and determine a test scenario pass/fail result. The key to obtaining the value of a form property is to use the PropertyInfo.GetValue() method. Unfortunately, there is a hidden issue—you should not call GetValue() directly from a thread that is not the main form thread. This issue is discussed in detail in “Section 2.2 Manipulating Form Properties.” If the hidden issue did not exist, you could get a form property like this: string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; theForm = LaunchApp(path, formName); // see Section 2.1 Type t = theForm.GetType(); PropertyInfo pi = t.GetProperty("Location"); Point p = (Point)pi.GetValue(theForm, null); Console.WriteLine("Form1 location = " + p.X + " " + p.Y); But because you are calling GetValue() from the test-harness thread instead of the main Form object thread, you should call Form.Invoke() with a delegate like this: if (theForm.InvokeRequired) { Delegate d = new GetFormPropertyValueHandler(GetFormPropertyValue); object[] o = new object[] { theForm, "Location" }; Point p = (Point)theForm.Invoke(d, o); Console.WriteLine("Form1 location = " + p.X + " " + p.Y); } CHAPTER 2 ■ REFLECTION-BASED UI TESTING 45 6633c02.qxd 4/3/06 1:53 PM Page 45 else { Console.WriteLine("Unexpected logic flow"); } where delegate object GetFormPropertyValueHandler(Form f, string propertyName); static object GetFormPropertyValue(Form f, string propertyName) { Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); return pi.GetValue(f, null); } In short, you call Form.Invoke() with a delegate argument. Control of execution is transferred to the delegate, which is in turn mapped to a helper method that calls the PropertyInfo.GetValue() method. This strategy solves the Invoke() issue. You can signifi- cantly increase the modularity of your test automation by wrapping up the code in this solution like this: delegate object GetFormPropertyValueHandler(Form f, string propertyName); static object GetFormPropertyValue(Form f, string propertyName) { if (f.InvokeRequired) { Delegate d = new GetFormPropertyValueHandler(GetFormPropertyValue); object[] o = new object[] { f, propertyName }; object iResult = f.Invoke(d, o); return iResult; } else { Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); object gResult = pi.GetValue(f, null); return gResult; } } This can be called in the following way: Point p = (Point)GetFormPropertyValue(theForm, "Location"); Console.WriteLine("Form location = " + p.X + " " + p.Y); CHAPTER 2 ■ REFLECTION-BASED UI TESTING46 6633c02.qxd 4/3/06 1:53 PM Page 46 This GetFormPropertyValue() wrapper is a bit tricky because it is self-referential. When called in the Main() method of your harness, InvokeRequired is initially true, because the calling thread does not own the form. Execution branches to the Form.Invoke() statement, which, in turn, calls the GetFormPropertyValueHandler() delegate that calls back into the associated GetFormPropertyValue() method. But on the second pass through the wrapper, InvokeRequired will be false because the call comes from the originating thread. Control transfers to the else part of the logic, where the PropertyInfo.GetValue() retrieves the form property. With this tech- nique, you can retrieve the value of any form property. For example: string title = (string)GetFormPropertyValue(theForm, "Text"); Console.WriteLine("Form title = " + title); Size size = (Size)GetFormPropertyValue(theForm, "Size"); Console.WriteLine("Form size = " + size.Height + " x " + size.Width); 2.4 Manipulating Control Properties Problem You want to set the value of a control property. Design Obtain a reference to the control you want to manipulate using the Form.GetType(), Type.GetField(), and FieldInfo.GetValue() methods. Then use the PropertyInfo.SetValue() method in conjunction with a method delegate to set the value of the target control. Solution if (theForm.InvokeRequired) { Delegate d = new SetControlPropertyValueHandler(SetControlPropertyValue); object[] o = new object[] { theForm, "textBox1", "Text", "foo" }; Console.WriteLine("Setting textBox1 to 'foo'"); theForm.Invoke(d, o); } else { Console.WriteLine("Unexpected logic flow"); } where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; CHAPTER 2 ■ REFLECTION-BASED UI TESTING 47 6633c02.qxd 4/3/06 1:53 PM Page 47 delegate void SetControlPropertyValueHandler(Form f, string controlName, string propertyName, object newValue); static void SetControlPropertyValue(Form f, string controlName, string propertyName, object newValue) { Type t1 = f.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(f); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName); //? pi.SetValue(ctrl, newValue, null); } Comments When writing lightweight reflection-based UI test automation, you may need to simulate user actions by manipulating properties of controls on the application form. Examples include setting the Text property value of a TextBox control to simulate a user typing and setting the Checked property value of a RadioButtonList item to true to simulate a user clicking on the item. The key to setting the value of a control’s property is to use the PropertyInfo.SetValue() method. Unfortunately, as described in Sections 2.2 “Manipulating Form Properties” and 2.3 “Accessing Form Properties,” there is a hidden issue—you should not call SetValue() directly from a thread that is not the main Form thread. If the hidden issue did not exist, you could set the value of a control like this: BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; Console.WriteLine("Setting textBox1 to 'foo'"); Type t1 = theForm.GetType(); FieldInfo fi = t1.GetField("textBox1", flags); object ctrl = fi.GetValue(theForm); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty("Text"); pi.SetValue(ctrl, "foo", null); The BindingFlags object is a filter for many of the methods in the System.Reflection namespace. In lightweight test-automation situations, you almost always filter for Public, NonPublic, Instance, and Static methods, as we’ve done in this example. Because this is such a common pattern, you’ll often find it convenient to declare a single class-scope BindingFlags object, rather than recode a new object for each call that requires a BindingFlags argument. To manipulate a control, you begin by getting a Type object from the parent Form object. This is the first of two Type objects you’ll need. Then you use the Type object to obtain a reference to a FieldInfo object by using the GetField() method. With this intermediate FieldInfo object, you can now get a reference to the actual control object by calling FieldInfo.GetValue(). This is not entirely intuitive but the pattern is always the same. Next, you use the control object and get its Type by calling GetType(). Then you can use this second Type object to get a PropertyInfo object using the GetProperty() method. At this point, you have references to the control object and one CHAPTER 2 ■ REFLECTION-BASED UI TESTING48 6633c02.qxd 4/3/06 1:53 PM Page 48 of its properties. Finally, you can manipulate the value of the control’s property by using the SetValue() method. The first two arguments passed to SetValue() are the control object to manipulate and the new value for the control’s property. The third argument represents optional index values. You only need this when you are dealing with indexed properties. This value should be a null reference for nonindexed properties, as is almost always the case for controls. Although some controls, such as the ListBox control, have components that are indexed (the Items property, for example), the control itself is not indexed. As described earlier, the hidden issue is that you should not directly call SetValue() on a control object from a thread that does not own the control’s parent Form object. Doing so can lead to complex thread synchronization problems. Because you are working from the test- harness thread instead of the Form thread, the Form.InvokeRequired property is always true. The recommended technique in situations like this is to call Form.Invoke(), passing a delegate object that is associated with a method that actually calls SetValue(). Implementing this idea gives you the code in this solution. You can significantly increase the modularity of this technique by wrapping the code up into a single method combined with a delegate object: static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; delegate void SetControlPropertyValueHandler(Form f, string controlName, string propertyName, object newValue); static void SetControlPropertyValue(Form f, string controlName, string propertyName, object newValue) { if (f.InvokeRequired) { //Console.WriteLine("in invoke req."); Delegate d = new SetControlPropertyValueHandler(SetControlPropertyValue); object[] o = new object[]{f, controlName, propertyName, newValue}; f.Invoke(d, o); } else { //Console.WriteLine("in else part"); Type t1 = f.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(f); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName); pi.SetValue(ctrl, newValue, null); } } CHAPTER 2 ■ REFLECTION-BASED UI TESTING 49 6633c02.qxd 4/3/06 1:53 PM Page 49 The method can be called like this: SetControlPropertyValue(theForm, "textBox1", "Text", "paper"); SetControlPropertyValue(theForm, "comboBox1", "Text", "rock"); This SetControlPropertyValue() wrapper improves the modularity of your automation code, but is somewhat tricky because it references itself. When SetControlPropertyValue() is called in the Main() method of your harness, InvokeRequired is initially true because the calling thread does not own the form. Execution branches to the Form.Invoke() statement, which, in turn, calls the SetControlPropertyValueHandler() delegate that calls back into the associated SetControlPropertyValue() method. But the second time through the wrapper, InvokeRequired will be false because the call now comes from the originating thread. Execution transfers to the else part of the logic, where the PropertyInfo.SetValue() changes the control’s property. If you remove the commented lines of code and run, you’ll see how the path of execution works. 2.5 Accessing Control Properties Problem You want to retrieve the properties of a control on a Windows form-based application. Design Obtain a reference to the control you want to manipulate using the Form.GetType(), Type.GetField(), and FieldInfo.GetValue() methods. Then use the PropertyInfo.GetValue() method in conjunction with a method delegate to retrieve the value of the target control. Solution if (theForm.InvokeRequired) { Delegate d = new GetControlPropertyValueHandler(GetControlPropertyValue); object[] o = new object[] { theForm, "textBox1", "Text" }; string txt = (string)theForm.Invoke(d, o); Console.WriteLine("textBox1 has " + txt); } else { Console.WriteLine("Unexpected logic flow"); } where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; CHAPTER 2 ■ REFLECTION-BASED UI TESTING50 6633c02.qxd 4/3/06 1:53 PM Page 50 delegate object GetControlPropertyValueHandler(Form f, string controlName, string propertyName); static object GetControlPropertyValue(Form f, string controlName, string propertyName) { Type t1 = f.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(f); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName); object result = pi.GetValue(ctrl, null); return result; } Comments When writing lightweight reflection-based UI test automation, you may want to retrieve proper- ties of controls on the application form. Examples include the Text property value of a TextBox control and the ObjectCollection property of a ListBox control. You must do this to verify the state of the AUT and determine a pass/fail test result. The key to obtaining the value of a con- trol’s property is to use the PropertyInfo.GetValue() method. But there is a hidden issue—you should not call GetValue() directly from a thread that is not the main Form thread. This issue is discussed in detail in Sections 2.2, 2.3, and 2.4. If the hidden issue did not exist, you could easily get a control property like this: // launch object theForm Type t1 = theForm.GetType(); FieldInfo fi = t1.GetField("textBox1", flags); object ctrl = fi.GetValue(theForm); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty("Text"); string txt = (string)pi.GetValue(ctrl, null); Console.WriteLine("TextBox1 Text is " + txt); To access the property of a control object, you start by getting a Type object from the par- ent Form object. This is the first of two Type objects you’ll need. Then you use that Type object to obtain a reference to a FieldInfo object by using the GetField() method. The flags argu- ment in this example is a BindingFlags object, as described in Section 2.4, and it acts as a filter. With the FieldInfo object, you can now get a reference to the actual control object by calling FieldInfo.GetValue(). Next, you use the control object and get its Type by calling GetType(). Then you can use this second Type object to get a PropertyInfo object using the GetProperty() method. At this point, you have references to the control object and one of its properties. Finally, you can manipulate the value of the control’s property by using the GetValue() method. CHAPTER 2 ■ REFLECTION-BASED UI TESTING 51 6633c02.qxd 4/3/06 1:53 PM Page 51 The GetValue() method accepts two arguments. The first argument to GetValue() is the parent control object. The second argument is an optional array of index values. You only need this when you are dealing with indexed properties. This value should be null for nonin- dexed properties, as is almost always the case for controls. Although some controls, such as the ListBox control, have components that are indexed (the Items property for example), the control property itself is not indexed. The hidden issue is that you should not directly call GetValue() on a Form object from a thread that does not own the form. Doing so can lead to thread problems. Because you are work- ing from the test-harness thread instead of the Form thread, the Form.InvokeRequired property is always true. The recommended technique in situations like this is to call Form.Invoke(), passing a delegate object that is associated with a method that actually calls PropertyInfo.GetValue(). You can significantly increase the modularity of this technique by wrapping the code into a single method in conjunction with a delegate: delegate object GetControlPropertyValueHandler(Form f, string controlName, string propertyName); static object GetControlPropertyValue(Form f, string controlName, string propertyName) { if (f.InvokeRequired) { Delegate d = new GetControlPropertyValueHandler(GetControlPropertyValue); object[] o = new object[] { f, controlName, propertyName }; object iResult = f.Invoke(d, o); return iResult; } else { Type t1 = f.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(f); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName); object gResult = pi.GetValue(ctrl, null); return gResult; } } with class-scope object static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; CHAPTER 2 ■ REFLECTION-BASED UI TESTING52 6633c02.qxd 4/3/06 1:53 PM Page 52 The logic behind this self-referential wrapping technique is explained in detail in the “Comments” part of Sections 2.2, 2.3, and 2.4. With this wrapper method, you can make clean calls in your test harness like this: string txt = (string)GetControlPropertyValue(theForm, "textBox1", "Text"); Console.WriteLine("TextBox1 holds " + txt); ListBox.ObjectCollection oc = (ListBox.ObjectCollection)GetControlPropertyValue(theForm, "listBox1", "Items"); if (oc.Count > 0) { string s = oc[0].ToString(); Console.WriteLine("The first line in listBox1 is " + s); } if (oc.Contains("The TextBox wins")) Console.WriteLine("Found 'The TextBox wins' in listBox1"); else Console.WriteLine("Did not find 'The TextBox wins' in listBox1"); Notice that for a ListBox control, you retrieve the Items property, which is a collection of type ListBox.ObjectCollection. This component is indexed so you can access each string in the collection or iterate through all the strings using square bracket syntax. 2.6 Invoking Methods Problem You want to invoke a method of a form-based application. Design Get a reference to the method you want to invoke using the Form.GetType() and Type.GetMethod() methods. Then use the MethodInfo.Invoke() method in conjunction with an AutoResetEvent object and a method delegate to call the target method. Solution if (theForm.InvokeRequired) { Delegate d = new InvokeMethodHandler(InvokeMethod); object[] parms = new object[] { null, EventArgs.Empty }; object[] o = new object[] { theForm, "button1_Click", parms }; theForm.Invoke(d, o); are.WaitOne(); } CHAPTER 2 ■ REFLECTION-BASED UI TESTING 53 6633c02.qxd 4/3/06 1:53 PM Page 53 else { Console.WriteLine("Unexpected logic flow"); } where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; static AutoResetEvent are = new AutoResetEvent(false); delegate void InvokeMethodHandler(Form f, string methodName, params object[] parms); static void InvokeMethod(Form f, string methodName, params object[] parms) { Type t = f.GetType(); MethodInfo mi = t.GetMethod(methodName, flags); mi.Invoke(f, parms); are.Set(); } Comments When writing lightweight reflection-based UI test automation, you usually need to invoke methods that are part of the application form to simulate user actions. Examples include invoking a button1_Click() method, which handles actions when a user clicks on a button1 control, and invoking a menuItem2_Click() method, which handles actions when a user clicks on a menuItem2 menu item. Notice that reflection-based UI automation simulates a button click by directly invoking the button control’s associated method rather than by firing an event. When a real user clicks on a button, it generates a Windows message that is processed by the control and turned into a managed event. This causes a particular method to be invoked. So, reflection-based UI automation will not catch the logic error if the AUT has the wrong method wired to a button click event. The key to invoking methods is to use the MethodInfo.Invoke() method. If there were no hidden issues, you could invoke a method like this: Type t = theForm.GetType(); MethodInfo mi = t.GetMethod("button1_Click", flags); mi.Invoke(theForm, new object[] { null, EventArgs.Empty }); where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; CHAPTER 2 ■ REFLECTION-BASED UI TESTING54 6633c02.qxd 4/3/06 1:53 PM Page 54 The BindingFlags object is a filter for many of the methods in the System.Reflection namespace and is discussed in Section 2.4. The MethodInfo.Invoke() method accepts two arguments. The first argument is the parent Form object that owns the method being invoked. The second argument is an object array containing the arguments that must be passed to the method being invoked. In this example, the button1_Click() method has a signature of private void button1_Click(object sender, System.EventArgs e) So you need to pass values for parameters sender and e, representing the object associ- ated with the button1_Click() method and optional event data the method might need. For lightweight UI test automation, you can ignore these parameters and simply pass null and EventArgs.Empty. As described in Sections 2.3, 2.3, and 2.4, there is a hidden issue—you should not call MethodInfo.Invoke() directly from a thread that is not the main Form thread. The solution to this hidden invoke issue is to call MethodInfo.Invoke() indirectly through a Delegate object: if (theForm.InvokeRequired) { Delegate d = new InvokeMethodHandler(InvokeMethod); object[] parms = new object[] { null, EventArgs.Empty }; object[] o = new object[] { theForm, "button1_Click", parms }; theForm.Invoke(d, o); } else { Console.WriteLine("Unexpected logic flow"); } where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; delegate void InvokeMethodHandler(Form f, string methodName, params object[] parms); static void InvokeMethod(Form f, string methodName, params object[] parms) { Type t = f.GetType(); MethodInfo mi = t.GetMethod(methodName, flags); mi.Invoke(f, parms); } This code will work most of the time; however, programmatically invoking a method has a second, very subtle, hidden issue involving synchronization. Suppose your test harness invokes a method on the AUT, and that method directly or indirectly spins off a new thread of execution. Before your test harness takes any further action, you must wait until control is returned to the test harness. There are two solutions to this timing problem. The first is a CHAPTER 2 ■ REFLECTION-BASED UI TESTING 55 6633c02.qxd 4/3/06 1:53 PM Page 55 crude but effective approach: place Thread.Sleep() statements in your test harness to slow the automation down. For example: if (theForm.InvokeRequired) { Delegate d = new InvokeMethodHandler(InvokeMethod); object[] parms = new object[] { null, EventArgs.Empty }; object[] o = new object[] { theForm, "button1_Click", parms }; theForm.Invoke(d, o); Thread.Sleep(2000); } else { Console.WriteLine("Unexpected logic flow"); } where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; delegate void InvokeMethodHandler(Form f, string methodName, params object[] parms); static void InvokeMethod(Form f, string methodName, params object[] parms) { Type t = f.GetType(); MethodInfo mi = t.GetMethod(methodName, flags); mi.Invoke(f, parms); Thread.Sleep(2000); } However, this crude approach has a big problem: there’s no way to determine how long to pause so you must make your delay times long. This leads to a test harness with multiple lengthy delays. A better solution to the timing problem is to use an AutoResetEvent object for synchronization. You declare a class scope object like static AutoResetEvent are = new AutoResetEvent(false); which creates an object that can have a value of signaled or not-signaled. The false argument means initialize the object to not-signaled. Then, whenever you want to pause your automa- tion, you insert the statement are.WaitOne(). This sets the value of the AutoResetEvent object to not-signaled. The current thread of execution halts until the AutoResetEvent object is set to signaled from an are.Set() statement. Putting these ideas together led to this code: CHAPTER 2 ■ REFLECTION-BASED UI TESTING56 6633c02.qxd 4/3/06 1:53 PM Page 56 if (theForm.InvokeRequired) { Delegate d = new InvokeMethodHandler(InvokeMethod); object[] parms = new object[] { null, EventArgs.Empty }; object[] o = new object[] { theForm, "button1_Click", parms }; theForm.Invoke(d, o); are.WaitOne(); } else { Console.WriteLine("Unexpected logic flow"); } where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; static AutoResetEvent are = new AutoResetEvent(false); delegate void InvokeMethodHandler(Form f, string methodName, params object[] parms); static void InvokeMethod(Form f, string methodName, params object[] parms) { Type t = f.GetType(); MethodInfo mi = t.GetMethod(methodName, flags); mi.Invoke(f, parms); are.Set(); } So, at the beginning of the code, the test-automation thread does not own the main test Form object thread, and the InvokeRequired property is true. Execution control is transferred to the InvokeMethodHandler() delegate, which in turn is associated with an InvokeMethod() helper method. InvokeMethod() actually performs the work by calling MethodInfo.Invoke(). For synchronization, calling AutoResetEvent.WaitOne() blocks the thread of execution, allow- ing the MethodInfo.Invoke() method to complete execution. Calling AutoResetEvent.Set() signals that the thread of execution can resume. You can greatly modularize this technique by wrapping the code in a single self-referential method in conjunction with a delegate and an AutoResetEvent object: delegate void InvokeMethodHandler(Form f, string methodName, params object[] parms); CHAPTER 2 ■ REFLECTION-BASED UI TESTING 57 6633c02.qxd 4/3/06 1:53 PM Page 57 static void InvokeMethod(Form f, string methodName, params object[] parms) { if (f.InvokeRequired) { Delegate d = new InvokeMethodHandler(InvokeMethod); f.Invoke(d, new object[] {f, methodName, parms}); are.WaitOne(); } else { Type t = f.GetType(); MethodInfo mi = t.GetMethod(methodName, flags); mi.Invoke(f, parms); are.Set(); } } where static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; static AutoResetEvent are = new AutoResetEvent(false); With this convenient wrapper method you can make clean calls: object[] parms = new object[] { null, EventArgs.Empty }; InvokeMethod(theForm, "button1_Click", parms ); The InvokeMethod() wrapper is self-referencing. On the initial call to InvokeMethod() from your test harness, InvokeRequired is true because the call is coming from your test harness. Control of execution transfers to the Form.Invoke() method, which passes control to the InvokeMethodHandler() delegate. The delegate is associated with the original InvokeMethod() method, so execution control reenters InvokeMethod(). The second time through the helper method, InvokeRequired will be false, so control is transferred to the else block where MethodInfo.Invoke() actually invokes the method passed in as an argument to the helper. 2.7 Example Program: ReflectionUITest This program (see Listing 2-1) combines several of the techniques in the chapter to demon- strate a lightweight reflection-based UI test-automation scenario. The scenario tests the “paper-rock-scissors” form application described in the introduction to this chapter. The test scenario launches the application, moves the form, and then simulates typing rock into the TextBox control and a user selecting scissors from the ComboBox control. Then the scenario sim- ulates a button click. The automation checks to see if the expected TextBox1 wins string is in the ListBox control, and determines a scenario pass or fail result. The scenario finishes by CHAPTER 2 ■ REFLECTION-BASED UI TESTING58 6633c02.qxd 4/3/06 1:53 PM Page 58 simulating a user selecting File ➤ Exit to close the application. Figure 2-1 in the introduction to this chapter shows the result of running this test scenario. This program assumes you have added project references to the System.Windows.Forms and System.Drawing namespaces. You can extend this automation scenario by using some of the techniques described in Chapter 1 and Chapter 4. For example, you can log test results to external storage, or parameterize the scenario to accept multiple input states. Listing 2-1. Program ReflectionUITest using System; using System.Reflection; using System.Windows.Forms; using System.Threading; using System.Drawing; namespace ReflectionUITest { class Class1 { static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; static AutoResetEvent are = new AutoResetEvent(false); [STAThread] static void Main(string[] args) { try { Console.WriteLine("\nStarting test scenario"); Console.WriteLine("\nLaunching Form1"); Form theForm = null; string formName = "AUT.Form1"; string path = "..\\..\\..\\AUT\\bin\\Debug\\AUT.exe"; theForm = LaunchApp(path, formName); Console.WriteLine("\nMoving Form1"); Point pt = new Point(320, 100); SetFormPropertyValue(theForm, "Location", pt); Console.WriteLine("\nSetting textBox1 to 'rock'"); SetControlPropertyValue(theForm, "textBox1", "Text", "rock"); Console.WriteLine("Setting comboBox1 to 'scissors'"); SetControlPropertyValue(theForm, "comboBox1", "Text", "scissors"); CHAPTER 2 ■ REFLECTION-BASED UI TESTING 59 6633c02.qxd 4/3/06 1:53 PM Page 59 Console.WriteLine("\nClicking button1"); object[] parms = new object[]{ null, EventArgs.Empty }; InvokeMethod(theForm, "button1_Click", parms); bool pass = true; Console.WriteLine("\nChecking listBox1 for 'TextBox wins'"); ListBox.ObjectCollection oc = (ListBox.ObjectCollection) GetControlPropertyValue(theForm, "listBox1", "Items"); string s = oc[0].ToString(); if (s.IndexOf("TextBox wins") == -1) pass = false; if (pass) Console.WriteLine("\n-- Scenario result = Pass --"); else Console.WriteLine("\n-- Scenario result = *FAIL* --"); Console.WriteLine("\nClicking File->Exit in 3 seconds"); Thread.Sleep(3000); InvokeMethod(theForm, "menuItem2_Click", parms); Console.WriteLine("\nEnd test scenario"); } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); } } // Main() static Form LaunchApp(string path, string formName) { Form result = null; Assembly a = Assembly.LoadFrom(path); Type t = a.GetType(formName); result = (Form)a.CreateInstance(t.FullName); AppState aps = new AppState(result); ThreadStart ts = new ThreadStart(aps.RunApp); Thread thread = new Thread(ts); thread.Start(); return result; } private class AppState { public readonly Form formToRun; public AppState(Form f) CHAPTER 2 ■ REFLECTION-BASED UI TESTING60 6633c02.qxd 4/3/06 1:53 PM Page 60 { this.formToRun = f; } public void RunApp() { Application.Run(formToRun); } } // class AppState delegate void SetFormPropertyValueHandler(Form f, string propertyName, object newValue); static void SetFormPropertyValue(Form f, string propertyName, object newValue) { if (f.InvokeRequired) { Delegate d = new SetFormPropertyValueHandler(SetFormPropertyValue); object[] o = new object[] { f, propertyName, newValue }; f.Invoke(d, o); are.WaitOne(); } else { Type t = f.GetType(); PropertyInfo pi = t.GetProperty(propertyName); pi.SetValue(f, newValue, null); are.Set(); } } delegate void SetControlPropertyValueHandler(Form f, string controlName, string propertyName, object newValue); static void SetControlPropertyValue(Form f, string controlName, string propertyName, object newValue) { if (f.InvokeRequired) { Delegate d = new SetControlPropertyValueHandler(SetControlPropertyValue); object[] o = new object[] { f, controlName, propertyName, newValue }; f.Invoke(d, o); are.WaitOne(); } CHAPTER 2 ■ REFLECTION-BASED UI TESTING 61 6633c02.qxd 4/3/06 1:53 PM Page 61 else { Type t1 = f.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(f); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName); pi.SetValue(ctrl, newValue, null); are.Set(); } } delegate void InvokeMethodHandler(Form f, string methodName, params object[] parms); static void InvokeMethod(Form f, string methodName, params object[] parms) { if (f.InvokeRequired) { Delegate d = new InvokeMethodHandler(InvokeMethod); f.Invoke(d, new object[] {f, methodName, parms}); are.WaitOne(); } else { Type t = f.GetType(); MethodInfo mi = t.GetMethod(methodName, flags); mi.Invoke(f, parms); are.Set(); } } delegate object GetControlPropertyValueHandler(Form f, string controlName, string propertyName); static object GetControlPropertyValue(Form f, string controlName, string propertyName) { if (f.InvokeRequired) { Delegate d = new GetControlPropertyValueHandler(GetControlPropertyValue); object[] o = new object[] { f, controlName, propertyName }; object iResult = f.Invoke(d, o); are.WaitOne(); return iResult; } CHAPTER 2 ■ REFLECTION-BASED UI TESTING62 6633c02.qxd 4/3/06 1:53 PM Page 62 else { Type t1 = f.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(f); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName); object gResult = pi.GetValue(ctrl, null); are.Set(); return gResult; } } } // Class1 } // ns CHAPTER 2 ■ REFLECTION-BASED UI TESTING 63 6633c02.qxd 4/3/06 1:53 PM Page 63 6633c02.qxd 4/3/06 1:53 PM Page 64 Windows-Based UI Testing 3.0 Introduction This chapter describes how to test an application through its user interface (UI) using low- level Windows-based automation. These techniques involve calling Win32 API functions such as FindWindow() and sending Windows messages such as WM_LBUTTONUP to the application under test (AUT). Although these techniques have been available to developers and testers for many years, the .NET programming environment dramatically simplifies the process. Figure 3-1 demonstrates the kind of lightweight test automation you can quickly create. Figure 3-1. Windows-based UI test run 65 CHAPTER 3 ■ ■ ■ 6633c03.qxd 4/3/06 1:58 PM Page 65 The dummy AUT is a color-mixer application. The key code for the application is void button1_Click(object sender, System.EventArgs e) { string tb = textBox1.Text; string cb = comboBox1.Text; if (tb == "" || cb == "") MessageBox.Show("You need 2 colors", "Error"); else { if (tb == cb) listBox1.Items.Add("Result is " + tb); else if (tb == "red" && cb == "blue" || tb == "blue" && cb =="red") listBox1.Items.Add("Result is purple"); else listBox1.Items.Add("Result is black"); } } Notice that the application may generate an error message box. Dealing with low-level constructs such as message boxes and the main menu are tasks that can be handled well by Win32 functions. The fundamental idea is that every Windows-based control is a window. Each control/window has a handle that can be used to access, manipulate, and examine the control/window. The three key categories of tasks in lightweight, low-level Windows-based UI automation are • Finding a window/control handle • Manipulating a window/control • Examining a window/control Keeping this task-organization structure in mind will help you arrange your test automation. The code in this chapter is written in a traditional procedural style rather than in an object- oriented style. This is a matter of personal preference, and you may want to recast the techniques to an OOP (object-oriented programming) style. Additionally, you may want to modularize the code solutions further by combining them into a .NET class library. The test automation harness that produced the test run shown in Figure 3-1 is presented in Section 3.10. 3.1 Launching the AUT Problem You want to launch a Windows form-based application so you can test it through its UI. Design Use the System.Diagnostics.Process.Start() method. CHAPTER 3 ■ WINDOWS-BASED UI TESTING66 6633c03.qxd 4/3/06 1:58 PM Page 66 Solution static void Main(string[] args) { try { Console.WriteLine("\nLaunching application under test"); string path = "..\\..\\..\\WinApp\\bin\\Debug\\WinApp.exe"; Process p = Process.Start(path); if (p == null) Console.WriteLine("Warning: process may already exist"); // run UI test scenario here Console.WriteLine("\nDone"); } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); } } There are several ways to launch a Windows form application so that you can test it through its UI using Windows-based techniques. The simplest way is to use the Process.Start() static method located in the System.Diagnostics namespace. Comments The Process.Start() method has four overloads. The overload used in this solution accepts a path to the AUT and returns a Process object that represents the resources associated with the application. You need to be a bit careful with the Process.Start() return value. A return of null does not necessarily indicate failure; null is also returned if an existing process is reused. Regard- less, a return of null is not good because your UI test automation will often become confused if more than one target application is running. This idea is explained more fully in Section 3.2. If you need to pass arguments to the AUT, you can use the Process.Start() overload that accepts a second argument, which represents command-line arguments to the application. For example: Process p = null; p = Process.Start("SomeApp.exe", "C:\\Somewhere\\Somefile.txt"); if (p == null) Console.WriteLine("Warning: process may already exist"); The Process.Start() method also supports an overload that accepts a ProcessStartInfo object as an argument. A ProcessStartInfo object can direct the AUT to launch and run in a variety of ways; however, this technique is rarely needed in a lightweight test automation sce- nario. The Process.Start() method is asynchronous, so when you use it to launch the AUT, be careful about attempting to access the application through your test harness until after you are sure the application has launched. This problem is discussed and solved in Section 3.2. CHAPTER 3 ■ WINDOWS-BASED UI TESTING 67 6633c03.qxd 4/3/06 1:58 PM Page 67 3.2 Obtaining a Handle to the Main Window of the AUT Problem You want to obtain a handle to the application main window. Design Use the FindWindow() Win32 API function with the .NET platform invoke (P/Invoke) mechanism. Solution class Class1 { [DllImport("user32.dll", EntryPoint="FindWindow", CharSet=CharSet.Auto)] static extern IntPtr FindWindow(string lpClassName, string lpWindowName); [STAThread] static void Main(string[] args) { try { // launch AUT; see Section 3.1 IntPtr mwh = IntPtr.Zero; // main window handle bool formFound = false; int attempts = 0; while (!formFound && attempts < 25) { if (mwh == IntPtr.Zero) { Console.WriteLine("Form not yet found"); Thread.Sleep(100); ++attempts; mwh = FindWindow(null, "Form1"); } else { Console.WriteLine("Form has been found"); formFound = true; } } if (mwh == IntPtr.Zero) throw new Exception("Could not find main window"); CHAPTER 3 ■ WINDOWS-BASED UI TESTING68 6633c03.qxd 4/3/06 1:58 PM Page 68 Console.WriteLine("\nDone"); } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); } } } // Class1 To manipulate and examine the state of a Windows application, you must obtain a handle to the application’s main window. A window handle is a system-generated value that you can think of as being both an ID for the associated window and a way to access the window. Comments In a .NET environment, a window handle is type System.IntPtr, which is a platform-specific type used to represent either a pointer (memory address) or a handle. To obtain a handle to the main window of an AUT, you can call the Win32 API FindWindow() function. The FindWindow() function is essentially a part of the Windows operating system, which is available to you. Because FindWindow() is part of Windows, it is written in traditional C++ and not managed code. The C++ signature for FindWindow() is HWND FindWindow(LPCTSTR lpClassName, LPCTSTR lpWindowName); This function accepts a window class name and a window name as arguments, and it returns a handle to the window. To call into unmanaged code like the FindWindow() function from C#, you can use a .NET mechanism called platform invoke (P/Invoke). P/Invoke func- tionality is contained in the System.Runtime.InteropServices namespace. The mechanism is very elegant. In essence, you create a C# wrapper, or alias for the Win32 function you want to use, and then call that alias. You start by placing a using System.Runtime.InteropServices; statement in your test harness so you can easily access P/Invoke functionality. Next you determine a C# signature for the unmanaged function you want to call. This really involves deter- mining C# data types that map to the return type and parameter types of the unman- aged function. In the case of FindWindow(), the unmanaged return type is HWND, which is a Win32 data type representing a handle to a window. As explained earlier, the corresponding C# data type is System.IntPtr. The Win32 FindWindow() function accepts two parameters of type LPCTSTR. Although the details are fairly deep, this is basically a Win32 data type that can be represented by a C# type string. ■Note One of the greatest productivity-enhancing improvements that .NET introduced to application develop- ment is a vastly simplified data type model. To use the P/Invoke mechanism, you must determine the C# equiv- alents to Win32 data types. A detailed discussion of the mappings between Win32 data types and .NET data types is outside the scope of this book, but fortunately most mappings are fairly obvious. For example, the Win32 data types LPCSTR, LPCTSTR, LPCWSTR, LPSTR, LPTSTR, and LPWSTR usually map to the C# string data type. CHAPTER 3 ■ WINDOWS-BASED UI TESTING 69 6633c03.qxd 4/3/06 1:58 PM Page 69 After determining the C# alias method signature, you can place a class-scope DllImport attribute with the C# method signature that corresponds to the Win32 function signature into your test harness: [DllImport("user32.dll", EntryPoint="FindWindow", CharSet=CharSet.Auto)] static extern IntPtr FindWindow(string lpClassName, string lpWindowName); The “user32.dll” argument specifies the DLL file where the unmanaged function you want to use is located. Because the DllImport attribute is expecting a DLL, the .dll extension is optional; however, including it makes your code more readable. The EntryPoint attribute specifies the name of the Win32 API function that you will be calling through the C# alias. If the C# method name is exactly the same as the Win32 function name, you may omit the EntryPoint argument. But again, putting the argument in the attribute makes your code easier to read and maintain. The CharSet argument is optional but should be used whenever the C# method alias has a return type or one or more parameters that are type char or string. Speci- fying CharSet.Auto essentially means to let the .NET Framework take care of all character type conversions, for example, ASCII to Unicode. The CharSet.Auto argument dramatically simpli- fies working with type char and type string. When you code the C# method alias for a Win32 function, you almost always use the static and extern modifiers because most Win32 functions are static functions rather than instance functions in C# terminology, and Win32 functions are external to your test harness. You may name the C# method anything you like but keeping the C# method name the same as the Win32 function name is the most readable approach. Similarly, you can name the C# parameters anything you like, but again, a good strategy is to make C# parameter names the same as their Win32 counterparts. With the P/Invoke plumbing in place, if a subtle timing issue did not exist, you could now get the handle to the main window of the AUT like this: IntPtr mwh = FindWindow(null, "Form1"); Before explaining the timing issue, let’s look at the method call. The second argument to FindWindow() is the window name. In help documentation, this value is sometimes called the window title or the window caption. In the case of a Windows form application, this will usually be the form name. The first argument to FindWindow() is the window class name. A window class name is a system-generated string that is used to register the window with the operating system. Note that the term “class name” in this context is an old pre-OOP term and is not at all related to the idea of a C# language class container structure. Window/control class names are not unique, so they have little value when trying to find a window/control. In this example, if you pass null as the window class name when calling FindWindow(), FindWindow() will return the handle of the first instance of a window with name "Form1". This means you should be very careful about having multiple AUTs active, because you may get the wrong window handle. If you attempt to obtain the application main window handle in the simple way just described, you are likely to run into a timing issue. The problem is that your AUT may not be fully launched and registered. A poor way to deal with this problem is to place Thread.Sleep() calls with large delays into your test harness to give the application time to launch. A better CHAPTER 3 ■ WINDOWS-BASED UI TESTING70 6633c03.qxd 4/3/06 1:58 PM Page 70 way to deal with this issue is to wrap the call to FindWindow() in a while loop with a small delay, checking to see if you get a valid window handle: IntPtr mwh = IntPtr.Zero; // main window handle bool formFound = false; while (!formFound) { if (mwh == IntPtr.Zero) { Console.WriteLine("Form not yet found"); Thread.Sleep(100); mwh = FindWindow(null, "Form1"); } else { Console.WriteLine("Form has been found"); formFound = true; } } You use a Boolean flag to control the while loop. If the value of the main window handle is IntPtr.Zero, then you delay the test automation by 100 milliseconds (one-tenth of a second) using the Thread.Sleep() method from the System.Threading namespace. This approach could lead to an infinite loop if the main window handle is never found, so in practice you will often want to add a counter to limit the maximum number of times you iterate through the loop: IntPtr mwh = IntPtr.Zero; // main window handle bool formFound = false; int attempts = 0; while (!formFound && attempts < 25) { if (mwh == IntPtr.Zero) { Console.WriteLine("Form not yet found"); Thread.Sleep(100); ++attempts; mwh = FindWindow(null, "Form1"); } else { Console.WriteLine("Form has been found"); formFound = true; } } if (mwh == IntPtr.Zero) throw new Exception("Could not find Main Window"); CHAPTER 3 ■ WINDOWS-BASED UI TESTING 71 6633c03.qxd 4/3/06 1:58 PM Page 71 If the value of the main window handle variable is still IntPtr.Zero after the while loop terminates, you know that the handle was never found, and you should abort the test run by throwing an exception. You can increase the modularity of your lightweight test harness by wrapping the code in this solution in a helper method. For example, if you write static IntPtr FindMainWindowHandle(string caption) { IntPtr mwh = IntPtr.Zero; bool formFound = false; int attempts = 0; do { mwh = FindWindow(null, caption); if (mwh == IntPtr.Zero) { Console.WriteLine("Form not yet found"); Thread.Sleep(100); ++attempts; } else { Console.WriteLine("Form has been found"); formFound = true; } } while (!formFound && attempts < 25); if (mwh != IntPtr.Zero) return mwh; else throw new Exception("Could not find Main Window"); } // FindMainWindowHandle() then you can make a clean call in your harness Main() method like this: Console.WriteLine("Finding main window handle"); IntPtr mwh = FindMainWindowHandle("Form1"); Console.WriteLine("Handle to main window is " + mwh); Depending on the complexity of your AUT, you may want to parameterize the delay time and the maximum number of attempts, leading to a helper signature such as static IntPtr FindMainWindowHandle(string caption, int delay, int maxTries) which can be called like this: CHAPTER 3 ■ WINDOWS-BASED UI TESTING72 6633c03.qxd 4/3/06 1:58 PM Page 72 Console.WriteLine("Finding main window handle"); int delay = 100; int maxTries = 25; IntPtr mwh = FindMainWindowHandle("Form1", delay, maxTries); Console.WriteLine("Handle to main window is " + mwh); 3.3 Obtaining a Handle to a Named Control Problem You want to obtain a handle to a control/window that has a window name. Design Use the FindWindowEx() Win32 API function with the .NET P/Invoke mechanism. Solution IntPtr mwh = IntPtr.Zero; // main window handle // obtain main window handle here; see Section 3.2 Console.WriteLine("Finding handle to textBox1"); IntPtr tb = FindWindowEx(mwh, IntPtr.Zero, null, ""); if (tb == IntPtr.Zero) throw new Exception("Unable to find textBox1"); else Console.WriteLine("Handle to textBox1 is " + tb); Console.WriteLine("Finding handle to button1"); IntPtr butt = FindWindowEx(mwh, IntPtr.Zero, null, "button1"); if (butt == IntPtr.Zero) throw new Exception("Unable to find button1"); else Console.WriteLine("Handle to button1 is " + butt); where a class-scope DllImport attribute is [DllImport("user32.dll", EntryPoint="FindWindowEx", CharSet=CharSet.Auto)] static extern IntPtr FindWindowEx(IntPtr hwndParent, IntPtr hwndChildAfter, string lpszClass, string lpszWindow); To access and manipulate a control on a form-based application, you must obtain a han- dle to the control. In a Windows environment, all GUI controls are themselves windows. So, a button control is a window, a textbox control is a window, and so forth. To get a handle to a control/window, you can use the FindWindowEx() Win32 API function. CHAPTER 3 ■ WINDOWS-BASED UI TESTING 73 6633c03.qxd 4/3/06 1:58 PM Page 73 Comments To call a Win32 function such as FindWindowEx() from a C# test harness, you can use the P/Invoke mechanism as described in Section 3.2. The Win32 FindWindowEx() function has this C++ signature: HWND FindWindowEx(HWND hwndParent, HWND hwndChildAfter, LPCTSTR lpszClass, LPCTSTR lpszWindow); The FindWindowEx() function accepts four arguments. The first argument is a handle to the parent window of the control you are seeking. The second argument is a handle to a control and directs FindWindowEx() where to begin searching; the search begins with the next child control. The third argument is the class name of the target control, and the fourth argument is the window name/title/caption of the target control. As discussed in Section 3.2, the C# equivalent to the Win32 type HWND is IntPtr and the C# equivalent to type LPCTSTR is string. Because the Win32 FindWindowEx() function is located in file user32.dll, you can insert this class-scope attribute and C# alias into the test harness: [DllImport("user32.dll", EntryPoint="FindWindowEx", CharSet=CharSet.Auto)] static extern IntPtr FindWindowEx(IntPtr hwndParent, IntPtr hwndChildAfter, string lpszClass, string lpszWindow); Notice that the C# alias method signature uses the same function name and same param- eter names as the Win32 function for code readability. With this P/Invoke plumbing in place, you can obtain a handle to a named control: // get main window handle in variable mwh; see Section 3.2 Console.WriteLine("Finding handle to textBox1"); IntPtr tb = FindWindowEx(mwh, IntPtr.Zero, null, ""); Console.WriteLine("Finding handle to button1"); IntPtr butt = FindWindowEx(mwh, IntPtr.Zero, null, "button1"); The first argument is the handle to the main window form that contains the target control. By specifying IntPtr.Zero as the second argument, you instruct FindWindowEx() to search all controls on the main form window. You ignore the target control class name by passing in null as the third argument. The fourth argument is the target control’s name/title/caption. You should not assume that a call to FindWindowEx() has succeeded. To check, you can test if the return handle has value IntPtr.Zero along the lines of if (tb == IntPtr.Zero) throw new Exception("Unable to find textBox1"); if (butt == IntPtr.Zero) throw new Exception("Unable to find button1"); So, just how do you determine a control name/title/caption? The simplest way is to use the Spy++ tool included with Visual Studio .NET. The Spy++ tool is indispensable for light- weight UI test automation. Figure 3-2 shows Spy++ after its window finder has been placed on the button1 control of the AUT shown in the foreground of Figure 3-1. CHAPTER 3 ■ WINDOWS-BASED UI TESTING74 6633c03.qxd 4/3/06 1:58 PM Page 74 Figure 3-2. The Spy++ tool In addition to a control’s caption, Spy++ provides other useful information such as the control’s class name, Windows events related to the control, and the control’s parent, child, and sibling controls. 3.4 Obtaining a Handle to a Non-Named Control Problem You want to obtain a handle to a control that does not have a window name. Design Write a FindWindowByIndex() helper method that finds the control by using its implied index value. CHAPTER 3 ■ WINDOWS-BASED UI TESTING 75 6633c03.qxd 4/3/06 1:58 PM Page 75 Solution static IntPtr FindWindowByIndex(IntPtr hwndParent, int index) { if (index == 0) return hwndParent; else { int ct = 0; IntPtr result = IntPtr.Zero; do { result = FindWindowEx(hwndParent, result, null, null); if (result != IntPtr.Zero) ++ct; } while (ct < index && result != IntPtr.Zero); return result; } } and then call like this: Console.WriteLine("Finding handle to listBox1"); IntPtr lb = FindWindowByIndex(mwh, 3); if (lb == IntPtr.Zero) throw new Exception("Unable to find listBox1"); else Console.WriteLine("Handle to listBox1 is " + lb); To access and manipulate a control on a form-based application, you must obtain a han- dle to the control. If the target control has a unique window name, then you can obtain its handle using the techniques in Section 3.3. But a control may not have a name/caption/title. Examples include empty textbox controls and empty listbox controls. Furthermore, controls may have nonunique names. To deal with these situations, you can write a helper method that uses the Win32 FindWindowEx() function to return a control handle based on the control’s order index value. Comments The index value of a control is implied rather than explicit. The idea is that each control on a form has a predecessor and a successor control (except for the first control, which has no predecessor, and the last control, which has no successor). This predecessor-successor rela- tionship can be used to find window handles. Before examining this control index order concept further, let’s imagine that we know the index value of a control and see how the FindWindowByIndex() helper method works to return the control handle. Suppose, for example, that an application has a listbox control, and the index of the control is 3. This means that index 0 represents the main form window, and indexes 1 and 2 represent predecessor controls to the listbox control. The FindWindowByIndex() helper CHAPTER 3 ■ WINDOWS-BASED UI TESTING76 6633c03.qxd 4/3/06 1:58 PM Page 76 method accepts two arguments. The first argument is a handle to the parent control, and the second is a control index. If the index argument is 0, the FindWindowByIndex() method immedi- ately returns the handle to the parent control. This design choice is arbitrary. The heart of the helper method is a call to FindWindwEx() inside a loop: int ct = 0; do { result = FindWindowEx(hwndParent, result, null, null); if (result != IntPtr.Zero) ++ct; } while (ct < index && result != IntPtr.Zero); Each call to FindWindowEx() returns a handle to the next available control because you pass in as arguments the current window handle, the result returned in the preceding iteration of the loop, null, and null again, as the first, second, third, and fourth arguments, respectively. As explained in Section 3.3, the second argument to FindWindowEx() directs the method where to begin searching, and passing null as the third and fourth arguments means to find the first available window/control regardless of class name or window name. If this loop executes n times, variable result will hold the handle of the nth window/control, or IntPtr.Zero if the control could not be found. So, if you know the index value of a control, you can get the control handle using the FindWindowByIndex() helper method. But just how do you determine a control’s implied index value? There are two simple ways to get this index value. First, if you have access to the AUT source code, you can get a control index value because the value is the order in which the control is added to the main form control. For example, suppose the AUT code contains this.Controls.Add(this.comboBox1); this.Controls.Add(this.button1); this.Controls.Add(this.listBox1); this.Controls.Add(this.textBox1); The implied index of the comboBox1 control is 1, the index of button1 is 2, the index of listBox1 is 3, and the index of textBox1 is 4. Note that the implied index value of a control is not the same as the control tab order. Now if you do not have access to the source code of the AUT, you can still determine the index value of each control by examining the predecessors and successors of the controls with the Spy++ tool as described in Section 3.3. The FindWindowByIndex() helper method gives you a way to deal with controls with nonunique names. Suppose your AUT has two buttons with the same label: this.Controls.Add(this.button1); // window name is "Click me" this.Controls.Add(this.button2); // window name also "Click me" You can still obtain handles to each button control: IntPtr butt1 = FindWindowByIndex(mwh, 1); IntPtr butt2 = FindWindowByIndex(mwh, 2); CHAPTER 3 ■ WINDOWS-BASED UI TESTING 77 6633c03.qxd 4/3/06 1:58 PM Page 77 3.5 Sending Characters to a Control Problem You want to send characters to a text-based control. Design Use the Win32 SendMessage() function with a WM_CHAR notification message. Solution // launch app; see Section 3.1 // get main window handle; see Section 3.2 // get handle to textBox1 as tb; see Sections 3.3 and 3.4 Console.WriteLine("Sending 'x' to textBox1"); uint WM_CHAR = 0x0102; SendMessage1(tb, WM_CHAR, 'x', 0); Console.WriteLine("Now adding 'foo' to textBox1"); string s = "foo"; foreach (char c in s) { SendMessage1(tb, WM_CHAR, c, 0); } where the class-scope DllImport attribute is [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern void SendMessage1(IntPtr hWnd, uint Msg, int wParam, int lParam); A common lightweight UI test automation task is to simulate a user typing characters into a UI control. One way to do this is to use the Win32 SendMessage() function with the .NET P/Invoke mechanism. Comments The SendMessage() function has this C++ signature: LRESULT SendMessage(HWND hWnd, UINT Msg, WPARAM wParam, LPARAM lParam); There are four parameters. The first parameter is a handle to the window/control that you are sending a Windows message to. The second parameter is the Windows message to send to the control. The third and fourth parameters are generic and their meaning and data type depend upon the Windows message. Similarly, the meaning and type of the return value for SendMessage() depend upon the message being sent. So, before you can create a C# signature CHAPTER 3 ■ WINDOWS-BASED UI TESTING78 6633c03.qxd 4/3/06 1:58 PM Page 78 alias for the C++ SendMessage() function, you need to examine the particular Windows mes- sage you will be sending. In this case, you want to send a WM_CHAR message. The WM_CHAR message is sent to the control that has keyboard focus when a key is pressed. WM_CHAR is actu- ally a Windows symbolic constant defined as 0x0102. If you look up “WM_CHAR” in the integrated Visual Studio .NET Help, you will find that wParam parameter specifies the character code of the key pressed. The lParam parameter specifies various key-state masks such as the repeat count, scan code, extended-key flag, context code, previous key-state flag, and transi- tion-state flag values. So, with this information in hand, you can create a C# signature like: [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern void SendMessage1(IntPtr hWnd, uint Msg, int wParam, int lParam); You use a C# method alias name of SendMessage1() rather than SendMessage() because there will be several different C# signatures depending on the particular Windows message passed to the SendMessage() function. As explained in Section 3.2, a C# IntPtr type corre- sponds to a C++ HWND type. All Windows messages are type uint, and the WM_CHAR message requires two int parameters for the scan code of the key pressed and a value for the key-state mask. With this code in place, you can send a character to a control like this: Console.WriteLine("Finding handle to textBox1"); IntPtr tb = FindWindowEx(mwh, IntPtr.Zero, null, ""); Console.WriteLine("Sending 'x' to textBox1"); uint WM_CHAR = 0x0102; SendMessage1(tb, WM_CHAR, 'x', 0); Notice that an implicit type conversion is occurring here. When you pass a character such as 'x' as the third argument to SendMessage(), the character will be implicitly converted to type int. You can increase the modularity of your test automation by wrapping the essential code into two helper methods: static void SendChar(IntPtr hControl, char c) { uint WM_CHAR = 0x0102; SendMessage1(hControl, WM_CHAR, c, 0); } static void SendChars(IntPtr hControl, string s) { foreach (char c in s) { SendChar(hControl, c); } } CHAPTER 3 ■ WINDOWS-BASED UI TESTING 79 6633c03.qxd 4/3/06 1:58 PM Page 79 Then you can make clean calls such as Console.WriteLine("Sending 'x' to textBox1"); SendChar(tb, 'x'); Console.WriteLine("Now adding 'foo' to textBox1"); SendChars(tb, "foo"); 3.6 Clicking on a Control Problem You want to automate a mouse click on a control. Design Use the Win32 PostMessage() function with WM_LBUTTONDOWN and WM_LBUTTONUP notification messages. Solution Console.WriteLine("Clicking on button1"); uint WM_LBUTTONDOWN = 0x0201; uint WM_LBUTTONUP = 0x0202; PostMessage1(butt, WM_LBUTTONDOWN, 0, 0); PostMessage1(butt, WM_LBUTTONUP, 0, 0); where a class-scope DllImport attribute is [DllImport("user32.dll", EntryPoint="PostMessage", CharSet=CharSet.Auto)] static extern bool PostMessage1(IntPtr hWnd, uint Msg, int wParam, int lParam); Comments A common lightweight UI test automation task is to simulate a user clicking on a UI control. One way to do this is to use the Win32 PostMessage() function with the .NET P/Invoke mecha- nism. The PostMessage() function has this C++ signature: BOOL PostMessage(HWND hWnd, UINT Msg, WPARAM wParam, LPARAM lParam); The PostMessage() function is closely related to the SendMessage() function described in Section 3.5. In lightweight test automation scenarios, you will use SendMessage() most often. The primary difference between SendMessage() and PostMessage() is that SendMessage() calls the specified procedure and does not return until after the procedure has processed the Win- dows message; PostMessage() returns without waiting for the message to be processed. In the CHAPTER 3 ■ WINDOWS-BASED UI TESTING80 6633c03.qxd 4/3/06 1:58 PM Page 80 case of a mouse button click action, you want control to return to the test automation harness without waiting for the thread to process the message. In practical terms, deciding whether to use SendMessage() or PostMessage() is difficult. You should consider the actions associated with the message you want to send; if the actions are very closely related and must happen more or less together, try PostMessage(), otherwise try SendMessage(). Regardless, you may have to experiment to get the desired effect. The PostMessage() function accepts four arguments: a handle to the window/control that you are posting a Windows message to, the Windows message to post to the control, and generic arguments whose data type and meaning depend upon the Windows message being posted. To create a C# signature alias for the C++ PostMessage() function, you need to exam- ine the particular Windows message you will be posting. In this instance, you want to post a WM_LBUTTONDOWN (mouse left button down) message followed by a WM_LBUTTONUP message. As with all Windows messages, WM_LBUTTONDOWN and WM_LBUTTONUP are symbolic constants, in this case 0x0201 and 0x0202, respectively. The wParam parameter for both messages indicates whether various virtual keys are down while the mouse button is being clicked. A value of 0 means no keys are down when the mouse button is clicked. The lParam parameter specifies where in the target control the mouse clicks at; a value of 0 means the upper-left corner of the control (the low byte is the x coordinate, and the high byte is the y coordinate). With this infor- mation in hand, you can create a C# signature such as [DllImport("user32.dll", EntryPoint="PostMessage", CharSet=CharSet.Auto)] static extern bool PostMessage1(IntPtr hWnd, uint Msg, int wParam, int lParam); Choose a C# method alias name of PostMessage1() rather than PostMessage() because there may be several different C# signatures depending on the particular Windows message passed to the PostMessage() function. As explained in Section 3.2, a C# IntPtr type corre- sponds to a C++ HWND type. After you have placed the P/Invoke plumbing in your lightweight test automation harness, you can simulate a user clicking on a control: // get button1 handle into variable 'butt'; see Section 3.3 Console.WriteLine("Clicking on button1"); uint WM_LBUTTONDOWN = 0x0201; uint WM_LBUTTONUP = 0x0202; PostMessage1(butt, WM_LBUTTONDOWN, 0, 0); PostMessage1(butt, WM_LBUTTONUP, 0, 0); Notice that you are ignoring the PostMessage1() return value. Win32 function return values are often used for error-checking, and you should take advantage of them. Somewhat surpris- ingly, however, Windows message return values are often not very helpful in lightweight UI test automation. The main reason for this is that the return values are generally intended for use by the message receiver; however, in a test automation situation, you are effectively the message sender. You can increase the modularity of your test automation by wrapping the essential control- click code into a helper method: CHAPTER 3 ■ WINDOWS-BASED UI TESTING 81 6633c03.qxd 4/3/06 1:58 PM Page 81 static void ClickOn(IntPtr hControl) { uint WM_LBUTTONDOWN = 0x0201; uint WM_LBUTTONUP = 0x0202; PostMessage1(hControl, WM_LBUTTONDOWN, 0, 0); // button down PostMessage1(hControl, WM_LBUTTONUP, 0, 0); // button up } Then you can make calls such as // get button1 handle into variable 'butt'; see Section 3.3 Console.WriteLine("Clicking on button1"); ClickOn(butt); In addition to using this solution to click on button controls, you can use the technique to give focus to a control in situations where a real user might do so, so that you can send charac- ters to the control. For example: // get comboBox1 handle into variable 'cb'; see Section 3.4 Console.WriteLine("Clicking on comboBox1 to set keyboard focus"); ClickOn(cb); Thread.Sleep(500); Console. WriteLine("Now sending 'foo' to comboBox1"); SendChars(cb, "foo"); 3.7 Dealing with Message Boxes Problem You want to deal with a message box, such as clicking it away. Design Treat the message box as a top-level window/control rather than as a child control, and use the Win32 FindWindow() function with the P/Invoke mechanism. Solution Console.WriteLine("Clicking button1"); ClickOn(butt); // see Section 3.6 // generates a message box with title "Error" and button "OK" CHAPTER 3 ■ WINDOWS-BASED UI TESTING82 6633c03.qxd 4/3/06 1:58 PM Page 82 Console.WriteLine("\nLooking for Message Box"); IntPtr mb = IntPtr.Zero; bool mbFound = false; int tries = 0; while (!mbFound && tries < 25) { mb = FindWindow(null, "Error"); ++tries; if (mb == IntPtr.Zero) { Console.WriteLine("Message Box window not yet found"); Thread.Sleep(100); } else { Console.WriteLine("Message Box found; handle = " + mb); mbFound = true; } } // while Console.WriteLine("Clicking away Message Box in 2.5 seconds"); Thread.Sleep(2500); IntPtr okButt = FindWindowEx(mb, IntPtr.Zero, null, "OK"); ClickOn(okButt); The key to dealing with message box windows is to realize that message box controls are not child controls of the application form. Message box controls are top-level windows, so you treat them just as you would the main application form. You can think of a message box as a tiny subprogram that runs independently from the AUT. So, first you get a handle to the mes- sage box, and then you get a handle to the “OK” button (or other control on the message box), which you can then manipulate. Comments The heart of the technique to obtain a handle to a message box is to call the Win32 API FindWindow() function using the .NET P/Invoke mechanism. This technique is described in detail in Section 3.2. To summarize, you create a C# alias for the Win32 FindWindow() function using a class-scope DllImport attribute. The C# FindWindow() method alias accepts the target message box class name (which is rarely useful), accepts the window name/title/caption, and returns a handle to the message box: IntPtr mb = IntPtr.Zero; mb = FindWindow(null, "Error"); After you obtain a handle to a message box, if you want to simulate a user clicking the box away as is usually the case, you obtain a handle to the OK button using the FindWindowEx() function as described in Section 3.3: CHAPTER 3 ■ WINDOWS-BASED UI TESTING 83 6633c03.qxd 4/3/06 1:58 PM Page 83 IntPtr okButt = FindWindowEx(mb, IntPtr.Zero, null, "OK"); The FindWindowEx() method accepts a handle to the parent control (the message box), a handle to the window/control to begin searching at (a value of IntPtr.Zero means search all child controls), the target control class name (rarely useful so we usually pass in null), and the target control name/title/caption. An alternative way to get the handle to the message box OK button is to use the FindWindowByIndex() helper method described in Section 3.4. For exam- ple, if the OK button is the only control on a message box, then you can get a handle to the button with IntPtr okButt = FindWindowByIndex(mb, 1); Because message box controls are top-level windows, a slight delay may occur before they are ready to be accessed. This is especially true if you are running your lightweight test automa- tion under a stress condition (reduced system resources). So, you can use the same idea as presented in Section 3.2—place the call to FindWindow() in a while loop with a slight delay, checking each time through the loop until the message box handle variable does not have a IntPtr.Zero value: Console.WriteLine("\nLooking for message box"); IntPtr mb = IntPtr.Zero; bool mbFound = false; int attempts = 0; while (!mbFound && attempts < 25) { mb = FindWindow(null, "Error"); ++attempts; if (mb == IntPtr.Zero) { Console.WriteLine("Message Box window not found yet . . . "); Thread.Sleep(100); } else { Console.WriteLine("Message Box window found with ptr = " + mb); mbFound = true; } } You can increase the modularity of this solution by wrapping the code up into a helper method: static IntPtr FindMessageBox(string caption) { IntPtr result = IntPtr.Zero; bool mbFound = false; int attempts = 0; CHAPTER 3 ■ WINDOWS-BASED UI TESTING84 6633c03.qxd 4/3/06 1:58 PM Page 84 do { result = FindWindow(null, caption); if (result == IntPtr.Zero) { Console.WriteLine("Message Box not yet found"); Thread.Sleep(100); ++attempts; } else { Console.WriteLine("Message Box has been found"); mbFound = true; } } while (!mbFound && attempts < 25); if (result != IntPtr.Zero) return result; else throw new Exception("Could not find Message Box"); } With this helper method, you can now make calls like this: Console.WriteLine("\nLooking for message box"); IntPtr mb = FindMessageBox("Error"); Console.WriteLine("Message box handle = " + mb); Except for some minor details such as the progress messages, this helper method is exactly the same as the helper method to find the main window handle presented in the discussion section of Section 3.2. Therefore, you could combine them both into a single helper: static IntPtr FindTopLevelWindow(string caption) { // code as shown previously } Then, make calls like this: // launch AUT here; see Section 3.1 Console.WriteLine("\nLooking for main window handle"); IntPtr mwh = FindTopLevelWindow ("Form1"); Console.WriteLine("Main window handle = " + mwh); ClickOn(butt); // see Section 3.6 Console.WriteLine("\nLooking for message box"); IntPtr mb = FindTopLevelWindow ("Error"); Console.WriteLine("Message box handle = " + mb); CHAPTER 3 ■ WINDOWS-BASED UI TESTING 85 6633c03.qxd 4/3/06 1:58 PM Page 85 This approach is more efficient than having two separate helper methods and makes technical sense, but the downside is that your code will be slightly harder to read and modify for specific message box or main window issues. A compromise approach is to write a single FindTopLevelWindow() method as shown previously and then write tiny wrapper methods such as static IntPtr FindMainWindowHandle(string caption, int delay, int maxTries) { return FindTopLevelWindow(caption, delay, maxTries); } static IntPtr FindMessageBox(string caption) { int delay = 100; int maxTries = 25; return FindTopLevelWindow(caption, delay, maxTries); } 3.8 Dealing with Menus Problem You want to simulate a user clicking on a main menu item such as Help ➤ About or File ➤ Exit. Design Use the Win32 API functions GetMenu(), GetSubMenu(), GetMenuItemID(), and SendMessage() with a WM_COMMAND Windows message. Solution // Launch AUT; See Section 3.1 // Get main window handle into mwh; See Section 3.2 IntPtr hMainMenu = GetMenu(mwh); Console.WriteLine("Handle to main menu is " + hMainMenu); IntPtr hHelp = GetSubMenu(hMainMenu, 2); Console.WriteLine("\nHandle to Help is " + hHelp); int iAbout = GetMenuItemID(hHelp, 0); Console.WriteLine("\nIndex to About is " + iAbout); uint WM_COMMAND = 0x0111; SendMessage2(mwh, WM_COMMAND, iAbout, IntPtr.Zero); CHAPTER 3 ■ WINDOWS-BASED UI TESTING86 6633c03.qxd 4/3/06 1:58 PM Page 86 where class-scope attributes are // Menu routines [DllImport("user32.dll")] static extern IntPtr GetMenu(IntPtr hWnd); [DllImport("user32.dll")] static extern IntPtr GetSubMenu(IntPtr hMenu, int nPos); [DllImport("user32.dll")] static extern int GetMenuItemID(IntPtr hMenu, int nPos); [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern void SendMessage2(IntPtr hWnd, uint Msg, int wParam, IntPtr lParam); Comments Suppose your AUT has a main menu structured like this: File Edit Help New Cut About Save Copy Update Print Paste Exit The preceding solution would simulate a user clicking on Help ➤ About. The GetMenu() function returns a handle to the main application menu. The GetSubMenu() function accepts a parent menu handle and a 0-based submenu index, and returns a handle to a submenu. In this example, if variable hMainMenu holds a handle to the main menu, then the call GetSubMenu(hMainMenu, 2) would return a handle to the Help part of the main menu, GetSubMenu(hMainMenu, 0) would return a handle to the File part of the main menu, and so on. After you have the submenu handle, the next step is to get an index value of the item you want to manipulate using the GetMenuItemID() function. If hHelp holds a handle to the Help part of the main menu, the call GetMenuItemID(hHelp, 0) returns the index of the About part of the submenu and GetMenuItemID(hHelp, 1) returns the index of the Update part of the submenu. After obtaining an index to the menu item you want to manipulate, the last step to simulate clicking on the menu item is to call the SendMessage() Win32 API function using a WM_COMMAND message. If variable iAbout holds the index of the About item in the Help submenu, then the statements uint WM_COMMAND = 0x0111; SendMessage2(mwh, WM_COMMAND, iAbout, IntPtr.Zero); will simulate a user clicking on Help ➤ About. CHAPTER 3 ■ WINDOWS-BASED UI TESTING 87 6633c03.qxd 4/3/06 1:58 PM Page 87 A common task in lightweight UI test automation scenarios is to simulate a user perform- ing a File ➤ Exit. If File is the first menu item, and Exit is the first submenu item under File, then the pattern is Console.WriteLine("\nDoing a File -> Exit in 2.5 seconds"); Thread.Sleep(2500); IntPtr hMainMenu = GetMenu(mwh); IntPtr hFile = GetSubMenu(hMainMenu, 0); int iExit = GetMenuItemID(hFile, 0); uint WM_COMMAND = 0x0111; SendMessage2(mwh, WM_COMMAND, iExit, IntPtr.Zero); The WM_COMMAND message is sent when the user selects a command item from a menu. Like all Windows messages, it is really just a constant; in this case, the value is 0x0111. Notice we are using a C# alias named SendMessage2(). As discussed in Section 3.2, the C# alias signatures for the Win32 SendMessage() and the PostMessage() functions depend on the message being sent/posted. In the case of WM_COMMAND, the wParam parameter represents two values. The high- order word of wParam specifies the notification code if the message comes from a control. If the message is from an accelerator, this value is 1; if the message is from a menu, this value is 0. The low-order word of wParam specifies the identifier of the menu item. In other words, the wParam parameter is type int. The lParam parameter specifies the handle to the control send- ing the message (if the message is from a control) or null if the message is not from a control (which is the case when notification comes from a user-initiated event). In other words, the lParam parameter is type IntPtr. Therefore, you create a DllImport attribute of [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern void SendMessage2(IntPtr hWnd, uint Msg, int wParam, IntPtr lParam); You could rely on .NET overloading and just use one common SendMessage() alias signature, but creating separate SendMessageX()-style alias signatures tends to be a bit more readable in general. The .NET alias signatures for GetMenu(), GetSubMenu(), and GetMenuItemID() follow their Win32 counterparts. Because these functions are used in only one way, you do not give them a MethodNameX()-style alias, such as GetMenu1() for example. The pattern to simulate more complex menu structures follows the same general pattern as the preceding solution. For example, suppose you have a menu structure like this: File Edit Help HelpItem0 [0] HelpItem1 [1] HelpItem2 [2] HelpItem2SubItem0 [0] HelpItem2SubItem1 [1] HelpItem3 [3] To simulate a user selecting Help ➤ HelpItem2 ➤ HelpItem2SubItem1, you could write this code: CHAPTER 3 ■ WINDOWS-BASED UI TESTING88 6633c03.qxd 4/3/06 1:58 PM Page 88 IntPtr hMainMenu = GetMenu(mwh); Console.WriteLine("Handle to main menu is " + hMainMenu); IntPtr hHelp = GetSubMenu(hMainMenu, 2); Console.WriteLine("\nHandle to Help is " + hHelp); IntPtr hSub = GetSubMenu(hHelp, 2); Console.WriteLine("\nHandle to HelpItem2 is " + hSub); int iSub = GetMenuItemID(hSub, 1); Console.WriteLine("\nIndex to HelpItem2SubItem1 is " + iSub); uint WM_COMMAND = 0x0111; SendMessage2(mwh, WM_COMMAND, iSub, IntPtr.Zero); 3.9 Checking Application State Problem You want to check the contents of a control on the AUT. Design Use the WM_GETTEXT message with the SendMessage() Win32 API function. Solution // launch AUT; see Section 3.1 // get handle to textBox1 into variable tb; see Section 3.3 // manipulate app; see Sections 3.5 and 3.6 Console.WriteLine("\nChecking the contents of textBox1"); uint WM_GETTEXT = 0x000D; byte[] buffer = new byte[256]; string text = null; int numFetched = SendMessage3(tb, WM_GETTEXT, 256, buffer); text = System.Text.Encoding.Unicode.GetString(buffer); Console.WriteLine("Fetched " + numFetched + " chars"); Console.WriteLine("TextBox1 contains = " + text); where [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern int SendMessage3(IntPtr hWndControl, uint Msg, int wParam, byte[] lParam); CHAPTER 3 ■ WINDOWS-BASED UI TESTING 89 6633c03.qxd 4/3/06 1:58 PM Page 89 Comments To determine a pass or fail test result when performing lightweight UI test automation, you must be able to programmatically examine the AUT to determine if the result state is what is expected. For example, you may need to determine if an expected text string is in a textbox control. The WM_GETTEXT message used with the SendMessage() function is one of several ways to retrieve the text from a textbox control or a combobox control. As described in Section 3.2, the C# method signature for SendMessage() depends on the particular message being sent. In the case of WM_GETTEXT, the wParam parameter is the maximum number of characters to fetch from the control, so you can use the C# int data type. The lParam parameter is a pointer to a buffer array to receive the text fetched from the control, so you can use a C# byte array. This leads to the following class-scope DllImport attribute to create a C# alias for SendMessage() when used in conjunction with a WM_GETTEXT message: [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern int SendMessage3(IntPtr hWndControl, uint Msg, int wParam, byte[] lParam); With this P/Invoke plumbing in place, you can prepare the call to retrieve the text in a textbox control like this: uint WM_GETTEXT = 0x000D; byte[] buffer = new byte[256]; string text = null; The Windows message WM_GETTEXT is just a symbolic constant with value 0x000D. You declare a byte array to hold the text in the control we are examining. In this example, placing a hard- coded 256 value in the automation is a simple approach but assumes the text in the target control is no longer than 256 bytes (or 128 Unicode characters). If you need to determine the actual size of the text in the target control, you can use the WM_GETTEXTLENGTH message first, and then use that return value to allocate your buffer array. Now, if variable tb holds the handle to textBox1, you can make the call to SendMessage() like this: int numFetched = SendMessage3(tb, WM_GETTEXT, 256, buffer); text = System.Text.Encoding.Unicode.GetString(buffer); This technique works for simple text-based controls. For example, you can retrieve the text in a combobox control in the exact same way. More complex controls, such as listbox con- trols, require a different approach: Console.WriteLine("\nChecking contents of listBox1 for 'foo'"); uint LB_FINDSTRING = 0x018F; int result = SendMessage4(lb, LB_FINDSTRING, -1, "foo"); if (result >= 0) Console.WriteLine("Found 'foo'"); else Console.WriteLine("Did not find 'foo'"); where CHAPTER 3 ■ WINDOWS-BASED UI TESTING90 6633c03.qxd 4/3/06 1:58 PM Page 90 [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern int SendMessage4(IntPtr hWnd, uint Msg, int wParam, string lParam); The LB_FINDSTRING message can be used to determine if a particular string is in a listbox control. The wParam parameter specifies which item in the listbox to begin searching at, with a value of -1 indicating to search the entire control. The lParam parameter is the target string to search for. The return value is a 0-based index of the location of the target string in the listbox, or -1 if the target is not found. 3.10 Example Program: WindowsUITest This program combines several of the techniques from this chapter to create a lightweight test automation harness to test the Windows application shown in the foreground of Figure 3-1. The program is a test scenario with test inputs hard-coded into the harness, rather than using test case inputs read from an external file. The test automation first launches the color-mixer AUT. Then the test scenario clicks the button control to generate an error message box. Next the automation clicks the error message box away and then simulates a user typing “red” and “blue” to the application. The automation clicks the button control again and then examines the listbox control, looking for an expected “Result is purple” string. The complete light- weight test harness is listed in Listing 3-1. When run, the output will be as shown in Figure 3-1 in the introduction section of this chapter. Listing 3-1. Program WindowsUITest using System; using System.Diagnostics; using System.Runtime.InteropServices; using System.Threading; namespace WindowsUITest { class Class1 { [STAThread] static void Main(string[] args) { try { Console.WriteLine("\nLaunching application under test"); string path = "..\\..\\..\\WinApp\\bin\\Debug\\WinApp.exe"; Process p = Process.Start(path); CHAPTER 3 ■ WINDOWS-BASED UI TESTING 91 6633c03.qxd 4/3/06 1:58 PM Page 91 Console.WriteLine("\nFinding main window handle"); IntPtr mwh = FindMainWindowHandle("Form1", 100, 25); Console.WriteLine("Main window handle = " + mwh); Console.WriteLine("\nFinding handles to textBox1, comboBox1"); Console.WriteLine(" button1, listBox1"); // you may want to add delays here to make sure Form has rendered IntPtr tb = FindWindowEx(mwh, IntPtr.Zero, null, ""); IntPtr cb = FindWindowByIndex(mwh, 1); IntPtr butt = FindWindowEx(mwh, IntPtr.Zero, null, "button1"); IntPtr lb = FindWindowByIndex(mwh, 3); if (tb == IntPtr.Zero || cb == IntPtr.Zero || butt == IntPtr.Zero || lb == IntPtr.Zero) throw new Exception("Unable to find all controls"); else Console.WriteLine("All control handles found"); Console.WriteLine("\nClicking button1"); ClickOn(butt); Console.WriteLine("Clicking away Error message box"); Thread.Sleep(1000); IntPtr mb = FindMessageBox("Error"); if (mb == IntPtr.Zero) throw new Exception("Unable to find message box"); IntPtr okButt = FindWindowEx(mb, IntPtr.Zero, null, "OK"); if (okButt == IntPtr.Zero) throw new Exception("Unable to find OK button"); ClickOn(okButt); Console.WriteLine("Typing 'red' and 'blue' to application"); SendChars(tb, "red"); ClickOn(cb); SendChars(cb, "blue"); Console.WriteLine("Clicking on button1"); ClickOn(butt); Console.WriteLine("\nChecking listBox1 for 'purple'"); uint LB_FINDSTRING = 0x018F; int result = SendMessage4(lb, LB_FINDSTRING, -1, "Result is purple"); if (result >= 0) Console.WriteLine("\nTest scenario result = Pass"); else Console.WriteLine("\nTest scenario result = *FAIL*"); CHAPTER 3 ■ WINDOWS-BASED UI TESTING92 6633c03.qxd 4/3/06 1:58 PM Page 92 Console.WriteLine("\nExiting app in 3 seconds . . . "); Thread.Sleep(3000); IntPtr hMainMenu = GetMenu(mwh); IntPtr hFile = GetSubMenu(hMainMenu, 0); int iExit = GetMenuItemID(hFile, 0); uint WM_COMMAND = 0x0111; SendMessage2(mwh, WM_COMMAND, iExit, IntPtr.Zero); Console.WriteLine("\nDone"); Console.ReadLine(); } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); } } // Main() static IntPtr FindTopLevelWindow(string caption, int delay, int maxTries) { IntPtr mwh = IntPtr.Zero; bool formFound = false; int attempts = 0; do { mwh = FindWindow(null, caption); if (mwh == IntPtr.Zero) { Console.WriteLine("Form not yet found"); Thread.Sleep(delay); ++attempts; } else { Console.WriteLine("Form has been found"); formFound = true; } } while (!formFound && attempts < maxTries); if (mwh != IntPtr.Zero) return mwh; else throw new Exception("Could not find Main Window"); } // FindTopLevelWindow() CHAPTER 3 ■ WINDOWS-BASED UI TESTING 93 6633c03.qxd 4/3/06 1:58 PM Page 93 static IntPtr FindMainWindowHandle(string caption, int delay, int maxTries) { return FindTopLevelWindow(caption, delay, maxTries); } static IntPtr FindMessageBox(string caption) { int delay = 100; int maxTries = 25; return FindTopLevelWindow(caption, delay, maxTries); } static IntPtr FindWindowByIndex(IntPtr hwndParent, int index) { if (index == 0) return hwndParent; else { int ct = 0; IntPtr result = IntPtr.Zero; do { result = FindWindowEx(hwndParent, result, null, null); if (result != IntPtr.Zero) ++ct; } while (ct < index && result != IntPtr.Zero); return result; } } // FindWindowByIndex() static void ClickOn(IntPtr hControl) { uint WM_LBUTTONDOWN = 0x0201; uint WM_LBUTTONUP = 0x0202; PostMessage1(hControl, WM_LBUTTONDOWN, 0, 0); PostMessage1(hControl, WM_LBUTTONUP, 0, 0); } static void SendChar(IntPtr hControl, char c) { uint WM_CHAR = 0x0102; SendMessage1(hControl, WM_CHAR, c, 0); } CHAPTER 3 ■ WINDOWS-BASED UI TESTING94 6633c03.qxd 4/3/06 1:58 PM Page 94 static void SendChars(IntPtr hControl, string s) { foreach (char c in s) { SendChar(hControl, c); } } // P/Invoke Aliases [DllImport("user32.dll", EntryPoint="FindWindow", CharSet=CharSet.Auto)] static extern IntPtr FindWindow(string lpClassName, string lpWindowName); [DllImport("user32.dll", EntryPoint="FindWindowEx", CharSet=CharSet.Auto)] static extern IntPtr FindWindowEx(IntPtr hwndParent, IntPtr hwndChildAfter, string lpszClass, string lpszWindow); // for WM_CHAR message [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern void SendMessage1(IntPtr hWnd, uint Msg, int wParam, int lParam); // for WM_COMMAND message [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern void SendMessage2(IntPtr hWnd, uint Msg, int wParam, IntPtr lParam); // for WM_LBUTTONDOWN and WM_LBUTTONUP messages [DllImport("user32.dll", EntryPoint="PostMessage", CharSet=CharSet.Auto)] static extern bool PostMessage1(IntPtr hWnd, uint Msg, int wParam, int lParam); // for WM_GETTEXT message [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern int SendMessage3(IntPtr hWndControl, uint Msg, int wParam, byte[] lParam); CHAPTER 3 ■ WINDOWS-BASED UI TESTING 95 6633c03.qxd 4/3/06 1:58 PM Page 95 // for LB_FINDSTRING message [DllImport("user32.dll", EntryPoint="SendMessage", CharSet=CharSet.Auto)] static extern int SendMessage4(IntPtr hWnd, uint Msg, int wParam, string lParam); // Menu routines [DllImport("user32.dll")] // static extern IntPtr GetMenu(IntPtr hWnd); [DllImport("user32.dll")] // static extern IntPtr GetSubMenu(IntPtr hMenu, int nPos); [DllImport("user32.dll")] // static extern int GetMenuItemID(IntPtr hMenu, int nPos); } // class } // ns CHAPTER 3 ■ WINDOWS-BASED UI TESTING96 6633c03.qxd 4/3/06 1:58 PM Page 96 Test Harness Design Patterns 4.0 Introduction One of the advantages of writing lightweight test automation instead of using a third-party testing framework is that you have great flexibility in how you can structure your test har- nesses. A practical way to classify test harness design patterns is to consider the type of test case data storage and the type of test-run processing. The three fundamental types of test case data storage are flat, hierarchical, and relational. For example, a plain-text file is usually flat storage; an XML file is typically hierarchical; and SQL data is often relational. The two funda- mental types of test-run processing are streaming and buffered. Streaming processing involves processing one test case at a time; buffered processing processes a collection of test cases at a time. This categorization leads to six fundamental test harness design patterns: • Flat test case data, streaming processing model • Flat test case data, buffered processing model • Hierarchical test case data, streaming processing model • Hierarchical test case data, buffered processing model • Relational test case data, streaming processing model • Relational test case data, buffered processing model Of course, there are many other ways to categorize, but thinking about test harness design in this way has proven to be effective in practice. Now, suppose you are developing a poker game application as shown in Figure 4-1. 97 CHAPTER 4 ■ ■ ■ 6633c04.qxd 4/3/06 1:56 PM Page 97 Figure 4-1. Poker Game AUT Let’s assume that the poker application references a PokerLib.dll library that houses classes to create and manipulate various poker objects. In particular, a Hand() constructor accepts a string argument such as “Ah Kh Qh Jh Th” (ace of hearts through ten of hearts), and a Hand.GetHandType() method returns an enumerated type with a string representation such as “RoyalFlush”. As described in Chapter 1, you need to thoroughly test the methods in the PokerLib.dll library. This chapter demonstrates how to test the poker library using each of the six fundamental test harness design patterns and explains the advantages and disadvantages of each pattern. For example, Section 4.3 uses this hierarchical test case data: CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS98 6633c04.qxd 4/3/06 1:56 PM Page 98 Ac Ad Ah As Tc FourOfAKindAces 4s 5s 6s 7s 3s StraightSevenHigh 5d 5c Qh 5s Qd FullHouseFivesOverQueens and uses a streaming processing model to produce this result: Pass 4s 5s 6s 7s 3s StraightSevenHigh StraightFlushSevenHigh *FAIL* Pass Although the techniques in this chapter demonstrate the six fundamental design patterns by testing a .NET class library, the patterns are general and apply to testing any type of software component. The streaming processing model, expressed in pseudo-code, is loop read a single test case from external store parse test case data into input(s) and expected(s) call component under test determine test case result save test case result to external store end loop CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 99 6633c04.qxd 4/3/06 1:56 PM Page 99 The buffered processing model, expressed in pseudo-code, is loop // 1. read all test cases read a single test case from external store into memory end loop loop // 2. run all test cases read a single test case from in-memory store parse test case data into input(s) and expected(s) call component under test determine test case result store test case result to in-memory store end loop loop // 3. save all results save test case result from in-memory store to external store end loop The streaming processing model is simpler than the buffered model, so it is often your best choice. However, in two common scenarios, you should consider using the buffered processing model. First, if the aspect in the system under test (SUT) involves file input/out- put, you often want to minimize test harness file operations. This is especially true if you are monitoring performance. Second, if you need to perform any preprocessing of your test case input (for example, pulling in and filtering test case data from more than one data store) or postprocessing of your test case results (for example, aggregating various test case category results), it’s almost always more convenient to have data in memory where you can process it. 4.1 Creating a Text File Data, Streaming Model Test Harness Problem You want to create a test harness that uses text file test case data and a streaming processing model. Design In one continuous processing loop, use a StreamReader object to read a test case data into memory, then parse the test case data into input and expected values using the String.Split() method, and call the component under test (CUT). Next, check the actual result with the expected result to determine a test case pass or fail. Then, write the results to external storage with a StreamWriter object. Do this for each test case. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS100 6633c04.qxd 4/3/06 1:56 PM Page 100 Solution Begin by creating a tagged and end-of-file delimited test case file: [id]=0001 [input]=Ac Ad Ah As Tc [expected]=FourOfAKindAces [id]=0002 [input]=4s 5s 6s 7s 3s [expected]=StraightSevenHigh [id]=0003 [input]=5d 5c Qh 5s Qd [expected]=FullHouseFivesOverQueens * Then process using StreamReader and StreamWriter objects: Console.WriteLine("\nBegin Text File Streaming model test run\n"); FileStream ifs = new FileStream("..\\..\\..\\TestCases.txt", FileMode.Open); StreamReader sr = new StreamReader(ifs); FileStream ofs = new FileStream("TextFileStreamingResults.txt", FileMode.Create); StreamWriter sw = new StreamWriter(ofs); string id, input, expected, blank, actual; while (sr.Peek() != '*') { id = sr.ReadLine().Split('=')[1]; input = sr.ReadLine().Split('=')[1]; expected = sr.ReadLine().Split('=')[1]; blank = sr.ReadLine(); string[] cards = input.Split(' '); Hand h = new Hand(cards[0], cards[1], cards[2], cards[3], cards[4]); actual = h.GetHandType().ToString(); sw.WriteLine("===================="); sw.WriteLine("ID = " + id); sw.WriteLine("Input = " + input); sw.WriteLine("Expected = " + expected); sw.WriteLine("Actual = " + actual); CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 101 6633c04.qxd 4/3/06 1:56 PM Page 101 if (actual == expected) sw.WriteLine("Pass"); else sw.WriteLine("*FAIL*"); } sw.WriteLine("===================="); sr.Close(); ifs.Close(); sw.Close(); ofs.Close(); Console.WriteLine("\nDone"); Comments You begin by creating a test case data file. As shown in the techniques in Chapter 1, you could structure the file with each test case on one line: 0001:Ac Ad Ah As Tc:FourOfAKindAces 0002:4s 5s 6s 7s 3s:StraightSevenHigh:deliberate error 0003:5d 5c Qh 5s Qd:FullHouseFivesOverQueens When using this approach, notice that the meaning of each part of the test case data is implied (the first item is the case ID, the second is the input, and the third is the expected result). A more flexible solution is to provide some structure to your test case data by adding tags such as "[id]" and "[input]". This allows you to easily perform rudimentary validity checks. For example: string temp = sr.ReadLine(); // should be the ID if (temp.StartsWith("[id]")) id = temp.Split('=')[1]; else throw new Exception("Invalid test case line"); You can perform validity checks on your test case data via a separate program that you run before you run the test harness, or you can perform validity checks inside the test harness itself. In addition to validity checks, structure tags also allow you to deal with test case data that has a variable number of inputs. This technique assumes that you have added a project reference to the PokerLib.dll library under test and that you have supplied appropriate using statements so you don’t have to fully qualify classes and objects: using System; using PokerLib; using System.IO; You should also always wrap your test harness code in try-catch-finally blocks: CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS102 6633c04.qxd 4/3/06 1:56 PM Page 102 static void Main(string[] args) { // Open any files here try { // main harness code here } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); } finally { // Close any open streams here } } // Main() When the code in this section is run with the preceding test case input data, the output is ==================== ID = 0001 Input = Ac Ad Ah As Tc Expected = FourOfAKindAces Actual = FourOfAKindAces Pass ==================== ID = 0002 Input = 4s 5s 6s 7s 3s Expected = StraightSevenHigh Actual = StraightFlushSevenHigh *FAIL* ==================== ID = 0003 Input = 5d 5c Qh 5s Qd Expected = FullHouseFivesOverQueens Actual = FullHouseFivesOverQueens Pass ==================== Test case #0002 is an intentional failure. Using a special character token in the test case data file to signal end-of-file is an old but effective technique. With such a token in place, you can use the StreamReader.Peek() method to check the next input character without actually consuming it from the associated stream. To create meaningful test cases, you must understand how the SUT works. This can be dif- ficult. Techniques to discover information about the SUT are discussed in Section 4.8. This solution represents a minimal test harness. You can extend the harness, for example, by adding CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 103 6633c04.qxd 4/3/06 1:56 PM Page 103 summary counters of the number of test cases that pass and the number that fail by using the techniques in Chapter 1. 4.2 Creating a Text File Data, Buffered Model Test Harness Problem You want to create a test harness that uses text file test case data and a buffered processing model. Design Read all test case data into an ArrayList collection that holds lightweight TestCase objects. Then iterate through the test cases ArrayList object, executing each test case and storing the results into a second ArrayList object that holds lightweight TestCaseResult objects. Finally, iterate through the results ArrayList object, saving the results to an external text file. Solution Begin by creating lightweight TestCase and TestCaseResult classes: class TestCase { public string id; public string input; public string expected; public TestCase(string id, string input, string expected) { this.id = id; this.input = input; this.expected = expected; } } // class TestCase class TestCaseResult { public string id; public string input; public string expected; public string actual; public string result; CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS104 6633c04.qxd 4/3/06 1:56 PM Page 104 public TestCaseResult(string id, string input, string expected, string actual, string result) { this.id = id; this.input = input; this.expected = expected; this.actual = actual; this.result = result; } } // class TestCaseResult Notice these class definitions use public data fields for simplicity. A reasonable alternative is to use a C# struct type instead of a class type. The data fields for the TestCase class should match the test case input data. The data fields for the TestCaseResult class should generally contain most of the fields in the TestCase class, the fields for the actual result of calling the CUT, and the test case pass or fail result. Because of this, a design option for you to consider is plac- ing a reference to a TestCase object in the definition of the TestCaseResult class. For example: class TestCaseResult { public TestCase tc; public string actual; public string result; public TestCaseResult(TestCase tc, string actual, string result) { this.tc = tc; this.actual = actual; this.result = result; } } // class TestCaseResult You may also want to include fields for the date and time when the test case was run. You process the test case data using three loop control structures and two ArrayList objects like this: Console.WriteLine("\nBegin Text File Buffered model test run\n"); FileStream ifs = new FileStream("..\\..\\..\\TestCases.txt", FileMode.Open); StreamReader sr = new StreamReader(ifs); FileStream ofs = new FileStream("TextFileBufferedResults.txt", FileMode.Create); StreamWriter sw = new StreamWriter(ofs); string id, input, expected = "", blank, actual; TestCase tc = null; TestCaseResult r = null; CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 105 6633c04.qxd 4/3/06 1:56 PM Page 105 // 1. read all test case data into memory ArrayList tcd = new ArrayList(); // test case data while (sr.Peek() != '*') { id = sr.ReadLine().Split('=')[1]; input = sr.ReadLine().Split('=')[1]; expected = sr.ReadLine().Split('=')[1]; blank = sr.ReadLine(); tc = new TestCase(id, input, expected); tcd.Add(tc); } sr.Close(); ifs.Close(); // 2. run all tests, store results to memory ArrayList tcr = new ArrayList(); // test case result for (int i = 0; i < tcd.Count; ++i) { tc = (TestCase)tcd[i]; string[] cards = tc.input.Split(' '); Hand h = new Hand(cards[0], cards[1], cards[2], cards[3], cards[4]); actual = h.GetHandType().ToString(); if (actual == tc.expected) r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "Pass"); else r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "*FAIL*"); tcr.Add(r); } // main processing loop // 3. emit all results to external storage for (int i = 0; i < tcr.Count; ++i) { r = (TestCaseResult)tcr[i]; sw.WriteLine("===================="); sw.WriteLine("ID = " + r.id); sw.WriteLine("Input = " + r.input); sw.WriteLine("Expected = " + r.expected); sw.WriteLine("Actual = " + r.actual); sw.WriteLine(r.result); } sw.WriteLine("===================="); sw.Close(); ofs.Close(); Console.WriteLine("\nDone"); CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS106 6633c04.qxd 4/3/06 1:56 PM Page 106 Comments The buffered processing model has three distinct phases. First, you read all test case data into memory. Although you can do this in many ways, experience has shown that your harness will be much easier to maintain if you create a very lightweight class for the test case data. Don’t get carried away and try to make a universal test case class that can accommodate any kind of test case input, however, because you’ll end up with a class that is so general it’s too awkward to use effectively. You have many choices of the kind of data structure to store your TestCase objects into. A System.Collections.ArrayList object is simple and effective. Because test case data is processed strictly sequentially in some situations, you may want to consider using a Stack or a Queue collection. In the second phase of the buffered processing model, you iterate through each test case in the ArrayList object that holds TestCase objects. After retrieving the current TestCase object, you execute the test and determine a result. Then you instantiate a new TestCaseResult object and add it to the ArrayList that holds TestCaseResult objects. Although it’s not a major issue, you do need to take some care to avoid confusing your objects. Notice that you’ll have two ArrayList objects, a TestCase object and a TestCaseResult object, both of which contain a test case ID, test case input, and expected result. In the third phase of the buffered processing model, you iterate through each test case result in the result ArrayList object and write information to an external text file. Of course, you can also easily emit results to an XML file, SQL database, or other external storage. If you run this code with the test case data file from Section 4.1 [id]=0001 [input]=Ac Ad Ah As Tc [expected]=FourOfAKindAces etc. you will get the identical output as in Section 4.1: ==================== ID = 0001 Input = Ac Ad Ah As Tc Expected = FourOfAKindAces Actual = FourOfAKindAces Pass ==================== etc. You can modularize this technique by writing three helper methods that wrap the code in the section. With these helper methods, your harness might look like: CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 107 6633c04.qxd 4/3/06 1:56 PM Page 107 class Class1 { static void Main(string[] args) { ArrayList tcd = null; // test case data ArrayList tcr = null; // test case results tcd = ReadData("..\\TestCases.txt"); tcr = RunTests(tcd); SaveResults(tcr, "..\\TestResults.txt"); } static ArrayList ReadData(string file) { // code here } static ArrayList RunTests(ArrayList testdata) { // code here } static void SaveResults(ArayList results, string file) { // code here } } class TestCase { // code here } class TestCaseResult { // code here } 4.3 Creating an XML File Data, Streaming Model Test Harness Problem You want to create a test harness that uses XML file test case data and a streaming processing model. Design In one continuous processing loop, use an XmlTextReader object to read a test case into mem- ory, then parse the test case data into input and expected values using the GetAttribute() and ReadElementString() methods, and call the CUT. Next, check the actual result with the CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS108 6633c04.qxd 4/3/06 1:56 PM Page 108 expected result to determine a test case pass or fail. Then, write the results to external storage using an XmlTextWriter object. Do this for each test case. Solution Begin by creating an XML test case file: Ac Ad Ah As Tc FourOfAKindAces 4s 5s 6s 7s 3s StraightSevenHigh 5d 5c Qh 5s Qd FullHouseFivesOverQueens Then process the test case data using XmlTextReader and XmlTextWriter objects: Console.WriteLine("\nBegin XML File Streaming model test run\n"); XmlTextReader xtr = new XmlTextReader("..\\..\\..\\TestCases.xml"); xtr.WhitespaceHandling = WhitespaceHandling.None; XmlTextWriter xtw = new XmlTextWriter("XMLFileStreamingResults.xml", System.Text.Encoding.UTF8); xtw.Formatting = Formatting.Indented; string id, input, expected, actual; xtw.WriteStartDocument(); xtw.WriteStartElement("TestResults"); // root node while (!xtr.EOF) // main loop { if (xtr.Name == "testcases" && !xtr.IsStartElement()) break; while (xtr.Name != "case" || !xtr.IsStartElement()) xtr.Read(); // go to a element if not there yet CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 109 6633c04.qxd 4/3/06 1:56 PM Page 109 id = xtr.GetAttribute("id"); xtr.Read(); // advance to input = xtr.ReadElementString("input"); // go to expected = xtr.ReadElementString("expected"); // go to xtr.Read(); // go to next or string[] cards = input.Split(' '); Hand h = new Hand(cards[0], cards[1], cards[2], cards[3], cards[4]); actual = h.GetHandType().ToString(); xtw.WriteStartElement("case"); xtw.WriteStartAttribute("id", null); xtw.WriteString(id); xtw.WriteEndAttribute(); xtw.WriteStartElement("input"); xtw.WriteString(input); xtw.WriteEndElement(); xtw.WriteStartElement("expected"); xtw.WriteString(expected); xtw.WriteEndElement(); xtw.WriteStartElement("actual"); xtw.WriteString(actual); xtw.WriteEndElement(); xtw.WriteStartElement("result"); if (actual == expected) xtw.WriteString("Pass"); else xtw.WriteString("*FAIL*"); xtw.WriteEndElement(); // xtw.WriteEndElement(); // } // main loop xtw.WriteEndElement(); // xtr.Close(); xtw.Close(); Console.WriteLine("\nDone"); The XmlTextReader.Read() method advances one XML node at a time through the XML file. Because XML is hierarchical, keeping track of exactly where you are within the file is a bit tricky. To write results, you use an XmlTextWriter object with the WriteStartElement(), the WriteString(), and the WriteEndElement() methods, along with the WriteStartAttribute() and WriteEndAttribute() methods. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS110 6633c04.qxd 4/3/06 1:56 PM Page 110 Comments The use of XML for test case storage has become very common. The key to understanding this technique is to understand the Read() and ReadElementString() methods of the System.Xml.XmlTextReader class. To an XmlTextReader object, an XML file is a sequence of nodes. For example, if you do not count whitespace, the XML file 99 has six nodes: the XML declaration, , , 99, , and . This means that the statement xtr.WhitespaceHandling = WhitespaceHandling.None; in your harness is critical because without it you would have to keep track of blank lines, tab characters, end-of-line sequences, and so on. The Read() method advances one node at a time. Unlike many Read() methods in other classes, the XmlTextReader.Read() method does not return significant data. The ReadElementString() method, on the other hand, returns the data between begin and end tags of its argument and advances to the next node after the end tag. Because XML attributes are not nodes, you have to extract attribute data using the GetAttribute() method. When run with the preceding test case data, this code produces the following as output: Ac Ad Ah As Tc FourOfAKindAces FourOfAKindAces Pass 4s 5s 6s 7s 3s StraightSevenHigh StraightFlushSevenHigh *FAIL* 5d 5c Qh 5s Qd FullHouseFivesOverQueens FullHouseFivesOverQueens Pass CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 111 6633c04.qxd 4/3/06 1:56 PM Page 111 Because XML is so flexible, you can use many alternative structures. For example, you can store all data as attributes: etc. This flexibility characteristic of XML is both a strength and a weakness. From a light- weight test automation point of view, the main disadvantage of XML is that you have to slightly modify your test harness code for every XML test case data structure. Processing an XML test case file with this loop structure: while (!xtr.EOF) // main loop { if (xtr.Name == "testcases" && !xtr.IsStartElement()) break; // process file here } may look a bit odd at first glance. The loop exits on end-of-file or when at the tag. But this structure is more readable than alternatives. When marching through the XML file, you can either Read() your way one node at a time or get a bit more sophisticated with code such as: while (xtr.Name != "testcase" || !xtr.IsStartElement() ) xtr.Read(); // advance to tag The choice of technique you use is purely a matter of style. Writing an XML element with XmlTextWriter tends to be a bit wordy but is straightforward. For example: xtw.WriteStartElement("alpha"); xtw.WriteStartElement("beta"); xtw.WriteString("b"); xtw.WriteEndElement(); // writes xtw.WriteEndElement(); // writes would create b Notice that the WriteEndElement() method does not accept an argument; the end element written is kept on an internal stack structure and popped off the stack. Writing an XML attribute follows a pattern similar to writing an element. For example: CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS112 6633c04.qxd 4/3/06 1:56 PM Page 112 xtw.WriteStartElement("alpha"); xtw.WriteStartAttribute("beta", null); xtw.WriteString("b"); xtw.WriteEndAttribute(); xtw.WriteEndElement(); produces as output: 4.4 Creating an XML File Data, Buffered Model Test Harness Problem You want to create a test harness that uses XML file test case data and a buffered processing model. Design To create a harness structure that uses a buffered processing model with XML test case data, you follow the same pattern as in Section 4.2 combined with the XML reading and writing techniques demonstrated in Section 4.3. You read all test case data into an ArrayList collection that holds lightweight TestCase objects, iterate through that ArrayList object, execute each test case, store the results into a second ArrayList object that holds lightweight TestCaseResult objects, and finally save the results to an external XML file. Solution With lightweight TestCase and TestCaseResult classes in place (see Section 4.2), you can write: Console.WriteLine("\nBegin XML File Buffered model test run\n"); XmlTextReader xtr = new XmlTextReader("..\\..\\..\\TestCases.xml"); xtr.WhitespaceHandling = WhitespaceHandling.None; XmlTextWriter xtw = new XmlTextWriter("XMLFileStreamingResults.xml", System.Text.Encoding.UTF8); xtw.Formatting = Formatting.Indented; string id, input, expected, actual; TestCase tc = null; TestCaseResult r = null; CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 113 6633c04.qxd 4/3/06 1:56 PM Page 113 // 1. read all test case data into memory ArrayList tcd = new ArrayList(); while (!xtr.EOF) // main loop { if (xtr.Name == "testcases" && !xtr.IsStartElement()) break; while (xtr.Name != "case" || !xtr.IsStartElement()) xtr.Read(); // advance to a element if not there yet id = xtr.GetAttribute("id"); xtr.Read(); // advance to input = xtr.ReadElementString("input"); // advance to expected = xtr.ReadElementString("expected"); // advance to tc = new TestCase(id, input, expected); tcd.Add(tc); xtr.Read(); // advance to next or } xtr.Close(); // 2. run all tests, store results to memory ArrayList tcr = new ArrayList(); for (int i = 0; i < tcd.Count; ++i) { tc = (TestCase)tcd[i]; string[] cards = tc.input.Split(' '); Hand h = new Hand(cards[0], cards[1], cards[2], cards[3], cards[4]); actual = h.GetHandType().ToString(); if (actual == tc.expected) r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "Pass"); else r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "*FAIL*"); tcr.Add(r); } // main processing loop // 3. emit all results to external storage xtw.WriteStartDocument(); xtw.WriteStartElement("TestResults"); // root node for (int i = 0; i < tcr.Count; ++i) { r = (TestCaseResult)tcr[i]; xtw.WriteStartElement("case"); xtw.WriteStartAttribute("id", null); xtw.WriteString(r.id); xtw.WriteEndAttribute(); CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS114 6633c04.qxd 4/3/06 1:56 PM Page 114 xtw.WriteStartElement("input"); xtw.WriteString(r.input); xtw.WriteEndElement(); xtw.WriteStartElement("expected"); xtw.WriteString(r.expected); xtw.WriteEndElement(); xtw.WriteStartElement("actual"); xtw.WriteString(r.actual); xtw.WriteEndElement(); xtw.WriteStartElement("result"); xtw.WriteString(r.result); xtw.WriteEndElement(); xtw.WriteEndElement(); // } xtw.WriteEndElement(); // xtw.Close(); Console.WriteLine("\nEnd test run\n"); Comments All the pertinent details to this technique are discussed in Sections 4.2 (buffered processing models) and 4.3 (reading and writing XML). If this code is run using the XML test case data file from Section 4.3: Ac Ad Ah As Tc FourOfAKindAces etc. the output will be identical to that produced by the technique code in Section 4.3: Ac Ad Ah As Tc FourOfAKindAces FourOfAKindAces Pass etc. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 115 6633c04.qxd 4/3/06 1:56 PM Page 115 Notice that this technique is starting to get a bit messy, mostly due to the large number of statements required to read and write XML. This makes it an excellent candidate for modular- ization by wrapping the code to read data, run tests, and save data into three helper methods. Furthermore, because the technique uses helper classes TestCase and TestCaseResult, recast- ing this solution to an OOP design is an attractive option. Such a design could take many forms, but here is one possibility: class XMLBufferedHarness { private ArrayList tcd = null; // test case data private ArrayList tcr = null; // test case results private XmlTextReader xtr = null; private XmlTextWriter xtw = null; public XMLBufferedHarness(string datafile, string resultfile) { // initialize tcd, tcr, xtr, xtw here } public void ReadData() { // use xtr to read datafile into tcd here } public void RunTests() { // run tests, store results to tcr here } public void SaveResults() { // save results to resultfile here } class TestCase { // see Section 4.2 } class TestCaseResult { // see Section 4.2 } } With this class in place, you can write very clean harness code like this: CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS116 6633c04.qxd 4/3/06 1:56 PM Page 116 static void Main(string[] args) { string data = "TestCases.xml"; string result = "TestResults.xml"; XMLBufferedHarness h = new XMLBufferedHarness(data, result); h.ReadData(); h.RunTests(); h.SaveResults(); } // Main() This approach has the advantage of being more modular than a non-OOP approach. However, the methods are very specific to a particular test scenario, meaning you’d have to significantly rewrite the methods for each CUT and associated XML test case file. This technique uses an XmlTextReader object to iterate through the XML test case data file and store test case data into memory. You have two significant alternatives: the XmlSerializer class and the XmlDocument class. The techniques to use these classes to read and parse test case data into memory are explained in Chapter 12. 4.5 Creating a SQL Database for Lightweight Test Automation Storage Problem You want to create a SQL database for a lightweight test automation harness. Design Write a lightweight T-SQL script and run it using the Query Analyzer or the osql.exe programs. Solution -- makeDbTestPoker.sql use master go if exists (select * from sysdatabases where name='dbTestPoker') drop database dbTestPoker go create database dbTestPoker go use dbTestPoker go CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 117 6633c04.qxd 4/3/06 1:56 PM Page 117 create table tblTestCases ( caseid char(4) primary key, input char(14) not null, expected varchar(35) not null, ) go insert into tblTestCases values('0001','Ac Ad Ah As Tc','FourOfAKindAces') insert into tblTestCases values('0002','4s 5s 6s 7s 3s','StraightSevenHigh') insert into tblTestCases values('0003','5d 5c Qh 5s Qd','FullHouseFivesOverQueens') go create table tblTestResults ( resultid int identity(1,1) primary key, caseid char(4) not null, input char(14) not null, expected varchar(35) not null, actual varchar(35) not null, result char(4) not null, runat datetime not null ) go Comments An alternative to using text files or XML files for your test case storage is to use a SQL database. SQL is particularly appropriate when you have many test cases (making the use of huge text files awkward) or when your SUT has a long development cycle (making management of many test case result files awkward). To run a SQL script, you can paste the code into the Query Analyzer program that ships with Microsoft SQL Server, and execute it directly. An alternative is to run the script using the osql.exe command-line program, which also ships with SQL Server. If the preceding script is saved as makeDbTestPoker.sql, you can run it like this: >osql -S(local) -E -i makeDbTestPoker.sql The -S switch specifies the name of the SQL Server machine. The -E switch means to use a trusted connection (explained later in this section). The -i switch specifies the name of the SQL script to run. The preceding script starts by setting the current database context to the “master” data- base, which is necessary to create or drop a database. Next, you check to see if the database dbTestPoker already exists by querying the sysdatabases system database. If dbTestPoker exists, then you drop it. Dropping a SQL database is surprisingly easy, so when using SQL for CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS118 6633c04.qxd 4/3/06 1:56 PM Page 118 test automation, be sure to back up your databases often. After creating database dbTestPoker, you switch context to that database. A common mistake is to forget to switch context, when all subsequent SQL commands will be directed at the master database. Next, you create a SQL table to hold test case data. The primary key argument to the caseid column means that each caseid value must be unique. The not null arguments mean that each test case must have an input and expected value. After creating the test case data table, you use the T-SQL insert command to populate the table. The last step is to create a table to hold test case results. Because each test run adds additional test results to the table, you usually want to include a column that holds the date and time when the test case result was added to the SQL database: runat datetime not null This technique creates a single database with a single test results table. An alternative approach is to create a new table for each test harness run. As a general rule, however, placing all harness run results into a single table is better than creating multiple tables—one table with thousands of rows of data is easier to manage than thousands of tables with any number of rows of data. If you do plan to put all test results into a single table, then you should create a column that uniquely identifies the test case result. The simplest way to do this is by adding an identity column to your test case data table definition: resultid int identity(1,1) primary key The identity(1,1) modifier instructs SQL Server to automatically generate an integer value for the resultid column, starting with value 1, and increasing by 1 on each insert operation. The technique in this section assumes that your test harness will be using a trusted connection. SQL Server requires a default Windows Authentication mode, which means in essence to integrate Windows security with SQL. This mode is the one used by a trusted con- nection. But SQL Server also supports an optional, additional SQL Authentication mode that can be used to gain access to SQL databases. The interaction between Windows Authentica- tion and SQL Authentication modes can be tricky and is outside the scope of this book. 4.6 Creating a SQL Data, Streaming Model Test Harness Problem You want to create a test harness that uses SQL test case data and a streaming processing model. Design In one continuous processing loop, use a SqlDataReader object from the System.Data.SqlClient namespace to read a test case into memory from SQL, then parse the test case data into input and expected values using the GetString() method, and call the CUT. Next, check the actual result with the expected result to determine a test case pass or fail. Write the results to external storage using a SqlCommand object with a SQL insert statement as an argument. Do this for each test case. This technique assumes you have previously prepared a SQL database with test case data and a table to hold test results. See Section 4.5 for details. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 119 6633c04.qxd 4/3/06 1:56 PM Page 119 Solution using System.Data.SqlClient; Console.WriteLine("\nBegin SQL Streaming model test run\n"); SqlConnection isc = new SqlConnection("Server=(local); Database=dbTestPoker;Trusted_Connection=yes"); SqlConnection osc = new SqlConnection("Server=(local); Database=dbTestPoker;Trusted_Connection=yes"); SqlCommand scSelect = new SqlCommand("SELECT * FROM tblTestCases", isc); isc.Open(); osc.Open(); SqlDataReader sdr; sdr = scSelect.ExecuteReader(); string caseid, input, expected, actual, result; while (sdr.Read()) // main loop { caseid = sdr.GetString(0); // parse input input = sdr.GetString(1); expected = sdr.GetString(2); string[] cards = input.Split(' '); Hand h = new Hand(cards[0], cards[1], cards[2], cards[3], cards[4]); actual = h.GetHandType().ToString(); if (actual == expected) // emit results result = "Pass"; else result = "FAIL"; string runat = DateTime.Now.ToString("s"); string insert = "INSERT INTO tblTestResults VALUES('" + caseid + "','" + input + "','" + expected + "','" + actual + "','" + result + "','" + runat + "')"; SqlCommand scInsert = new SqlCommand(insert, osc); scInsert.ExecuteNonQuery(); } // while sdr.Close(); isc.Close(); osc.Close(); Console.WriteLine("\nEnd test run\n"); CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS120 6633c04.qxd 4/3/06 1:56 PM Page 120 Comments Although there are several ways to iterate through a SQL table, the simplest is to use the SqlDataReader class. A SqlDataReader object gives you a way of reading a forward-only stream of rows from a SQL Server database. Notice that to create a SqlDataReader object, you use a factory mechanism by calling the ExecuteReader() method of the SqlCommand object, rather than directly by using a constructor and the new keyword. You also must prepare the SqlCommand object by passing in a T-SQL select statement to the SqlCommand constructor, so that the resulting SqlDataReader object knows how to traverse through the rows of its associated table. If the code in this section is run with the input data from Section 4.5: insert into tblTestCases values('0001','Ac Ad Ah As Tc','FourOfAKindAces') insert into tblTestCases values('0002','4s 5s 6s 7s 3s','StraightSevenHigh') insert into tblTestCases values('0003','5d 5c Qh 5s Qd','FullHouseFivesOverQueens') then table tblTestResults in database dbTestPoker will hold this result data: resultid caseid input expected =================================================================== 1 0001 Ac Ad Ah As Tc FourOfAKindAces 2 0002 4s 5s 6s 7s 3s StraightSevenHigh 3 0003 5d 5c Qh 5s Qd FullHouseFivesOverQueens actual result runat =================================================================== FourOfAKindAces Pass 2006-06-15 07:50:20.000 StraightFlushSevenHigh FAIL 2006-06-15 07:50:20.000 FullHouseFivesOverQueens Pass 2006-06-15 07:50:20.000 The values in the runat column will be the date and time when the results were inserted into the SQL table. You use the SqlDataReader.GetString() method to extract each column value as a string. The GetString() method accepts a zero-based column index rather than a column name as a string as you might expect. So you must write caseid = sdr.GetString(0); rather than caseid = sdr.GetString("caseid"); which would be more readable. If you insert all test case results into one SQL table rather than creating a new table to hold the results of each test run, you usually should time-stamp the result. A simple way to do this is to fetch the current system date and time: string runat = DateTime.Now.ToString("s"); The "s" argument will format the DateTime object into a sortable pattern such as: '2006-09-20T11:46:41.000' CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 121 6633c04.qxd 4/3/06 1:56 PM Page 121 SQL Server understands this format, and a C# string variable in this format is converted into a SQL datetime data type automatically when you insert it into a datetime column. The technique used in this solution to insert a row of data into the SQL test results table is rather ugly. If you were inserting literals into the results table, code might look like this: string insert = "INSERT INTO tblTestResults VALUES('0001', 'Ac Ad Ah As Kc', 'FourOfAKindAces', 'FourOfAKindAces', 'Pass', '2006-09-20 11:46:41.000')"; But because you are inserting values stored in variables, you have to build up a fairly complex insert string like this: string insert = "INSERT INTO tblTestResults VALUES('" + caseid + "','" + input + "','" + expected + "','" + actual + "','" + result + "','" + runat + "')"; Creating such SQL strings can be an error-prone process, so you must be careful when coding them. An alternative to writing such long strings is to create a SQL stored procedure and then call it from your harness. For example, if your SQL database creation script contains this user stored procedure T-SQL code: create procedure usp_insert @caseid char(4), @input char(14), @expected varchar(35), @actual varchar(35), @result char(4), @runat datetime as insert into tblTestResults values(@caseid, @input, @expected, @actual, @result, @runat) go then you can prepare a SqlCommand object like this: SqlConnection osc = new SqlConnection("Server=(local); Database=dbTestPoker; Trusted_Connection=yes"); SqlCommand sp = new SqlCommand("usp_insert", osc); sp.CommandType = CommandType.StoredProcedure; SqlParameter paramCaseID = sp.Parameters.Add("@caseid", SqlDbType.Char, 4); SqlParameter paramInput = sp.Parameters.Add("@input", SqlDbType.Char, 14); SqlParameter paramExpected = sp.Parameters.Add("@expected", SqlDbType.VarChar, 35); CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS122 6633c04.qxd 4/3/06 1:56 PM Page 122 SqlParameter paramActual = sp.Parameters.Add("@actual", SqlDbType.VarChar, 35); SqlParameter paramResult = sp.Parameters.Add("@result", SqlDbType.Char, 4); SqlParameter paramRunAt = sp.Parameters.Add("@runat", SqlDbType.DateTime); osc.Open(); And then in the main processing loop, you can insert test case results in SQL like this: // read caseid, input, expected from test case data here // run test and get actual, result here string runat = DateTime.Now.ToString("s"); paramCaseID.Value = caseid; paramInput.Value = input; paramExpected.Value = expected; paramActual.Value = actual; paramResult.Value = result; paramRunAt.Value = runat; sp.ExecuteNonQuery(); // insert using usp_insert This technique has the advantage of reducing complexity by eliminating an ugly SQL insert command string, but has the disadvantage of increasing complexity by adding many more lines of code to your test harness. 4.7 Creating a SQL Data, Buffered Model Test Harness Problem You want to create a test harness that uses SQL test case data and a buffered processing model. Design To create a harness structure that uses a buffered processing model with SQL test case data, you follow the same pattern as in Section 4.2 combined with the SQL reading and writing techniques demonstrated in Section 4.6. You use a SqlDataReader object to read all test case data into an ArrayList collection that holds lightweight TestCase objects. Next, you iterate through that ArrayList object, execute each test case, and store the results into a second ArrayList object that holds lightweight TestCaseResult objects. Then you save the results to an external SQL database. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 123 6633c04.qxd 4/3/06 1:56 PM Page 123 Solution With lightweight TestCase and TestCaseResult classes in place (see Section 4.2), you can write: Console.WriteLine("\nBegin SQL Buffered model test run\n"); SqlConnection isc = new SqlConnection("Server=(local); Database=dbTestPoker; Trusted_Connection=yes"); SqlConnection osc = new SqlConnection("Server=(local); Database=dbTestPoker;Trusted_Connection=yes"); isc.Open(); osc.Open(); SqlCommand scSelect = new SqlCommand("SELECT * FROM tblTestCases", isc); SqlDataReader sdr = scSelect.ExecuteReader(); string caseid, input, expected = "", actual; TestCase tc = null; // see Section 4.2 TestCaseResult r = null; // 1. read all test case data into memory ArrayList tcd = new ArrayList(); while (sdr.Read()) // main loop { caseid = sdr.GetString(0); input = sdr.GetString(1); expected = sdr.GetString(2); tc = new TestCase(caseid, input, expected); tcd.Add(tc); } isc.Close(); // 2. run all tests, store results to memory ArrayList tcr = new ArrayList(); for (int i = 0; i < tcd.Count; ++i) { tc = (TestCase)tcd[i]; string[] cards = tc.input.Split(' '); Hand h = new Hand(cards[0], cards[1], cards[2], cards[3], cards[4]); actual = h.GetHandType().ToString(); if (actual == tc.expected) r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "Pass"); else r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "FAIL"); tcr.Add(r); } // main processing loop CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS124 6633c04.qxd 4/3/06 1:56 PM Page 124 // 3. emit all results to external SQL storage for (int i = 0; i < tcr.Count; ++i) { r = (TestCaseResult)tcr[i]; string runat = DateTime.Now.ToString("s"); string insert = "INSERT INTO tblTestResults VALUES('" + r.id + "','" + r.input + "','" + r.expected + "','" + r.actual + "','" + r.result + "','" + runat + "')"; SqlCommand scInsert = new SqlCommand(insert, osc); scInsert.ExecuteNonQuery(); } osc.Close(); Console.WriteLine("\nDone"); Comments All the pertinent details to this technique are discussed in Sections 4.2 and 4.4 (buffered pro- cessing models), and Section 4.6 (reading and writing SQL). If the following code is run using the SQL test case data file from Section 4.5: insert into tblTestCases values('0001','Ac Ad Ah As Tc','FourOfAKindAces') insert into tblTestCases values('0002','4s 5s 6s 7s 3s','StraightSevenHigh') insert into tblTestCases values('0003','5d 5c Qh 5s Qd','FullHouseFivesOverQueens') then the output will be identical to that produced by the technique in Section 4.6: resultid caseid input expected =================================================================== 1 0001 Ac Ad Ah As Tc FourOfAKindAces 2 0002 4s 5s 6s 7s 3s StraightSevenHigh 3 0003 5d 5c Qh 5s Qd FullHouseFivesOverQueens actual result runat =================================================================== FourOfAKindAces Pass 2006-06-15 07:50:20.000 StraightFlushSevenHigh FAIL 2006-06-15 07:50:20.000 FullHouseFivesOverQueens Pass 2006-06-15 07:50:20.000 Using a buffered test automation-processing model makes it easy for you to perform test case data filtering or test case results filtering. For example, suppose you want to filter your test cases so that only certain suites of tests are run rather than all your tests. Test suite means a collection of test cases, usually a subset of a larger set of tests. Following are examples of common test suite categorizations: CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 125 6633c04.qxd 4/3/06 1:56 PM Page 125 • Developer Regression Tests (DRTs): A set of tests run on some new code (typically a set of classes or methods) before a developer checks in the code to the main build system. Designed to verify that the new code has not broken existing functionality. • Build Verification Tests (BVTs): A set of tests run on a new build of the SUT immediately after the build process. Designed to verify that the new build has minimal functionality and can be released to the test team for further testing. • Daily Test Runs (DTRs): A set of tests run by the test team every day. Designed to verify that previous functionality is still correct, uncover new functionality and performance bugs, and so on. • Weekly Test Runs (WTRs): A set of tests that is more extensive than Daily Test Run test cases but only run once a week due primarily to time constraints. • Milestone Test Runs (MTRs): A comprehensive set of tests run before the release of a major or minor milestone. May require several days to run. • Full Test Pass (FTP): Running every test case available. Typically requires several days to run. Of course, there are many variations on these categories of test suites, but the general prin- ciple is that you’ll have many test cases and you’ll run various subsets of test cases at different times. This holds true whether you are working in a traditional spiral software development methodology environment or in any of a number of currently fashionable methodologies, such as test-driven development, extreme programming, agile development, and so on. 4.8 Discovering Information About the SUT Problem You want to discover information about the SUT so that you can create meaningful test cases. Solution One of the greatest challenges of software testing in almost any environment is discovering the essential information about the SUT (SUT) so that you can test it meaningfully. There are six primary ways to perform system discovery in a .NET environment: • Read traditional specification documents. • Examine SUT source code. • Write experimental stub programs. • Use XML auto-documentation. • Examine .NET intermediate language code. • Use reflection techniques. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS126 6633c04.qxd 4/3/06 1:56 PM Page 126 Comments In a very small production environment where developers test their own code, system discov- ery may not be an issue. As the size of a development effort increases, however, the discovery process becomes more difficult. The most common approach is for you to read traditional written specification documents that describe the SUT. In theory at least, every system has a set of documents, usually written by senior developers, managers, or architects, that com- pletely and precisely describes the SUT. In reality, of course, such specification documents are often out-of-date, incomplete, or even nonexistent. Regardless, examining traditional specifi- cation documents is an important way to determine how to create meaningful test cases. You can examine the source code of the SUT to gain insights on how to test your system, although in some cases, this may not be possible for security or legal reasons. Even when source code examination is possible, reviewing the source code for a complex SUT can be enormously time consuming. When you have access to system source code while developing test cases, the situation is sometimes called white box or clear box testing. When you do not have access to source code, the situation is sometimes called black box testing. When you have partial access to system source code, for example, the signatures of methods but not the body of the method, the situation is sometimes called gray box testing. These labels are some of the most overused but least-useful terms in software testing. However, the principles behind these labels are important. You cannot test every possible input to a system (see Chapter 10 for dis- cussions of this idea), so the more you know about your SUT, the better your test cases will be. Although there has been much research in the area of automatic test case generation, cur- rently test case development is still for the most part a human activity where experience and intuition play a big role. A third discovery mechanism available to you is to experiment with the SUT by creating small stub programs. Again, this is not always possible for legal and security reasons and even when possible, it may not be a realistic technique: large software systems can be so complex that trying to understand them through experimentation just requires too much time. The development environment is often so dynamic that by the time you’ve figured a part of the system out, it has changed. This is not to say that experimentation is not important. On the contrary, initial experimentation with stub programs is usually the key first step when devel- oping lightweight test automation. The Visual Studio .NET IDE allows developers to add XML-based comments into their source code and have an XML-based document created automatically at project build time. In source code files, lines that begin with “///” and that precede user-defined items such as classes, delegates, interfaces, fields, events, properties, methods, or namespace declarations, can be processed as comments and placed in a file. There is a recommended set of tags. For example, the tag is used to describe parameters. When used, the compiler verifies that the parameter exists and that all parameters are described in the documentation. This mecha- nism requires developers to expend extra effort, but the payoff is that system specs are always up to date. Because .NET-compliant languages compile to an intermediate language, a terrific way to expose information about a SUT is to examine the SUT’s intermediate language. The .NET envi- ronment provides developers and testers with a tool named ILDASM. The ILDASM tool parses .NET Framework .exe or .dll assemblies and shows the information in human-readable format. ILDASM also displays namespaces and types, including their interfaces. The use of ILDASM for system discovery is essential for any lightweight test automation situation. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 127 6633c04.qxd 4/3/06 1:56 PM Page 127 The sixth primary way for you to discover information about the SUT is through the .NET reflection mechanism. Reflection means the process of programmatically obtaining information about .NET assemblies and the types defined within them. Using classes in the System.Reflection namespace, you can easily write short utility scripts that expose a wide range of data about the SUT. For example: Console.WriteLine("\nBegin Reflection Discovery"); string assembly = "..\\..\\..\\LibUnderTest\\PokerLib.dll"; Assembly a = Assembly.LoadFrom(assembly); Console.WriteLine("Assembly name = " + a.GetName()); Type[] tarr = a.GetTypes(); BindingFlags flags = BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.Instance; foreach(Type t in tarr) { Console.WriteLine(" Type name = " + t.Name); MemberInfo[] members = t.GetMembers(flags); foreach (MemberInfo mi in members) // fields, methods, ctors, etc. { if (mi.MemberType == MemberTypes.Field) Console.WriteLine(" (Field) member name = " + mi.Name); } // each member MethodInfo[] miarr = t.GetMethods(); // public only foreach (MethodInfo mi in miarr) { Console.WriteLine(" Method name = " + mi.Name); Console.WriteLine(" Return type = " + mi.ReturnType); ParameterInfo[] piarr = mi.GetParameters(); foreach (ParameterInfo pi in piarr) { Console.WriteLine(" Parameter name = " + pi.Name); Console.WriteLine(" Parameter type = " + pi.ParameterType); } } // each method } // each Type Console.WriteLine("\nDone"); This example loads the PokerLib.dll assembly and then iterates through each type (classes, enumerations, interfaces, and so on) in the assembly. Then for each type, you iterate through each member (fields, methods, properties, constructors, and so on), printing some information if you hit a field. After iterating through the members, you iterate through each method, printing the method’s name, return type, parameter names, and parameter types. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS128 6633c04.qxd 4/3/06 1:56 PM Page 128 4.9 Example Program: PokerLibTest This demonstration program combines several of the techniques in this chapter to create a lightweight test automation harness to test the PokerLib.dll library described in Section 4.1. The harness reads test case data from a SQL database, processes test cases using a buffered model, and emits test results to an XML file. If the test case input is caseid input expected ================================================== 0001 Ac Ad Ah As Tc FourOfAKindAces 0002 4s 5s 6s 7s 3s StraightSevenHigh 0003 5d 5c Qh 5s Qd FullHouseFivesOverQueens then the resulting XML output (where the runat attribute will be the value of the date and time the harness executed) is Ac Ad Ah As Tc FourOfAKindAces FourOfAKindAces Pass 4s 5s 6s 7s 3s StraightSevenHigh StraightFlushSevenHigh FAIL 5d 5c Qh 5s Qd FullHouseFivesOverQueens FullHouseFivesOverQueens Pass The complete lightweight test harness is presented in Listing 4-1. CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 129 6633c04.qxd 4/3/06 1:56 PM Page 129 Listing 4-1. Program PokerLibTest using System; using System.Collections; using System.Data.SqlClient; using System.Xml; using PokerLib; namespace PokerLibTest { class Class1 { [STAThread] static void Main(string[] args) { try { Console.WriteLine("\nBegin PokerLibTest run\n"); SqlConnection isc = new SqlConnection("Server=(local); Database=dbTestPoker;Trusted_Connection=yes"); isc.Open(); SqlCommand scSelect = new SqlCommand("SELECT * FROM tblTestCases", isc); SqlDataReader sdr = scSelect.ExecuteReader(); string caseid, input, expected = "", actual; TestCase tc = null; TestCaseResult r = null; // 1. read all test case data from SQL into memory ArrayList tcd = new ArrayList(); while (sdr.Read()) { caseid = sdr.GetString(0); input = sdr.GetString(1); expected = sdr.GetString(2); tc = new TestCase(caseid, input, expected); tcd.Add(tc); } isc.Close(); // 2. run all tests, store results to memory ArrayList tcr = new ArrayList(); for (int i = 0; i < tcd.Count; ++i) { tc = (TestCase)tcd[i]; string[] cards = tc.input.Split(' '); CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS130 6633c04.qxd 4/3/06 1:56 PM Page 130 Hand h = new Hand(cards[0], cards[1], cards[2], cards[3], cards[4]); actual = h.GetHandType().ToString(); string runat = DateTime.Now.ToString("s"); if (actual == tc.expected) r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "Pass", runat); else r = new TestCaseResult(tc.id, tc.input, tc.expected, actual, "FAIL", runat); tcr.Add(r); } // 3. emit all results to external XML storage XmlTextWriter xtw = new XmlTextWriter("PokerLibResults.xml", System.Text.Encoding.UTF8); xtw.Formatting = Formatting.Indented; xtw.WriteStartDocument(); xtw.WriteStartElement("TestResults"); // root node for (int i = 0; i < tcr.Count; ++i) { r = (TestCaseResult)tcr[i]; xtw.WriteStartElement("case"); xtw.WriteStartAttribute("id", null); xtw.WriteString(r.id); xtw.WriteEndAttribute(); xtw.WriteStartAttribute("runat", null); xtw.WriteString(r.runat); xtw.WriteEndAttribute(); xtw.WriteStartElement("input"); xtw.WriteString(r.input); xtw.WriteEndElement(); xtw.WriteStartElement("expected"); xtw.WriteString(r.expected); xtw.WriteEndElement(); xtw.WriteStartElement("actual"); xtw.WriteString(r.actual); xtw.WriteEndElement(); xtw.WriteStartElement("result"); xtw.WriteString(r.result); xtw.WriteEndElement(); xtw.WriteEndElement(); // } xtw.WriteEndElement(); // xtw.Close(); CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS 131 6633c04.qxd 4/3/06 1:56 PM Page 131 Console.WriteLine("\nDone"); Console.ReadLine(); } catch(Exception ex) { Console.WriteLine("Fatal error: " + ex.Message); Console.ReadLine(); } } // Main() class TestCase { public string id; public string input; public string expected; public TestCase(string id, string input, string expected) { this.id = id; this.input = input; this.expected = expected; } } // class TestCase class TestCaseResult { public string id; public string input; public string expected; public string actual; public string result; public string runat; public TestCaseResult(string id, string input, string expected, string actual, string result, string runat) { this.id = id; this.input = input; this.expected = expected; this.actual = actual; this.result = result; this.runat = runat; } } // class TestCaseResult } // Class1 } // ns CHAPTER 4 ■ TEST HARNESS DESIGN PATTERNS132 6633c04.qxd 4/3/06 1:56 PM Page 132 Web Application Testing PART 2 ■ ■ ■ 6633c05.qxd 4/3/06 1:56 PM Page 133 6633c05.qxd 4/3/06 1:56 PM Page 134 Request-Response Testing 5.0 Introduction The most fundamental type of Web application testing is request-response testing. You pro- grammatically send an HTTP request to a Web server, and then after the Web server processes the request and sends an HTTP response (usually in the form of an HTML page), you capture the response and examine it for an expected value. The request-response actions normally occur together, meaning that in a lightweight test automation situation, it is unusual for you to send an HTTP request and not retrieve the response, or to retrieve an HTTP response from a request you did not create. Accordingly, most of the techniques in this chapter show you how to send an HTTP request and fetch the HTTP response, or how to examine an HTTP response for an expected value. Consider the simple ASP.NET Web application shown in Figure 5-1. Figure 5-1. Web AUT 135 CHAPTER 5 ■ ■ ■ 6633c05.qxd 4/3/06 1:56 PM Page 135 The code that produced the Web application shown in Figure 5-1 is

Request-Response

Choose one: red blue green

Notice that for simplicity, the C# logic and HTML display code are combined in the same file rather than the more usual approach of storing them in separate files using the ASP.NET code-behind mechanism (as when you create a Web application using Visual Studio .NET). This ASP.NET Web application is coded in C#, but the request-response testing techniques in this chapter will work for ASP.NET applications written in any .NET-compliant language. To test this application manually, you select a color from the Choose One drop-down list and click the Send button. The drop-down value is sent as part of an HTTP request to the ASP.NET Web server. The server processes the request and constructs an HTTP response. The response is returned to the Internet Explorer (IE) client where the HTML is rendered in friendly form as shown in Figure 5-1. You have to visually examine the result for some indica- tion that the HTTP response was correct (the message in the text box control in this case). Manually testing a Web application in this way is slow, inefficient, error-prone, and tedious. A better approach is to write lightweight test automation. An automated request-response test programmatically sends an HTTP request that con- tains the same information as the result of a user selecting a drop-down value, and the test programmatically examines the HTTP response for data that indicates a correct response as shown in Figure 5-2. CHAPTER 5 ■ REQUEST-RESPONSE TESTING136 6633c05.qxd 4/3/06 1:56 PM Page 136 Figure 5-2. Request-response test run The .NET Framework provides you with three fundamental ways and two low-level ways to send an HTTP request and retrieve the corresponding HTTP response. Listed from easiest- to-use but least-flexible to hardest-to-use but most-flexible, following are the five ways to send and retrieve HTTP data: • WebClient: Particularly simple to use but does not allow you to send authentication credentials. • WebRequest - WebResponse: Gives you more flexibility, including the ability to send authentication credentials. • HttpWebRequest - HttpWebResponse: Gives you full control at the expense of a slight increase in complexity. • TcpClient: A low-level class available to you, but except in unusual situations, it isn’t needed for lightweight request-response test automation. • Socket: A very low-level class not often used in lightweight test automation. The .NET Framework also has an HttpRequest class, but it’s a base class that is not intended to be used directly. The techniques in this chapter use the three higher-level classes (WebClient, WebRequest - WebResponse, and HttpWebRequest - HttpWebResponse). The TcpClient and Socket classes are explained in Chapter 8. The test harness that produced the test run shown in Figure 5-2 is presented in Section 5.12. CHAPTER 5 ■ REQUEST-RESPONSE TESTING 137 6633c05.qxd 4/3/06 1:56 PM Page 137 5.1 Sending a Simple HTTP GET Request and Retrieving the Response Problem You want to send a simple HTTP GET request and retrieve the HTTP response. Design Create an instance of the WebClient class and use its DownloadData() method. Solution string uri = "http://server/path/WebForm.aspx"; WebClient wc = new WebClient(); Console.WriteLine("Sending an HTTP GET request to " + uri); byte[] bResponse = wc.DownloadData(uri); string strResponse = Encoding.ASCII.GetString(bResponse); Console.WriteLine("HTTP response is: "); Console.WriteLine(strResponse); Comments The WebClient class is part of the System.Net namespace, which is accessible by default from the console application. Using the WebClient.DownloadData() method to fetch an HTTP response is particularly simple, but DownLoadData() only returns a byte array that must be converted into a string using the System.Text.Encoding.ASCII.GetString() method. An alternative is to use the WebClient.OpenRead() method and associate it with a stream: string uri = " http://server/path/WebForm.aspx"; WebClient wc = new WebClient(); Console.WriteLine("Sending an HTTP GET request to " + uri); Stream st = wc.OpenRead(uri); StreamReader sr = new StreamReader(st); string res = sr.ReadToEnd(); sr.Close(); st.Close(); Console.WriteLine("HTTP Response is "); Console.WriteLine(res); The WebClient class is most useful when you are testing static HTML pages rather than ASP.NET Web applications. This code may be used to examine an ASP.NET application response but to expand this code into an automated test, you need to examine the HTTP response for an expected value. The techniques in this section are used in Section 5.8 to programmatically determine an ASP.NET Web application ViewState value. The techniques in Section 5.11 show you how to examine an HTTP response for an expected value. CHAPTER 5 ■ REQUEST-RESPONSE TESTING138 6633c05.qxd 4/3/06 1:56 PM Page 138 5.2 Sending an HTTP Request with Authentication and Retrieving the Response Problem You want to send an HTTP request with network authentication credentials and retrieve the HTTP response. Design Create a WebRequest object and create a NetworkCredential object. Assign the NetworkCredential object to the Credentials property of WebRequest object and fetch the HTTP response using the WebRequest.GetResponse() method. Solution string uri = " http://server/path/WebForm.aspx"; WebRequest wreq = WebRequest.Create(uri); string uid = "someDomainUserID"; string pwd = "theDomainPassword"; string domain = "theDomainName"; NetworkCredential nc = new NetworkCredential(uid, pwd, domain); wreq.Credentials = nc; Console.WriteLine("Sending authenticated request to " + uri); WebResponse wres = wreq.GetResponse(); Stream st = wres.GetResponseStream(); StreamReader sr = new StreamReader(st); string res = sr.ReadToEnd(); sr.Close(); st.Close(); Console.WriteLine("HTTP Response is "); Console.WriteLine(res); Comments If you need to send an HTTP request with network authentication credentials (user ID, domain, and password), you can use the WebRequest and WebResponse classes. These classes are located in the System.Web namespace, which is not accessible by default in a console application, so you have to add a project reference to file System.Web.dll. Notice that a WebRequest object is created using a factory mechanism with the Create() method rather than the more usual con- structor approach using the new keyword. After creating a NetworkCredential object, you can attach that object to the WebRequest object. The WebResponse object is returned by a call to the WebRequest.GetResponse() method; there is no explicit “Send” method as you might have expected. The response stream can be associated, like any stream, to a StreamReader object so that you can fetch the entire HTTP response as a string using the ReadToEnd() method. CHAPTER 5 ■ REQUEST-RESPONSE TESTING 139 6633c05.qxd 4/3/06 1:56 PM Page 139 The WebRequest and WebResponse classes are actually abstract base classes. In practical terms, you’ll use WebRequest - WebResponse for relatively simple HTTP requests that require authentication. If authentication isn’t necessary, the WebClient class is often a better choice. If you need to send an HTTP POST request, the HttpWebRequest and HttpWebResponse classes are often a better choice. The WebRequest and WebResponse classes support asynchronous calls, but this is rarely needed in lightweight test automation situations. The code in this section may be used to examine an ASP.NET application response, but to expand this code into an automated test, you need to examine the HTTP response for an expected value as described in Section 5.11. 5.3 Sending a Complex HTTP GET Request and Retrieving the Response Problem You want to send an HTTP GET Request and have full control over the request properties. Design Create an instance of an HttpWebRequest class and fetch the HTTP response using the GetResponse() method. Solution string uri = " http://server/path/WebForm.aspx"; HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri); req.Method = "GET"; req.MaximumAutomaticRedirections = 3; req.Timeout = 5000; Console.WriteLine("Sending HTTP request"); HttpWebResponse res = (HttpWebResponse)req.GetResponse(); Stream resst = res.GetResponseStream(); StreamReader sr = new StreamReader(resst); Console.WriteLine("HTTP Response is: "); Console.WriteLine(sr.ReadToEnd()); sr.Close(); resst.Close(); Comments The HttpWebRequest and HttpWebResponse classes are your best all around choice for sending and receiving HTTP data in lightweight test automation scenarios. They support a wide range of useful properties. These classes are located in the System.Net namespace, which is accessible by default in a console application. Notice that an HttpWebRequest object is created using a fac- tory mechanism with the Create() method rather than the more usual constructor approach CHAPTER 5 ■ REQUEST-RESPONSE TESTING140 6633c05.qxd 4/3/06 1:56 PM Page 140 using the new keyword. Also, there is no explicit “Send” method as you might have expected; an HttpWebResponse object is returned by a call to the HttPWebRequest.GetResponse() method. You can associate the response stream to a StreamReader object so that you can retrieve the entire HTTP response as a string using the ReadToEnd() method. You can also retrieve the HTTP response line-by-line using the StreamReader.ReadLine() method. This technique shows how you can limit the number of request redirections and set a timeout value. Following are a few of the HttpWebRequest properties that are most useful for lightweight test automation: • AllowAutoRedirect: Gets or sets a value that indicates whether the request should follow redirection responses. • CookieContainer: Gets or sets the cookies associated with the request. • Credentials: Provides authentication information for the request. • KeepAlive: Gets or sets a value indicating whether to make a persistent connection to the Internet resource. • MaximumAutomaticRedirections: Gets or sets the maximum number of redirects that the request will follow. • Proxy: Gets or sets proxy information for the request. • SendChunked: Gets or sets a value indicating whether to send data in segments to the Internet resource. • Timeout: Gets or sets the timeout value for a request. • UserAgent: Gets or sets the value of the User-Agent HTTP header. The purpose of each of these properties is fairly obvious from their names, and they are fully documented in case you need to use them. 5.4 Retrieving an HTTP Response Line-by-Line Problem You want to retrieve an HTTP response line-by-line rather than as an entire string. Design Obtain the HTTP response stream using the HttpWebRequest.GetResponse() method and pass that stream to a StreamReader() constructor. Then use the StreamReader.ReadLine() method inside a while loop. Solution // send an HTTP request using the WebClient class, // the WebRequest class, or the HttpWebRequest class CHAPTER 5 ■ REQUEST-RESPONSE TESTING 141 6633c05.qxd 4/3/06 1:56 PM Page 141 Stream st = null; // attach Stream st to an HTTP response using the // WebClient.OpenRead() method, the WebRequest.GetResponseStream() // method, or the HttpWebRequest.GetResponse() method StreamReader sr = new StreamReader(st); string line = null; Console.WriteLine("HTTP response line-by-line: "); while ((line = sr.ReadLine()) != null) { Console.WriteLine(line); } sr.Close(); st.Close(); Comments Each of the three fundamental ways to send an HTTP request (WebClient, WebRequest, HttpWebRequest) supports a method that returns their associated HTTP response as a Stream object. The Stream object can be associated to a StreamReader object that has several ways to fetch stream data. Using the StreamReader.ReadToEnd() method, you can retrieve the HTTP response as one big string. This is fine for most test automation situations, but sometimes you want to retrieve the HTTP response a line at a time. For instance, if the response is very large, you may not want to store it into one huge string. Or if you are searching the response for a tar- get string, searching line-by-line is sometimes more efficient. To search line-by-line, you can use the StreamReader.ReadLine() method in conjunction with a while loop. The ReadLine() method returns a string consisting of everything up to and including a newline character, or null if no characters are available. In addition to fetching an HTTP response stream a line at a time, you can also retrieve the response a block of characters at a time: // attach response stream to Stream st // associate st to StreamReader sr char[] block = new char[3]; int ct = 0; while ((ct = sr.Read(block, 0, 3)) != 0) { for (int i = 0; i < ct; i++) Console.Write(block[i] + " "); } Code like this is useful when you want to examine the HTTP response at the character level rather than at the line or string level. In this example, you prepare a character array block of size 3 to hold the response. The StreamReader.Read() method reads 3 characters (or as many charac- ters as are available in the stream), stores the characters into an array block starting at position 0, CHAPTER 5 ■ REQUEST-RESPONSE TESTING142 6633c05.qxd 4/3/06 1:56 PM Page 142 and returns the actual number of characters read. If 0 characters are read, that means the stream has been exhausted and you can exit the while loop. Notice that a degenerative case is defined when you declare a character array of size 1; in this situation, you are reading a single character at a time. 5.5 Sending a Simple HTTP POST Request to a Classic ASP Web Page Problem You want to send a simple HTTP POST request to a classic ASP page/script and retrieve the resulting HTTP response. Design Create an instance of the HttpWebRequest class. Set the object’s Method property to "POST" and the ContentType property to "application/x-www-form-urlencoded". Add the POST data to the request using the GetRequestStream() method and the Stream.Write() method. Fetch the HTTP response using the HttpWebRequest.GetResponse() method. Solution string url = "http://localhost/TestAuto/Ch5/classic.asp"; string data = "inputBox1=orange"; byte[] buffer = Encoding.ASCII.GetBytes(data); HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url); req.Method = "POST"; req.ContentType = "application/x-www-form-urlencoded"; req.ContentLength = buffer.Length; Stream reqst = req.GetRequestStream(); reqst.Write(buffer, 0, buffer.Length); reqst.Flush(); reqst.Close(); Console.WriteLine("\nPosting 'orange'"); HttpWebResponse res = (HttpWebResponse)req.GetResponse(); Stream resst = res.GetResponseStream(); StreamReader sr = new StreamReader(resst); Console.WriteLine("\nGrabbing HTTP response\n"); Console.WriteLine(sr.ReadToEnd()); sr.Close(); resst.Close(); Console.WriteLine("Done"); CHAPTER 5 ■ REQUEST-RESPONSE TESTING 143 6633c05.qxd 4/3/06 1:56 PM Page 143 Comments Suppose you have an HTML page form like this:

Enter color:

And you have a related classic ASP page/script like this:

You submitted:

<% strColor = Request.Form("inputBox1") Response.Write(strColor) %>

Bye

If a user loads page classic.html into a Web client such as IE, an “Enter color:” prompt and a text field are displayed. After entering some text and clicking on the submit button, an HTTP request containing the HTML form data is sent to the Web server. The Web server accepts the POST request and runs the classic.asp script. The script grabs the value entered in the text field and inserts it into the HTML result stream, which is then sent as an HTTP response back to the client (where the HTML would be rendered in human-friendly form). To send an HTTP request directly to page/script classic.asp and retrieve the HTTP response, the most flexible option is to use the HttpWebRequest class. The key is to first set up data to post as a string of name-value pairs connected with &: string data = "inputBox1=orange&inputBox2=green"; Next, you must convert the post data from type string into a byte array using the System. Text.Encoding.ASCII.GetBytes() method because all HTTP data is transferred as bytes. After creating an HttpWebRequest object, you must set the request object’s Method property to "POST" and the ContentType to "application/x-www-form-urlencoded". You can think of the ContentType value as a magic string that tells the Web server to interpret the HTTP request data as HTML form data. You must set the value of the ContentLength property to the length of the post data stored in the byte array. Notice that because the ContentLength property is required, you must prepare the post data before setting up the HttpWebRequest object. After setting up the request, CHAPTER 5 ■ REQUEST-RESPONSE TESTING144 6633c05.qxd 4/3/06 1:56 PM Page 144 you obtain the request stream using the HttpWebRequest.GetRequestStream() method so that you can add the post data into the stream. You do this by writing to the stream like this: reqst.Write(buffer, 0, buffer.Length); You specify what byte array to write into the request stream, the starting position within the byte array, and the number of bytes to write. If you use the Length property as the number of bytes to write, you will write the entire byte array to the request stream. Now you can send the HTTP request and retrieve the HTTP response as a string using a StreamReader object. If the preceding solution is run, the output is Posting 'orange' Grabbing HTTP response:

You submitted:

orange

Bye

Done This technique uses the HttpWebRequest and HttpWebResponse classes, but you can use the WebClient class or the WebRequest and WebResponse classes, too. The technique in this section is useful to examine an HTTP response from a classic ASP Web application, but to extend the solution into test automation, you must search the HTTP response for an expected value as discussed in Section 5.11. This technique assumes that the POST data string does not contain any characters that may be misinterpreted by the Web server such as blank spaces and ampersands. To deal with such characters see Section 5.7. This solution also assumes that the HTTP request-response does not travel through a proxy server. To deal with proxy servers, see the “Comments” section in Section 5.6. 5.6 Sending an HTTP POST Request to an ASP.NET Web Application Problem You want to send an HTTP POST request to an ASP.NET Web application and retrieve the result- ing HTTP response. CHAPTER 5 ■ REQUEST-RESPONSE TESTING 145 6633c05.qxd 4/3/06 1:56 PM Page 145 Design Create an HttpWebRequest object. Set the object’s Method property to "POST" and the ContentType property to "application/x-www-form-urlencoded". Concatenate the application’s ViewState value to the POST data. If your Web application is running on ASP.NET 2.0, you must also con- catenate the application’s EventValidation value to the POST data. Add the POST data to the request using the GetRequestStream() method and the Stream.Write() method. Fetch the HTTP response using the HttpWebRequest.GetResponse() method. Solution string url = "http://localhost/TestAuto/Ch5/WebForm.aspx"; string data = "TextBox1=red&TextBox2=empty&Button1=clicked"; string vs = "dDwtMTQwNDA4NDA4ODs7PeWiylVlaimBKuqooykeHvDojL2i"; vs = HttpUtility.UrlEncode(vs); data += "&__VIEWSTATE=" + vs; byte[] buffer = Encoding.ASCII.GetBytes(data); HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url); req.Method = "POST"; req.ContentType = "application/x-www-form-urlencoded"; req.ContentLength = buffer.Length; Stream reqst = req.GetRequestStream(); reqst.Write(buffer, 0, buffer.Length); reqst.Flush(); reqst.Close(); HttpWebResponse res = (HttpWebResponse)req.GetResponse(); Stream resst = res.GetResponseStream(); StreamReader sr = new StreamReader(resst); Console.WriteLine("\nGrabbing HTTP response:\n"); Console.WriteLine(sr.ReadToEnd()); sr.Close(); resst.Close(); Console.WriteLine("Done"); Comments Suppose you have this ASP.NET Web application named WebForm1.aspx:

Color Commenter

Enter color:

My comment:

This Web application was created manually rather than by using Visual Studio .NET, which would have resulted in the C# logic code and the HTML display code being in different files. This fact does not affect how you automate the application. The Web application has two text fields and a button control. The user enters a string such as “red” into the TextBox1 control. Clicking on the Button1 control sends an HTTP request to the Web server. The ASP.NET server logic checks the value in TextBox1 and creates an HTTP response page that displays a short message such as “Roses are red” in TextBox2. If the code in this solution executes, the output is Sending TextBox1=red Grabbing the HTTP response:

Color Commenter

CHAPTER 5 ■ REQUEST-RESPONSE TESTING 147 6633c05.qxd 4/3/06 1:56 PM Page 147

Enter color:

My comment:

Done Notice that the value attribute of the TextBox2 input tag is "Roses are red". To send an HTTP request directly to an ASP.NET Web application and retrieve the HTTP response, your most flexible option is to use the HttpWebRequest class. The first step is to set up data to post a string of name-value pairs: string data = "TextBox1=red&TextBox2=empty&Button1=clicked"; The "TextBox1=red" is self-explanatory. The "TextBox2=empty" and the "Button1=clicked" part of the post data are there to keep the ViewState value synchronized between the client test automation program and the ASP.NET Web server. Every ASP.NET Web application has a ViewState value that represents the state of the application after each request-response round trip. The ViewState value is a Base64-encoded string. By creating and maintaining a ViewState value, the Web server can maintain application state between successive HTTP requests. You must determine the ViewState value and add it to the post data: string vs = "dDwtMTQwNDA4NDA4ODs7PuWdy3VjanmrKIqoo7kBHkDzjH2p"; vs = HttpUtility.UrlEncode(vs); data += "&__VIEWSTATE=" + vs; Because the ViewState value can have characters that may confuse the ASP.NET Web server (&, for example), you should apply the HttpUtility.UrlEncode() method to the ViewState value. The HttpUtility class is contained in the System.Web namespace, which is not accessible by default to a console application, so you’ll have to add a project reference to System.Web.dll (see Section 5.7 for details). Note that two underscore characters appear before the VIEWSTATE. You have two ways to determine an initial ViewState value. The first, as demonstrated here, is to manually find the value by simply launching IE (or another client), loading the WebForm.aspx application, and then choosing View ➤ Source. The second way to determine an initial ViewState value is to programmatically send an HTTP request to WebForm.aspx and then programmatically grab the ViewState value from the HTTP response. This technique is explained in Section 5.8. Exactly which components of an ASP.NET application contribute to the ViewState value and how the ViewState value is calculated by the ASP.NET Web server is not fully documented, so it requires some trial and error to determine exactly what to place in the post data string. For instance, in this example, you can leave out the "TextBox2=empty" portion of the string, how- ever the "Button1=clicked" is necessary. The "empty" and "clicked" string values are arbitrary. In other words, you can type "Button1=foo" or even "Button1=" and the ViewState value will CHAPTER 5 ■ REQUEST-RESPONSE TESTING148 6633c05.qxd 4/3/06 1:56 PM Page 148 remain synchronized, and your automation will succeed. Using string constants such as "empty" and "clicked" makes your code more readable at the expense of possibly misleading code reviewers into thinking there is something special about those values. When adding the ViewState value to a post data string, the position of the ViewState value does not matter. How- ever, your code will be more readable if you place the ViewState value at the end of the string. In ASP.NET 2.0, a new EventValidation feature was added for security against fraudulent postbacks. The framework posts encrypted data, which is part of the __EVENTVALIDATION hidden field. The hidden field is generated as the last element in the Web application form element. So in an ASP.NET 2.0 environment, you have to add the EventValidation value to the POST data like this: string ev = "d+waMTswVDA4NDA4OQs7buWdy3VwbjkrKIqoo7kBHkDzjH2p"; ev = HttpUtility.UrlEncode(ev); data += "&__EVENTVALIDATION=" + ev; After building up the post data string, you must convert it into a byte array using the System.Text.Encoding.ASCII.GetBytes() method, because all HTTP traffic works at the byte level. Next, you must set the request object’s Method property to "POST" and the ContentType to "application/x-www-form-urlencoded". The ContentType value is a string that tells the Web server that the HTTP request data should be interpreted as HTML form data. Then, you need to set the value of the ContentLength property to the length of the post data stored in the byte array. After setting up the request, you can obtain the request stream using the HttpWebRequest.GetRequestStream() method so that you can add the post data into the HTTP request stream: reqst.Write(buffer, 0, buffer.Length); You specify which byte array to write into the stream, the starting position within the byte array, and the total number of bytes to write. Finally, you are ready to send the HTTP request and retrieve the HTTP response: HttpWebResponse res = (HttpWebResponse)req.GetResponse(); Stream resst = res.GetResponseStream(); You can then fetch the response as a string using a StreamReader object. The technique presented here is useful to examine an HTTP response from an ASP.NET Web application, but to extend the solution into true test automation, you must search the response for an expected value. If you must deal with a proxy server, you can easily add the optional Proxy property to an HttpWebRequest object: // instantiate HttpWebRequest req here string proxy = "someProxyMachineNameOrIPAddress"; req.Proxy = new WebProxy(proxy, true); You pass the name or IP address of the proxy server machine as a string to a WebProxy con- structor and attach the resulting object to the HttpWebRequest object. The Boolean argument specifies whether or not to ignore the proxy server for local addresses; true means ignore the proxy server for local addresses. You can significantly increase the modularity of and extend your test automation by factor- ing the code in this section into a helper method: CHAPTER 5 ■ REQUEST-RESPONSE TESTING 149 6633c05.qxd 4/3/06 1:56 PM Page 149 private static bool ResponseHasTarget(string uri, string postData, string target) { // create HttpWebRequest // add postData to request stream // obtain HttpResponse stream // attach response to StreamReader object sr string result = sr.ReadToEnd(); if (result.IndexOf(target) >= 0) return true; else return false; } The helper accepts the URI of the Web application (such as "http://server/path/WebForm. aspx"), the data that is to be posted to the application (such as "TextBox1=red&TextBox2=blue"), and a target string (such as "The result is purple"). The method returns true if the HTTP response associated with the HTTP request contains the target string and returns false if the target string is not in the HTTP response. The example program in Section 5.12 has a complete implementation of the helper method ResponseHasTarget(). 5.7 Dealing with Special Input Characters Problem You want to handle special characters such as “&” in your HTTP POST data. Design Use the HttpUtility.UrlEncode() method to convert potentially troublesome characters into their character-entity equivalents. Solution string badValueForTextBox1 = "this&that"; string goodValueForTextBox1 = HttpUtility.UrlEncode(badValueForTextBox1); string data = "TextBox1=" + goodValueForTextBox1; Comments If you place characters (such as blank spaces) and/or punctuation (such as “&”) into an HTTP request stream, the receiving ASP.NET Web server may misinterpret them. URL encoding converts characters that are not allowed in a URL into character-entity equivalents. For example, when embedded in a string to be transmitted in a URL, the characters “<” and “>” are encoded as %3c and %3d, respectively. CHAPTER 5 ■ REQUEST-RESPONSE TESTING150 6633c05.qxd 4/3/06 1:56 PM Page 150 The HttpUtility.UrlEncode() method handles the mapping of potentially troublesome characters into a three-character sequence starting with “%”. The UrlEncode() method is located inside the System.Web namespace. Suppose you have an ASP.NET Web application that contains this code: if (TextBox1.Text == "this&that") TextBox2.Text = "Oh really"; else TextBox2.Text = "unknown input"; To test this logic, you need to post the string “this&that” to the Web application. If you try this directly as string data = "TextBox1=this&that"; you will get an HTTP response with "unknown input" as the TextBox2 attribute instead of "Oh really" because the Web server will get confused by the “&” character embedded in the POST data. To solve this issue, you can use the HttpUtility.UrlEncode() method that converts the “&” character to the sequence “%26”. When the Web server receives the HTTP request, the %26 will be URL decoded into a “&” character and your automation logic will succeed. One strategy you can employ is to always apply HttpUtility.UrlEncode() to your input POST data even when it does not contain troublesome characters: string anyValue = "whatever"; anyValue = HttpUtility.UrlEncode(anyValue); string data = "TextBox1=" + anyValue; The downside to this strategy is that when testing, you sometimes want to actually send troublesome characters. One approach is to create two harnesses: one that always performs a UrlEncode() on the value part of name-value post pairs and one that never performs a UrlEncode(). Another approach is to parameterize your harness to read input POST data and expected values from an external test case data store and to include a value in the test case data to indicate whether the input data should be URL encoded or not: 001!TextBox1!red!noencode!Roses are red 002!TextBox1!this&that!encode!Oh really 003!TextBox1!this&that!noencode!unknown input You then use branching logic to determine whether you should URL-encode or not: while ((line = sr.ReadLine()) != null) // test loop { tokens = line.Split('!'); if (tokens[3] == "encode") input = HttpUtility.UrlEncode(tokens[2]); else input = tokens[2]; data = tokens[1] + "=" + input; // etc. } CHAPTER 5 ■ REQUEST-RESPONSE TESTING 151 6633c05.qxd 4/3/06 1:56 PM Page 151 5.8 Programmatically Determining a ViewState Value and an EventValidation Value Problem You want to programmatically determine an initial ViewState value (and an initial EventValidation value under ASP.NET 2.0) for an ASP.NET Web application. Design Use a WebClient object to send a simple, initial probing HTTP request to the application. Fetch the HTTP probe response and parse out the ViewState value (and the EventValidation value under ASP.NET 2.0) using the String.IndexOf() and String.SubString() methods. Solution If you are running under ASP.NET 1.1: string uri = "http://server/path/WebForm.aspx"; WebClient wc = new WebClient(); Stream st = wc.OpenRead(uri); StreamReader sr = new StreamReader(st); string res = sr.ReadToEnd(); sr.Close(); st.Close(); int start = res.IndexOf("__VIEWSTATE", 0) + 20; int end = res.IndexOf("\"", start); string vs = res.Substring(start, (end-start)); Console.WriteLine("ViewState = " + vs); If you are running under ASP.NET 2.0: string uri = "http://server/path/WebForm.aspx"; WebClient wc = new WebClient(); Stream st = wc.OpenRead(uri); StreamReader sr = new StreamReader(st); string res = sr.ReadToEnd(); sr.Close(); st.Close(); int startVS = res.IndexOf("__VIEWSTATE", 0) + 37; int endVS = res.IndexOf("\"", startVS); string vs = res.Substring(startVS, (endVS-startVS)); Console.WriteLine("ViewState = " + vs); CHAPTER 5 ■ REQUEST-RESPONSE TESTING152 6633c05.qxd 4/3/06 1:56 PM Page 152 int startEV = res.IndexOf("__EVENTVALIDATION", 0) + 49; int endEV = res.IndexOf("\"", startEV); string ev = res.Substring(startEV, (endEV-startEV)); Console.WriteLine("EventValidation = " + ev); Comments Before you can programmatically send an HTTP request to an ASP.NET Web application, you must determine the application’s ViewState value (and the application’s EventValidation value if you are running under ASP.NET 2.0). These are Base64-encoded values that represent the state of the Web application after each request-response round trip. This built-in mecha- nism is similar to how Web developers must maintain state in classic ASP Web pages by using HTML hidden input values. Although you can manually determine the ViewState value of an ASP.NET Web application by launching the application in a client such as IE and then choos- ing View ➤ Source, many times a better technique is to programmatically determine the ViewState value. The idea is to send an HTTP request for the Web application, retrieve the HTTP response into a string, and then parse the ViewState value out from the response string. After instantiating a WebClient object and attaching a stream to the HTTP response with the OpenRead() method, you fetch the entire response string into variable res using the ReadToEnd() method. The ViewState value is embedded in an HTML input tag: To extract the ViewState value in ASP.NET 1.1, you first get the location within the entire response string where the identifying "__VIEWSTATE" string occurs: int start = res.IndexOf("__VIEWSTATE", 0) + 20; If you add 20 to that index value, the index will point to the double-quote character just before the ViewState value. (Note: two underscores appear before VIEWSTATE). Next, you get an index pointing to the double-quote character that is just after the ViewState value: int end = res.IndexOf("\"", start); Notice you need to escape the double-quote character. After you have the indexes of the two double-quote characters that delimit the ViewState value, you can extract and save the ViewState value using the SubString() method: string vs = res.Substring(start, (end-start)); You can increase the modularity of your lightweight test automation code by recasting this solution into a method that accepts a URI string and returns a ViewState string: CHAPTER 5 ■ REQUEST-RESPONSE TESTING 153 6633c05.qxd 4/3/06 1:56 PM Page 153 private static string ViewState(string uri) { try { WebClient wc = new WebClient(); Stream st = wc.OpenRead(uri); StreamReader sr = new StreamReader(st); string res = sr.ReadToEnd(); sr.Close(); st.Close(); int start = res.IndexOf("__VIEWSTATE", 0) + 20; int end = res.IndexOf("\"", start); string vs = res.Substring(start, (end-start)); return vs; } catch { throw new Exception("Fatal error finding ViewState"); } } With this helper method, you can append a ViewState value to a POST data string: string uri = "http://server/path/WebForm.aspx"; string postData = "TextBox1=red&"; string vs = ViewState(uri); vs = HttpUtility.UrlEncode(vs); postData += "__VIEWSTATE=" + vs; Because a ViewState value may contain characters such as “&” that need to be URL encoded, you must apply the HttpUtility.UrlEncode() method to the ViewState value at some point. A design decision you’ll have to make is whether to apply UrlEncode() inside your helper method or outside the helper as you’ve done here. You must also be careful where you place the connecting “&” characters. The code in this solution is referred to as “brittle.” Brittle code makes assumptions about external dependencies and will break if those dependencies change. Notice the hard-coded 20 value in the code. The external dependency here is the way in which an ASP.NET 1.1 Web server returns the ViewState value to the calling client. The 20 assumes that exactly 20 charac- ters appear between the start of "__VIEWSTATE" and the double-quote character that appears before the ViewState value. If, for example, a future modification to ASP.NET results in charac- ters other than the two underscore characters in front of VIEWSTATE, then your test automation will break. Writing brittle code is almost always unacceptable in a development environment, but you can often get away with it in lightweight test automation. The idea is that lightweight automation is supposed to be quick and easy, which means you are willing to accept the con- sequences of brittle code—if the automation breaks, then you’ll just have to fix it. CHAPTER 5 ■ REQUEST-RESPONSE TESTING154 6633c05.qxd 4/3/06 1:56 PM Page 154 In ASP.NET 2.0, the ViewState value occurs 37 characters after the start of the "__VIEWSTATE" string because the 17-character string id="__VIEWSTATE" (with a preceding blank space) is added to the HTML hidden input element. Additionally, the new EventValidation value occurs 49 char- acters after the start of the "__EVENTVALIDATION" string. So, in an ASP.NET 2.0 environment, you can either write separate ViewState() and EventValidation() methods that programmatically fetch their values, or you can combine the logic into a single ViewStateEventValidation() method that returns a URL-encoded string containing both values like this: private static string ViewStateAndEventValidation(string uri) { try { WebClient wc = new WebClient(); Stream st = wc.OpenRead(uri); StreamReader sr = new StreamReader(st); string res = sr.ReadToEnd(); sr.Close(); st.Close(); int startVS = res.IndexOf("__VIEWSTATE", 0) + 37; int endVS = res.IndexOf("\"", startVS); string vs = res.Substring(startVS, (endVS-startVS)); vs = HttpUtility.UrlEncode(vs); int startEV = res.IndexOf("__EVENTVALIDATION", 0) + 49; int endEV = res.IndexOf("\"", startEV); string ev = res.Substring(startEV, (endEV-startEV)); ev = HttpUtility.UrlEncode(ev); return "&__VIEWSTATE=" + vs + "&__EVENTVALIDATION=" + ev; } catch { throw new Exception("Fatal error finding ViewState or EventValidation"); } } With this method, setting up POST data to an ASP.NET 2.0 Web application looks like this: string uri = "http://server/path/WebForm.aspx"; string postData = "TextBox1=red"; postData += ViewStateAndEventValidation(uri); Executing this code results in the variable postData having a value resembling "TextBox1=red&__VIEWSTATE=%2fQazwJ&__EVENTVALIDATION=%2fMaR4d8j=" where the actual values for ViewState and EventValidation depend on the particular Web AUT. CHAPTER 5 ■ REQUEST-RESPONSE TESTING 155 6633c05.qxd 4/3/06 1:56 PM Page 155 5.9 Dealing with CheckBox and RadioButtonList Controls Problem You want to send an HTTP request indicating a CheckBox or RadioButtonList control is checked. Design Modify the POST data string to include a name-value pair with the ID of the control you want to manipulate and the new value of the control. Solution string url = "http://server/path/WebForm.aspx"; string data = "CheckBox1=checked&RadioButtonList1=Alpha"; string viewstate = HttpUtility.UrlEncode("dDwtMTQ2MzgwNTQ2MD=="); data += "&__VIEWSTATE=" + viewstate; // send data to Web application here Comments Two of the most common ASP.NET Web application controls are CheckBox and RadioButtonList. Suppose you have this Web application:

CheckBox and RadioButtonList

CHAPTER 5 ■ REQUEST-RESPONSE TESTING156 6633c05.qxd 4/3/06 1:56 PM Page 156

Check or not:

Select one: Alpha Beta

My obervations:

This Web application checks to determine whether the CheckBox1 control is checked and whether the radio button with value Alpha or with Beta is selected, and prints a brief diagnos- tic message in a TextBox control. To programmatically send an HTTP request that corresponds to CheckBox1 being checked and Alpha being selected in RadioButtonList1, you can set up a POST data string like this: string data = "CheckBox1=checked&RadioButtonList1=Alpha"; data += "&TextBox1=empty&TextBox2=empty&Button1=clicked"; If you submit this data to the Web application, the HTTP response includes the following:

My obervations:

To indicate that CheckBox1 is unchecked, you send an HTTP request with a name-value pair that has no value component: "CheckBox1=". Similarly, to indicate that none of the RadioButtonList1 options are selected, you send "RadioButtonList1=". 5.10 Dealing with DropDownList Controls Problem You want to send an HTTP request indicating the selected value of a DropDownList control. CHAPTER 5 ■ REQUEST-RESPONSE TESTING 157 6633c05.qxd 4/3/06 1:56 PM Page 157 Design Modify the POST data string to include the ID of the DropDownList control and the selected value you want to indicate has been chosen in name-value form. Solution string url = "http://server/path/WebForm.aspx"; string data = "DropDownList1=SomeOption"; string viewstate = HttpUtility.UrlEncode("dDwtMTQ2MzgwNTQ2MD=="); data += "&__VIEWSTATE=" + viewstate; // send data to Web application here Comments A common Web control used in ASP.NET Web applications is the DropDownList control. For example, suppose you have this application:

DropDownList

Choose one: ant bug cat

You chose:

CHAPTER 5 ■ REQUEST-RESPONSE TESTING158 6633c05.qxd 4/3/06 1:56 PM Page 158 This application grabs the selected value on control DropDownList1 and displays that value in TextBox1. To programmatically send an HTTP request that corresponds to "bug" selected in DropDownList1, you can set up a POST data string like this: string data = "DropDownList1=bug&TextBox1=empty"; data += "&Button1=clicked"; If you submit this data to the Web application, the HTTP response includes the following:

You chose:

5.11 Determining a Request-Response Test Result Problem You want to determine whether a request-response test case passes or fails. Design Read the HTTP response a line at a time using the StreamReader.ReadLine() method. Parse each line of the HTTP response using the String.IndexOf() method for an identifying target string that unambiguously determines a pass or fail test result. Solution // set up url here // set up post data in byte array buffer here HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url); req.Method = "POST"; req.ContentType = "application/x-www-form-urlencoded"; req.ContentLength = buffer.Length; // write buffer into request stream here HttpWebResponse res = (HttpWebResponse)req.GetResponse(); // get response stream and associate to StreamReader sr here string expected = "someTargetString"; bool expectedFound = false; string line = null; while ((line = sr.ReadLine()) != null && !expectedFound) { if (line.IndexOf(expected) >= 0) { Console.WriteLine("expected value found"); expectedFound = true; } } CHAPTER 5 ■ REQUEST-RESPONSE TESTING 159 6633c05.qxd 4/3/06 1:56 PM Page 159 if (expectedFound) Console.WriteLine("Pass"); else Console.WriteLine("Fail"); Comments The essence of performing a request-response test of an ASP.NET Web application is to send an HTTP request to the application, retrieve the HTTP response, and examine the response for an identifying expected value. The following sets up the request, sends the request, and asso- ciates the response with a Stream object: HttpWebResponse res = (HttpWebResponse)req.GetResponse(); Stream resst = res.GetResponseStream(); // fetch HTTP response You then create a StreamReader object from the stream so that you access the response stream: StreamReader sr = new StreamReader(resst); You need to assign a target string to search for in the HTTP response: string expected = "someTargetString"; The expected string is some string that, if found in the HTTP response, will uniquely identify a correct response. This is not always easy to specify. For example, suppose you have a Web appli- cation with a DropDownList control that has options “red”, “blue”, and “green”. If the user selects “red” from the control, a message such as “apples are red” is displayed in a TextBox control. If you naively use the string “red” as an expected target, you will always get a pass result because “red” will be in the HTML

下载文档,方便阅读与编辑

文档的实际排版效果,会与网站的显示效果略有不同!!

需要 6 金币 [ 分享文档获得金币 ] 1 人已下载

下载文档

相关文档