Learning Perl SIXTH EDITION Learning Perl Randal L. Schwartz, brian d foy, and Tom Phoenix Beijing • Cambridge • Farnham • Köln • Sebastopol • Tokyo Learning Perl, Sixth Edition by Randal L. Schwartz, brian d foy, and Tom Phoenix Copyright © 2011 Randal L. Schwartz, brian d foy, and Tom Phoenix. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com. Editor: Simon St.Laurent Production Editor: Kristen Borg Copyeditor: Audrey Doyle Proofreader: Kiel Van Horn Indexer: John Bickelhaupt Cover Designer: Karen Montgomery Interior Designer: David Futato Illustrator: Robert Romano Printing History: November 1993: First Edition. July 1997: Second Edition. July 2001: Third Edition. July 2005: Fourth Edition. July 2008: Fifth Edition. June 2011: Sixth Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Learning Perl, the image of a llama, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information con- tained herein. ISBN: 978-1-449-30358-7 [LSI] 1308077187 Table of Contents Preface .................................................................... xiii 1. Introduction ........................................................... 1 Questions and Answers 1 Is This the Right Book for You? 1 Why Are There So Many Footnotes? 2 What About the Exercises and Their Answers? 3 What Do Those Numbers Mean at the Start of the Exercise? 4 What If I’m a Perl Course Instructor? 4 What Does “Perl” Stand For? 4 Why Did Larry Create Perl? 5 Why Didn’t Larry Just Use Some Other Language? 5 Is Perl Easy or Hard? 6 How Did Perl Get to Be So Popular? 7 What’s Happening with Perl Now? 7 What’s Perl Really Good For? 8 What Is Perl Not Good For? 8 How Can I Get Perl? 9 What Is CPAN? 10 How Can I Get Support for Perl? 10 Are There Any Other Kinds of Support? 10 What If I Find a Bug in Perl? 12 How Do I Make a Perl Program? 12 A Simple Program 13 What’s Inside That Program? 15 How Do I Compile My Perl Program? 16 A Whirlwind Tour of Perl 17 Exercises 18 2. Scalar Data ........................................................... 21 Numbers 21 v All Numbers Have the Same Format Internally 22 Floating-Point Literals 22 Integer Literals 22 Nondecimal Integer Literals 23 Numeric Operators 23 Strings 24 Single-Quoted String Literals 25 Double-Quoted String Literals 25 String Operators 26 Automatic Conversion Between Numbers and Strings 27 Perl’s Built-in Warnings 28 Scalar Variables 29 Choosing Good Variable Names 30 Scalar Assignment 31 Binary Assignment Operators 31 Output with print 32 Interpolation of Scalar Variables into Strings 32 Creating Characters by Code Point 34 Operator Precedence and Associativity 34 Comparison Operators 36 The if Control Structure 37 Boolean Values 38 Getting User Input 39 The chomp Operator 39 The while Control Structure 40 The undef Value 41 The defined Function 42 Exercises 42 3. Lists and Arrays ........................................................ 43 Accessing Elements of an Array 44 Special Array Indices 45 List Literals 46 The qw Shortcut 46 List Assignment 48 The pop and push Operators 49 The shift and unshift Operators 50 The splice Operator 50 Interpolating Arrays into Strings 51 The foreach Control Structure 53 Perl’s Favorite Default: $_ 54 The reverse Operator 54 The sort Operator 54 vi | Table of Contents The each Operator 55 Scalar and List Context 55 Using List-Producing Expressions in Scalar Context 57 Using Scalar-Producing Expressions in List Context 58 Forcing Scalar Context 59
in List Context 59 Exercises 60 4. Subroutines ........................................................... 63 Defining a Subroutine 63 Invoking a Subroutine 64 Return Values 64 Arguments 66 Private Variables in Subroutines 68 Variable-Length Parameter Lists 69 A Better &max Routine 69 Empty Parameter Lists 70 Notes on Lexical (my) Variables 71 The use strict Pragma 72 The return Operator 74 Omitting the Ampersand 74 Non-Scalar Return Values 76 Persistent, Private Variables 76 Exercises 78 5. Input and Output ...................................................... 81 Input from Standard Input 81 Input from the Diamond Operator 83 The Invocation Arguments 85 Output to Standard Output 86 Formatted Output with printf 89 Arrays and printf 90 Filehandles 91 Opening a Filehandle 93 Binmoding Filehandles 95 Bad Filehandles 96 Closing a Filehandle 96 Fatal Errors with die 97 Warning Messages with warn 99 Automatically die-ing 99 Using Filehandles 100 Changing the Default Output Filehandle 100 Reopening a Standard Filehandle 101 Table of Contents | vii Output with say 102 Filehandles in a Scalar 103 Exercises 104 6. Hashes .............................................................. 107 What Is a Hash? 107 Why Use a Hash? 109 Hash Element Access 110 The Hash As a Whole 112 Hash Assignment 113 The Big Arrow 114 Hash Functions 115 The keys and values Functions 115 The each Function 116 Typical Use of a Hash 118 The exists Function 118 The delete Function 118 Hash Element Interpolation 119 The %ENV hash 119 Exercises 120 7. In the World of Regular Expressions ...................................... 121 What Are Regular Expressions? 121 Using Simple Patterns 122 Unicode Properties 123 About Metacharacters 123 Simple Quantifiers 124 Grouping in Patterns 125 Alternatives 127 Character Classes 128 Character Class Shortcuts 129 Negating the Shortcuts 131 Exercises 131 8. Matching with Regular Expressions ...................................... 133 Matches with m// 133 Match Modifiers 134 Case-Insensitive Matching with /i 134 Matching Any Character with /s 134 Adding Whitespace with /x 135 Combining Option Modifiers 135 Choosing a Character Interpretation 136 Other Options 138 viii | Table of Contents Anchors 138 Word Anchors 140 The Binding Operator =~ 141 Interpolating into Patterns 142 The Match Variables 143 The Persistence of Captures 144 Noncapturing Parentheses 145 Named Captures 146 The Automatic Match Variables 147 General Quantifiers 149 Precedence 150 Examples of Precedence 151 And There’s More 152 A Pattern Test Program 152 Exercises 153 9. Processing Text with Regular Expressions ................................. 155 Substitutions with s/// 155 Global Replacements with /g 156 Different Delimiters 157 Substitution Modifiers 157 The Binding Operator 157 Nondestructive Substitutions 157 Case Shifting 158 The split Operator 159 The join Function 160 m// in List Context 161 More Powerful Regular Expressions 161 Nongreedy Quantifiers 162 Matching Multiple-Line Text 164 Updating Many Files 164 In-Place Editing from the Command Line 166 Exercises 168 10. More Control Structures ................................................ 169 The unless Control Structure 169 The else Clause with unless 170 The until Control Structure 170 Expression Modifiers 171 The Naked Block Control Structure 172 The elsif Clause 173 Autoincrement and Autodecrement 174 The Value of Autoincrement 175 Table of Contents | ix The for Control Structure 176 The Secret Connection Between foreach and for 178 Loop Controls 178 The last Operator 179 The next Operator 179 The redo Operator 181 Labeled Blocks 182 The Conditional Operator ?: 182 Logical Operators 184 The Value of a Short Circuit Operator 184 The defined-or Operator 185 Control Structures Using Partial-Evaluation Operators 186 Exercises 188 11. Perl Modules ......................................................... 189 Finding Modules 189 Installing Modules 190 Using Your Own Directories 191 Using Simple Modules 193 The File::Basename Module 194 Using Only Some Functions from a Module 195 The File::Spec Module 196 Path::Class 197 CGI.pm 198 Databases and DBI 199 Dates and Times 200 Exercises 201 12. File Tests ............................................................ 203 File Test Operators 203 Testing Several Attributes of the Same File 207 Stacked File Test Operators 208 The stat and lstat Functions 210 The localtime Function 211 Bitwise Operators 212 Using Bitstrings 213 Exercises 214 13. Directory Operations .................................................. 215 Moving Around the Directory Tree 215 Globbing 216 An Alternate Syntax for Globbing 217 Directory Handles 218 x | Table of Contents Recursive Directory Listing 220 Manipulating Files and Directories 221 Removing Files 221 Renaming Files 223 Links and Files 224 Making and Removing Directories 229 Modifying Permissions 230 Changing Ownership 231 Changing Timestamps 231 Exercises 232 14. Strings and Sorting .................................................... 235 Finding a Substring with index 235 Manipulating a Substring with substr 236 Formatting Data with sprintf 238 Using sprintf with “Money Numbers” 238 Interpreting Non-Decimal Numerals 240 Advanced Sorting 240 Sorting a Hash by Value 244 Sorting by Multiple Keys 245 Exercises 246 15. Smart Matching and given-when ........................................ 247 The Smart Match Operator 247 Smart Match Precedence 250 The given Statement 251 Dumb Matching 254 Using when with Many Items 256 Exercises 257 16. Process Management .................................................. 259 The system Function 259 Avoiding the Shell 261 The Environment Variables 263 The exec Function 263 Using Backquotes to Capture Output 264 Using Backquotes in a List Context 267 External Processes with IPC::System::Simple 268 Processes as Filehandles 269 Getting Down and Dirty with Fork 271 Sending and Receiving Signals 272 Exercises 274 Table of Contents | xi 17. Some Advanced Perl Techniques . ....................................... 277 Slices 277 Array Slice 279 Hash Slice 281 Trapping Errors 282 Using eval 282 More Advanced Error Handling 286 autodie 288 Picking Items from a List with grep 289 Transforming Items from a List with map 290 Fancier List Utilities 291 Exercises 293 A. Exercise Answers . .................................................... 295 B. Beyond the Llama . ................................................... 331 C. A Unicode Primer . .................................................... 343 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 xii | Table of Contents Preface Welcome to the sixth edition of Learning Perl, updated for Perl 5.14 and its latest features. This book is still good even if you are still using Perl 5.8 (although, it’s been a long time since it was released; have you thought about upgrading?). If you’re looking for the best way to spend your first 30 to 45 hours with the Perl programming language, you’ve found it. In the pages that follow, you’ll find a carefully paced introduction to the language that is the workhorse of the Internet, as well as the language of choice for system administrators, web hackers, and casual programmers around the world. We can’t give you all of Perl in just a few hours. The books that promise that are probably fibbing a bit. Instead, we’ve carefully selected a useful subset of Perl for you to learn, good for programs from one to 128 lines long, which end up being about 90% of the programs in use out there. And when you’re ready to go on, you can get Inter- mediate Perl, which picks up where this book leaves off. We’ve also included a number of pointers for further education. Each chapter is small enough so you can read it in an hour or two. Each chapter ends with a series of exercises to help you practice what you’ve just learned, with the answers in Appendix A for your reference. Thus, this book is ideally suited for a classroom “Introduction to Perl” course. We know this directly because the material for this book was lifted almost word-for-word from our flagship “Learning Perl” course, delivered to thousands of students around the world. However, we’ve designed the book for self- study as well. Perl lives as the “toolbox for Unix,” but you don’t have to be a Unix guru, or even a Unix user, to read this book. Unless otherwise noted, everything we’re saying applies equally well to Windows ActivePerl from ActiveState and pretty much every other modern implementation of Perl. Although you don’t need to know a single thing about Perl to begin reading this book, we recommend that you already have familiarity with basic programming concepts such as variables, loops, subroutines, and arrays, and the all-important “editing a source code file with your favorite text editor.” We don’t spend any time trying to explain those concepts. Although we’re pleased that we’ve had many reports of people xiii successfully picking up Learning Perl and grasping Perl as their first programming lan- guage, of course we can’t promise the same results for everyone. Typographical Conventions The following font conventions are used in this book: Constant width is used for method names, function names, variables, and attributes. It is also used for code examples. Constant width bold is used to indicate user input. Constant width italic is used to indicate a replaceable item in code (e.g., filename, where you are sup- posed to substitute an actual filename). Italic is used for filenames, URLs, hostnames, commands in text, important words on first mention, and emphasis. Footnotes are used to attach parenthetical notes that you should not read on your first (or perhaps second or third) reading of this book. Sometimes lies are spoken to simplify the presentation, and the footnotes restore the lie to truth. Often the material in the footnote will be advanced material not even mentioned anywhere else in the book. [2] at the start of an exercise’s text represents our (very rough) estimate of how many minutes you can expect to spend on that particular exercise. Code Examples This book is here to help you get your job done. You are invited to copy the code in the book and adapt it for your own needs. Rather than copying by hand, however, we encourage you to download the code from http://www.learning-perl.com. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of ex- amples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a sig- nificant amount of example code from this book into your product’s documentation does require permission. xiv | Preface We appreciate, but do not require, attribution. An attribution usually includes the title, authors, publisher, and ISBN. For example: “Learning Perl, 6th edition, by Randal L. Schwartz, brian d foy, and Tom Phoenix (O’Reilly). Copyright 2011 Randal L. Schwartz, brian d foy, and Tom Phoenix, 978-1-449-30358-7.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com Safari® Books Online Safari Books Online is an on-demand digital library that lets you easily search over 7,500 technology and creative reference books and videos to find the answers you need quickly. With a subscription, you can read any page and watch any video from our library online. Read books on your cell phone and mobile devices. Access new titles before they are available for print, and get exclusive access to manuscripts in development and post feedback for the authors. Copy and paste code samples, organize your favorites, down- load chapters, bookmark key sections, create notes, print out pages, and benefit from tons of other time-saving features. O’Reilly Media has uploaded this book to the Safari Books Online service. To have full digital access to this book and others on similar topics from O’Reilly and other pub- lishers, sign up for free at http://my.safaribooksonline.com. How to Contact Us We have tested and verified all the information in this book to the best of our abilities, but you may find that features have changed or that we have let errors slip through the production of the book. Please let us know of any errors that you find, as well as sug- gestions for future editions, by writing to: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for the book, where we’ll list examples, errata, and any additional information. It also offers a downloadable set of text files (and a couple of Perl pro- grams) that are useful, but not required, when doing some of the exercises. You can access this page at: http://www.learning-perl.com Preface | xv or go to the O’Reilly page at: http://oreilly.com/catalog/0636920018452/ To comment or ask technical questions about this book, send email to: bookquestions@oreilly.com For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia History of This Book For the curious, here’s how Randal tells the story of how this book came about: After I had finished the first Programming perl book with Larry Wall (in 1991), I was approached by Taos Mountain Software in Silicon Valley to produce a training course. This included having me deliver the first dozen or so courses and train their staff to continue offering the course. I wrote the course for them* and delivered it for them as promised. On the third or fourth delivery of that course (in late 1991), someone came up to me and said, “You know, I really like Programming perl, but the way the material is pre- sented in this course is so much easier to follow—you oughta write a book like this course.” It sounded like an opportunity to me, so I started thinking about it. I wrote to Tim O’Reilly with a proposal based on an outline that was similar to the course I was presenting for Taos—although I had rearranged and modified a few of the chapters based on observations in the classroom. I think that was my fastest proposal acceptance in history—I got a message from Tim within fifteen minutes, saying “We’ve been waiting for you to pitch a second book—Programming perl is selling like gang- busters.” That started the effort over the next 18 months to finish the first edition of Learning Perl. During that time, I was starting to see an opportunity to teach Perl classes outside Silicon Valley,† so I created a class based on the text I was writing for Learning Perl. I gave a dozen classes for various clients (including my primary contractor, Intel Oregon), and used the feedback to fine-tune the book draft even further. * In the contract, I retained the rights to the exercises, hoping someday to reuse them in some other way, like in the magazine columns I was writing at the time. The exercises are the only things that leapt from the Taos course to the book. † My Taos contract had a no-compete clause, so I had to stay out of Silicon Valley with any similar courses, which I respected for many years. xvi | Preface The first edition hit the streets on the first day of November 1993,‡ and became a smashing success, frequently even outpacing Programming perl book sales. The back-cover jacket of the first book said “written by a leading Perl trainer.” Well, that became a self-fulfilling prophecy. Within a few months, I was starting to get email from all over the United States asking me to teach at their site. In the following seven years, my company became the leading worldwide on-site Perl training company, and I had personally racked up (literally) a million frequent-flier miles. It didn’t hurt that the Web started taking off about then, and the webmasters and webmistresses picked Perl as the language of choice for content management, interaction through CGI, and maintenance. For two years, I worked closely with Tom Phoenix in his role as lead trainer and content manager for Stonehenge, giving him charter to experiment with the “Llama” course by moving things around and breaking things up. When we had come up with what we thought was the best major revision of the course, I contacted O’Reilly and said, “It’s time for a new book!” And that became the third edition. Two years after writing the third edition of the Llama, Tom and I decided it was time to push our follow-on “advanced” course out into the world as a book, for people writing programs that are “100 to 10,000 lines of code.” And together we created the first Alpaca book, released in 2003. But fellow instructor brian d foy was just getting back from the conflict in the Gulf, and had noticed that we could use some rewriting in both books, because our courseware still needed to track the changing needs of the typical student. So, he pitched the idea to O’Reilly to take on rewriting both the Llama and the Alpaca one final time before Perl 6 (we hope). This edition of the Llama reflects those changes. brian has really been the lead writer here, working with my occasional guidance, and has done a brilliant job of the usual “herding cats” that a multiple-writer team generally feels like. On December 18, 2007, the Perl 5 Porters released Perl 5.10, a significant new version of Perl with several new features. The previous version, 5.8, had focused on the un- derpinnings of Perl and its Unicode support. The latest version, starting from the stable 5.8 foundation, was able to add completely new features, some of which it borrowed from the development of Perl 6 (not yet released). Some of these features, such as named captures in regular expressions, are much better than the old ways of doing things, thus perfect for Perl beginners. We hadn’t thought about a fifth edition of this book, but Perl 5.10 was so much better that we couldn’t resist. Since then, Perl has been under constant improvement and is keeping a regular release cycle. We didn’t have a chance to update this book for Perl 5.12 because development proceeded too quickly. We’re pleased to offer this update for Perl 5.14, and are amazed that there’s now a sixth edition. ‡ I remember that date very well, because it was also the day I was arrested at my home for computer-related- activities around my Intel contract, a series of felony charges for which I was later convicted. Preface | xvii Changes from the Previous Edition The text is updated for the latest version, Perl 5.14, and some of the code only works with that version. We note in the text when we are talking about a Perl 5.14 feature, and we mark those code sections with a special use statement that ensures you’re using the right version: use 5.014; # this script requires Perl 5.14 or greater If you don’t see that use 5.014 in a code example (or a similar statement with a different version), it should work all the way back to Perl 5.8. To see which version of Perl you have, try the -v command-line switch: $ perl -v Here’s some of the new features from Perl 5.14 that we cover, and where appropriate, we still show you the old ways of doing the same thing: • We include Unicode examples and features where appropriate. If you haven’t star- ted playing with Unicode, we include a primer in Appendix C. You have to bite the bullet sometime, so it might as well be now. You’ll see Unicode throughout the book, most notably in the chapters on Scalars (Chapter 2), Input/Output (Chapter 5), and Sorting (Chapter 14). • There is more information in the regular expression chapters, covering the new features from Perl 5.14 to deal with Unicode case-folding. The regular expression operators have new /a, /u, and /l switches. We now cover matching by Unicode properties with the \p{} and \P{} regular expression features. • Perl 5.14 adds a nondestructive substitution operator (Chapter 9), which turns out to be really handy. • Smart matching and given-when has mutated a bit since their introduction in Perl 5.10, so we update Chapter 15 to cover the new rules. • We updated and expanded Perl Modules (Chapter 11) to include the latest news, including the zero-conf cpanm tool. We add some more module examples as well. • Some of the items previously in Appendix B, the advanced-but-not-demonstrated features, move into the main text. Notably, that includes the fat arrow => moving into Hashes (Chapter 6) and splice moving into Lists and Arrays (Chapter 3). Acknowledgments From Randal I want to thank the Stonehenge trainers past and present (Joseph Hall, Tom Phoenix, Chip Salzenberg, brian d foy, and Tad McClellan) for their willingness to go out and teach in front of classrooms week after week and to come back with their notes about xviii | Preface what’s working (and what’s not), so we could fine-tune the material for this book. I especially want to single out my co-author and business associate, Tom Phoenix, for having spent many, many hours working to improve Stonehenge’s Llama course and to provide the wonderful core text for most of this book. And brian d foy for being the lead writer beginning with the fourth edition, and taking that eternal to-do item out of my inbox so that it would finally happen. I also want to thank everyone at O’Reilly, especially our very patient editor and overseer for previous editions, Allison Randal (no relation, but she has a nicely spelled last name), current editor Simon St.Laurent, and Tim O’Reilly himself for taking a chance on me in the first place with the Camel and Llama books. I am also absolutely indebted to the thousands of people who have purchased the past editions of the Llama so that I could use the money to stay “off the streets and out of jail,” and to those students in my classrooms who have trained me to be a better trainer, and to the stunning array of Fortune 1000 clients who have purchased our classes in the past and will continue to do so into the future. As always, a special thanks to Lyle and Jack, for teaching me nearly everything I know about writing. I won’t ever forget you guys. From Tom I’ve got to echo Randal’s thanks to everyone at O’Reilly. For the third edition of this book, Linda Mui was our editor, and I still thank her, for her patience in pointing out which jokes and footnotes were most excessive, while pointing out that she is in no way to blame for the ones that remain. Both she and Randal have guided me through the process of writing, and I am grateful. In a previous edition, Allison Randal took charge; now Simon St.Laurent has become the editor. My thanks go to each of them in recognition of their unique contributions. And another echo with regard to Randal and the other Stonehenge trainers, who hardly ever complained when I unexpectedly updated the course materials to try out a new teaching technique. You folks have contributed many different viewpoints on teaching methods that I would never have seen. For many years, I worked at the Oregon Museum of Science and Industry (OMSI), and I’d like to thank the folks there for letting me hone my teaching skills as I learned to build a joke or two into every activity, explosion, or dissection. To the many folks on Usenet who have given me your appreciation and encouragement for my contributions there, thanks. As always, I hope this helps. Also to my many students, who have shown me with their questions (and befuddled looks) when I needed to try a new way of expressing a concept. I hope that the present edition helps to relieve any remaining puzzlement. Preface | xix Of course, deep thanks are due especially to my co-author, Randal, for giving me the freedom to try various ways of presenting the material both in the classroom and here in the book, as well as for the push to make this material into a book in the first place. And without fail, I must say that I am indeed inspired by your ongoing work to ensure that no one else becomes ensnared by the legal troubles that have stolen so much of your time and energy; you’re a fine example. To my wife, Jenna, thanks for being a cat person, and everything thereafter. From brian I have to thank Randal first, since I learned Perl from the first edition of this book, and then had to learn it again when he asked me to start teaching for Stonehenge in 1998. Teaching is often the best way to learn. Since then, Randal has mentored me not only in Perl but several other things he thought I needed to learn—like the time he decided that we could use Smalltalk instead of Perl for a demonstration at a web conference. I’m always amazed at the breadth of his knowledge. He’s the one who told me to start writing about Perl. Now I’m helping out on the book where I started. I’m honored, Randal. I probably only actually saw Tom Phoenix for less than two weeks in the entire time I worked for Stonehenge, but I had been teaching his version of Stonehenge’s Learning Perl course for years. That version turned into the third edition of this book. By teaching Tom’s new version, I found new ways to explain almost everything, and learned even more corners of Perl. When I convinced Randal that I should help out on the Llama update, I was anointed as the maker of the proposal to the publisher, the keeper of the outline, and the version control wrangler. Our editor, Allison Randal, helped me get all of those set up and endured my frequent emails without complaining. After Allison went on to other things, Simon St. Laurent has been extremely helpful in the role of editor and inside guy at O’Reilly, patiently waiting for the right phase of the moon to suggest another update. From All of Us Thanks to our reviewers, David H. Adler, Alan Haggai Alavi, Andy Armstrong, Dave Cross, Chris Devers, Paul Fenwick, Stephen B. Jenkins, Matthew Musgrove, Jacinta Richardson, Steve Peters, Peter Scott, Wil Wheaton, and Karl Williamson, for providing comments on the draft of this book. Thanks also to our many students who have let us know what parts of the course material have needed improvement over the years. It’s because of you that we’re all so proud of it today. xx | Preface Thanks to the many Perl Mongers who have made us feel at home as we’ve visited your cities. Let’s do it again sometime. And finally, our sincerest thanks to our friend Larry Wall, for having the wisdom to share his really cool and powerful toys with the rest of the world so that we can all get our work done just a little bit faster, easier, and with more fun. Preface | xxi CHAPTER 1 Introduction Welcome to the Llama book! This is the sixth edition of a book that has been enjoyed by over half a million readers since 1993. At least, we hope they’ve enjoyed it. It’s a sure thing that we enjoyed writing it.* Questions and Answers You probably have some questions about Perl, and maybe even some about this book; especially if you’ve already flipped through it to see what’s coming. So, we’ll use this chapter to answer them, including how to find answers that we don’t provide. Is This the Right Book for You? If you’re anything like us, you probably didn’t get to browse this book before you bought it. As we finish up this edition, the bookstore Borders is closing many of its stores and other booksellers aren’t doing much better. You might be reading this book in a digital form that you downloaded, or as HTML in Safari Books Online. How can you find out if this book is the one you want to buy if you can’t look at it first? How can we warn you off if you need to buy the book to read this paragraph? This is not a reference book. It’s a tutorial on the very basics of Perl, which is just enough for you to create simple programs mostly for your own use. We don’t cover every detail of every topic, and we spread out some of the topics over several chapters so you pick up concepts as you need them. * To be sure, the first edition was written by Randal L. Schwartz, the second by Randal and Tom Christiansen, then one by Randal and Tom Phoenix, and now three by Randal, Tom Phoenix, and brian d foy. So, whenever we say “we” in this edition, we mean that last group. Now, if you’re wondering how we can say that we’ve enjoyed writing it (in the past tense) when we’re still on the first page, that’s easy: we started at the end, and worked our way backward. It sounds like a strange way to do it, we know. But, honestly, once we finished writing the index, the rest was hardly any trouble at all. 1 Our intended readers are people who know at least a little bit about programming and just need to learn Perl. We assume that you have at least some background in using a terminal, editing files, and running programs—just not Perl programs. You already know about variables and subroutines and the like, but you just need to see how Perl does it. This doesn’t mean that the absolute beginner, having never touched a terminal program or written a single line of code, will be completely lost. You might not catch everything we say the first time you go through the book, but many beginners have used the book with only minor frustrations. The trick is to not worry about everything you might be missing and to focus on just the core concepts we present. You might take a little longer than an experienced programmer, but you have to start somewhere. And, this shouldn’t be the only Perl book you ever read. It’s just a tutorial. It’s not comprehensive. It gets you started in the right direction so you can go on to our other books, Intermediate Perl (at the time of this writing, the second edition is forthcoming) and Mastering Perl, when you are ready. The definitive reference for Perl is Program- ming Perl, also known as the “Camel book.” We should also note that even though this book covers up to Perl 5.14, it’s still useful even if you have an earlier version. You might miss out on some of the cool new features, but you’ll still learn how to use basic Perl. The least recent version that we’ll think about, however, is Perl 5.8, even though that was released almost 10 years ago. Why Are There So Many Footnotes? Thank you for noticing. There are a lot of footnotes in this book. Ignore them. They’re needed because Perl is chock-full of exceptions to its rules. This is a good thing, as real life is chock-full of exceptions to rules. But it means that we can’t honestly say, “The fizzbin operator frobnicates the hoozi- static variables” without a footnote giving the exceptions.† We’re pretty honest, so we have to write the footnotes. But you can be honest without reading them. (It’s funny how that works out.) The footnotes are extra information that you don’t need for the core concepts. Many of the exceptions have to do with portability. Perl began on Unix systems, and it still has deep roots in Unix. But wherever possible, we’ve tried to show when some- thing may behave unexpectedly, whether that’s because it’s running on a non-Unix system or for another reason. We hope that readers who know nothing about Unix will nevertheless find this book a good introduction to Perl. (And they’ll learn a little about Unix along the way, at no extra charge.) † Except on Tuesdays, during a power outage, when you hold your elbow at a funny angle during the equinox, or when use integer is in effect inside a loop block being called by a prototyped subroutine prior to Perl version 5.12. 2 | Chapter 1: Introduction And many of the other exceptions have to do with the old “80/20” rule. By that we mean that 80% of the behavior of Perl can be described in 20% of the documentation, and the other 20% of the behavior takes up the other 80% of the documentation. So, to keep this book small, we’ll talk about the most common, easy-to-talk-about behavior in the main text, and hint in the direction of the other stuff in the footnotes (which are in a smaller font, so we can say more in the same space).‡ Once you’ve read the book all the way through without reading the footnotes, you’ll probably want to look back at some sections for reference. At that point, or if you become unbearably curious along the way, go ahead and read the notes. A lot of them are just computer jokes anyway. What About the Exercises and Their Answers? The exercises are at the end of each chapter because, between the three of us, we’ve presented this same course material to several thousand students.§ We have carefully crafted these exercises to give you the chance to make mistakes as well. It’s not that we want you to make mistakes, but you need to have the chance. That’s because you are going to make most of these mistakes during your Perl programming career, and it may as well be now. Any mistake that you make while reading this book you won’t make again when you’re writing a program on a deadline. And we’re always here to help you out if something goes wrong, in the form of Appendix A, which has our answers for each exercise and a little text to go with it, explaining the mistakes you made and a few you didn’t. Check out the answers when you’re done with the exercises. Try not to peek at the answer until you’ve given the problem a good try, though. You’ll learn better if you figure it out rather than read about it. Don’t knock your head re- peatedly against the wall if you don’t figure out a solution: move on to the next chapter and don’t worry too much about it. Even if you never make any mistakes, you should look at the answers when you’re done; the accompanying text will point out some details of the program that might not be obvious at first. If you want additional exercises, check out the Learning Perl Student Workbook, which adds several exercises for each chapter. ‡ We even discussed doing the entire book as a footnote to save the page count, but footnotes on footnotes started to get a bit crazy. § Not all at once. Questions and Answers | 3 What Do Those Numbers Mean at the Start of the Exercise? Each exercise has a number in square brackets in front of the exercise text, looking something like this: 1. [2] What does the number 2 inside square brackets mean, when it appears at the start of an exercise’s text? That number is our (very rough) estimate of how many minutes you can expect to spend on that particular exercise. It’s rough, so don’t be too surprised if you’re all done (with writing, testing, and debugging) in half that time, or not done in twice that long. On the other hand, if you’re really stuck, we won’t tell anyone that you peeked at Appen- dix A to see what our answer looked like. What If I’m a Perl Course Instructor? If you’re a Perl instructor who has decided to use this as your textbook (as many have over the years), you should know that we’ve tried to make each set of exercises short enough that most students could do the whole set in 45 minutes to an hour, with a little time left over for a break. Some chapters’ exercises should be quicker, and some may take longer. That’s because, once we had written all of those little numbers in square brackets, we discovered that we don’t know how to add (luckily we know how to make computers do it for us). We also have a companion book, the Learning Perl Student Workbook, which has ad- ditional exercises for each chapter. If you get the version of the workbook for the fourth edition, you will have to adjust the chapter order because in this edition, we have added a chapter and moved another. What Does “Perl” Stand For? Perl is sometimes called the “Practical Extraction and Report Language,” although it has also been called a “Pathologically Eclectic Rubbish Lister,” among other expan- sions. It’s actually a backronym, not an acronym, since Larry Wall, Perl’s creator, came up with the name first and the expansion later. That’s why “Perl” isn’t in all caps. There’s no point in arguing which expansion is correct: Larry endorses both. You may also see “perl” with a lowercase p in some writing. In general, “Perl” with a capital P refers to the language and “perl” with a lowercase p refers to the actual inter- preter that compiles and runs your programs. In the house style, we write the names of programs like perl. 4 | Chapter 1: Introduction Why Did Larry Create Perl? Larry created Perl in the mid-1980s when he was trying to produce some reports from a Usenet-news-like hierarchy of files for a bug-reporting system, and awk ran out of steam. Larry, being the lazy programmer that he is,‖ decided to overkill the problem with a general-purpose tool that he could use in at least one other place. The result was Perl version zero. Why Didn’t Larry Just Use Some Other Language? There’s no shortage of computer languages, is there? But, at the time, Larry didn’t see anything that really met his needs. If one of the other languages of today had been available back then, perhaps Larry would have used one of those. He needed something with the quickness of coding available in shell or awk programming, and with some of the power of more advanced tools like grep, cut, sort, and sed,# without having to resort to a language like C. Perl tries to fill the gap between low-level programming (such as in C or C++ or as- sembly) and high-level programming (such as “shell” programming). Low-level pro- gramming is usually hard to write and ugly, but fast and unlimited; it’s hard to beat the speed of a well-written low-level program on a given machine. And there’s not much you can’t do there. High-level programming, at the other extreme, tends to be slow, hard, ugly, and limited; there are many things you can’t do at all with the shell or batch programming if there’s no command on your system that provides the needed func- tionality. Perl is easy, nearly unlimited, mostly fast, and kind of ugly. Let’s take another look at those four claims we just made about Perl. First, Perl is easy. As you’ll see, though, this means it’s easy to use. It’s not especially easy to learn. If you drive a car, you spent many weeks or months learning how, and now it’s easy to drive. When you’ve been programming Perl for about as many hours as it took you to learn to drive, Perl will be easy for you.* Perl is nearly unlimited. There are very few things you can’t do with Perl. You wouldn’t want to write an interrupt-microkernel-level device driver in Perl (even though that’s been done), but most things that ordinary folks need most of the time are good tasks for Perl, from quick little one-off programs to major industrial-strength applications. ‖ We’re not insulting Larry by saying he’s lazy; laziness is a virtue. The wheelbarrow was invented by someone who was too lazy to carry things; writing was invented by someone who was too lazy to memorize; Perl was invented by someone who was too lazy to get the job done without inventing a whole new computer language. #Don’t worry if you don’t know what these are. All that matters is that they were the programs Larry had in his Unix toolbox, but they weren’t up to the tasks at hand. * But we hope you’ll crash less often with the car. What Does “Perl” Stand For? | 5 Perl is mostly fast. That’s because nobody is developing Perl who doesn’t also use it— so we all want it to be fast. If someone wants to add a feature that would be really cool but would slow down other programs, Larry is almost certain to refuse the new feature until we find a way to make it quick enough. Perl is kind of ugly. This is true. The symbol of Perl has become the camel, from the cover of the venerable Camel book (also known as Programming Perl), a cousin of this book’s Llama (and her sister, the Alpaca). Camels are kind of ugly, too. But they work hard, even in tough conditions. Camels are there to get the job done despite all diffi- culties, even when they look bad and smell worse and sometimes spit at you. Perl is a little like that. Is Perl Easy or Hard? Perl is easy to use, but sometimes hard to learn. This is a generalization, of course. In designing Perl, Larry made many trade-offs. When he’s had the chance to make some- thing easier for the programmer at the expense of being more difficult for the student, he’s decided in the programmer’s favor nearly every time. That’s because you’ll learn Perl only once, but you’ll use it again and again.† Perl has any number of conveniences that let the programmer save time. For example, most functions will have a default; frequently, the default is the way that you’ll want to use the function. So you’ll see lines of Perl code like these:‡ while (<>) { chomp; print join("\t", (split /:/)[0, 2, 1, 5] ), "\n"; } Written out in full, without using Perl’s defaults and shortcuts, that snippet would be roughly ten or twelve times longer, so it would take much longer to read and write. It would be harder to maintain and debug, too, with more variables. If you already know some Perl, and you don’t see the variables in that code, that’s part of the point. They’re all being used by default. But to have this ease at the programmer’s tasks means paying the price when you’re learning; you have to learn those defaults and shortcuts. A good analogy is the proper and frequent use of contractions in English. Sure, “will not” means the same as “won’t.” But most people say “won’t” rather than “will not” because it saves time, and because everybody knows it and it makes sense. Similarly, Perl’s “contractions” abbreviate common “phrases” so that they can be “spoken” quicker and understood by the maintainer as a single idiom, rather than a series of unrelated steps. † If you’re going to use a programming language for only a few minutes each week or month, you’d prefer one that is easier to learn, since you’ll have forgotten nearly all of it from one use to the next. Perl is for people who are programmers for at least twenty minutes per day, and probably most of that in Perl. ‡ We won’t explain it all here, but this example pulls some data from an input file or files in one format and writes some of it out in another format. All of its features are covered in this book. 6 | Chapter 1: Introduction Once you become familiar with Perl, you may find yourself spending less time trying to get shell quoting (or C declarations) right, and more time surfing the Web because Perl is a great tool for leverage. Perl’s concise constructs allow you to create (with minimal fuss) some very cool one-up solutions or general tools. Also, you can drag those tools along to your next job because Perl is highly portable and readily available, so you’ll have even more time to surf. Perl is a very high-level language. That means that the code is quite dense; a Perl pro- gram may be around a quarter to three-quarters as long as the corresponding program in C. This makes Perl faster to write, faster to read, faster to debug, and faster to main- tain. It doesn’t take much programming before you realize that, when the entire sub- routine is small enough to fit onscreen all at once, you don’t have to keep scrolling back and forth to see what’s going on. Also, since the number of bugs in a program is roughly proportional to the length of the source code§ (rather than being proportional to the program’s functionality), the shorter source in Perl will mean fewer bugs on average. Like any language, Perl can be “write-only”—it’s possible to write programs that are impossible to read. But with proper care, you can avoid this common accusation. Yes, sometimes Perl looks like CPAN line-noise to the uninitiated, but to the seasoned Perl programmer, it looks like the notes of a grand symphony. If you follow the guidelines of this book, your programs should be easy to read and easy to maintain, and they probably won’t win The Obfuscated Perl Contest. How Did Perl Get to Be So Popular? After playing with Perl a bit, adding stuff here and there, Larry released it to the com- munity of Usenet readers, commonly known as “the Net.” The users on this ragtag fugitive fleet of systems around the world (tens of thousands of them) gave him feed- back, asking for ways to do this, that, or the other thing, many of which Larry had never envisioned his little Perl handling. But as a result, Perl grew, and grew, and grew. It grew in features. It grew in portability. What was once a little language available on only a couple of Unix systems now has thousands of pages of free online documentation, dozens of books, several mainstream Usenet newsgroups (and a dozen newsgroups and mailing lists outside the mainstream) with an uncountable number of readers, and implementations on nearly every system in use today—and don’t forget this Llama book as well. What’s Happening with Perl Now? Larry Wall doesn’t write the code these days, but he still guides the development and makes the big decisions. Perl is mostly maintained by a hardy group of people § With a sharp jump when any one section of the program exceeds the size of your screen. What Does “Perl” Stand For? | 7 called the Perl 5 Porters. You can follow their work and discussions on the perl5-porters@perl.org mailing list. As we write this (March 2011), there is a lot happening with Perl. For the past couple of years, many people have been working on the next major version of Perl: Perl 6. In short, Perl 6 is a completely different language now, even to the point that it’s main implementation goes by the name Rakudo. In 2000, Perl 6 started as something that might replace Perl 5, which had been in the doldrums with long lag times in the releases of Perl 5.6, 5.8, and 5.10. Through various accidents and tangents, it turned out that as Perl 5 hotted up again, Perl 6 bogged down. Ironic, perhaps? However, Perl 5 development was also revitalized and now has monthly releases of experimental versions and roughly yearly releases of new maintenance versions. The last edition of this book covered 5.10, and there wasn’t time to update it before Perl 5.12 came out. Now this book is available right around the time Perl 5.14 should be released, with the Perl 5 Porters already thinking about Perl 5.16. What’s Perl Really Good For? Perl is good for quick-and-dirty programs that you whip up in three minutes. Perl is also good for long-and-extensive programs that will take a dozen programmers three years to finish. Of course, you’ll probably find yourself writing many programs that take you less than an hour to complete, from the initial plan to the fully tested code. Perl is optimized for problems which are about 90% working with text and about 10% everything else. That description seems to fit most programming tasks that pop up these days. In a perfect world, every programmer would know every language; you’d always be able to choose the best language for each project. Most of the time, you’d choose Perl.‖ Although the Web wasn’t even a twinkle in Tim Berners-Lee’s eye when Larry created Perl, it was a marriage made on the Net. Some claim that the deployment of Perl in the early 1990s permitted people to move lots of content into HTML format very rapidly, and the Web couldn’t exist without content. Of course, Perl is the darling language for small CGI scripting (programs run by a web server) as well—so much so that many of the uninformed still make statements like “Isn’t CGI just Perl?” or “Why would you use Perl for something other than CGI?” We find those statements amusing. What Is Perl Not Good For? So, if it’s good for so many things, what is Perl not good for? Well, you shouldn’t choose Perl if you’re trying to make an opaque binary. That’s a program that you could give ‖ Don’t just take our word for it, though. If you want to know whether Perl is better than language X, learn them both and try them both, then see which one you use most often. That’s the one that’s best for you. In the end, you’ll understand Perl better because of your study of language X, and vice versa, so it will be time well spent. 8 | Chapter 1: Introduction away or sell to someone who then can’t see your secret algorithms in the source, and thus can’t help you maintain or debug your code either. When you give someone your Perl program, you’ll normally be giving them the source, not an opaque binary. If you’re wishing for an opaque binary, though, we have to tell you that they don’t exist. If someone can install and run your program, they can turn it back into source code. Granted, this won’t necessarily be the same source that you started with, but it will be some kind of source code. The real way to keep your secret algorithm a secret is, alas, to apply the proper number of attorneys; they can write a license that says “you can do this with the code, but you can’t do that. And if you break our rules, we’ve got the proper number of attorneys to ensure that you’ll regret it.” How Can I Get Perl? You probably already have it. At least, we find Perl wherever we go. It ships with many systems, and system administrators often install it on every machine at their site. But if you can’t find it already on your system, you can still get it for free. It comes pre- installed with most Linux or *BSD systems, Mac OS X, and some others. Companies such as ActiveState (http://www.activestate.com) provide pre-built and enhanced dis- tributions for several platforms, including Windows. You can also get Strawberry Perl for Windows (http://www.strawberryperl.com), which comes with all the same stuff as regular Perl plus extra tools to compile and install third-party modules. Perl is distributed under two different licenses. For most people, since you’ll merely be using it, either license is as good as the other. If you’ll be modifying Perl, however, you’ll want to read the licenses more closely, because they put some small restrictions on distributing the modified code. For people who won’t modify Perl, the licenses essen- tially say, “It’s free—have fun with it.” In fact, it’s not only free, but it runs rather nicely on nearly everything that calls itself Unix and has a C compiler. You download it, type a command or two, and it starts configuring and building itself. Or, better yet, you get your system administrator to type those two commands and install it for you.# Besides Unix and Unix-like systems, people addicted to Perl have ported it to other systems, such as Mac OS X, VMS, OS/2, even MS/DOS, and every modern species of Windows—and probably even more by the time you read this.* Many of these ports of Perl come with an installation program that’s even easier to use than the process for installing Perl on Unix. Check for links in the “ports” section on CPAN. #If system administrators can’t install software, what good are they? If you have trouble convincing your admin to install Perl, offer to buy a pizza. We’ve never met a sysadmin who could say no to a free pizza, or at least counteroffer with something just as easy to get. * And no, as we write this, it won’t fit in your Blackberry—it’s just too darn big, even stripped down. We’ve heard rumors that it runs on WinCE though. How Can I Get Perl? | 9 What Is CPAN? CPAN is the Comprehensive Perl Archive Network, your one-stop shopping for Perl. It has the source code for Perl itself, ready-to-install ports of Perl to all sorts of non- Unix systems,† examples, documentation, extensions to Perl, and archives of messages about Perl. In short, CPAN is comprehensive. CPAN is replicated on hundreds of mirror machines around the world; start at http:// search.cpan.org/ to browse or search the archive. If you don’t have access to the Net, you might find a CD-ROM or DVD-ROM with all of the useful parts of CPAN on it; check with your local technical bookstore. Look for a recently minted archive, though. Since CPAN changes daily, an archive from two years ago is an antique. Better yet, get a kind friend with Net access to burn you one with today’s CPAN. How Can I Get Support for Perl? Well, you get the complete source—so you get to fix the bugs yourself! That doesn’t sound so good, does it? But it really is a good thing. Since there’s no “source code escrow” on Perl, anyone can fix a bug—in fact, by the time you’ve found and verified a bug, someone else has probably already got a fix for it. There are thou- sands of people around the world who help maintain Perl. Now, we’re not saying that Perl has a lot of bugs. But it’s a program, and every program has at least one bug. To see why it’s so useful to have the source to Perl, imagine that instead of using Perl, you licensed a programming language called Forehead from a giant, powerful corporation owned by a zillionaire with a bad haircut. (This is all hy- pothetical. Everyone knows there’s no such programming language as Forehead.) Now think of what you can do when you find a bug in Forehead. First, you can report it. Second, you can hope—hope that they fix the bug, hope that they fix it soon, hope that they won’t charge too much for the new version. You can hope that the new version doesn’t add new features with new bugs, and hope that the giant company doesn’t get broken up in an antitrust lawsuit. But with Perl, you’ve got the source. In the rare and unlikely event that you can’t get a bug fixed any other way, you can hire a programmer or ten and get to work. For that matter, if you buy a new machine that Perl doesn’t yet run on, you can port it yourself. Or if you need a feature that doesn’t yet exist, well, you know what to do. Are There Any Other Kinds of Support? Sure. One of our favorites is the Perl Mongers. This is a worldwide association of Perl users’ groups; see http://www.pm.org/ for more information. There’s probably a group † It’s nearly always better to compile Perl from the source on Unix systems. Other systems may not have a C compiler and other tools needed for compilation, so CPAN has binaries for these. 10 | Chapter 1: Introduction near you with an expert or someone who knows an expert. If there’s no group, you can easily start one. Of course, for the first line of support, you shouldn’t neglect the documentation. Be- sides the included documentation, you can also read the documentation on CPAN, http://www.cpan.org, as well as other sites—http://perldoc.perl.org has HTML and PDF versions of the Perl documentation, and http://faq.perl.org/ has the latest version of the perlfaq. Another authoritative source is the book Programming Perl, commonly called “the Camel book” because of its cover animal (just as this book is known as “the Llama book”). The Camel book contains the complete reference information, some tutorial stuff, and a bunch of miscellaneous information about Perl. There’s also a separate pocket-sized Perl 5 Pocket Reference by Johan Vromans (O’Reilly) that’s handy to keep at hand (or in your pocket). If you need to ask a question of someone, there are newsgroups on Usenet and any number of mailing lists.‡ At any hour of the day or night, there’s a Perl expert awake in some time zone answering questions on Usenet’s Perl newsgroups—the sun never sets on the Perl empire. This means that if you ask a question, you’ll often get an answer within minutes. And if you didn’t check the documentation and FAQ first, you’ll get flamed within minutes. The official Perl newsgroups on Usenet are located in the comp.lang.perl.* part of the hierarchy. As of this writing, there are five of them, but they change from time to time. You (or whoever is in charge of Perl at your site) should generally subscribe to comp.lang.perl.announce, which is a low-volume newsgroup just for important an- nouncements about Perl, including any security-related announcements. Ask your local expert if you need help with Usenet. Also, a few web communities have sprung up around Perl discussions. One very popular one, known as The Perl Monastery (http://www.perlmonks.org), has seen quite a bit of participation from many Perl book and column authors, including at least two of the authors of this book. There is also good Perl support on Stack Overflow (http://www .stackoverflow.com). You can also check out http://learn.perl.org/ and its associated mailing list, beginners@perl.org. Many well-known Perl programmers also have blogs that regularly feature Perl-related posts, most of which you can read through Perlsphere, http://perl sphere.net/. If you find yourself needing a support contract for Perl, there are a number of firms who are willing to charge as much as you’d like. In most cases, these other support avenues will take care of you for free. ‡ Many mailing lists are listed at http://lists.perl.org. How Can I Get Perl? | 11 What If I Find a Bug in Perl? The first thing to do when you find a bug is to check the documentation§ again.‖ Perl has so many special features and exceptions to rules that you may have discovered a feature, not a bug. Also, check that you don’t have an older version of Perl; maybe you found something that’s been fixed in a more recent version. Once you’re 99% certain that you’ve found a real bug, ask around. Ask someone at work, at your local Perl Mongers meeting, or at a Perl conference. Chances are, it’s still a feature, not a bug. Once you’re 100% certain that you’ve found a real bug, cook up a test case. (What, you haven’t done so already?) The ideal test case is a tiny self-contained program that any Perl user could run to see the same (mis-)behavior as you’ve found. Once you’ve got a test case that clearly shows the bug, use the perlbug utility (which comes with Perl) to report the bug. That will normally send email from you to the Perl developers, so don’t use perlbug until you’ve got your test case ready. Once you’ve sent off your bug report, if you’ve done everything right, it’s not unusual to get a response within minutes. Typically, you can apply a simple patch and get right back to work. Of course, you may (at worst) get no response at all; the Perl developers are under no obligation to read your bug reports. But all of us love Perl, so nobody likes to let a bug escape our notice. How Do I Make a Perl Program? It’s about time you asked (even if you didn’t). Perl programs are text files; you can create and edit them with your favorite text editor. You don’t need any special development environment, although there are some commercial ones available from various vendors. We’ve never used any of these enough to recommend them (but long enough to stop using them). Besides, your environment is a personal choice. Ask three programmers what you should use and you’ll get eight answers. You should generally use a programmers’ text editor, rather than an ordinary editor. What’s the difference? Well, a programmers’ text editor will let you do things that programmers need, like indenting or un-indenting a block of code, or finding the matching closing curly brace for a given opening curly brace. On Unix systems, the two most popular programmers’ editors are emacs and vi (and their variants and clones). BBEdit and TextMate are good editors for Mac OS X, and a lot of people have said nice things about UltraEdit and PFE (Programmer’s Favorite Editor) on Windows. The § Even Larry admits to consulting the documentation from time to time. ‖ Maybe even two or three times. Many times, we’ve gone into the documentation looking to explain a particular unexpected behavior and found some new little nuance that ends up on a slide or in a magazine article. 12 | Chapter 1: Introduction perlfaq3 documentation lists several other editors, too. Ask your local expert about text editors on your system. For the simple programs you’ll write for the exercises in this book, none of which should be more than about 20 or 30 lines of code, any text editor will be fine. Some beginners try to use a word processor instead of a text editor. We recommend against this—it’s inconvenient at best and impossible at worst. But we won’t try to stop you. Be sure to tell the word processor to save your file as “text only”; the word pro- cessor’s own format will almost certainly be unusable. Most word processors will probably also tell you that your Perl program is spelled incorrectly and you should use fewer semicolons. In some cases, you may need to compose the program on one machine, then transfer it to another to run it. If you do this, be sure that the transfer uses “text” or “ASCII” mode, and not “binary” mode. This step is needed because of the different text formats on different machines. Without it, you may get inconsistent results—some versions of Perl actually abort when they detect a mismatch in the line endings. A Simple Program According to the oldest rule in the book, any book about a computer language that has Unix-like roots has to start with showing the “Hello, world” program. So, here it is in Perl: #!/usr/bin/perl print "Hello, world!\n"; Let’s imagine that you’ve typed that into your text editor. (Don’t worry yet about what the parts mean and how they work. You’ll see about those in a moment.) You can generally save that program under any name you wish. Perl doesn’t require any special kind of filename or extension, and it’s better not to use an extension at all.# But some systems may require an extension like .plx (meaning PerL eXecutable); see your sys- tem’s release notes for more information. You may also need to do something so that your system knows it’s an executable pro- gram (that is, a command). What you’ll do depends upon your system; maybe you won’t have to do anything more than save the program in a certain place. (Your current directory will generally be fine.) On Unix systems, you mark a program as being exe- cutable using the chmod command, perhaps like this: $ chmod a+x my_program #Why is it better to have no extension? Imagine that you’ve written a program to calculate bowling scores and you’ve told all of your friends that it’s called bowling.plx. One day you decide to rewrite it in C. Do you still call it by the same name, implying that it’s still written in Perl? Or do you tell everyone that it has a new name? (And don’t call it bowling.c, please!) The answer is that it’s none of their business what language it’s written in, if they’re merely using it. So it should have simply been called bowling in the first place. How Do I Make a Perl Program? | 13 The dollar sign (and space) at the start of the line represents the shell prompt, which will probably look different on your system. If you’re used to using chmod with a num- ber like 755 instead of a symbolic parameter like a+x, that’s fine too, of course. Either way, it tells the system that this file is now a program. Now you’re ready to run it: $ ./my_program The dot and slash at the start of this command mean to find the program in the current working directory. That’s not needed in all cases, but you should use it at the start of each command invocation until you fully understand what it’s doing.* If everything worked, it’s a miracle. More often, you’ll find that your program has a bug. Edit and try again—but you don’t need to use chmod each time, as that should “stick” to the file. (Of course, if the bug is that you didn’t use chmod correctly, you’ll probably get a “permission denied” message from your shell.) There’s another way to write this simple program in Perl 5.10 or later, and we might as well get that out of the way right now. Instead of print, we use say, which does almost the same thing, but with less typing. It adds the newline for us, meaning that we can save some time forgetting to add it ourselves. Since it’s a new feature and you might not be using Perl 5.10 yet, we include a use 5.010 statement that tells Perl that we used new features: #!/usr/bin/perl use 5.010; say "Hello World!"; This program only runs under Perl 5.10 or later. When we introduce Perl 5.10 or later features in this book, we’ll explicitly say they are new features in the text and include that use 5.010 statement to remind you. Perl actually thinks about the minor version as a three digit number, so ensure that you say use 5.010 and not use 5.10 (which Perl thinks is 5.100, a version we definitely don’t have yet!). Typically, we only require the earliest version of Perl for the features that we need. This book covers up to Perl 5.14, so in many of the new features we preface the examples to remind you to add this line: use 5.014; * In short, it’s preventing your shell from running another program (or shell built-in) of the same name. A common mistake among beginners is to name their first program test. Many systems already have a program (or shell built-in) with that name; that’s what the beginners run instead of their program. 14 | Chapter 1: Introduction What’s Inside That Program? Like other “free-form” languages, Perl generally lets you use insignificant whitespace (like spaces, tabs, and newlines) at will to make your program easier to read. Most Perl programs use a fairly standard format, though, much like most of what we show here.† We strongly encourage you to properly indent your programs, as that makes your program easier to read; a good text editor will do most of the work for you. Good comments also make a program easier to read. In Perl, comments run from a pound sign (#) to the end of the line. (There are no “block comments” in Perl.‡) We don’t use many comments in the programs in this book because the surrounding text explains their workings, but you should use comments as needed in your own programs. So another way (a very strange way, it must be said) to write that same “Hello, world” program might be like this: #!/usr/bin/perl print # This is a comment "Hello, world!\n" ; # Don't write your Perl code like this! That first line is actually a very special comment. On Unix systems,§ if the very first two characters on the first line of a text file are #!, then what follows is the name of the program that actually executes the rest of the file. In this case, the program is stored in the file /usr/bin/perl. This #! line is actually the least portable part of a Perl program because you’ll need to find out what goes there for each machine. Fortunately, it’s almost always either /usr/ bin/perl or /usr/local/bin/perl. If that’s not it, you’ll have to find where your system is hiding perl, then use that path. On some Unix systems, you might use a shebang line that finds perl for you: #!/usr/bin/env perl If perl is not in any of the directories in your search path, you might have to ask your local system administrator or somebody using the same system as you. Beware though, that finds the first perl, which might not be the one that you wanted. On non-Unix systems, it’s traditional (and even useful) to make the first line say #! perl. If nothing else, it tells your maintenance programmer as soon as he gets ready to fix it that it’s a Perl program. If that #! line is wrong, you’ll generally get an error from your shell. This may be some- thing unexpected, like “file not found” or “bad interpreter”. It’s not your program that’s † There is some general advice (not rules!) in the perlstyle documentation. ‡ But there are a number of ways to fake them. See the perlfaq portions of the documentation. § Most modern ones, anyway. The “sh-bang” mechanism, pronounced “sheh-bang” as in “the whole shebang”, was introduced somewhere in the mid-1980s, and that’s pretty ancient, even on the extensively long Unix timeline. How Do I Make a Perl Program? | 15 not found, though; it’s that /usr/bin/perl wasn’t where it should have been. We’d make the message clearer if we could, but it’s not coming from Perl; it’s the shell that’s complaining. Another problem you could have is that your system doesn’t support the #! line at all. In that case, your shell (or whatever your system uses) will probably try to run your program all by itself, with results that may disappoint or astonish you. If you can’t figure out what some strange error message is telling you, search for it in the perldiag documentation. The “main” program consists of all the ordinary Perl statements (not including anything in subroutines, which you’ll see later). There’s no “main” routine, as there is in lan- guages like C or Java. In fact, many programs don’t even have routines (in the form of subroutines). There’s also no required variable declaration section, as there is in some other lan- guages. If you’ve always had to declare your variables, you may be startled or unsettled by this at first. But it allows us to write quick-and-dirty Perl programs. If your program is only two lines long, you don’t want to have to use one of those lines just to declare your variables. If you really want to declare your variables, that’s a good thing; you’ll see how to do that in Chapter 4. Most statements are an expression followed by a semicolon.‖ Here’s the one you’ve seen a few times so far: print "Hello, world!\n"; As you may have guessed by now, this line prints the message Hello, world! At the end of that message is the shortcut \n, which is probably familiar to you if you’ve used another language like C, C++, or Java; it means a newline character. When that’s prin- ted after the message, the print position drops down to the start of the next line, al- lowing the following shell prompt to appear on a line of its own, rather than being attached to the message. Every line of output should end with a newline character. We’ll see more about the newline shortcut and other so-called backslash escapes in the next chapter. How Do I Compile My Perl Program? Just run your Perl program. The perl interpreter compiles and runs your program in one user step: $ perl my_program When you run your program, Perl’s internal compiler first runs through your entire source, turning it into internal bytecodes, which is an internal data structure represent- ing the program. Perl’s bytecode engine takes over and actually runs the bytecode. If ‖ You only need semicolons to separate statements, not terminate them. 16 | Chapter 1: Introduction there’s a syntax error on line 200, you’ll get that error message before you start running line 2.# If you have a loop that runs 5,000 times, it’s compiled just once; the actual loop can then run at top speed. And there’s no runtime penalty for using as many comments and as much whitespace as you need to make your program easy to understand. You can even use calculations involving only constants, and the result is a constant com- puted once as the program is beginning—not each time through a loop. To be sure, this compilation does take time—it’s inefficient to have a voluminous Perl program that does one small quick task (out of many potential tasks, say) and then exits because the runtime for the program will be dwarfed by the compile time. But the compiler is very fast; normally the compilation will be a tiny percentage of the runtime. An exception might be if you were writing a program run as a CGI script, where it may be called hundreds or thousands of times every minute. (This is a very high usage rate. If it were called a few hundreds or thousands of times per day, like most programs on the Web, we probably wouldn’t worry too much about it.) Many of these programs have very short runtimes, so the issue of recompilation may become significant. If this is an issue for you, you’ll want to find a way to keep your program in memory between invocations. The mod_perl extension to the Apache web server (http://perl.apache .org) or Perl modules like CGI::Fast can help you. What if you could save the compiled bytecodes to avoid the overhead of compilation? Or, even better, what if you could turn the bytecodes into another language, like C, and then compile that? Well, both of these things are possible in some cases, but they probably won’t make most programs any easier to use, maintain, debug, or install, and they may even make your program slower. A Whirlwind Tour of Perl So, you want to see a real Perl program with some meat? (If you don’t, just play along for now.) Here you are: #!/usr/bin/perl @lines = `perldoc -u -f atan2`; foreach (@lines) { s/\w<([^>]+)>/\U$1/g; print; } Now, the first time you see Perl code like this, it can seem pretty strange. (In fact, every time you see Perl code like this, it can seem pretty strange.) But let’s take it line by line, and see what this example does. These explanations are very brief; this is a whirlwind tour, after all. We’ll see all of this program’s features in more detail during the rest of this book. You’re not really supposed to understand the whole thing until later. #Unless line 2 happens to be a compile-time operation, like a BEGIN block or a use invocation. A Whirlwind Tour of Perl | 17 The first line is the #! line, as you saw before. You might need to change that line for your system, as we showed you earlier. The second line runs an external command, named within backquotes (` `). (The backquote key is often found next to the number 1 on full-sized American keyboards. Be sure not to confuse the backquote with the single quote, '.) The command we used is perldoc -u -f atan2; try typing that in your command line to see what its output looks like. The perldoc command is used on most systems to read and display the documen- tation for Perl and its associated extensions and utilities, so it should normally be avail- able.* This command tells you something about the trigonometric function atan2; we’re using it here just as an example of an external command whose output we wish to process. The output of that command in the backquotes is saved in an array variable called @lines. The next line of code starts a loop that will process each one of those lines. Inside the loop, the statements are indented. Although Perl doesn’t require this, good programmers do. The first line inside the loop body is the scariest one; it says s/\w<([^>]+)>/\U$1/g;. Without going into too much detail, we’ll just say that this can change any line that has a special marker made with angle brackets (< >), and there should be at least one of those in the output of the perldoc command. The next line, in a surprise move, prints out each (possibly modified) line. The resulting output should be similar to what perldoc -u -f atan2 would do on its own, but there will be a change where any of those markers appear. Thus, in the span of a few lines, we’ve run another program, saved its output in memory, updated the memory items, and printed them out. This kind of program is a fairly common use of Perl, where one type of data is converted to another. Exercises Normally, each chapter will end with some exercises, with the answers in Appen- dix A. But you don’t need to write the programs needed to complete this section— those are supplied within the chapter text. If you can’t get these exercises to work on your machine, double-check your work and then consult your local expert. Remember that you may need to tweak each program a little, as described in the text: * If perldoc is not available, that probably means that your system doesn’t have a command-line interface, and your Perl can’t run commands (like perldoc!) in backquotes or via a piped open, which you’ll see in Chapter 14. In that case, you should simply skip the exercises that use perldoc. 18 | Chapter 1: Introduction 1. [7] Type in the “Hello, world” program and get it to work! You may name it any- thing you wish, but a good name might be ex1-1, for simplicity, as it’s exercise 1 in Chapter 1. This is a program that even an experienced programmer would write, mostly to test the setup of a system. If you can run this program, your perl is working. 2. [5] Type the command perldoc -u -f atan2 at a command prompt and note its output. If you can’t get that to work, find out from a local administrator or the documentation for your version of Perl about how to invoke perldoc or its equiv- alent. You’ll need this for the next exercise anyway. 3. [6] Type in the second example program (from the previous section) and see what it prints. Hint: be careful to type those punctuation marks exactly as shown! Do you see how it changed the output of the command? Exercises | 19 CHAPTER 2 Scalar Data In English, as in many other spoken languages, you’re used to distinguishing between singular and plural. As a computer language designed by a human linguist, Perl is sim- ilar. As a general rule, when Perl has just one of something, that’s a scalar.* A scalar is the simplest kind of data that Perl manipulates. Most scalars are either a number (like 255 or 3.25e20) or a string of characters (like hello† or the Gettysburg Address). Al- though you may think of numbers and strings as very different things, Perl uses them nearly interchangeably. You can act on a scalar value with operators (like addition or concatenation), generally yielding a scalar result. You can store a scalar value in a scalar variable. You can read scalars from files and devices, and write to them as well. Numbers Although a scalar is most often either a number or a string, it’s useful to look at numbers and strings separately for the moment. We’ll cover numbers first, and then move on to strings. * This has little to do with the similar term from mathematics or physics in that a scalar is a single thing; there are no “vectors” in Perl. † If you have been using other programming languages, you may think of hello as a collection of five characters, rather than as a single thing. But in Perl, a string is a single scalar value. Of course, you can access the individual characters when you need to; you’ll see how to do that in later chapters. 21 All Numbers Have the Same Format Internally As you’ll see in the next few paragraphs, you can specify both integers (whole numbers, like 255 or 2,001) and floating-point numbers (real numbers with decimal points, like 3.14159, or 1.35 × 1,025). But internally, Perl computes with double-precision floating- point values.‡ This means that there are no integer values internal to Perl—an integer constant in the program is treated as the equivalent floating-point value.§ You probably won’t notice the conversion (or care much), but you should stop looking for distinct integer operations (as opposed to floating-point operations) because there aren’t any.‖ Floating-Point Literals A literal is how you represent a value in your Perl source code. A literal is not the result of a calculation or an I/O operation; it’s data that you type directly into your program. Perl’s floating-point literals should look familiar to you. Numbers with and without decimal points are allowed (including an optional plus or minus prefix), as well as tacking on a power-of-10 indicator (exponential notation) with E notation. For example: 1.25 255.000 255.0 7.25e45 # 7.25 times 10 to the 45th power (a big number) –6.5e24 # negative 6.5 times 10 to the 24th # (a big negative number) –12e–24 # negative 12 times 10 to the –24th # (a very small negative number) –1.2E–23 # another way to say that the E may be uppercase Integer Literals Integer literals are also straightforward, as in: 0 2001 –40 255 61298040283768 ‡ A double-precision floating-point value is whatever the C compiler that compiled Perl used for a double declaration. While the size may vary from machine to machine, most modern systems use the IEEE-754 format, which suggests 15 digits of precision and a range of at least 1e-100 to 1e100. § Well, Perl will sometimes use internal integers in ways that are not visible to the programmer. That is, the only difference you should generally be able to see is that your program runs faster. And who could complain about that? ‖ Okay, there is the integer pragma. But using that is beyond the scope of this book. And yes, some operations compute an integer from a given floating-point number, as you’ll see later. But that’s not what we’re talking about here. 22 | Chapter 2: Scalar Data That last one is a little hard to read. Perl allows you to add underscores for clarity within integer literals, so you can also write that number with embedded underscores to make it easier to read: 61_298_040_283_768 It’s the same value; it merely looks different to us human beings. You might have thought that commas should be used for this purpose, but commas are already used for a more-important purpose in Perl (as you’ll see in Chapter 3). Nondecimal Integer Literals Like many other programming languages, Perl allows you to specify numbers in other ways than base 10 (decimal). Octal (base 8) literals start with a leading 0, hexadecimal (base 16) literals start with a leading 0x, and binary (base 2) literals start with a leading 0b.# The hex digits A through F (or a through f) represent the conventional digit values of 10 through 15. For example: 0377 # 377 octal, same as 255 decimal 0xff # FF hex, also 255 decimal 0b11111111 # also 255 decimal Although these values look different to us humans, they’re all three the same number to Perl. It makes no difference to Perl whether you write 0xFF or 255.000, so choose the representation that makes the most sense to you and your maintenance programmer (by which we mean the poor chap who gets stuck trying to figure out what you meant when you wrote your code. Most often, this poor chap is you, and you can’t recall why you did what you did three months ago). When a nondecimal literal is more than about four characters long, it may be hard to read, so underscores are handy: 0x1377_0B77 0x50_65_72_7C Numeric Operators Perl provides the typical ordinary addition, subtraction, multiplication, and division operators, and so on. For example: 2 + 3 # 2 plus 3, or 5 5.1 – 2.4 # 5.1 minus 2.4, or 2.7 3 * 12 # 3 times 12 = 36 14 / 2 # 14 divided by 2, or 7 #The “leading zero” indicator works only for literals—not for automatic string-to-number conversions, which you’ll see later in this chapter in “Automatic Conversion Between Numbers and Strings” on page 27. You can convert a data string that looks like an octal or hex value into a number with oct() or hex(). Although there’s no bin() function for converting binary values, oct() can do that for strings beginning with 0b. Numbers | 23 10.2 / 0.3 # 10.2 divided by 0.3, or 34 10 / 3 # always floating-point divide, so 3.3333333... Perl also supports a modulus operator (%). The value of the expression 10 % 3 is the remainder when 10 is divided by 3, which is 1. Both values are first reduced to their integer values, so 10.5 % 3.2 is computed as 10 % 3.* Additionally, Perl provides the FORTRAN-like exponentiation operator, which many have yearned for in Pascal and C. The operator is represented by the double asterisk, such as 2**3, which is two to the third power, or eight.† In addition, there are other numeric operators, which we’ll introduce as we need them. Strings Strings are sequences of characters, such as hello or ☃★๛. Strings may contain any combination of any characters.‡ The shortest possible string has no characters, and is called the empty string. The longest string fills all of your available memory (although you wouldn’t be able to do much with that). This is in accordance with the principle of “no built-in limits” that Perl follows at every opportunity. Typical strings are print- able sequences of letters and digits and punctuation. However, the ability to have any character in a string means you can create, scan, and manipulate raw binary data as strings—something with which many other utilities would have great difficulty. For example, you could update a graphical image or compiled program by reading it into a Perl string, making the change, and writing the result back out. Perl has full support for Unicode, and your string can contain any of the valid Unicode characters. However, because of Perl’s history, it doesn’t automatically interpret your source code as Unicode. If you want to use Unicode literally in your program, you need to add the utf8 pragma:§ use utf8; For the rest of this book, we assume that you’re using that pragma. In some cases it won’t matter, but if you see characters outside the ASCII range in the source, you’ll need it. Also, you should ensure that you save your files with the UTF-8 encoding. If you skipped our advice about Unicode from the Preface, you might want to go through Appendix C to learn more about Unicode. * The result of a modulus operator when a negative number (or two) is involved can vary between Perl implementations. Beware. † You can’t normally raise a negative number to a noninteger exponent. Math geeks know that the result would be a complex number. To make that possible, you’ll need the help of the Math::Complex module. ‡ Unlike C or C++, there’s nothing special about the NUL character in Perl because Perl uses length counting, not a null byte, to determine the end of the string. § It’s probably a good practice to always include this in your program unless you know why you wouldn’t want to. 24 | Chapter 2: Scalar Data Like numbers, strings have a literal representation, which is the way you represent the string in a Perl program. Literal strings come in two different flavors: single-quoted string literals and double-quoted string literals. Single-Quoted String Literals A single-quoted string literal is a sequence of characters enclosed in single quotes, the ' character. The single quotes are not part of the string itself—they’re just there to let Perl identify the beginning and the ending of the string. Any character other than a single quote or a backslash between the quote marks (including newline characters, if the string continues on to successive lines) stands for itself inside a string. To get a backslash, put two backslashes in a row, and to get a single quote, put a backslash followed by a single quote. In other words: 'fred' # those four characters: f, r, e, and d 'barney' # those six characters '' # the null string (no characters) '⅚∞☃☠' # Some "wide" Unicode characters 'Don\'t let an apostrophe end this string prematurely!' 'the last character is a backslash: \\' 'hello\n' # hello followed by backslash followed by n 'hello there' # hello, newline, there (11 characters total) '\'\\' # single quote followed by backslash Note that Perl does not interpret the \n within a single-quoted string as a newline, but as the two characters backslash and n. Only when the backslash is followed by another backslash or a single quote does it have special meaning. Double-Quoted String Literals A double-quoted string literal is a sequence of characters, although this time enclosed in double quotes. But now the backslash takes on its full power to specify certain control characters, or even any character at all through octal and hex representations. Here are some double-quoted strings: "barney" # just the same as 'barney' "hello world\n" # hello world, and a newline "The last character of this string is a quote mark: \"" "coke\tsprite" # coke, a tab, and sprite "\x{2668}" # Unicode HOT SPRINGS character code point Note that the double-quoted literal string "barney" means the same six-character string to Perl as does the single-quoted literal string 'barney'. It’s like what you saw with numeric literals, where you saw that 0377 was another way to write 255.0. Perl lets you write the literal in the way that makes more sense to you. Of course, if you wish to use a backslash escape (like \n to mean a newline character), you’ll need to use the double quotes. Strings | 25 The backslash can precede many different characters to mean different things (generally called a backslash escape). The nearly complete list of double-quoted string escapes is given in Table 2-1. Table 2-1. Double-quoted string backslash escapes Construct Meaning \n Newline \r Return \t Tab \f Formfeed \b Backspace \a Bell \e Escape (ASCII escape character) \007 Any octal ASCII value (here, 007 = bell) \x7f Any hex ASCII value (here, 7f = delete) \x{2744} Any hex Unicode code point (here, U+2744 = snowflake) \cC A “control” character (here, Ctrl-C) \\ Backslash \“ Double quote \l Lowercase next letter \L Lowercase all following letters until \E \u Uppercase next letter \U Uppercase all following letters until \E \Q Quote nonword characters by adding a backslash until \E \E End \L, \U, or \Q Another feature of double-quoted strings is that they are variable interpolated, meaning that some variable names within the string are replaced with their current values when the strings are used. You haven’t formally been introduced to what a variable looks like yet, so we’ll get back to that later in this chapter. String Operators You can concatenate, or join, string values with the . operator. (Yes, that’s a single period.) This does not alter either string, any more than 2+3 alters either 2 or 3. The resulting (longer) string is then available for further computation or assignment to a variable. For example: 26 | Chapter 2: Scalar Data "hello" . "world" # same as "helloworld" "hello" . ' ' . "world" # same as 'hello world' 'hello world' . "\n" # same as "hello world\n" Note that you must explicitly use the concatenation operator, unlike in some other languages where you merely have to stick the two values next to each other. A special string operator is the string repetition operator, consisting of the single lowercase letter x. This operator takes its left operand (a string) and makes as many concatenated copies of that string as indicated by its right operand (a number). For example: "fred" x 3 # is "fredfredfred" "barney" x (4+1) # is "barney" x 5, or "barneybarneybarneybarneybarney" 5 x 4.8 # is really "5" x 4, which is "5555" That last example is worth noting carefully. The string repetition operator wants a string for a left operand, so the number 5 is converted to the string "5" (using rules described in detail later), giving a one-character string. The x copies the new string four times, yielding the four-character string 5555. Note that if you had reversed the order of the operands, as 4 x 5, you would have made five copies of the string 4, yielding 44444. This shows that string repetition is not commutative. The copy count (the right operand) is first truncated to an integer value (4.8 becomes 4) before being used. A copy count of less than one results in an empty (zero-length) string. Automatic Conversion Between Numbers and Strings For the most part, Perl automatically converts between numbers and strings as needed. How does it know which it should use? It all depends upon the operator that you apply to the scalar value. If an operator expects a number (like + does), Perl will see the value as a number. If an operator expects a string (like . does), Perl will see the value as a string. So, you don’t need to worry about the difference between numbers and strings; just use the proper operators, and Perl will make it all work. When you use a string value where an operator needs a number (say, for multiplication), Perl automatically converts the string to its equivalent numeric value, as if you had entered it as a decimal floating-point value. So "12" * "3" gives the value 36. Trailing nonnumber stuff and leading whitespace are discarded, so "12fred34" * " 3" will also give 36 without any complaints.‖ At the extreme end of this, something that isn’t a number at all converts to zero. This would happen if you used the string "fred" as a number. ‖ Unless you turn on warnings, which we’ll show in a moment. Strings | 27 The trick of using a leading zero to mean an octal value works for literals, but never for automatic conversion, which is always base-10:# 0377 # that's octal for 255 decimal '0377' # that's 377 decimal Likewise, if a numeric value is given when a string value is needed (say, for string con- catenation), the numeric value is expanded into whatever string would have been prin- ted for that number. For example, if you want to concatenate the string Z followed by the result of 5 multiplied by 7,* you can say this simply as: "Z" . 5 * 7 # same as "Z" . 35, or "Z35" In other words, you don’t really have to worry about whether you have a number or a string (most of the time). Perl performs all the conversions for you.† Perl’s Built-in Warnings Perl can be told to warn you when it sees something suspicious going on in your pro- gram. With Perl 5.6 and later, you can turn on warnings with a pragma (but be careful because it won’t work for people with earlier versions of Perl):‡ #!/usr/bin/perl use warnings; You can use the -w option on the command line, which turns on warnings everywhere in your program:§ $ perl -w my_program You can also specify the command-line switches on the shebang line: #!/usr/bin/perl -w That works even on non-Unix systems, where it’s traditional to write something like this, since the path to Perl doesn’t generally matter: #!perl -w #If you have a numeric string in octal or hexadecimal, or even binary, you may convert it with the oct() or hex() functions. See “Interpreting Non-Decimal Numerals” on page 240. * You’ll see about precedence and parentheses shortly. † And if you’re worried about efficiency, don’t be. Perl generally remembers the result of a conversion so that it’s done only once. ‡ The warnings pragma actually allows lexical warnings, but you’ll have to see the perllexwarn documentation to find out about those. The advantage of warnings over -w is that you only turn on warnings for the file in which you use the pragma, whereas -w turns on for the entire program. § This might include modules that you use but didn’t write yourself, so you might see warnings from other people’s code. 28 | Chapter 2: Scalar Data Now, Perl will warn you if you use '12fred34' as if it were a number: Argument "12fred34" isn't numeric Perl still turns the non-numeric '12fred34' into 12 using its normal rules even though you get the warning. Of course, warnings are generally meant for programmers, not for end users. If the warning won’t be seen by a programmer, it probably won’t do you any good. And warnings won’t change the behavior of your program, except that now it gripes once in a while. If you get a warning message you don’t understand, you can get a longer description of the problem with the diagnostics pragma. The perldiag documentation has both the short warning and the longer diagnostic description, and is the source of diagnostics’ helpfulness: #!/usr/bin/perl use diagnostics; When you add the use diagnostics pragma to your program, it may seem to you that your program now pauses for a moment whenever you launch it. That’s because your program has to do a lot of work (and gobble a chunk of memory) just in case you want to read the documentation as soon as Perl notices your mistakes, if any. This leads to a nifty optimization that can speed up your program’s launch (and memory footprint) with no adverse impact on users: once you no longer need to read the documentation about the warning messages produced by your program, remove the use diagnostics pragma. (It’s even better if you fix your program to avoid causing the warnings. But it’s sufficient merely to finish reading the output.) A further optimization can be had by using one of Perl’s command-line options, -M, to load the pragma only when needed instead of editing the source code each time to enable and disable diagnostics: $ perl -Mdiagnostics ./my_program Argument "12fred34" isn't numeric in addition (+) at ./my_program line 17 (#1) (W numeric) The indicated string was fed as an argument to an operator that expected a numeric value instead. If you're fortunate the message will identify which operator was so unfortunate. Note the (W numeric) in the message. The W says that the message is a warning and the numeric is the class of warning. In this case, you know to look for something dealing with a number. As we run across situations in which Perl will usually be able to warn us about a mistake in your code, we’ll point them out. But you shouldn’t count on the text or behavior of any warning staying exactly the same in future Perl releases. Scalar Variables A variable is a name for a container that holds one or more values. As you’ll see, a scalar variable holds exactly one value, and in upcoming chapters you’ll see other types of Scalar Variables | 29 variables, such as arrays and hashes, that can hold many values. The name of the var- iable stays the same throughout your program, but the value or values in that variable can change over and over again throughout the execution of the program. A scalar variable holds a single scalar value, as you’d expect. Scalar variable names begin with a dollar sign, called the sigil, followed by a Perl identifier: a letter or underscore, and then possibly more letters, or digits, or underscores. Another way to think of it is that it’s made up of alphanumerics and underscores, but can’t start with a digit. Up- percase and lowercase letters are distinct: the variable $Fred is a different variable from $fred. And all of the letters, digits, and underscores are significant, so all of these refer to different variables: $name $Name $NAME $a_very_long_variable_that_ends_in_1 $a_very_long_variable_that_ends_in_2 $A_very_long_variable_that_ends_in_2 $AVeryLongVariableThatEndsIn2 Perl doesn’t restrict itself to ASCII for variable names, either. If you enable the utf8 pragma, you can use a much wider range of alphabetic or numeric characters in your identifiers: $résumé $coördinate Perl uses the sigils to distinguish things that are variables from anything else that you might type in the program. You don’t have to know the names of all the Perl functions and operators to choose your variable name. Furthermore, Perl uses the sigil to denote what you’re doing with that variable. The $ sigil really means “single item” or “scalar.” Since a scalar variable is always a single item, it always gets the “single item” sigil. In Chapter 3, you’ll see the “single item” sigil used with another type of variable, the array. Choosing Good Variable Names You should generally select variable names that mean something regarding the purpose of the variable. For example, $r is probably not very descriptive but $line_length is. If you are using a variable for only two or three lines close together, you might call some- thing simple, like $n, but a variable you use throughout a program should probably have a more descriptive name to not only remind you want it does, but let other people know what it does.‖ ‖ Most of your program will make sense to you because you’re the one who invented it. However, someone else isn’t going to know why a name like $srly makes sense to you. 30 | Chapter 2: Scalar Data Similarly, properly placed underscores can make a name easier to read and understand, especially if your maintenance programmer has a different spoken language back- ground than you have. For example, $super_bowl is a better name than $superbowl, since that last one might look like $superb_owl. Does $stopid mean $sto_pid (storing a process ID of some kind?) or $s_to_pid (converting something to a process ID?) or $stop_id (the ID for some kind of “stop” object?) or is it just a stopid misspelling? Most variable names in our Perl programs are all lowercase, like most of the ones you’ll see in this book. In a few special cases, uppercase letters are used. Using all caps (like $ARGV) generally indicates that there’s something special about that variable. When a variable’s name has more than one word, some say $underscores_are_cool, while others say $giveMeInitialCaps. Just be consistent.# You can name your variables with all up- percase, but you might end up using a special variable reserved for Perl. If you avoid all uppercase names you won’t have that problem.* Of course, choosing good or poor names makes no difference to Perl. You could name your program’s three most important variables $OOO000OOO, $OO00OO00, and $O0O0O0O0O and Perl wouldn’t be bothered—but in that case, please, don’t ask us to maintain your code. Scalar Assignment The most common operation on a scalar variable is assignment, which is the way to give a value to a variable. The Perl assignment operator is the equals sign (much like other languages), which takes a variable name on the left side, and gives it the value of the expression on the right. For example: $fred = 17; # give $fred the value of 17 $barney = 'hello'; # give $barney the five-character string 'hello' $barney = $fred + 3; # give $barney the current value of $fred plus 3 (20) $barney = $barney * 2; # $barney is now $barney multiplied by 2 (40) Notice that last line uses the $barney variable twice: once to get its value (on the right side of the equals sign), and once to define where to put the computed expression (on the left side of the equals sign). This is legal, safe, and rather common. In fact, it’s so common that you can write it using a convenient shorthand, as you’ll see in the next section. Binary Assignment Operators Expressions like $fred = $fred + 5 (where the same variable appears on both sides of an assignment) occur frequently enough that Perl (like C and Java) has a shorthand for the operation of altering a variable—the binary assignment operator. Nearly all binary #There is some advice in the perlstyle documentation. * You can see all of Perl’s special variables in the perlvar documentation. Scalar Variables | 31 operators that compute a value have a corresponding binary assignment form with an appended equals sign. For example, the following two lines are equivalent: $fred = $fred + 5; # without the binary assignment operator $fred += 5; # with the binary assignment operator These are also equivalent: $barney = $barney * 3; $barney *= 3; In each case, the operator alters the existing value of the variable in some way, rather than simply overwriting the value with the result of some new expression. Another common assignment operator is made with the string concatenate operator ( . ); this gives us an append operator ( .= ): $str = $str . " "; # append a space to $str $str .= " "; # same thing with assignment operator Nearly all binary operators are valid this way. For example, a raise to the power of operator is written as **=. So, $fred **= 3 means “raise the number in $fred to the third power, placing the result back in $fred”. Output with print It’s generally a good idea to have your program produce some output; otherwise, someone may think it didn’t do anything. The print operator makes this possible: it takes a scalar argument and puts it out without any embellishment onto standard out- put. Unless you’ve done something odd, this will be your terminal display. For example: print "hello world\n"; # say hello world, followed by a newline print "The answer is "; print 6 * 7; print ".\n"; You can give print a series of values, separated by commas: print "The answer is ", 6 * 7, ".\n"; This is really a list, but we haven’t talked about lists yet, so we’ll put that off for later. Interpolation of Scalar Variables into Strings When a string literal is double-quoted, it is subject to variable interpolation† (besides being checked for backslash escapes). This means that any scalar variable‡ name in the string is replaced with its current value. For example: † This has nothing to do with mathematical or statistical interpolation. ‡ And some other variable types, but you won’t see those until later. 32 | Chapter 2: Scalar Data $meal = "brontosaurus steak"; $barney = "fred ate a $meal"; # $barney is now "fred ate a brontosaurus steak" $barney = 'fred ate a ' . $meal; # another way to write that As you see on the last line above, you can get the same results without the double quotes, but the double-quoted string is often the more convenient way to write it. If the scalar variable has never been given a value,§ the empty string is used instead: $barney = "fred ate a $meat"; # $barney is now "fred ate a " Don’t bother with interpolating if you have just the one lone variable: print "$fred"; # unneeded quote marks print $fred; # better style There’s nothing really wrong with putting quote marks around a lone variable,‖ but the other programmers will laugh at you behind your back, or maybe even to your face. Variable interpolation is also known as double-quote interpolation because it happens when double-quote marks (but not single quotes) are used. It happens for some other strings in Perl, which we’ll mention as we get to them. To put a real dollar sign into a double-quoted string, precede the dollar sign with a backslash, which turns off the dollar sign’s special significance: $fred = 'hello'; print "The name is \$fred.\n"; # prints a dollar sign Alternatively, you could avoid using double quotes around the problematic part of the string: print 'The name is $fred' . "\n"; # so does this The variable name will be the longest possible variable name that makes sense at that part of the string. This can be a problem if you want to follow the replaced value im- mediately with some constant text that begins with a letter, digit, or underscore.# § This is actually the special undefined value, undef, which you’ll see a little later in this chapter. If warnings are turned on, Perl will complain about interpolating the undefined value. ‖ Well, it may interpret the value as a string, rather than a number. In a few rare cases that may be needed, but nearly always it’s just a waste of typing. #There are some other characters that may be a problem as well. If you need a left square bracket or a left curly brace just after a scalar variable’s name, precede it with a backslash. You may also do that if the variable’s name is followed by an apostrophe or a pair of colons, or you could use the curly brace method described in the main text. Output with print | 33 34 | Chapter 2: Scalar Data Unicode, see Appendix C. said ordinal value to denote the numeric position in ASCII. To pick up anything you might have missed about * We’ll use code point throughout the book because we’re assuming Unicode. In ASCII, we might have just addition. tiplication first. Because of this, you say multiplication has a higher precedence than Fortunately, Perl chooses the common mathematical definition, performing the mul- did the multiplication first (as you were taught in math class), you’d get 2+12, or 14. or the multiplication first? If you did the addition first, you’d get 5*4, or 20. But if you happen first. For example, in the expression 2+3*4, do you perform the addition first Operator precedence determines which operations in a complex group of operations Operator Precedence and Associativity "\x{03B1}\x{03C9}" representation in \x{}: That might be more work than interpolating them directly by putting the hexadecimal "$alpha$omega" You can interpolate these into double-quoted strings just like any other variable: ;( 'א' )$code_point = ord point: You can go the other way with the ord() function, which turns a character into its code $omega = chr( 0x03C9 ); $alpha = chr( hex('03B1') ); $alef = chr( 0x05D0 ); it’s easier to create them by their code point* with the chr() function: on your system and the editor you’re using, but sometimes, instead of typing them out, How you get these characters into your program depends .א keyboard, such as é, å, α, or Sometimes you want to create strings with characters that may not appear on your Creating Characters by Code Point print 'fred ate ' . $n . ' ' . $what . "s.\n"; # an especially difficult way print "fred ate $n $what" . "s.\n"; # another way to do it print "fred ate $n ${what}s.\n"; # now uses $what print "fred ate $n $whats.\n"; # not the steaks, but the value of $whats $n = 3; $what = "brontosaurus steak"; a concatenation operator: braces. Or, you can end that part of the string and start another part of the string with a manner similar to the shell. Simply enclose the name of the variable in a pair of curly acters, which is not what you want. Perl provides a delimiter for the variable name in As Perl scans for variable names, it considers those characters as additional name char- Parentheses have the highest precedence. Anything inside parentheses is completely computed before the operator outside of the parentheses is applied (just like you learned in math class). So if you really want the addition before the multiplication, you can say (2+3)*4, yielding 20. Also, if you wanted to demonstrate that multiplication is performed before addition, you could add a decorative but unnecessary set of paren- theses, as in 2+(3*4). While precedence is simple for addition and multiplication, you start running into problems when faced with, say, string concatenation compared with exponentiation. The proper way to resolve this is to consult the official, accept-no-substitutes Perl operator precedence chart in the perlop documentation, which we partially show in Table 2-2.† Table 2-2. Associativity and precedence of operators (highest to lowest) Associativity Operators left parentheses and arguments to list operators left -> ++ -- (autoincrement and autodecrement) right ** right \ ! ~ + - (unary operators) left =~ !~ left * / % x left + - . (binary operators) left << >> named unary operators (-X filetests, rand) < <= > >= lt le gt ge (the “unequal” ones) == != <=> eq ne cmp (the “equal” ones) left & left | ^ left && left || .. ... right ?: (conditional operator) right = += -= .= (and similar assignment operators) left , => list operators (rightward) † C programmers: rejoice! The operators that are available in both Perl and C have the same precedence and associativity in both. Output with print | 35 Associativity Operators right not left and left or xor In the chart, any given operator has higher precedence than all of the operators listed below it, and lower precedence than all of the operators listed above it. Operators at the same precedence level resolve according to rules of associativity instead. Just like precedence, associativity resolves the order of operations when two operators of the same precedence compete for three operands: 4 ** 3 ** 2 # 4 ** (3 ** 2), or 4 ** 9 (right associative) 72 / 12 / 3 # (72 / 12) / 3, or 6/3, or 2 (left associative) 36 / 6 * 3 # (36/6)*3, or 18 In the first case, the ** operator has right associativity, so the parentheses are implied on the right. Comparatively, the * and / operators have left associativity, yielding a set of implied parentheses on the left. So should you just memorize the precedence chart? No! Nobody actually does that. Instead, just use parentheses when you don’t remember the order of operations, or when you’re too busy to look in the chart. After all, if you can’t remember it without the parentheses, your maintenance programmer is going to have the same trouble. So be nice to your maintenance programmer: you may be that person one day. Comparison Operators To compare numbers, Perl has logical comparison operators that may remind you of algebra: < <= == >= > !=. Each of these returns a true or false value. You’ll find out more about those return values in the next section. Some of these may be different than you’d use in other languages. For example, == is used for equality, not a single = sign because that’s used for assignment. And != is used for inequality testing because <> is used for another purpose in Perl. And you’ll need >= and not => for “greater than or equal to” because the latter is used for another purpose in Perl. In fact, nearly every sequence of punctuation is used for something in Perl. So, if you get writer’s block, just let the cat walk across the keyboard, and debug the result. To compare strings, Perl has an equivalent set of string comparison operators which look like funny little words: lt, le, eq, ge, gt, and ne. These compare two strings char- acter-by-character to see whether they’re the same, or whether one comes first in standard string sorting order. Note that the order of characters in ASCII or Unicode is not an order that might make sense to you. You’ll see how to fix that in Chapter 14. The comparison operators (for both numbers and strings) are given in Table 2-3. 36 | Chapter 2: Scalar Data Table 2-3. Numeric and string comparison operators Comparison Numeric String Equal == eq Not equal != ne Less than < lt Greater than > gt Less than or equal to <= le Greater than or equal to >= ge Here are some example expressions using these comparison operators: 35 != 30 + 5 # false 35 == 35.0 # true '35' eq '35.0' # false (comparing as strings) 'fred' lt 'barney' # false 'fred' lt 'free' # true 'fred' eq "fred" # true 'fred' eq 'Fred' # false ' ' gt '' # true The if Control Structure Once you can compare two values, you’ll probably want your program to make deci- sions based upon that comparison. Like all similar languages, Perl has an if control structure that only executes if its condition returns a true value: if ($name gt 'fred') { print "'$name' comes after 'fred' in sorted order.\n"; } If you need an alternative choice, the else keyword provides that as well: if ($name gt 'fred') { print "'$name' comes after 'fred' in sorted order.\n"; } else { print "'$name' does not come after 'fred'.\n"; print "Maybe it's the same string, in fact.\n"; } You must have those block curly braces around the conditional code, unlike C (whether you know C or not). It’s a good idea to indent the contents of the blocks of code as we show here; that makes it easier to see what’s going on. If you’re using a programmers’ text editor (as we show in Chapter 1), it should do most of that work for you. The if Control Structure | 37 Boolean Values You may actually use any scalar value as the conditional of the if control structure. That’s handy if you want to store a true or false value into a variable, like this: $is_bigger = $name gt 'fred'; if ($is_bigger) { ... } But how does Perl decide whether a given value is true or false? Perl doesn’t have a separate Boolean datatype, like some languages have. Instead, it uses a few simple rules:‡ • If the value is a number, 0 means false; all other numbers mean true. • Otherwise, if the value is a string, the empty string ('') means false; all other strings mean true. • Otherwise (that is, if the value is another kind of scalar than a number or a string), convert it to a number or a string and try again.§ There’s one trick hidden in those rules. Because the string '0' is the exact same scalar value as the number 0, Perl has to treat them both the same. That means that the string '0' is the only non-empty string that is false. If you need to get the opposite of any Boolean value, use the unary not operator, !. If what follows is a true value, it returns false; if what follows is false, it returns true: if (! $is_bigger) { # Do something when $is_bigger is not true } Here’s a handy trick. Since the ! changes true to false and false to true, and since Perl doesn’t have a separate Boolean type, the ! has to return some scalar to represent true and false. It turns out that 1 and 0 are good enough values, so some people like to standardize their values to just those values. To do that, they double up the ! to turn true into false into true again (or the other way around): $still_true = !! 'Fred'; $still_false = !! '0'; However, this idiom isn’t documented to always return exactly the values 1 or 0, but we don’t think that behavior will change any time soon. ‡ These aren’t the rules that Perl uses, of course. These are some rules that you can use to get essentially the same result, though. § This means that undef (which you’ll see soon) means false, while all references (which we cover in Intermediate Perl) are true. 38 | Chapter 2: Scalar Data Getting User Input At this point, you’re probably wondering how to get a value from the keyboard into a Perl program. Here’s the simplest way: use the line-input operator, .‖ Each time you use in a place where Perl expects a scalar value, Perl reads the next complete text line from standard input (up to the first newline), and uses that string as the value of . Standard input can mean many things, but unless you do something uncommon, it means the keyboard of the user who invoked your program (probably you). If there’s nothing waiting for to read (typically the case, unless you type ahead a complete line), the Perl program will stop and wait for you to enter some characters followed by a newline (return).# The string value of typically has a newline character on the end of it,* so you could do something like this: $line = ; if ($line eq "\n") { print "That was just a blank line!\n"; } else { print "That line of input was: $line"; } But in practice, you don’t often want to keep the newline, so you need the chomp() operator. The chomp Operator The first time you read about the chomp() operator, it seems terribly overspecialized. It works on a variable. The variable has to hold a string, and if the string ends in a newline character, chomp() removes the newline. That’s (nearly) all it does. For example: $text = "a line of text\n"; # Or the same thing from chomp($text); # Gets rid of the newline character But it turns out to be so useful, you’ll put it into nearly every program you write. As you see, it’s the best way to remove a trailing newline from a string in a variable. In fact, there’s an easier way to use chomp() because of a simple rule: any time that you need a variable in Perl, you can use an assignment instead. First, Perl does the assignment. ‖ This is actually a line-input operator working on the filehandle STDIN, but we can’t tell you about that until we get to filehandles (in Chapter 5). #To be honest, it’s normally your system that waits for the input; perl waits for your system. Although the details depend upon your system and its configuration, you can generally correct your mistyping with a backspace key before you press return—your system handles that, not perl itself. If you need more control over the input, get the Term::ReadLine module from CPAN. * The exception is if the standard input stream somehow runs out in the middle of a line. But that’s not a proper text file, of course! The chomp Operator | 39 Then it uses the variable in whatever way you requested. So the most common use of chomp() looks like this: chomp($text = ); # Read the text, without the newline character $text = ; # Do the same thing... chomp($text); # ...but in two steps At first glance, the combined chomp() may not seem to be the easy way, especially if it seems more complex! If you think of it as two operations—read a line, then chomp() it—then it’s more natural to write it as two statements. But if you think of it as one operation—read just the text, not the newline—it’s more natural to write the one statement. And since most other Perl programmers are going to write it that way, you may as well get used to it now. chomp() is actually a function. As a function, it has a return value, which is the number of characters removed. This number is hardly ever useful: $food = ; $betty = chomp $food; # gets the value 1 - but you knew that! As you see, you may write chomp() with or without the parentheses. This is another general rule in Perl: except in cases where it changes the meaning to remove them, parentheses are always optional. If a line ends with two or more newlines,† chomp() removes only one. If there’s no newline, it does nothing, and returns zero. The while Control Structure Like most algorithmic programming languages, Perl has a number of looping struc- tures.‡ The while loop repeats a block of code as long as a condition is true: $count = 0; while ($count < 10) { $count += 2; print "count is now $count\n"; # Gives values 2 4 6 8 10 } As always in Perl, the truth value here works like the truth value in the if test. Also like the if control structure, the block curly braces are required. The conditional expression is evaluated before the first iteration, so the loop may be skipped completely if the condition is initially false. † This situation can’t arise if you’re reading a line at a time, but it certainly can when you have set the input separator ($/) to something other than newline, or used the read function, or perhaps have glued some strings together yourself. ‡ Every programmer eventually creates an infinite loop by accident. If your program keeps running and running, though, you can generally stop it in the same way you’d stop any other program on your system. Often, typing Control-C will stop a runaway program; check with your system’s documentation to be sure. 40 | Chapter 2: Scalar Data The undef Value What happens if you use a scalar variable before you give it a value? Nothing serious, and definitely nothing fatal. Variables have the special undef value before they are first assigned, which is just Perl’s way of saying, “Nothing here to look at—move along, move along.” If you try to use this “nothing” as a “numeric something,” it acts like zero. If you try to use it as a “string something,” it acts like the empty string. But undef is neither a number nor a string; it’s an entirely separate kind of scalar value. Because undef automatically acts like zero when used as a number, it’s easy to make a numeric accumulator that starts out empty: # Add up some odd numbers $n = 1; while ($n < 10) { $sum += $n; $n += 2; # On to the next odd number } print "The total was $sum.\n"; This works properly when $sum was undef before the loop started. The first time through the loop $n is one, so the first line inside the loop adds one to $sum. That’s like adding one to a variable that already holds zero (because you’re using undef as if it were a number). So now it has the value 1. After that, since it’s been initialized, addition works in the traditional way. Similarly, you could have a string accumulator that starts out empty: $string .= "more text\n"; If $string is undef, this will act as if it already held the empty string, putting "more text \n" into that variable. But if it already holds a string, the new text is simply appended. Perl programmers frequently use a new variable in this way, letting it act as either zero or the empty string as needed. Many operators return undef when the arguments are out of range or don’t make sense. If you don’t do anything special, you’ll get a zero or a null string without major con- sequences. In practice, this is hardly a problem. In fact, most programmers will rely upon this behavior. But you should know that when warnings are turned on, Perl will typically warn about unusual uses of the undefined value, since that may indicate a bug. For example, simply copying undef from one variable into another isn’t a problem, but trying to print it generally causes a warning. The undef Value | 41 The defined Function One operator that can return undef is the line-input operator, . Normally, it will return a line of text. But if there is no more input, such as at end-of-file, it returns undef to signal this.§ To tell whether a value is undef and not the empty string, use the defined function, which returns false for undef, and true for everything else: $madonna = ; if ( defined($madonna) ) { print "The input was $madonna"; } else { print "No input available!\n"; } If you’d like to make your own undef values, you can use the obscurely named undef operator: $madonna = undef; # As if it had never been touched Exercises See “Answers to Exercises” on page 296 for answers to the following exercises: 1. [5] Write a program that computes the circumference of a circle with a radius of 12.5. Circumference is 2π times the radius (approximately 2 times 3.141592654). The answer you get should be about 78.5. 2. [4] Modify the program from the previous exercise to prompt for and accept a radius from the person running the program. So, if the user enters 12.5 for the radius, she should get the same number as in the previous exercise. 3. [4] Modify the program from the previous exercise so that, if the user enters a number less than zero, the reported circumference will be zero, rather than negative. 4. [8] Write a program that prompts for and reads two numbers (on separate lines of input) and prints out the product of the two numbers multiplied together. 5. [8] Write a program that prompts for and reads a string and a number (on separate lines of input) and prints out the string the number of times indicated by the num- ber on separate lines. (Hint: use the x operator.) If the user enters “fred” and “3”, the output should be three lines, each saying “fred”. If the user enters “fred” and “299792,” there may be a lot of output. § Normally, there’s no “end-of-file” when the input comes from the keyboard, but input may have been redirected to come from a file. Or the user may have pressed the key that the system recognizes to indicate end-of-file. 42 | Chapter 2: Scalar Data CHAPTER 3 Lists and Arrays If a scalar is the “singular” in Perl, as we described it at the beginning of Chapter 2, the “plural” in Perl is represented by lists and arrays. A list is an ordered collection of scalars. An array is a variable that contains a list. People tend to use the terms interchangeably, but there’s a big difference. The list is the data and the array is the variable that stores the data. You can have a list value that isn’t in an array, but every array variable holds a list (although that list may be empty). Fig- ure 3-1 represents a list, whether it’s stored in an array or not. Since lists and arrays share many of the same operations, just like scalar values and variables do, we’ll treat them in parallel. Don’t forget their differences though. Figure 3-1. A list with five elements 43 Each element of an array or list is a separate scalar value. These values are ordered— that is, they have a particular sequence from the first to the last element. The elements of an array or a list are indexed by small integers starting at zero* and counting by ones, so the first element of any array or list is always element zero. Since each element is an independent scalar value, a list or array may hold numbers, strings, undef values, or any mixture of different scalar values. Nevertheless, it’s com- mon to have all elements of the same type, such as a list of book titles (all strings) or a list of cosines (all numbers). Arrays and lists can have any number of elements. The smallest one has no elements, while the largest can fill all of available memory. Once again, this is in keeping with Perl’s philosophy of “no unnecessary limits.” Accessing Elements of an Array If you’ve used arrays in another language, you won’t be surprised to find that Perl provides a way to subscript an array in order to refer to an element by a numeric index. The array elements are numbered using sequential integers,† beginning at zero and increasing by one for each element, like this: $fred[0] = "yabba"; $fred[1] = "dabba"; $fred[2] = "doo"; The array name itself (in this case, fred) is from a completely separate namespace than scalars use; you can have a scalar variable named $fred in the same program, and Perl will treat them as different things and won’t be confused.‡ (Your maintenance pro- grammer might be confused, though, so don’t capriciously make all of your variable names the same!) You can use an array element like $fred[2] in every place§ where you could use any other scalar variable like $fred. For example, you can get the value from an array ele- ment or change that value by the same sort of expressions you used in Chapter 2: print $fred[0]; $fred[2] = "diddley"; $fred[1] .= "whatsis"; * Array and list indices always start at zero in Perl, unlike in some other languages. In early Perl, it was possible to change the starting number of array and list indexing (not for just one array or list, but for all of them at once!). Larry later realized that this was a misfeature, and its (ab)use is now strongly discouraged. But, if you’re terminally curious, look up the $[ variable in the perlvar documentation. † Yes, you can use the negative ones too, but we’ll show that later. ‡ The syntax is always unambiguous—tricky perhaps, but unambiguous. § Well, almost. The most notable exception is that the control variable of a foreach loop, which you’ll see in “The foreach Control Structure” on page 53, must be a simple scalar. And there are others, like the “indirect object slot” and “indirect filehandle slot” for print and printf. 44 | Chapter 3: Lists and Arrays Of course, the subscript may be any expression that gives a numeric value. If it’s not an integer already, Perl will automatically truncate it (not round!) to the next lower integer: $number = 2.71828; print $fred[$number – 1]; # Same as printing $fred[1] If the subscript indicates an element that would be beyond the end of the array, the corresponding value will be undef. This is just as with ordinary scalars; if you’ve never stored a value into the variable, it’s undef: $blank = $fred[ 142_857 ]; # unused array element gives undef $blanc = $mel; # unused scalar $mel also gives undef Special Array Indices If you store into an array element that is beyond the end of the array, the array is automatically extended as needed—there’s no limit on its length, as long as there’s available memory for Perl to use.‖ If Perl needs to create the intervening elements, it creates them as undef values: $rocks[0] = 'bedrock'; # One element... $rocks[1] = 'slate'; # another... $rocks[2] = 'lava'; # and another... $rocks[3] = 'crushed rock'; # and another... $rocks[99] = 'schist'; # now there are 95 undef elements Sometimes, you need to find out the last element index in an array. For the array of rocks, the last element index is $#rocks.# That’s not the same as the number of elements, though, because there’s an element number zero: $end = $#rocks; # 99, which is the last element's index $number_of_rocks = $end + 1; # okay, but you'll see a better way later $rocks[ $#rocks ] = 'hard rock'; # the last rock Using the $#name value as an index, like that last example, happens often enough that Larry has provided a shortcut: negative array indices count from the end of the array. But don’t get the idea that these indices “wrap around.” If you have three elements in the array, the valid negative indices are –1 (the last element), –2 (the middle element), and –3 (the first element). If you try –4 and beyond, you just get undef. In the real world, nobody seems to use any of these except –1, though: $rocks[ –1 ] = 'hard rock'; # easier way to do that last example $dead_rock = $rocks[–100]; # gets 'bedrock' $rocks[ –200 ] = 'crystal'; # fatal error! ‖ This isn’t strictly true. The largest array index is the size of a signed integer, so, up to now, you can only have 2,147,483,647 entries. At the risk of repeating history, “that should be enough for anyone.” #Blame this ugly syntax on the C shell. Fortunately, you don’t have to look at this very often in the real world. Special Array Indices | 45 List Literals A list literal (the way you represent a list value within your program) is a list of comma- separated values enclosed in parentheses. These values form the elements of the list. For example: (1, 2, 3) # list of three values 1, 2, and 3 (1, 2, 3,) # the same three values (the trailing comma is ignored) ("fred", 4.5) # two values, "fred" and 4.5 ( ) # empty list - zero elements (1..100) # list of 100 integers That last one uses the .. range operator, which you see here for the first time. That operator creates a list of values by counting from the left scalar up to the right scalar by ones.* For example: (1..5) # same as (1, 2, 3, 4, 5) (1.7..5.7) # same thing; both values are truncated (5..1) # empty list; .. only counts "uphill" (0, 2..6, 10, 12) # same as (0, 2, 3, 4, 5, 6, 10, 12) ($m..$n) # range determined by current values of $m and $n (0..$#rocks) # the indices of the rocks array from the previous section As you can see from those last two items, the elements of a list literal are not necessarily constants—they can be expressions that will be newly evaluated each time the literal is used. For example: ($m, 17) # two values: the current value of $m, and 17 ($m+$o, $p+$q) # two values Of course, a list may have any scalar values, like this typical list of strings: ("fred", "barney", "betty", "wilma", "dino") The qw Shortcut It turns out that lists of simple words (like the previous example) are frequently needed in Perl programs. The qw shortcut makes it easy to generate them without typing a lot of extra quote marks: qw( fred barney betty wilma dino ) # same as above, but less typing qw stands for “quoted words” or “quoted by whitespace,” depending upon whom you ask. Either way, Perl treats it like a single-quoted string (so, you can’t use \n or $fred inside a qw list as you would in a double-quoted string). The whitespace (characters like spaces, tabs, and newlines) disappear and whatever is left becomes the list of items. Since whitespace is insignificant, here’s another (but unusual) way to write that same list: * The range operator only counts up, unfortunately, but Perl has a way around that. 46 | Chapter 3: Lists and Arrays qw(fred barney betty wilma dino) # same as above, but pretty strange whitespace Since qw is a form of quoting, though, you can’t put comments inside a qw list. Some people like to format their lists with one element per line, which makes it easy to read as a column: qw( fred barney betty wilma dino ) The previous two examples have used parentheses as the delimiter, but Perl actually lets you choose any punctuation character as the delimiter. Here are some of the com- mon ones: qw! fred barney betty wilma dino ! qw/ fred barney betty wilma dino / qw# fred barney betty wilma dino # # like in a comment! Sometimes the two delimiters can be different. If the opening delimiter is one of those “left” characters, the corresponding “right” character is the proper closing delimiter: qw( fred barney betty wilma dino ) qw{ fred barney betty wilma dino } qw[ fred barney betty wilma dino ] qw< fred barney betty wilma dino > If you need to include the closing delimiter within the string as one of the characters, you probably picked the wrong delimiter. But even if you can’t or don’t want to change the delimiter, you can still include the character using the backslash: qw! yahoo\! google ask msn ! # include yahoo! as an element As in single-quoted strings, two consecutive backslashes contribute one single back- slash to the item: qw( This as a \\ real backslash ); Now, although the Perl motto is “There’s More Than One Way To Do It,” you may well wonder why anyone would need all of those different ways! Well, you’ll see later that there are other kinds of quoting where Perl uses this same rule, and it can come in handy in many of those. But even here, it could be useful if you need a list of Unix filenames: qw{ /usr/dict/words /home/rootbeer/.ispell_english } List Literals | 47 That list would be quite inconvenient to read, write, and maintain if you could only use the / as a delimiter. List Assignment In much the same way as you can assign scalar values to variables, you can assign list values to variables: ($fred, $barney, $dino) = ("flintstone", "rubble", undef); All three variables in the list on the left get new values, just as if you did three separate assignments. Since the list on the right side is built up before the assignment starts, this makes it easy to swap two variables’ values in Perl:† ($fred, $barney) = ($barney, $fred); # swap those values ($betty[0], $betty[1]) = ($betty[1], $betty[0]); But what happens if the number of variables (on the left side of the equals sign) isn’t the same as the number of values (from the right side)? In a list assignment, extra values are silently ignored—Perl figures that if you wanted those values stored somewhere, you would have told it where to store them. Alternatively, if you have too many vari- ables, the extras get the value undef:‡ ($fred, $barney) = qw< flintstone rubble slate granite >; # two ignored items ($wilma, $dino) = qw[flintstone]; # $dino gets undef Now that you can assign lists, you could build up an array of strings with a line of code like this:§ ($rocks[0], $rocks[1], $rocks[2], $rocks[3]) = qw/talc mica feldspar quartz/; But when you wish to refer to an entire array, Perl has a simpler notation. Just use the at sign (@) before the name of the array (and no index brackets after it) to refer to the entire array at once. You can read this as “all of the,” so @rocks is “all of the rocks.”‖ This works on either side of the assignment operator: @rocks = qw/ bedrock slate lava /; @tiny = ( ); # the empty list @giant = 1..1e5; # a list with 100,000 elements @stuff = (@giant, undef, @giant); # a list with 200,001 elements $dino = "granite"; @quarry = (@rocks, "crushed rock", @tiny, $dino); † As opposed to languages like C, in which there is no easy way to do this in general. C programmers use an auxiliary swap variable to temporarily hold the value, possibly managed via a macro. ‡ Well, that’s true for scalar variables. Array variables get an empty list, as you’ll see in a moment. § We’re cheating by assuming that the rocks array is empty before this statement. If there were a value in $rocks[7], say, this assignment wouldn’t affect that element. ‖ Larry claims that he chose the dollar and at sign because they can be read as $calar (scalar) and @rray (array). If you don’t get that, or can’t remember it that way, no big deal. 48 | Chapter 3: Lists and Arrays That last assignment gives @quarry the five-element list (bedrock, slate, lava, crushed rock, granite), since @tiny contributes zero elements to the list. (In particular, it doesn’t add an undef item into the list—but you could do that explicitly, as we did with @stuff earlier.) It’s also worth noting that an array name expands to the list it contains. An array doesn’t become an element in the list, because these arrays can contain only scalars, not other arrays.# The value of an array variable that has not yet been assigned is ( ), the empty list. Just as new, empty scalars start out with undef, new, empty arrays start out with the empty list. When an array is copied to another array, it’s still a list assignment. The lists are simply stored in arrays. For example: @copy = @quarry; # copy a list from one array to another The pop and push Operators You could add new items to the end of an array by simply storing them as elements with new, larger indices. But real Perl programmers don’t use indices.* So in the next few sections, we’ll present some ways to work with an array without using indices. One common use of an array is as a stack of information, where you add new values to and remove old values from the righthand side of the list, like a stack of plates in a cafeteria.† The righthand side is the end with the “last” items in the array, the end with the highest index values. These operations occur often enough to have their own special functions. The pop operator takes the last element off of an array and returns it: @array = 5..9; $fred = pop(@array); # $fred gets 9, @array now has (5, 6, 7, 8) $barney = pop @array; # $barney gets 8, @array now has (5, 6, 7) pop @array; # @array now has (5, 6). (The 7 is discarded.) That last example uses pop “in a void context,” which is merely a fancy way of saying the return value isn’t going anywhere. There’s nothing wrong with using pop in this way, if that’s what you want. #But in Intermediate Perl, we’ll show you a special kind of scalar called a reference that lets you make what are informally called “lists of lists,” among other interesting and useful structures. But in that case, you’re still not really storing a list into a list; you’re storing a reference to an array. * Of course, we’re joking, but there’s a kernel of truth in this joke. Indexing into arrays is not using Perl’s strengths. If you use the pop, push, and similar operators that avoid using indexing, your code will generally be faster than if you use many indices, and you avoid “off-by-one” errors, often called “fencepost” errors. Occasionally, a beginning Perl programmer (wanting to see how Perl’s speed compares to C’s) will take, say, a sorting algorithm optimized for C (with many array index operations), rewrite it straightforwardly in Perl (again, with many index operations) and wonder why it’s so slow. The answer is that using a Stradivarius violin to pound nails should not be considered a sound construction technique. † The other way is a queue, where you add to the end but take from the front. List Assignment | 49 If the array is empty, pop leaves it alone (since there is no element to remove) and returns undef. You may have noticed that you can use pop with or without parentheses. This is a general rule in Perl: as long you don’t change the meaning by removing the parentheses, they’re optional.‡ The converse operation is push, which adds an element (or a list of elements) to the end of an array: push(@array, 0); # @array now has (5, 6, 0) push @array, 8; # @array now has (5, 6, 0, 8) push @array, 1..10; # @array now has those 10 new elements @others = qw/ 9 0 2 1 0 /; push @array, @others; # @array now has those five new elements (19 total) Note that the first argument to push or the only argument for pop must be an array variable—pushing and popping would not make sense on a literal list. The shift and unshift Operators The push and pop operators do things to the end of an array (or the right side of an array, or the portion with the highest subscripts, depending upon how you like to think of it). Similarly, the unshift and shift operators perform the corresponding actions on the “start” of the array (or the “left” side of an array, or the portion with the lowest subscripts). Here are a few examples: @array = qw# dino fred barney #; $m = shift(@array); # $m gets "dino", @array now has ("fred", "barney") $n = shift @array; # $n gets "fred", @array now has ("barney") shift @array; # @array is now empty $o = shift @array; # $o gets undef, @array is still empty unshift(@array, 5); # @array now has the one-element list (5) unshift @array, 4; # @array now has (4, 5) @others = 1..3; unshift @array, @others; # @array now has (1, 2, 3, 4, 5) Analogous to pop, shift returns undef if you give it an empty array variable. The splice Operator The push-pop and shift-unshift operators work with the ends of the array, but what if you need to remove or add elements to the middle? That’s where the splice operator comes in. It takes up to four arguments, two of which are optional. The first argument is always the array and the second argument is the position where you want to start. If you only use those two arguments, Perl removes all of the elements from your starting position to the end and returns them to you: ‡ You might recognize that this is a tautology. 50 | Chapter 3: Lists and Arrays @array = qw( pebbles dino fred barney betty ); @removed = splice @array, 2; # remove everything after fred # @removed is qw(fred barney betty) # @array is qw(pebbles dino) You can use a third argument to specify a length. Read that sentence again because many people assume that the third argument is an ending position, but no, it’s a length. That way you can remove elements from the middle and leave some at the end: @array = qw( pebbles dino fred barney betty ); @removed = splice @array, 1, 2; # remove dino, fred # @removed is qw(dino fred) # @array is qw(pebbles barney betty) The fourth argument is a replacement list. At the same time that you take some elements out, you can put others in. The replacement list does not need to be the same size as the slice that you are removing: @array = qw( pebbles dino fred barney betty ); @removed = splice @array, 1, 2, qw(wilma); # remove dino, fred # @removed is qw(dino fred) # @array is qw(pebbles wilma # barney betty) You don’t have to remove any elements. If you specify a length of 0, you remove no elements but still insert the “replacement” list: @array = qw( pebbles dino fred barney betty ); @removed = splice @array, 1, 0, qw(wilma); # remove nothing # @removed is qw() # @array is qw(pebbles wilma dino # fred barney betty) Notice that wilma shows up before dino. Perl inserted the replacement list starting at index 1 and moved everything else over. splice might not seem like a big deal to you, but this is a hard thing to do in some languages, and many people developed complicated techniques, such as linked lists, that take a lot of programmer attention to get right. Interpolating Arrays into Strings As with scalars, you can interpolate array values into a double-quoted string. Perl ex- pands the array and automatically adds spaces between the elements, putting the whole result in the string§ upon interpolation: @rocks = qw{ flintstone slate rubble }; print "quartz @rocks limestone\n"; # prints five rocks separated by spaces § Actually, the separator is the value of the special $" variable, which is a space by default. Interpolating Arrays into Strings | 51 There are no extra spaces added before or after an interpolated array; if you want those, you’ll have to put them in yourself: print "Three rocks are: @rocks.\n"; print "There's nothing in the parens (@empty) here.\n"; If you forget that arrays interpolate like this, you’ll be surprised when you put an email address into a double-quoted string: $email = "fred@bedrock.edu"; # WRONG! Tries to interpolate @bedrock Although you probably intended to have an email address, Perl sees the array named @bedrock and tries to interpolate it. Depending on your version of Perl, you’ll probably just get a warning:‖ Possible unintended interpolation of @bedrock To get around this problem, you either escape the @ in a double-quoted string or use a single-quoted string: $email = "fred\@bedrock.edu"; # Correct $email = 'fred@bedrock.edu'; # Another way to do that A single element of an array interpolates into its value, just as you’d expect from a scalar variable: @fred = qw(hello dolly); $y = 2; $x = "This is $fred[1]'s place"; # "This is dolly's place" $x = "This is $fred[$y-1]'s place"; # same thing Note that the index expression evaluates as an ordinary expression, as if it were outside a string. It is not variable-interpolated first. In other words, if $y contains the string "2*4", we’re still talking about element 1, not element 7, because the string "2*4" as a number (the value of $y used in a numeric expression) is just plain 2.# If you want to follow a simple scalar variable with a left square bracket, you need to delimit the square bracket so that it isn’t considered part of an array reference, as follows: @fred = qw(eating rocks is wrong); $fred = "right"; # we are trying to say "this is right[3]" print "this is $fred[3]\n"; # prints "wrong" using $fred[3] print "this is ${fred}[3]\n"; # prints "right" (protected by braces) print "this is $fred"."[3]\n"; # right again (different string) print "this is $fred\[3]\n"; # right again (backslash hides it) ‖ Some Perl versions before 5.6 actually made this a fatal error, but they changed it to a warning because that was too annoying. That shouldn’t be a problem if you’re using a recent version of Perl. #Of course, if you turn on warnings, Perl is likely to remind you that "2*4" is a pretty funny-looking number. 52 | Chapter 3: Lists and Arrays The foreach Control Structure It’s handy to be able to process an entire array or list, so Perl provides a control structure to do just that. The foreach loop steps through a list of values, executing one iteration (time through the loop) for each value: foreach $rock (qw/ bedrock slate lava /) { print "One rock is $rock.\n"; # Prints names of three rocks } The control variable ($rock in that example) takes on a new value from the list for each iteration. The first time through the loop, it’s "bedrock"; the third time, it’s "lava". The control variable is not a copy of the list element—it actually is the list element. That is, if you modify the control variable inside the loop, you modify the element itself, as shown in the following code snippet. This is useful, and supported, but it would surprise you if you weren’t expecting it: @rocks = qw/ bedrock slate lava /; foreach $rock (@rocks) { $rock = "\t$rock"; # put a tab in front of each element of @rocks $rock .= "\n"; # put a newline on the end of each } print "The rocks are:\n", @rocks; # Each one is indented, on its own line What is the value of the control variable after the loop has finished? It’s the same as it was before the loop started. Perl automatically saves and restores the value of the con- trol variable of a foreach loop. While the loop is running, there’s no way to access or alter that saved value. So after the loop is done, the variable has the value it had before the loop, or undef if it hadn’t had a value: $rock = 'shale'; @rocks = qw/ bedrock slate lava /; foreach $rock (@rocks) { ... } print "rock is still $rock\n"; # 'rock is still shale' That means that if you want to name your loop control variable $rock, you don’t have to worry that maybe you’ve already used that name for another variable. After we in- troduce subroutines to you in Chapter 4, we’ll show you a better way to handle that. The foreach Control Structure | 53 Perl’s Favorite Default: $_ If you omit the control variable from the beginning of the foreach loop, Perl uses its favorite default variable, $_. This is (mostly) just like any other scalar variable, except for its unusual name. For example: foreach (1..10) { # Uses $_ by default print "I can count to $_!\n"; } Although this isn’t Perl’s only default by a long shot, it’s Perl’s most common default. You’ll see many other cases in which Perl will automatically use $_ when you don’t tell it to use some other variable or value, thereby saving the programmer from the heavy labor of having to think up and type a new variable name. So as not to keep you in suspense, one of those cases is print, which will print $_ if given no other argument: $_ = "Yabba dabba doo\n"; print; # prints $_ by default The reverse Operator The reverse operator takes a list of values (which may come from an array) and returns the list in the opposite order. So if you were disappointed that the range operator (..) only counts upward, this is the way to fix it: @fred = 6..10; @barney = reverse(@fred); # gets 10, 9, 8, 7, 6 @wilma = reverse 6..10; # gets the same thing, without the other array @fred = reverse @fred; # puts the result back into the original array The last line is noteworthy because it uses @fred twice. Perl always calculates the value being assigned (on the right) before it begins the actual assignment. Remember that reverse returns the reversed list; it doesn’t affect its arguments. If the return value isn’t assigned anywhere, it’s useless: reverse @fred; # WRONG - doesn't change @fred @fred = reverse @fred; # that's better The sort Operator The sort operator takes a list of values (which may come from an array) and sorts them in the internal character ordering. For strings, that would be in code point order.* In pre-Unicode Perls, the sort order was based on ASCII, but Unicode maintains that same order as well as defining the order of many more characters. So, the code point order is a strange place where all of the capital letters come before all of the lowercase letters, where the numbers come before the letters, and the punctuation marks—well, those * The Unicode sorting assumes that you have no locale in effect, but you have to do something special to turn that on so you’re probably not using locales. 54 | Chapter 3: Lists and Arrays are here, there, and everywhere. But sorting in that order is just the default behavior; you’ll see in Chapter 14 how to sort in whatever order you’d like. The sort operator takes an input list, sorts it, and outputs a new list: @rocks = qw/ bedrock slate rubble granite /; @sorted = sort(@rocks); # gets bedrock, granite, rubble, slate @back = reverse sort @rocks; # these go from slate to bedrock @rocks = sort @rocks; # puts sorted result back into @rocks @numbers = sort 97..102; # gets 100, 101, 102, 97, 98, 99 As you can see from that last example, sorting numbers as if they were strings may not give useful results. But, of course, any string that starts with 1 has to sort before any string that starts with 9, according to the default sorting rules. And like what happened with reverse, the arguments themselves aren’t affected. If you want to sort an array, you must store the result back into that array: sort @rocks; # WRONG, doesn't modify @rocks @rocks = sort @rocks; # Now the rock collection is in order The each Operator Starting with Perl 5.12, you can use the each operator on arrays. Before that version, you could only use each with hashes, but we don’t show you those until Chapter 5. Every time that you call each on an array, it returns two values for the next element in the array—the index of the value and the value itself: use 5.012; @rocks = qw/ bedrock slate rubble granite /; while( my( $index, $value ) = each @rocks ) { say "$index: $value"; } If you wanted to do this without each, you have to iterate through all of the indices of the array and use the index to get the value: @rocks = qw/ bedrock slate rubble granite /; foreach $index ( 0 .. $#rocks ) { print "$index: $rocks[$index]\n"; } Depending on your task, one or the other may be more convenient for you. Scalar and List Context This is the most important section in this chapter. In fact, it’s the most important section in the entire book. In fact, it wouldn’t be an exaggeration to say that your entire career in using Perl will depend upon understanding this section. So if you’ve gotten away with skimming the text up to this point, this is where you should really pay attention. Scalar and List Context | 55 That’s not to say that this section is in any way difficult to understand. It’s actually a simple idea: a given expression may mean different things depending upon where it appears and how you use it. This is nothing new to you; it happens all the time in natural languages. For example, in English,† suppose someone asked you what the word “read”‡ means. It has different meanings depending on how it’s used. You can’t identify the meaning until you know the context. The context refers to how you use an expression. You’ve actually already seen some contextual operations with numbers and strings. When you do numbery sorts of things, you get numeric results. When you do stringy sorts of things, you get string results. And, it’s the operator that decides what you are doing, not the values. The * in 2*3 is numeric multiplication, while the x in <2×3> is string replication. The first gives you 8 while the second gives you 222. That’s context at work for you. As Perl is parsing your expressions, it always expects either a scalar value or a list value.§ What Perl expects is called the context of the expression:‖ 42 + something # The something must be a scalar sort something # The something must be a list Even if something is the exact same sequence of characters, in one case it may give a single, scalar value, while in the other, it may give a list.# Expressions in Perl always return the appropriate value for their context. For example, how about the “name”* of an array. In a list context, it gives the list of elements. But in a scalar context, it returns the number of elements in the array: @people = qw( fred barney betty ); @sorted = sort @people; # list context: barney, betty, fred $number = 42 + @people; # scalar context: 42 + 3 gives 45 Even ordinary assignment (to a scalar or a list) causes different contexts: @list = @people; # a list of three people $n = @people; # the number 3 † If you aren’t a native speaker of English, this analogy may not be obvious to you. But context sensitivity happens in every spoken language, so you may be able to think of an example in your own language. ‡ Or maybe they were asking what the word “red” means, if they were speaking rather than writing a book. It’s ambiguous either way. As Douglas Hofstadter said, no language can express every thought unambiguously, especially this one. § Unless, of course, Perl is expecting something else entirely. There are other contexts that aren’t covered here. In fact, nobody knows how many contexts Perl uses; the biggest brains in all of Perl haven’t agreed on an answer to that yet. ‖ This is no different than what you’re used to in human languages. If I make a grammatical mistake, you notice it right away because you expect certain words in places certain. Eventually, you’ll read Perl this way, too, but at first you have to think about it. #The list may be just one element long, of course. It could also be empty, or it could have any number of elements. * Well, the true name of the array @people is just people. The @ sign is just a qualifier. 56 | Chapter 3: Lists and Arrays But please don’t jump to the conclusion that scalar context always gives the number of elements that would have been returned in list context. Most list-producing expres- sions† return something much more interesting. Not only that, but you can’t make any general rules to apply what you know about some expressions to others. Each expression can make up its own rules. Or, really, follow the overall rule that isn’t very helpful to you: do the thing that makes the most sense for that context. Perl is very much a language that tries to do the most common, mostly right thing for you. Using List-Producing Expressions in Scalar Context There are many expressions that you will typically use to produce a list. If you use one in a scalar context, what do you get? See what the author of that operation says about it. Usually, that person is Larry, and usually the documentation gives the whole story. In fact, a big part of learning Perl is actually learning how Larry thinks.‡ Therefore, once you can think like Larry does, you know what Perl should do. But while you’re learning, you’ll probably need to look into the documentation. Some expressions don’t have a scalar-context value at all. For example, what should sort return in a scalar context? You wouldn’t need to sort a list to count its elements, so until someone implements something else, sort in a scalar context always returns undef. Another example is reverse. In a list context, it gives a reversed list. In a scalar context, it returns a reversed string (or reversing the result of concatenating all the strings of a list, if given one):§ @backwards = reverse qw/ yabba dabba doo /; # gives doo, dabba, yabba $backwards = reverse qw/ yabba dabba doo /; # gives oodabbadabbay At first, it’s not always obvious whether an expression is being used in a scalar or a list context. But, trust us, it will become second nature for you eventually. † But with regard to the point of this section, there’s no difference between a “list-producing” expression and a “scalar-producing” one; any expression can produce a list or a scalar, depending upon context. So when we say “list-producing expressions,” we mean expressions that are typically used in a list context and therefore might surprise you when they’re used unexpectedly in a scalar context (like reverse or @fred). ‡ This is only fair, since while writing Perl he tried to think like you do to predict what you would want! § One of us once cornered Larry in an elevator and asked him what problem he was solving with this, but he looked as far off into the distance as he could in an elevator and said, “It seemed like a good idea at the time.” Scalar and List Context | 57 Here are some common contexts to start you off: $fred = something; # scalar context @pebbles = something; # list context ($wilma, $betty) = something; # list context ($dino) = something; # still list context! Don’t be fooled by the one-element list; that last one is a list context, not a scalar one. The parentheses are significant here, making the fourth of those different than the first. If you assign to a list (no matter the number of elements), it’s a list context. If you assign to an array, it’s a list context. Here are some other expressions you’ve seen, and the contexts they provide. First, some that provide scalar context to something: $fred = something; $fred[3] = something; 123 + something something + 654 if (something) { ... } while (something) { ... } $fred[something] = something; And here are some that provide a list context: @fred = something; ($fred, $barney) = something; ($fred) = something; push @fred, something; foreach $fred (something) { ... } sort something reverse something print something Using Scalar-Producing Expressions in List Context Going this direction is straightforward: if an expression doesn’t normally have a list value, the scalar value is automatically promoted to make a one-element list: @fred = 6 * 7; # gets the one-element list (42) @barney = "hello" . ' ' . "world"; Well, there’s one possible catch: @wilma = undef; # OOPS! Gets the one-element list (undef) # which is not the same as this: @betty = ( ); # A correct way to empty an array Since undef is a scalar value, assigning undef to an array doesn’t clear the array. The better way to do that is to assign an empty list.‖ ‖ Well, in most real-world algorithms, if the variable is declared in the proper scope, you do not need to explicitly empty it. So this type of assignment is rare in well-written Perl programs. You’ll learn about scoping in Chapter 4. 58 | Chapter 3: Lists and Arrays Forcing Scalar Context On occasion, you may need to force scalar context where Perl is expecting a list. In that case, you can use the fake function scalar. It’s not a true function because it just tells Perl to provide a scalar context: @rocks = qw( talc quartz jade obsidian ); print "How many rocks do you have?\n"; print "I have ", @rocks, " rocks!\n"; # WRONG, prints names of rocks print "I have ", scalar @rocks, " rocks!\n"; # Correct, gives a number Oddly enough, there’s no corresponding function to force list context. It turns out you almost never need it. Trust us on this, too. in List Context One previously seen operator that returns a different value in an array context is the line-input operator, . As we described earlier, returns the next line of input in a scalar context. Now, in list context, this operator returns all of the remaining lines up to the end-of-file. It returns each line as a separate element of the list. For example: @lines = ; # read standard input in list context When the input is coming from a file, this will read the rest of the file. But how can there be an end-of-file when the input comes from the keyboard? On Unix and similar systems, including Linux and Mac OS X, you’ll normally type a Control-D# to indicate to the system that there’s no more input; the special character itself is never seen by Perl,* even though it may be echoed to the screen. On DOS/Windows systems, use Ctrl-Z instead.† You’ll need to check the documentation for your system or ask your local expert if it’s different from these. If the person running the program types three lines, then presses the proper keys needed to indicate end-of-file, the array ends up with three elements. Each element will be a string that ends in a newline, corresponding to the three newline-terminated lines entered. #This is merely the default; it can be changed by the stty command. But it’s pretty dependable—we’ve never seen a Unix system where a different character was used to mean end-of-file from the keyboard. * It’s the OS that “sees” the Control key and reports “end-of-file” to the application. † There’s a bug affecting some ports of Perl for DOS/Windows where the first line of output to the terminal following the use of Control-Z is obscured. On these systems, you can work around this problem by simply printing a blank line ("\n") after reading the input. in List Context | 59 Wouldn’t it be nice if, having read those lines, you could chomp the newlines all at once? It turns out that if you give chomp an array holding a list of lines, it will remove the newlines from each item in the list. For example: @lines = ; # Read all the lines chomp(@lines); # discard all the newline characters But the more common way to write that is with code similar to what you used earlier: chomp(@lines = ); # Read the lines, not the newlines Although you’re welcome to write your code either way in the privacy of your own cubicle, most Perl programmers will expect the second, more compact, notation. It may be obvious to you (but it’s not obvious to everyone) that once these lines of input have been read, they can’t be reread.‡ Once you’ve reached end-of-file, there’s no more input out there to read. And what happens if the input is coming from a 400 MB logfile? The line input operator reads all of the lines, gobbling up lots of memory.§ Perl tries not to limit you in what you can do, but the other users of your system (not to mention your system adminis- trator) are likely to object. If the input data is large, you should generally find a way to deal with it without reading it all into memory at once. Exercises See “Answers to Exercises” on page 298 for answers to the following exercises: 1. [6] Write a program that reads a list of strings on separate lines until end-of-input and prints out the list in reverse order. If the input comes from the keyboard, you’ll probably need to signal the end of the input by pressing Control-D on Unix, or Control-Z on Windows. 2. [12] Write a program that reads a list of numbers (on separate lines) until end-of- input and then prints for each number the corresponding person’s name from the list shown below. (Hardcode this list of names into your program. That is, it should appear in your program’s source code.) For example, if the input numbers were 1, 2, 4, and 2, the output names would be fred, betty, dino, and betty: fred betty barney dino wilma pebbles bamm-bamm ‡ Well, yes, if the input is from a source upon which you can seek, then you’ll be able to go back and read again. But that’s not what we’re talking about here. § Typically, that’s much more memory than the size of the file, too. That is, a 400MB file will typically take up at least a full gigabyte of memory when read into an array. This is because Perl will generally waste memory to save time. This is a good trade-off; if you’re short of memory, you can buy more; if you’re short on time, you’re hosed. 60 | Chapter 3: Lists and Arrays 3. [8] Write a program that reads a list of strings (on separate lines) until end-of-input. Then it should print the strings in code point order. That is, if you enter the strings fred, barney, wilma, betty, the output should show barney betty fred wilma. Are all of the strings on one line in the output or on separate lines? Could you make the output appear in either style? Exercises | 61 CHAPTER 4 Subroutines You’ve already seen and used some of the built-in system functions, such as chomp, reverse, print, and so on. But, as other languages do, Perl has the ability to make subroutines, which are user-defined functions.* These let you recycle one chunk of code many times in one program. The name of a subroutine is another Perl identifier (letters, digits, and underscores, but they can’t start with a digit) with a sometimes-optional ampersand (&) in front. There’s a rule about when you can omit the ampersand and when you cannot; you’ll see that rule by the end of the chapter. For now, just use it every time that it’s not forbidden, which is always a safe rule. We’ll tell you every place where it’s forbidden, of course. The subroutine name comes from a separate namespace, so Perl won’t be confused if you have a subroutine called &fred and a scalar called $fred in the same program— although there’s no reason to do that under normal circumstances. Defining a Subroutine To define your own subroutine, use the keyword sub, the name of the subroutine (without the ampersand), then the block of code in curly braces which makes up the body of the subroutine. Something like this: sub marine { $n += 1; # Global variable $n print "Hello, sailor number $n!\n"; } You may put your subroutine definitions anywhere in your program text, but pro- grammers who come from a background of languages like C or Pascal like to put them * Perl doesn’t generally make the distinction that Pascal programmers are used to, between functions, which return a value, and procedures, which don’t. But a subroutine is always user-defined, while a function may or may not be. That is, you may use the word function as a synonym for subroutine, or it may mean one of Perl’s built-in functions. That’s why this chapter is titled Subroutines; it’s about the ones you may define, not the built-ins. Mostly. 63 at the start of the file. Others may prefer to put them at the end of the file so that the main part of the program appears at the beginning. It’s up to you. In any case, you don’t normally need any kind of forward declaration.† Subroutine definitions are global; without some powerful trickiness, there are no private subroutines.‡ If you have two subroutine definitions with the same name,§ the later one overwrites the earlier one. Although, if you have warnings enabled, Perl will tell you when you do that. It’s gen- erally considered bad form, or the sign of a confused maintenance programmer. As you may have noticed in the previous example, you may use any global variables within the subroutine body. In fact, all of the variables you’ve seen so far are global; that is, they are accessible from every part of your program. This horrifies linguistic purists, but the Perl development team formed an angry mob with torches and ran them out of town years ago. You’ll see how to make private variables in the section “Private Variables in Subroutines” on page 68. Invoking a Subroutine You invoke a subroutine from within an expression by using the subroutine name (with the ampersand):‖ &marine; # says Hello, sailor number 1! &marine; # says Hello, sailor number 2! &marine; # says Hello, sailor number 3! &marine; # says Hello, sailor number 4! Most often, you refer to the invocation as simply calling the subroutine. You’ll also see other ways that you may call the subroutine as you go on in this chapter. Return Values You always invoke a subroutine as part of an expression, even if you don’t use the result of the expression. When you invoked &marine earlier, you were calculating the value of the expression containing the invocation, but then throwing away the result. † Unless your subroutine is being particularly tricky and declares a “prototype,” which dictates how a compiler will parse and interpret its invocation arguments. This is rare—see the perlsub documentation for more information. ‡ If you wish to be powerfully tricky, read the Perl documentation about coderefs stored in private (lexical) variables. § We don’t talk about subroutines of the same name in different packages until Intermediate Perl. ‖ And frequently a pair of parentheses, even if empty. As written, the subroutine inherits the caller’s @_ value, which we’ll show you shortly. So don’t stop reading here, or you’ll write code with unintended effects! 64 | Chapter 4: Subroutines Many times, you call a subroutine and actually do something with the result. This means that you do something with the return value of the subroutine. All Perl subroutines have a return value—there’s no distinction between those that return val- ues and those that don’t. Not all Perl subroutines have a useful return value, however. Since you can call Perl subroutines in a way that needs a return value, it’d be a bit wasteful to have to declare special syntax to “return” a particular value for the majority of the cases. So Larry made it simple. As Perl chugs along in a subroutine, it calculates values as part of its series of actions. Whatever calculation is last performed in a sub- routine is automatically also the return value. For example, this subroutine has an addition as the last expression: sub sum_of_fred_and_barney { print "Hey, you called the sum_of_fred_and_barney subroutine!\n"; $fred + $barney; # That's the return value } The last evaluated expression in the body of this subroutine is the sum of $fred and $barney, so the sum of $fred and $barney is the return value. Here’s that in action: $fred = 3; $barney = 4; $wilma = &sum_of_fred_and_barney; # $wilma gets 7 print "\$wilma is $wilma.\n"; $betty = 3 * &sum_of_fred_and_barney; # $betty gets 21 print "\$betty is $betty.\n"; That code produces this output: Hey, you called the sum_of_fred_and_barney subroutine! $wilma is 7. Hey, you called the sum_of_fred_and_barney subroutine! $betty is 21. That print statement is just a debugging aid, so you can see that you called the sub- routine. You normally take in those sorts of statements when you’re ready to deploy your program. But suppose you added another print to the end of the subroutine, like this: sub sum_of_fred_and_barney { print "Hey, you called the sum_of_fred_and_barney subroutine!\n"; $fred + $barney; # That's not really the return value! print "Hey, I'm returning a value now!\n"; # Oops! } The last expression evaluated is not the addition anymore; it’s now the print statement, whose return value is normally 1, meaning “printing was successful,”# but that’s not #The return value of print is true for a successful operation and false for a failure. You’ll see how to determine the kind of failure in Chapter 5. Return Values | 65 the return value you actually wanted. So be careful when adding additional code to a subroutine, since the last expression evaluated will be the return value. So, what happened to the sum of $fred and $barney in that second (faulty) subroutine? You didn’t put it anywhere, so Perl discarded it. If you had requested warnings, Perl (noticing that there’s nothing useful about adding two variables and discarding the result) would likely warn you about something like “a useless use of addition in a void context.” The term void context is just a fancy way of saying that you aren’t using the answer, whether that means storing it in a variable or using it any other way. “The last evaluated expression” really means the last expression that Perl evaluates, rather than the last statement in the subroutine. For example, this subroutine returns the larger value of $fred or $barney: sub larger_of_fred_or_barney { if ($fred > $barney) { $fred; } else { $barney; } } The last evaluated expression is either $fred or $barney, so the value of one of those variables becomes the return value. You don’t know if the return value will be $fred or $barney until you see what those variables hold at runtime. These are all rather trivial examples. It gets better when you can pass values that are different for each invocation into a subroutine instead of relying on global variables. In fact, that’s coming right up. Arguments That subroutine called larger_of_fred_or_barney would be much more useful if it didn’t force you to use the global variables $fred and $barney. If you wanted to get the larger value from $wilma and $betty, you currently have to copy those into $fred and $barney before you can use larger_of_fred_or_barney. And if you had something useful in those variables, you’d have to first copy those to other variables, say $save_fred and $save_barney. And then, when you’re done with the subroutine, you’d have to copy those back to $fred and $barney again. Luckily, Perl has subroutine arguments. To pass an argument list to the subroutine, simply place the list expression, in parentheses, after the subroutine invocation, like this: $n = &max(10, 15); # This sub call has two parameters Perl passes the list to the subroutine; that is, Perl makes the list available for the sub- routine to use however it needs to. Of course, you have to store this list somewhere, so Perl automatically stores the parameter list (another name for the argument list) in the 66 | Chapter 4: Subroutines special array variable named @_ for the duration of the subroutine. You can access this array to determine both the number of arguments and the value of those arguments. This means that the first subroutine parameter is in $_[0], the second one is stored in $_[1], and so on. But—and here’s an important note—these variables have nothing whatsoever to do with the $_ variable, any more than $dino[3] (an element of the @dino array) has to do with $dino (a completely distinct scalar variable). It’s just that the parameter list must be in some array variable for your subroutine to use it, and Perl uses the array @_ for this purpose. Now, you could write the subroutine &max to look a little like the subroutine &larger_of_fred_or_barney, but instead of using $fred you could use the first subrou- tine parameter ($_[0]), and instead of using $barney, you could use the second subroutine parameter ($_[1]). And so you could end up with something like this: sub max { # Compare this to &larger_of_fred_or_barney if ($_[0] > $_[1]) { $_[0]; } else { $_[1]; } } Well, as we said, you could do that. But it’s pretty ugly with all of those subscripts, and hard to read, write, check, and debug, too. You’ll see a better way in a moment. There’s another problem with this subroutine. The name &max is nice and short, but it doesn’t remind us that this subroutine works properly only if called with exactly two parameters: $n = &max(10, 15, 27); # Oops! max ignores the extra parameters since it never looks at $_[2]. Perl doesn’t care whether there’s something in there or not. Perl doesn’t care about insufficient parameters either—you simply get undef if you look beyond the end of the @_ array, as with any other array. You’ll see how to make a better &max, which works with any number of parameters, later in this chapter. The @_ variable is private to the subroutine;* if there’s a global value in @_, Perl saves it before it invokes the next subroutine and restores its previous value upon return from that subroutine.† This also means that a subroutine can pass arguments to another subroutine without fear of losing its own @_ variable—the nested subroutine invocation gets its own @_ in the same way. Even if the subroutine calls itself recursively, each * Unless there’s an ampersand in front of the name for the invocation, and no parentheses (or arguments) afterward, in which case the @_ array is inherited from the caller’s context. That’s generally a bad idea, but is occasionally useful. † You might recognize that this is the same mechanism as used with the control variable of the foreach loop, as seen in Chapter 3. In either case, the variable’s value is saved and automatically restored by Perl. Arguments | 67 invocation gets a new @_, so @_ is always the parameter list for the current subroutine invocation. Private Variables in Subroutines But if Perl can give you a new @_ for every invocation, can’t it give you variables for your own use as well? Of course it can. By default, all variables in Perl are global variables; that is, they are accessible from every part of the program. But you can create private variables called lexical variables at any time with the my operator: sub max { my($m, $n); # new, private variables for this block ($m, $n) = @_; # give names to the parameters if ($m > $n) { $m } else { $n } } These variables are private (or scoped) to the enclosing block; any other $m or $n is totally unaffected by these two. And that goes the other way, too—no other code can access or modify these private variables, by accident or design.‡ So, you could drop this sub- routine into any Perl program in the world and know that you wouldn’t mess up that program’s $m and $n (if any).§ It’s also worth pointing out that, inside those if’s blocks, you don’t need a semicolon after the return value expression. Although Perl allows you to omit the last semicolon in a block,‖ in practice you omit it only when the code is so simple that you can write the block in a single line. You can make the subroutine in the previous example even simpler. Did you notice that the list ($m, $n) shows up twice? You can apply the my operator to a list of variables enclosed in parentheses you use in a list assignment, so it’s customary to combine those first two statements in the subroutine: my($m, $n) = @_; # Name the subroutine parameters That one statement creates the private variables and sets their values, so the first parameter now has the easier-to-use name $m and the second has $n. Nearly every sub- routine starts with a line much like that one, naming its parameters. When you see that line, you’ll know that the subroutine expects two scalar parameters, which you’ll call $m and $n inside the subroutine. ‡ Advanced programmers will realize that a lexical variable’s data may be accessible by reference from outside its scope, but never by name. We show that in Intermediate Perl. § Of course, if that program already had a subroutine called &max, you’d mess that up. ‖ The semicolon is really a statement separator, not a statement terminator. 68 | Chapter 4: Subroutines Variable-Length Parameter Lists In real-world Perl code, subroutines often have parameter lists of arbitrary length. That’s because of Perl’s “no unnecessary limits” philosophy that you’ve already seen. Of course, this is unlike many traditional programming languages, which require every subroutine to be strictly typed; that is, to permit only a certain predefined number of parameters of predefined types. It’s nice that Perl is so flexible, but (as you saw with the &max routine earlier) that may cause problems when you call a subroutine with a different number of arguments than it expects. Of course, you can easily check that the subroutine has the right number of arguments by examining the @_ array. For example, you could have written &max to check its ar- gument list like this:# sub max { if (@_ != 2) { print "WARNING! &max should get exactly two arguments!\n"; } # continue as before... . . . } That if test uses the “name” of the array in a scalar context to find out the number of array elements, as you saw in Chapter 3. But in real-world Perl programming, virtually no one really uses this sort of check; it’s better to make your subroutines adapt to the parameters. A Better &max Routine Rewrite &max to allow for any number of arguments, so you can call it like this: $maximum = &max(3, 5, 10, 4, 6); sub max { my($max_so_far) = shift @_; # the first one is the largest yet seen foreach (@_) { # look at the remaining arguments if ($_ > $max_so_far) { # could this one be bigger yet? $max_so_far = $_; } } $max_so_far; } #As soon as you learn about warn in Chapter 5, you’ll see that you can use it to turn improper usage like this into a proper warning. Or perhaps you’ll decide that this case is severe enough to warrant using die, described in the same chapter. Variable-Length Parameter Lists | 69 This code uses what has often been called the “high-water mark” algorithm; after a flood, when the waters have surged and receded for the last time, the high-water mark shows where the highest water was seen. In this routine, $max_so_far keeps track of our high-water mark, the largest number yet seen, in the $max_so_far variable. The first line sets $max_so_far to 3 (the first parameter in the example code) by shifting that parameter from the parameter array, @_. So @_ now holds (5, 10, 4, 6), since you removed the 3. And the largest number yet seen is the only one yet seen: 3, the first parameter. Next, the foreach loop steps through the remaining values in the parameter list, from @_. The control variable of the loop is, by default, $_. (But, remember, there’s no au- tomatic connection between @_ and $_; it’s just a coincidence that they have such similar names.) The first time through the loop, $_ is 5. The if test sees that it is larger than $max_so_far, so it sets $max_so_far to 5—the new high-water mark. The next time through the loop, $_ is 10. That’s a new record high, so you store it in $max_so_far as well. The next time, $_ is 4. The if test fails, since that’s no larger than $max_so_far, which is 10, so you skip the body of the if. Finally, $_ is 6, and you skip the body of the if again. And that was the last time through the loop, so the loop is done. Now, $max_so_far becomes the return value. It’s the largest number you’ve seen, and you’ve seen them all, so it must be the largest from the list: 10. Empty Parameter Lists That improved &max algorithm works fine now, even if there are more than two pa- rameters. But what happens if there are none? At first, it may seem too esoteric to worry about. After all, why would someone call &max without giving it any parameters? But maybe someone wrote a line like this one: $maximum = &max(@numbers); And the array @numbers might sometimes be an empty list; perhaps it was read in from a file that turned out to be empty, for example. So you need to know: what does &max do in that case? The first line of the subroutine sets $max_so_far by using shift on @_, the (now empty) parameter array. That’s harmless; the array is left empty, and shift returns undef to $max_so_far. Now the foreach loop wants to iterate over @_, but since that’s empty, you execute the loop body zero times. 70 | Chapter 4: Subroutines So in short order, Perl returns the value of $max_so_far—undef—as the return value of the subroutine. In some sense, that’s the right answer because there is no largest (non)value in an empty list. Of course, whoever called this subroutine should be aware that the return value may be undef—or they could simply ensure that the parameter list is never empty. Notes on Lexical (my) Variables Those lexical variables can actually be used in any block, not merely in a subroutine’s block. For example, they can be used in the block of an if, while, or foreach: foreach (1..10) { my($square) = $_ * $_; # private variable in this loop print "$_ squared is $square.\n"; } The variable $square is private to the enclosing block; in this case, that’s the block of the foreach loop. If there’s no enclosing block, the variable is private to the entire source file. For now, your programs aren’t going to use more than one source file,* so this isn’t an issue. But the important concept is that the scope of a lexical variable’s name is limited to the smallest enclosing block or file. The only code that can say $square and mean that variable is the code inside that textual scope. This is a big win for maintain- ability—if you find a wrong value in $square, you should also find the culprit within a limited amount of source code. As experienced programmers have learned (often the hard way), limiting the scope of a variable to a page of code, or even to a few lines of code, really speeds along the development and testing cycle. Note also that the my operator doesn’t change the context of an assignment: my($num) = @_; # list context, same as ($num) = @_; my $num = @_; # scalar context, same as $num = @_; In the first one, $num gets the first parameter, as a list-context assignment; in the second, it gets the number of parameters, in a scalar context. Either line of code could be what the programmer wanted; you can’t tell from that one line alone, and so Perl can’t warn you if you use the wrong one. (Of course, you wouldn’t have both of those lines in the same subroutine, since you can’t have two lexical variables with the same name de- clared in the same scope; this is just an example.) So, when reading code like this, you can always tell the context of the assignment by seeing what the context would be without the word my. * We cover reuseable libraries and modules in Intermediate Perl. Notes on Lexical (my) Variables | 71 Remember that without the parentheses, my only declares a single lexical variable:† my $fred, $barney; # WRONG! Fails to declare $barney my($fred, $barney); # declares both Of course, you can use my to create new, private arrays as well:‡ my @phone_number; Any new variable will start out empty—undef for scalars, or the empty list for arrays. In regular Perl programming, you’ll probably use my to introduce any new variable in a scope. In Chapter 3, you saw that you could define your own control variable with the foreach structure. You can make that a lexical variable, too: foreach my $rock (qw/ bedrock slate lava /) { print "One rock is $rock.\n"; # Prints names of three rocks } This is important in the next section, where you start using a feature that makes you declare all your variables. The use strict Pragma Perl tends to be a pretty permissive language.§ But maybe you want Perl to impose a little discipline; that can be arranged with the use strict pragma. A pragma is a hint to a compiler, telling it something about the code. In this case, the use strict pragma tells Perl’s internal compiler that it should enforce some good pro- gramming rules for the rest of this block or source file. Why would this be important? Well, imagine that you’re composing your program and you type a line like this one: $bamm_bamm = 3; # Perl creates that variable automatically Now, you keep typing for a while. After that line has scrolled off the top of the screen, you type this line to increment the variable: $bammbamm += 1; # Oops! Since Perl sees a new variable name (the underscore is significant in a variable name), it creates a new variable and increments that one. If you’re lucky and smart, you’ve turned on warnings, and Perl can tell you that you used one or both of those global variable names only a single time in your program. But if you’re merely smart, you used each name more than once, and Perl won’t be able to warn you. † As usual, turning on warnings will generally report this abuse of my. Using the strict pragma, which we’ll see in a moment, should forbid it outright. ‡ Or hashes, which you’ll see in Chapter 6. § Bet you hadn’t noticed. 72 | Chapter 4: Subroutines To tell Perl that you’re ready to be more restrictive, put the use strict pragma at the top of your program (or in any block or file where you want to enforce these rules): use strict; # Enforce some good programming rules Starting with Perl 5.12, you implicitly use this pragma when you declare a minimum Perl version: use 5.012; # loads strict for you Now, among other restrictions,‖ Perl will insist that you declare every new variable, usually done with my:# my $bamm_bamm = 3; # New lexical variable Now if you try to spell it the other way, Perl recognizes the problems and complains that you haven’t declared any variable called $bammbamm, so your mistake is automati- cally caught at compile time: $bammbamm += 1; # No such variable: Compile time fatal error Of course, this applies only to new variables; you don’t need to declare Perl’s built-in variables, such as $_ and @_.* If you add use strict to an already written program, you’ll generally get a flood of warning messages, so it’s better to use it from the start, when it’s needed. Most people recommend that programs that are longer than a screenful of text generally need use strict. And we agree. From here on, we’ll write most (but not all) of our examples as if use strict is in effect, even where we don’t show it. That is, we’ll generally declare variables with my where it’s appropriate. But, even though we don’t always do so here, we encourage you to include use strict in your programs as often as possible. You’ll thank us in the long run. ‖ To learn about the other restrictions, see the documentation for strict. The documentation for any pragma is under that pragma’s name, so the command perldoc strict (or your system’s native documentation method) should find it for you. In brief, the other restrictions require that you quote strings in most cases, and that references be true (hard) references. (We don’t talk about references, soft or hard, until Intermediate Perl). Neither of these restrictions should affect beginners in Perl. #There are some other ways to declare variables, too. * And, at least in some circumstances, you don’t want to declare $a and $b, because Perl uses them internally for sort, which you’ll see in Chapter 14. So if you’re testing this feature, use other variable names than those two. The fact that use strict doesn’t forbid these two is one of the most frequently reported nonbugs in Perl. The use strict Pragma | 73 The return Operator What if you want to stop your subroutine right away? The return operator immediately returns a value from a subroutine: my @names = qw/ fred barney betty dino wilma pebbles bamm-bamm /; my $result = &which_element_is("dino", @names); sub which_element_is { my($what, @array) = @_; foreach (0..$#array) { # indices of @array's elements if ($what eq $array[$_]) { return $_; # return early once found } } –1; # element not found (return is optional here) } You’re asking this subroutine to find the index of dino in the array @names. First, the my declaration names the parameters: there’s $what, which is what you’re searching for, and @array, an array of values to search within. That’s a copy of the array @names, in this case. The foreach loop steps through the indices of @array (the first index is 0, and the last one is $#array, as you saw in Chapter 3). Each time through the foreach loop, you check to see whether the string in $what is equal† to the element from @array at the current index. If it’s equal, you return that index at once. This is the most common use of the keyword return in Perl—to return a value immediately, without executing the rest of the subroutine. But what if you never found that element? In that case, the author of this subroutine has chosen to return –1 as a “value not found” code. It would be more Perlish, perhaps, to return undef in that case, but this programmer used –1. Saying return –1 on that last line would be correct, but the word return isn’t really needed. Some programmers like to use return every time there’s a return value, as a means of documenting that it is a return value. For example, you might use return when the return value is not the last line of the subroutine, such as in the subroutine &larger_of_fred_or_barney, earlier in this chapter. You don’t really need it, but it doesn’t hurt anything either. However, many Perl programmers believe it’s just an extra seven characters of typing. Omitting the Ampersand As promised, now we’ll tell you the rule for when you can omit the ampersand on a subroutine call. If the compiler sees the subroutine definition before invocation, or if Perl can tell from the syntax that it’s a subroutine call, the subroutine can be called † You noticed the string equality test, eq, instead of the numeric equality test, ==, didn’t you? 74 | Chapter 4: Subroutines without an ampersand, just like a built-in function. (But there’s a catch hidden in that rule, as you’ll see in a moment.) This means that if Perl can see that it’s a subroutine call without the ampersand, from the syntax alone, that’s generally fine. That is, if you’ve got the parameter list in parentheses, it’s got to be a function‡ call: my @cards = shuffle(@deck_of_cards); # No & necessary on &shuffle Or, if Perl’s internal compiler has already seen the subroutine definition, that’s generally okay, too. In that case, you can even omit the parentheses around the argument list: sub division { $_[0] / $_[1]; # Divide first param by second } my $quotient = division 355, 113; # Uses &division This works because of the rule that you may always omit parentheses when they don’t change the meaning of the code. But don’t put that subroutine declaration after the invocation or the compiler won’t know what the attempted invocation of division is all about. The compiler has to see the definition before the invocation in order to use the subroutine call as if it were a built-in. Otherwise, the compiler doesn’t know what to do with that expression. That’s not the catch, though. The catch is this: if the subroutine has the same name as a Perl built-in, you must use the ampersand to call your version. With an ampersand, you’re sure to call the subroutine; without it, you can get the subroutine only if there’s no built-in with the same name: sub chomp { print "Munch, munch!\n"; } &chomp; # That ampersand is not optional! Without the ampersand, you’d be calling the built-in chomp, even though you’ve defined the subroutine &chomp. So, the real rule to use is this one: until you know the names of all Perl’s built-in functions, always use the ampersand on function calls. That means that you will use it for your first hundred programs or so. But when you see someone else has omitted the ampersand in his own code, it’s not necessarily a mistake; perhaps he simply knows that Perl has no built-in with that name.§ When programmers plan to call their subroutines as if they were calling Perl’s built-ins, often when writing modules, they often use prototypes to tell Perl about the parameters to expect. Making modules is an advanced topic, though; when you’re ready for that, see Perl’s ‡ In this case, the function is the subroutine &shuffle. But it may be a built-in function, as you’ll see in a moment. § Then again, maybe it is a mistake; you can search the perlfunc and perlop documentation for that name, though, to see whether it’s the same as a built-in. And Perl will usually be able to warn you about this when you have warnings turned on. The return Operator | 75 documentation (in particular, the perlmod and perlsub documents) for more informa- tion about subroutine prototypes and making modules.‖ Non-Scalar Return Values A scalar isn’t the only kind of return value a subroutine may have. If you call your subroutine in a list context,# it can return a list of values. Suppose you want to get a range of numbers (as from the range operator, ..), except that you want to be able to count down as well as up. The range operator only counts upward, but that’s easily fixed: sub list_from_fred_to_barney { if ($fred < $barney) { # Count upwards from $fred to $barney $fred..$barney; } else { # Count downwards from $fred to $barney reverse $barney..$fred; } } $fred = 11; $barney = 6; @c = &list_from_fred_to_barney; # @c gets (11, 10, 9, 8, 7, 6) In this case, the range operator gives you the list from 6 to 11, then reverse reverses the list so that it goes from $fred (11) to $barney (6), just as we wanted. The least you can return is nothing at all. A return with no arguments will return undef in a scalar context or an empty list in a list context. This can be useful for an error return from a subroutine, signaling to the caller that a more meaningful return value is unavailable. Persistent, Private Variables With my, you were able to make variables private to a subroutine, although each time you called the subroutine you had to define them again. With state, you can still have private variables scoped to the subroutine but Perl will keep their values between calls. ‖ Or, continue your education with Intermediate Perl. #You can detect whether a subroutine is being evaluated in a scalar or list context using the wantarray function, which lets you easily write subroutines with specific list or scalar context values. 76 | Chapter 4: Subroutines Going back to the first example in this chapter, you had a subroutine named marine that incremented a variable: sub marine { $n += 1; # Global variable $n print "Hello, sailor number $n!\n"; } Now that you know about strict, you add that to your program and realize that your use of the global variable $n is now a compilation error. You can’t make $n a lexical variable with my because it wouldn’t retain its value between calls. Declaring our variable with state tells Perl to retain the variable’s value between calls to the subroutine and to make the variable private to the subroutine. This feature showed up in Perl 5.10: use 5.010; sub marine { state $n = 0; # private, persistent variable $n $n += 1; print "Hello, sailor number $n!\n"; } Now you can get the same output while being strict-clean and not using a global variable. The first time you call the subroutine, Perl declares and initializes $n. Perl ignores the statement on all subsequent calls. Between calls, Perl retains the value of $n for the next call to the subroutine. You can make any variable type a state variable; it’s not just for scalars. Here’s a sub- routine that remembers its arguments and provides a running sum by using a state array: use 5.010; running_sum( 5, 6 ); running_sum( 1..3 ); running_sum( 4 ); sub running_sum { state $sum = 0; state @numbers; foreach my $number ( @_ ) { push @numbers, $number; $sum += $number; } say "The sum of (@numbers) is $sum"; } Persistent, Private Variables | 77 This outputs a new sum each time you call it, adding the new arguments to all of the previous ones: The sum of (5 6) is 11 The sum of (5 6 1 2 3) is 17 The sum of (5 6 1 2 3 4) is 21 There’s a slight restriction on arrays and hashes as state variables, though. You can’t initialize them in list contexts as of Perl 5.10: state @array = qw(a b c); # Error! This gives you an error that hints that you might be able to do it in a future version of Perl, but as of Perl 5.14, you still can’t: Initialization of state variables in list context currently forbidden ... Exercises See “Answers to Exercises” on page 299 for answers to the following exercises: 1. [12] Write a subroutine, named total, which returns the total of a list of numbers. Hint: the subroutine should not perform any I/O; it should simply process its pa- rameters and return a value to its caller. Try it out in this sample program, which merely exercises the subroutine to see that it works. The first group of numbers should add up to 25. my @fred = qw{ 1 3 5 7 9 }; my $fred_total = total(@fred); print "The total of \@fred is $fred_total.\n"; print "Enter some numbers on separate lines: "; my $user_total = total(); print "The total of those numbers is $user_total.\n"; 2. [5] Using the subroutine from the previous problem, make a program to calculate the sum of the numbers from 1 to 1,000. 3. [18] Extra credit exercise: write a subroutine, called &above_average, which takes a list of numbers and returns the ones which are above the average (mean). (Hint: make another subroutine that calculates the average by dividing the total by the number of items.) Try your subroutine in this test program. my @fred = above_average(1..10); print "\@fred is @fred\n"; print "(Should be 6 7 8 9 10)\n"; my @barney = above_average(100, 1..10); print "\@barney is @barney\n"; print "(Should be just 100)\n"; 4. [10] Write a subroutine named greet that welcomes the person you name by telling them the name of the last person it greeted: greet( "Fred" ); greet( "Barney" ); 78 | Chapter 4: Subroutines This sequence of statements should print: Hi Fred! You are the first one here! Hi Barney! Fred is also here! 5. [10] Modify the previous program to tell each new person the names of all the people it has previously greeted: greet( "Fred" ); greet( "Barney" ); greet( "Wilma" ); greet( "Betty" ); This sequence of statements should print: Hi Fred! You are the first one here! Hi Barney! I've seen: Fred Hi Wilma! I've seen: Fred Barney Hi Betty! I've seen: Fred Barney Wilma Exercises | 79 CHAPTER 5 Input and Output You’ve already seen how to do some input/output (I/O) in order to make some of the earlier exercises possible. But now you’ll learn more about those operations by covering the 80% of the I/O you’ll need for most programs. If you’re already familiar with the workings of standard input, output, and error streams, you’re ahead of the game. If not, we’ll get you caught up by the end of this chapter. For now, just think of “standard input” as being “the keyboard,” and “standard output” as being “the display screen.” Input from Standard Input Reading from the standard input stream is easy. You’ve been doing it already with the operator.* Evaluating this operator in a scalar context gives you the next line of input: $line = ; # read the next line chomp($line); # and chomp it chomp($line = ); # same thing, more idiomatically Since the line-input operator will return undef when you reach end-of-file, this is handy for dropping out of loops: while (defined($line = )) { print "I saw $line"; } * What we’re calling the line-input operator here, , is actually a line-input operator (represented by the angle brackets) around a filehandle. You’ll learn about filehandles later in this chapter. 81 There’s a lot going on in that first line: you’re reading the input into a variable, checking that it’s defined, and if it is (meaning that we haven’t reached the end of the input) you’re running the body of the while loop. So, inside the body of the loop, you’ll see each line, one after another, in $line.† This is something you’ll want to do fairly often, so naturally Perl has a shortcut for it. The shortcut looks like this: while () { print "I saw $_"; } Now, to make this shortcut, Larry chose some useless syntax. That is, this is literally saying, “Read a line of input, and see if it’s true. (Normally it is.) And if it is true, enter the while loop, but throw away that line of input!” Larry knew that it was a useless thing to do; nobody should ever need to do that in a real Perl program. So, Larry took this useless syntax and made it useful. What this is actually saying is that Perl should do the same thing as you saw in our earlier loop: it tells Perl to read the input into a variable, and (as long as the result was defined, so you haven’t reached end-of file) then enter the while loop. However, instead of storing the input into $line, Perl uses its favorite default variable, $_, just as if you had written this: while (defined($_ = )) { print "I saw $_"; } Now, before you go any further, we must be very clear about something: this shortcut works only if you write it just like that. If you put a line-input operator anywhere else (in particular, as a statement all on its own), it won’t read a line into $_ by default. It works only if there’s nothing but the line-input operator in the conditional of a while loop.‡ If you put anything else into the conditional expression, this shortcut won’t apply. There’s otherwise no other connection between the line-input operator () and Perl’s favorite default variable ($_). In this case, though, it just happens that Perl is storing the input in that variable. On the other hand, evaluating the line-input operator in a list context gives you all of the (remaining) lines of input as a list—each element of the list is one line: foreach () { print "I saw $_"; } † You probably noticed that you never chomped that input. In this kind of a loop, you can’t really put chomp into the conditional expression, so it’s often the first item in the loop body, when it’s needed. You’ll see examples of that in the next section. ‡ Well, okay, the conditional of a for loop is just a while conditional in disguise, so it works there, too. 82 | Chapter 5: Input and Output Once again, there’s no connection between the line-input operator and Perl’s favorite default variable. In this case, though, the default control variable for foreach is $_. So in this loop, you see each line of input in $_, one after the other. That may sound familiar, and for good reason: that’s the same behavior the while loop would do. Isn’t it? The difference is under the hood. In the while loop, Perl reads a single line of input, puts it into a variable, and runs the body of the loop. Then, it goes back to find another line of input. But in the foreach loop, you’re using the line-input operator in a list context (since foreach needs a list to iterate through); you read all of the input before the loop can start running. That difference will become apparent when the input is coming from your 400 MB web server logfile! It’s generally best to use code like the while loop’s shortcut, which will process input a line at a time, whenever possible. Input from the Diamond Operator Another way to read input is with the diamond§ operator: <>. This is useful for making programs that work like standard Unix‖ utilities, with respect to the invocation argu- ments (which we’ll see in a moment). If you want to make a Perl program that can be used like the utilities cat, sed, awk, sort, grep, lpr, and many others, the diamond op- erator will be your friend. If you want to make anything else, the diamond operator probably won’t help. The invocation arguments to a program are normally a number of “words” on the com- mand line after the name of the program.# In this case, they give the names of the files your program will process in sequence: $ ./my_program fred barney betty That command means to run the command my_program (which will be found in the current directory), and that it should process file fred, followed by file barney, followed by file betty. § The diamond operator was named by Larry’s daughter, Heidi, when Randal went over to Larry’s house one day to show off the new training materials he’d been writing and complained that there was no spoken name for “that thing.” Larry didn’t have a name for it, either. Heidi (eight years old at the time) quickly chimed in, “That’s a diamond, Daddy.” So the name stuck. Thanks, Heidi! ‖ But not just on Unix systems. Many other systems have adopted this way of using invocation arguments. #Whenever a program is started, it has a list of zero or more invocation arguments, supplied by whatever program is starting it. Often this is the shell, which makes up the list depending upon what you type on the command line. But you’ll see later that you can invoke a program with pretty much any strings as the invocation arguments. Because they often come from the shell’s command line, they are sometimes called “command-line arguments” as well. Input from the Diamond Operator | 83 If you give no invocation arguments, the program should process the standard input stream. Or, as a special case, if you give just a hyphen as one of the arguments, that means standard input as well.* So, if the invocation arguments had been fred - betty, that would have meant that the program should process file fred, followed by the standard input stream, followed by file betty. The benefit of making your programs work like this is that you may choose where the program gets its input at run time; for example, you won’t have to rewrite the program to use it in a pipeline (which we’ll show more later). Larry put this feature into Perl because he wanted to make it easy for you to write your own programs that work like standard Unix utilities—even on non-Unix machines. Actually, he did it so he could make his own programs work like standard Unix utilities; since some vendors’ utilities don’t work just like others’, Larry could make his own utilities, deploy them on a number of machines, and know that they’d all have the same behavior. Of course, this meant porting Perl to every machine he could find. The diamond operator is actually a special kind of line-input operator. But instead of getting the input from the keyboard, it comes from the user’s choice of input:† while (defined($line = <>)) { chomp($line); print "It was $line that I saw!\n"; } So, if you run this program with the invocation arguments fred, barney, and betty, it will say something like: “It was [a line from file fred] that I saw!”, “It was [another line from file fred] that I saw!”, on and on until it reaches the end of file fred. Then, it will automatically go on to file barney, printing out one line after another, and then on through file betty. Note that there’s no break when you go from one file to another; when you use the diamond, it’s as if the input files have been merged into one big file.‡ The diamond will return undef (and we’ll drop out of the while loop) only at the end of all of the input. In fact, since this is just a special kind of line-input operator, you may use the same shortcut you saw earlier to read the input into $_ by default: while (<>) { chomp; print "It was $_ that I saw!\n"; } * Here’s a possibly unfamiliar Unix fact: most of those standard utilities, like cat and sed, use this same convention, where a hyphen stands for the standard input stream. † Which may or may not include getting input from the keyboard. ‡ If it matters to you, or even if it doesn’t, the current file’s name is kept in Perl’s special variable $ARGV. This name may be "-" instead of a real filename if the input is coming from the standard input stream, though. 84 | Chapter 5: Input and Output This works like the loop above, but with less typing. And you may have noticed that you use the default for chomp; without an argument, chomp works on $_. Every little bit of saved typing helps! Since you generally use the diamond operator to process all of the input, it’s typically a mistake to use it in more than one place in your program. If you find yourself putting two diamonds into the same program, especially using the second diamond inside the while loop that is reading from the first one, it’s almost certainly not going to do what you would like.§ In our experience, when beginners put a second diamond into a pro- gram, they meant to use $_ instead. Remember, the diamond operator reads the input, but the input itself is (generally, by default) found in $_. If the diamond operator can’t open one of the files and read from it, it’ll print an al- legedly helpful diagnostic message, such as: can't open wimla: No such file or directory The diamond operator will then go on to the next file automatically, much like what you’d expect from cat or another standard utility. The Invocation Arguments Technically, the diamond operator isn’t looking literally at the invocation arguments— it works from the @ARGV array. This array is a special array that is preset by the Perl interpreter as the list of the invocation arguments. In other words, this is just like any other array (except for its funny, all-caps name), but when your program starts, @ARGV is already stuffed full of the list of invocation arguments.‖ You can use @ARGV just like any other array; you could shift items off of it, perhaps, or use foreach to iterate over it. You could even check to see if any arguments start with a hyphen, so that you could process them as invocation options (like Perl does with its own -w option).# The diamond operator looks in @ARGV to determine what filenames it should use. If it finds an empty list, it uses the standard input stream; otherwise it uses the list of files that it finds. This means that after your program starts and before you start using the § If you reinitialize @ARGV before using the second diamond, then you’re on solid ground. We’ll see @ARGV in the next section. ‖ C programmers may be wondering about argc(there isn’t one in Perl), and what happened to the program’s own name (that’s found in Perl’s special variable $0, not @ARGV). Also, depending upon how you’ve invoked your program, there may be a little more happening than we say here. See the perlrun documentation for the full details. #If you need more than just one or two such options, you should almost certainly use a module to process them in a standard way. See the documentation for the Getopt::Long and Getopt::Std modules, which are part of the standard distribution. The Invocation Arguments | 85 diamond, you’ve got a chance to tinker with @ARGV. For example, you can process three specific files, regardless of what the user chose on the command line: @ARGV = qw# larry moe curly #; # force these three files to be read while (<>) { chomp; print "It was $_ that I saw in some stooge-like file!\n"; } You’ll see more about @ARGV in Chapter 14, when we show you how to translate its values to the right encoding.* Output to Standard Output The print operator takes a list of values and sends each item (as a string, of course) to standard output in turn, one after another. It doesn’t add any extra characters before, after, or in between the items;† if you want spaces between items and a newline at the end, you have to say so: $name = "Larry Wall"; print "Hello there, $name, did you know that 3+4 is ", 3+4, "?\n"; Of course, that means that there’s a difference between printing an array and interpo- lating an array: print @array; # print a list of items print "@array"; # print a string (containing an interpolated array) That first print statement will print a list of items, one after another, with no spaces in between. The second one will print exactly one item, which is the string you get by interpolating @array into the empty string—that is, it prints the contents of @array, separated by spaces.‡ So, if @array holds qw/ fred barney betty /,§ the first one prints fredbarneybetty, while the second prints fred barney betty separated by spaces. But before you decide to always use the second form, imagine that @array is a list of unchomped lines of input. That is, imagine that each of its strings has a trailing newline character. Now, the first print statement prints fred, barney, and betty on three sep- arate lines. But the second one prints this: fred barney betty * See Appendix C if you need to brush up on encodings. † Well, it doesn’t add anything extra by default, but this default (like so many others in Perl) may be changed. Changing these defaults will likely confuse your maintenance programmer, though, so avoid doing so except in small, quick-and-dirty programs, or (rarely) in a small section of a normal program. See the perlvar documentation to learn about changing the defaults. ‡ Yes, the spaces are another default; see the $" variable in the perlvar documentation. § You know that we mean a three-element list here, right? This is just Perl notation. 86 | Chapter 5: Input and Output Do you see where the spaces come from? Perl is interpolating an array, so it puts spaces between the elements. So, we get the first element of the array (fred and a newline character), then a space, then the next element of the array (barney and a newline character), then a space, then the last element of the array (betty and a newline char- acter). The result is that the lines seem to have become indented, except for the first one. Every week or two, a mailing list or forum has a message with a subject line something like “Perl indents everything after the first line.” Without even reading the message, we can immediately see that the program used double quotes around an array containing unchomped strings. “Did you perhaps put an array of unchomped strings inside double quotes?” we ask, and the answer is always yes. Generally, if your strings contain newlines, you simply want to print them, after all: print @array; But if they don’t contain newlines, you generally want to add one at the end: print "@array\n"; So, if you use the quote marks, you’re (generally) adding the \n at the end of the string anyway; this should help you to remember which is which. It’s normal for your program’s output to be buffered. That is, instead of sending out every little bit of output at once, your program saves the output until there’s enough to bother with. If (for example) you want to save the output to disk, it’s (relatively) slow and inefficient to spin the disk every time you add one or two characters to the file. Generally, then, the output will go into a buffer that is flushed (that is, actually written to disk, or wher- ever) only when the buffer gets full, or when the output is otherwise finished (such as at the end of runtime). Usually, that’s what you want. But if you (or a program) may be waiting impatiently for the output, you may wish to take that performance hit and flush the output buffer each time you print. See the Perl documentation for more information on controlling buffering in that case. Since print is looking for a list of strings to print, it evaluates its arguments in list context. Since the diamond operator (as a special kind of line-input operator) returns a list of lines in a list context, these can work well together: print <>; # source code for 'cat' print sort <>; # source code for 'sort' Well, to be fair, the standard Unix commands cat and sort do have some additional functionality that these replacements lack. But you can’t beat them for the price! You can now reimplement all of your standard Unix utilities in Perl and painlessly port them to any machine that has Perl, whether that machine is running Unix or not. And you Output to Standard Output | 87 can be sure that the programs on every different type of machine will nevertheless have the same behavior.‖ What might not be obvious is that print has optional parentheses, which can some- times cause confusion. Remember the rule that parentheses in Perl may always be omitted—except when doing so would change the meaning of a statement. So, here are two ways to print the same thing: print("Hello, world!\n"); print "Hello, world!\n"; So far, so good. But another rule in Perl is that if the invocation of print looks like a function call, then it is a function call. It’s a simple rule, but what does it mean for something to look like a function call? In a function call, there’s a function name immediately# followed by parentheses around the function’s arguments, like this: print (2+3); That looks like a function call, so it is a function call. It prints 5, but it returns a value like any other function. The return value of print is a true or false value, indicating the success of the print. It nearly always succeeds, unless you get some I/O error, so the $result in the following statement will normally be 1: $result = print("hello world!\n"); But what if you use the result in some other way? Suppose you decide to multiply the return value times four: print (2+3)*4; # Oops! When Perl sees this line of code, it prints 5, just as you asked. Then it takes the return value from print, which is 1, and multiplies that times 4. It then throws away the prod- uct, wondering why you didn’t tell it to do something else with it. And at this point, someone looking over your shoulder says, “Hey, Perl can’t do math! That should have printed 20, rather than 5!” This is the problem with the optional parentheses; sometimes we humans forget where the parentheses really belong. When there are no parentheses, print is a list operator, printing all of the items in the following list; that’s generally what you’d expect. But when the first thing after print is a left parenthesis, print is a function call, and it will ‖ In fact, the PPT (Perl Power Tools) project, whose goal was to implement all of the classic Unix utilities in Perl, completed nearly all the utilities (and most of the games!) but got bogged down when they got to reimplementing the shell. The PPT project has been helpful because it has made these standard utilities available on many non-Unix machines. #We say “immediately” here because Perl won’t permit a newline character between the function name and the opening parenthesis in this kind of function call. If there is a newline there, Perl sees your code as making a list operator, rather than a function call. This is the kind of piddling technical detail that we mention only for completeness. If you’re terminally curious, see the full story in the documentation. 88 | Chapter 5: Input and Output print only what’s found inside the parentheses. Since that line had parentheses, it’s the same to Perl as if you’d said this: ( print(2+3) ) * 4; # Oops! Fortunately, Perl itself can almost always help you with this, if you ask for warnings— so use -w, or use warnings, at least during program development and debugging. Actually, this rule—“If it looks like a function call, it is a function call”—applies to all list functions* in Perl, not just to print. It’s just that you’re most likely to notice it with print. If print (or another function name) is followed by an open parenthesis, make sure that the corresponding close parenthesis comes after all of the arguments to that function. Formatted Output with printf You may wish to have a little more control with your output than print provides. In fact, you may be accustomed to the formatted output of C’s printf function. Fear not! Perl provides a comparable operation with the same name. The printf operator takes a format string followed by a list of things to print. The format† string is a fill-in-the-blanks template showing the desired form of the output: printf "Hello, %s; your password expires in %d days!\n", $user, $days_to_die; The format string holds a number of so-called conversions; each conversion begins with a percent sign (%) and ends with a letter. (As you’ll see in a moment, there may be significant extra characters between these two symbols.) There should be the same number of items in the following list as there are conversions; if these don’t match up, it won’t work correctly. In the example above, there are two items and two conversions, so the output might look something like this: Hello, merlyn; your password expires in 3 days! There are many possible printf conversions, so we’ll take time here to describe just the most common ones. Of course, the full details are available in the perlfunc documentation. To print a number in what’s generally a good way, use %g,‡ which automatically chooses floating-point, integer, or even exponential notation, as needed: printf "%g %g %g\n", 5/2, 51/17, 51 ** 17; # 2.5 3 1.0683e+29 * Functions that take zero or one arguments don’t suffer from this problem. † Here, we’re using “format” in the generic sense. Perl has a report-generating feature called “formats” that we won’t even be mentioning (except in this footnote) until Appendix B, and then only to say that we really aren’t going to talk about them. So, you’re on your own there. Just wanted to keep you from getting lost. ‡“General” numeric conversion. Or maybe “Good conversion for this number,” or “Guess what I want the output to look like.” Formatted Output with printf | 89 The %d format means a decimal§ integer, truncated as needed: printf "in %d days!\n", 17.85; # in 17 days! Note that this is truncated, not rounded; you’ll see how to round off a number in a moment. In Perl, you most often use printf for columnar data, since most formats accept a field width. If the data won’t fit, the field will generally be expanded as needed: printf "%6d\n", 42; # output like ````42 (the ` symbol stands for a space) printf "%2d\n", 2e3 + 1.95; # 2001 The %s conversion means a string, so it effectively interpolates the given value as a string, but with a given field width: printf "%10s\n", "wilma"; # looks like `````wilma A negative field width is left-justified (in any of these conversions): printf "%-15s\n", "flintstone"; # looks like flintstone````` The %f conversion (floating-point) rounds off its output as needed, and even lets you request a certain number of digits after the decimal point: printf "%12f\n", 6 * 7 + 2/3; # looks like ```42.666667 printf "%12.3f\n", 6 * 7 + 2/3; # looks like ``````42.667 printf "%12.0f\n", 6 * 7 + 2/3; # looks like ``````````43 To print a real percent sign, use %%, which is special in that it uses no element from the list:‖ printf "Monthly interest rate: %.2f%%\n", 5.25/12; # the value looks like "0.44%" Arrays and printf Generally, you won’t use an array as an argument to printf. That’s because an array may hold any number of items, and a given format string will work with only a certain fixed number of items. But there’s no reason you can’t whip up a format string on the fly, since it may be any expression. This can be tricky to get right, though, so it may be handy (especially when debugging) to store the format into a variable: § There’s also %x for hexadecimal and %o for octal if you need those. But we really say “decimal” here as a memory aid: %d for decimal integer. ‖ Maybe you thought you could simply put a backslash in front of the percent sign. Nice try, but no. The reason that won’t work is that the format is an expression, and the expression "\%" means the one-character string '%'. Even if we got a backslash into the format string, printf wouldn’t know what to do with it. Besides, C programmers are used to printf working like this. 90 | Chapter 5: Input and Output my @items = qw( wilma dino pebbles ); my $format = "The items are:\n" . ("%10s\n" x @items); ## print "the format is >>$format<<\n"; # for debugging printf $format, @items; This uses the x operator (which you learned about in Chapter 2) to replicate the given string a number of times given by @items (which is being used in a scalar context). In this case, that’s 3, since there are 3 items, so the resulting format string is the same as if you wrote it as "The items are:\n%10s\n%10s\n%10s\n". And the output prints each item on its own line, right-justified in a 10-character column, under a heading line. Pretty cool, huh? But not cool enough because you can even combine these: printf "The items are:\n".("%10s\n" x @items), @items; Note that here you have @items being used once in a scalar context, to get its length, and once in a list context, to get its contents. Context is important. Filehandles A filehandle is the name in a Perl program for an I/O connection between your Perl process and the outside world. That is, it’s the name of a connection, not necessarily the name of a file. Indeed, Perl has evolved that there might not even be a file behind that filehandle. Before Perl 5.6, all filehandle names were barewords, and Perl 5.6 added the ability to store a filehandle reference in a normal scalar variable. We’ll show you the bareword versions first since Perl still uses those for its special filehandles, and catch up with the scalar variable versions later in this chapter. You name these filehandles just like other Perl identifiers: letters, digits, and under- scores (but not starting with a digit). The bareword filehandles don’t have any prefix character, so Perl might confuse them with present or future reserved words, or with labels, which you’ll see in Chapter 10. Once again, as with labels, the recommendation from Larry is that you use all uppercase letters in the name of your filehandle—not only does it stand out better, but it also guarantees that your program won’t fail when Perl introduces a future (always lowercase) reserved word. But there are also six special filehandle names that Perl already uses for its own pur- poses: STDIN, STDOUT, STDERR, DATA, ARGV, and ARGVOUT.# Although you may choose any filehandle name you like, you shouldn’t choose one of those six unless you intend to use that one’s special properties.* #Some people hate typing in all caps, even for a moment, and will try spelling these in lowercase, like stdin. Perl may even let you get away with that from time to time, but not always. The details of when these work and when they fail are beyond the scope of this book. But the important thing is that programs that rely upon this kindness will one day break, so it is best to avoid lowercase here. * In some cases, you could (re)use these names without a problem. But your maintenance programmer may think that you’re using the name for its built-in features, and thus may be confused. Filehandles | 91 Maybe you recognized some of those names already. When your program starts, STDIN is the filehandle naming the connection between the Perl process and wherever the program should get its input, known as the standard input stream. This is generally the user’s keyboard unless the user asked for something else to be the source of input, such as a file or the output of another program through a pipe.† There’s also the standard output stream, which is STDOUT. By default, this one goes to the user’s display screen, but the user may send the output to a file or to another program, as you’ll see shortly. These standard streams come to you from the Unix “standard I/O” library, but they work in much the same way on most modern operating systems.‡ The general idea is that your program should blindly read from STDIN and blindly write to STDOUT, trusting in the user (or generally whichever program is starting your program) to have set those up. In that way, the user can type a command like this one at the shell prompt: $ ./your_program wilma That command tells the shell that the program’s input should be read from the file dino, and the output should go to the file wilma. As long as the program blindly reads its input from STDIN, processes it (in whatever way we need), and blindly writes its output to STDOUT, this will work just fine. And at no extra charge, the program will work in a pipeline. This is another concept from Unix, which lets us write command lines like this one: $ cat fred barney | sort | ./your_program | grep something | lpr Now, if you’re not familiar with these Unix commands, that’s okay. This line says that the cat command should print out all of the lines of file fred followed by all of the lines of file barney. Then that output should be the input of the sort command, which sorts those lines and passes them on to your_program. After it has done its processing, your_program sends the data on to grep, which discards certain lines in the data, send- ing the others on to the lpr command, which should print everything that it gets on a printer. Whew! Pipelines like that are common in Unix and many other systems today because they let you build powerful, complex commands out of simple, standard building blocks. Each building block does one thing very well, and it’s your job to use them together in the right way. There’s one more standard I/O stream. If (in the previous example) your_program had to emit any warnings or other diagnostic messages, those shouldn’t go down the pipe- line. The grep command is set to discard anything that it hasn’t specifically been told † The defaults we speak of in this chapter for the three main I/O streams are what the Unix shells do by default. But it’s not just shells that launch programs, of course. You’ll see in Chapter 14 what happens when you launch another program from Perl. ‡ If you’re not already familiar with how your non-Unix system provides standard input and output, see the perlport documentation or the documentation for that system’s equivalent to the Unix shell (the program that runs programs based upon your keyboard input). 92 | Chapter 5: Input and Output to look for, and so it will most likely discard the warnings. Even if it did keep the warnings, you probably don’t want to pass them downstream to the other programs in the pipeline. So that’s why there’s also the standard error stream: STDERR. Even if the standard output is going to another program or file, the errors will go to wherever the user desires. By default, the errors will generally go to the user’s display screen,§ but the user may send the errors to a file with a shell command like this one: $ netstat | ./your_program 2>/tmp/my_errors Opening a Filehandle So you’ve seen that Perl provides three filehandles—STDIN, STDOUT, and STDERR—which are automatically open to files or devices established by the program’s parent process (probably the shell). When you need other filehandles, use the open operator to tell Perl to ask the operating system to open the connection between your program and the outside world. Here are some examples: open CONFIG, 'dino'; open CONFIG, 'fred'; open LOG, '>>logfile'; The first one opens a filehandle called CONFIG to a file called dino. That is, the (existing) file dino will be opened and whatever it holds will come into our program through the filehandle named CONFIG. This is similar to the way that data from a file could come in through STDIN if the command line had a shell redirection like $selected_output"; Note the space after the greater-than. Perl ignores this,# but it keeps unexpected things from happening if $selected_output were ">passwd" for example (which would make an append instead of a write). In modern versions of Perl (starting with Perl 5.6), you can use a “three-argument” open: open CONFIG, '<', 'dino'; open BEDROCK, '>', $file_name; open LOG, '>>', &logfile_name(); The advantage here is that Perl never confuses the mode (the second argument) with some part of the filename (the third argument), which has nice advantages for security. Since they are separate arguments, Perl doesn’t have a chance to get confused. The three-argument form has another big advantage. Along with the mode, you can specify an encoding. If you know that your input file is UTF-8, you can specify that by putting a colon after the file mode and naming the encoding: open CONFIG, '<:encoding(UTF-8)', 'dino'; If you want to write your data to a file with a particular encoding, you do the same thing with one of the write modes: open BEDROCK, '>:encoding(UTF-8)', $file_name; open LOG, '>>:encoding(UTF-8)', &logfile_name(); There’s a shortcut for this. Instead of the full encoding(UTF-8), you might sometimes see :utf8. This isn’t really a shortcut for the full version because it doesn’t care if the input (or output) is valid UTF-8. If you use encoding(UTF-8), you ensure that the data is encoded correctly. The :utf8 takes whatever it gets and marks it as a UTF-8 string even if it isn’t, which might cause problems later. Still, you might see people do some- thing like this: open BEDROCK, '>:utf8', $file_name; # probably not right With the encoding() form, you can specify other encodings too. You can get a list of all of the encodings that your perl understands with a Perl one-liner: % perl -MEncode -le "print for Encode->encodings(':all')" #Yes, this means that if your filename were to have leading whitespace, that would also be ignored by Perl. See perlfunc and perlopentut if you’re worried about this. 94 | Chapter 5: Input and Output You should be able to use any of the names from that list as an encoding for reading or writing a file. Not all encodings are available on every machine since the list depends on what you’ve installed (or excluded). If you want a little-endian version of UTF-16: open BEDROCK, '>:encoding(UTF-16LE)', $file_name; Or perhaps Latin-1: open BEDROCK, '>:encoding(iso-8859-1)', $file_name; There are other layers* that perform transformations on the input or output. For in- stance, you sometimes need to handle files that have DOS line endings, where each line ends with a carriage-return/linefeed (CR-LF) pair (also normally written as "\r\n"). Unix line endings only use the newlines. When you try to use one on the other, odd things can happen. The :crlf encoding takes care of that.† When you want to ensure you get a CR-LF at the end of each line, you can set that encoding on the file: open BEDROCK, '>:crlf', $file_name; Now when you print to each line, this layer translates each newline to a CR-LF, al- though be careful since if you already have a CR-LF, you’ll end up with two carriage returns in a row. You can do the same thing to read a file which might have DOS line endings: open BEDROCK, '<:crlf', $file_name; Now when you read a file, Perl will translate all CR-LF to just newlines. Binmoding Filehandles You don’t have to know the encoding ahead of time, or even specify it if you already know it. In older Perls, if you didn’t want to translate line endings, such as a random value in a binary file that happens to have the same ordinal value as the newline, you used binmode to turn off line ending processing:‡ binmode STDOUT; # don't translate line endings binmode STDERR; # don't translate line endings * A layer is slightly different from an encoding because it doesn’t really have to do anything. You can stack layers (which is how they get their name) to get different effects. † The :crlf encoding is already the default on Windows. ‡ Much like you’d set binary mode in FTP, if you remember what that is. Opening a Filehandle | 95 Starting with Perl 5.6, you could specify a layer§ as the second argument to binmode. If you want to output Unicode to STDOUT, you want to ensure that STDOUT knows how to handle what it gets: binmode STDOUT, ':encoding(UTF-8)'; If you don’t do that, you might get a warning (even without turning on warnings) because STDOUT doesn’t know how you’d like to encode it: Wide character in print at test line 1. You can use binmode with either input or output handles. If you expect UTF-8 on standard input, you can tell Perl to expect that: binmode STDERR, ':encoding(UTF-8)'; Bad Filehandles Perl can’t actually open a file all by itself. Like any other programming language, Perl can merely ask the operating system to let us open a file. Of course, the operating system may refuse, because of permission settings, an incorrect filename, or other reasons. If you try to read from a bad filehandle (that is, a filehandle that isn’t properly open or a closed network connection), you’ll see an immediate end-of-file. (With the I/O meth- ods you’ll see in this chapter, end-of-file will be indicated by undef in a scalar context or an empty list in a list context.) If you try to write to a bad filehandle, the data is silently discarded. Fortunately, these dire consequences are easy to avoid. First of all, if you ask for warn- ings with -w or the warnings pragma, Perl will generally be able to tell you with a warning when it sees that you’re using a bad filehandle. But even before that, open always tells you if it succeeded or failed by returning true for success or false for failure. So you could write code like this: my $success = open LOG, '>>', 'logfile'; # capture the return value if ( ! $success ) { # The open failed ... } Well, you could do it like that, but there’s another way that you’ll see in the next section. Closing a Filehandle When you are finished with a filehandle, you may close it with the close operator like this: close BEDROCK; § Perl 5.6 called it a discipline, but that name changed in favor of layer. 96 | Chapter 5: Input and Output Closing a filehandle tells Perl to inform the operating system that you’re done with the given data stream, so it should write any last output data to disk in case someone is waiting for it.‖ Perl automatically closes a filehandle if you reopen it (that is, if you reuse the filehandle name in a new open) or if you exit the program.# Because of this, many simple Perl programs don’t bother with close. But it’s there if you want to be tidy, with one close for every open. In general, it’s best to close each filehandle soon after you’re done with it, though the end of the program often arrives soon enough.* Fatal Errors with die Step aside for a moment. You need some stuff that isn’t directly related to (or limited to) I/O, but is more about getting out of a program earlier than normal. When a fatal error happens inside Perl (for example, if you divide by zero, use an invalid regular expression, or call a subroutine that you haven’t declared), your program stops with an error message telling why.† But this functionality is available to you with the die function, so you can make your own fatal errors. The die function prints out the message you give it (to the standard error stream, where such messages should go) and makes sure that your program exits with a nonzero exit status. You may not have known it, but every program that runs on Unix (and many other modern operating systems) has an exit status, telling whether it was successful or not. Programs that run other programs (like the make utility program) look at that exit status to see that everything happened correctly. The exit status is just a single byte, so it can’t say much; traditionally, it is 0 for success and a nonzero value for failure. Perhaps 1 means a syntax error in the command arguments, while 2 means that something went wrong during processing and 3 means the configuration file couldn’t be found; the ‖ If you know much about I/O systems, you’ll know there’s more to the story. Generally, though, when a filehandle is closed, here’s what happens. If there’s input remaining in a file, it’s ignored. If there’s input remaining in a pipeline, the writing program may get a signal that the pipeline is closed. If there’s output going to a file or pipeline, the buffer is flushed (that is, pending output is sent on its way). If the filehandle had a lock, the lock is released. See your system’s I/O documentation for further details. #Any exit from the program will close all filehandles, but if Perl itself breaks, it can’t flush the pending output buffers. That is to say, if you accidentally crash your program by dividing by zero, for example, Perl itself is still running. Perl will ensure that data you’ve written actually gets output in that case. But if Perl itself can’t run (because you ran out of memory or caught an unexpected signal), the last few pieces of output may not be written to disk. Usually, this isn’t a big issue. * Closing a filehandle will flush any output buffers and release any locks on the file. Since someone else may be waiting for those things, a long-running program should generally close each filehandle as soon as possible. But many of our programs will take only one or two seconds to run to completion, so this may not matter. Closing a filehandle also releases possibly limited resources, so it’s more than just being tidy. † Well, it does this by default, but errors may be trapped with an eval block, as you’ll see in Chapter 16. Fatal Errors with die | 97 details differ from one command to the next. But 0 always means that everything worked. When the exit status shows failure, a program like make knows not to go on to the next step. So you could rewrite the previous example, perhaps something like this: if ( ! open LOG, '>>', 'logfile' ) { die "Cannot create logfile: $!"; } If the open fails, die terminates the program and tells you that it cannot create the logfile. But what’s that $! in the message? That’s the human-readable complaint from the system. In general, when the system refuses to do something you’ve requested (such as opening a file), $! will give you a reason (perhaps “permission denied” or “file not found,” in this case). This is the string that you may have obtained with perror in C or a similar language. This human-readable complaint message is available in Perl’s special variable $!.‡ It’s a good idea to include $! in the message when it could help the user to figure out what he or she did wrong. But if you use die to indicate an error that is not the failure of a system request, don’t include $!, since it will generally hold an unrelated message left over from something Perl did internally. It will hold a useful value only immediately after a failed system request. A successful request won’t leave anything useful there. There’s one more thing that die will do for you: it will automatically append the Perl program name and line number§ to the end of the message, so you can easily identify which die in your program is responsible for the untimely exit. The error message from the previous code might look like this, if $! contained the message permission denied: Cannot create logfile: permission denied at your_program line 1234. That’s pretty helpful—in fact, you always seem to want more information in your error messages than you included the first time around. If you don’t want the line number and file revealed, make sure the dying words have a newline on the end. That is, another way you could use die is with a trailing newline on the message: if (@ARGV < 2) { die "Not enough arguments\n"; } If there aren’t at least two command-line arguments, that program will say so and quit. It won’t include the program name and line number, since the line number is of no use to the user; this is the user’s error, after all. As a rule of thumb, put the newline on ‡ On some non-Unix operating systems, $! may say something like error number 7, leaving it up to the user to look that one up in the documentation. On Windows and VMS, the variable $^E may have additional diagnostic information. § If the error happened while reading from a file, the error message will include the “chunk number” (usually the line number) from the file and the name of the filehandle as well, since those are often useful in tracking down a bug. 98 | Chapter 5: Input and Output messages that indicate a usage error and leave it off when the error might be something you want to track down during debugging.‖ You should always check the return value of open, since the rest of the program is relying upon its success. Warning Messages with warn Just as die can indicate a fatal error that acts like one of Perl’s built-in errors (like dividing by zero), you can use the warn function to cause a warning that acts like one of Perl’s built-in warnings (like using an undef value as if it were defined, when warnings are enabled). The warn function works just like die does, except for that last step—it doesn’t actually quit the program. But it adds the program name and line number if needed, and it prints the message to standard error, just as die would.# And having talked about death and dire warnings, we now return you to your regularly scheduled I/O instructional material. Read on. Automatically die-ing Starting with Perl 5.10, the autodie pragma is part of the Standard Library. So far in the examples, you checked the return value of open and handled the error yourself: if ( ! open LOG, '>>', 'logfile' ) { die "Cannot create logfile: $!"; } That can get a bit tedious if you have to do that every time you want to open a filehandle. Instead, you can use the autodie pragma once in your program and automatically get the die if your open fails: use autodie; open LOG, '>>', 'logfile'; ‖ The program’s name is in Perl’s special variable $0, so you may wish to include that in the string: "$0:Not enough arguments\n". This is useful if the program may be used in a pipeline or shell script, for example, where it’s not obvious which command is complaining. You can change $0 during the execution of the program, however. You might also want to look into the special __FILE__ and __LINE__ tokens (or the caller function) to get the information that is being left out by adding the newline, so you can print it in your own choice of format. #You can’t trap warnings with an eval block like you can with fatal errors. See the documentation for the __WARN__ pseudosignal (in the perlvar documentation for %SIG) if you need to trap a warning. Fatal Errors with die | 99 This pragma works by recognizing which Perl built-ins are system calls, which might fail for reasons beyond your program’s control. When one of those system calls fails, autodie magically invokes the die on your behalf. Its error message looks close to what you might choose yourself: Can't open('>>', 'logfile'): No such file or directory at test line 3 Using Filehandles Once a filehandle is open for reading, you can read lines from it just like you can read from standard input with STDIN. So, for example, to read lines from the Unix password file: if ( ! open PASSWD, "/etc/passwd") { die "How did you get logged in? ($!)"; } while () { chomp; ... } In this example, the die message uses parentheses around $!. Those are merely paren- theses around the message in the output. (Sometimes a punctuation mark is just a punctuation mark.) As you can see, what we’ve been calling the “line-input operator” is really made of two components; the angle brackets (the real line-input operator) are around an input filehandle. You can use a filehandle open for writing or appending with print or printf, appearing immediately after the keyword but before the list of arguments: print LOG "Captain's log, stardate 3.14159\n"; # output goes to LOG printf STDERR "%d percent complete.\n", $done/$total * 100; Did you notice that there’s no comma between the filehandle and the items to be prin- ted?* This looks especially weird if you use parentheses. Either of these forms is correct: printf (STDERR "%d percent complete.\n", $done/$total * 100); printf STDERR ("%d percent complete.\n", $done/$total * 100); Changing the Default Output Filehandle By default, if you don’t give a filehandle to print (or to printf, as everything we say here about one applies equally well to the other), the output will go to STDOUT. But that * If you got straight As in freshman English or Linguistics, when we say that this is called “indirect object syntax,” you may say, “Ah, of course! I see why there’s no comma after the filehandle name—it’s an indirect object!” We didn’t get straight As; we don’t understand why there’s no comma; we merely omit it because Larry told us that we should omit the comma. 100 | Chapter 5: Input and Output default may be changed with the select operator. Here we’ll send some output lines to BEDROCK: select BEDROCK; print "I hope Mr. Slate doesn't find out about this.\n"; print "Wilma!\n"; Once you’ve selected a filehandle as the default for output, it stays that way. But it’s generally a bad idea to confuse the rest of the program, so you should generally set it back to STDOUT when you’re done.† Also by default, the output to each filehandle is buffered. Setting the special $| variable to 1 will set the currently selected filehandle (that is, the one selected at the time that the variable is modified) to always flush the buffer after each output operation. So if you wanted to be sure that the logfile gets its entries at once, in case you might be reading the log to monitor progress of your long- running program, you could use something like this: select LOG; $| = 1; # don't keep LOG entries sitting in the buffer select STDOUT; # ... time passes, babies learn to walk, tectonic plates shift, and then... print LOG "This gets written to the LOG at once!\n"; Reopening a Standard Filehandle We mentioned earlier that if you were to reopen a filehandle (that is, if you were to open a filehandle FRED when you’ve already got an open filehandle named FRED), the old one would be closed for you automatically. And we said that you shouldn’t reuse one of the six standard filehandle names unless you intended to get that one’s special features. And we also said that the messages from die and warn, along with Perl’s in- ternally generated complaints, go automatically to STDERR. If you put those three pieces of information together, you now have an idea about how you could send error mes- sages to a file, rather than to your program’s standard error stream:‡ # Send errors to my private error log if ( ! open STDERR, ">>/home/barney/.error_log") { die "Can't open error log for append: $!"; } † In the unlikely case that STDOUT might not be the selected filehandle, you could save and restore the filehandle, using the technique shown in the documentation for select in the perlfunc documentation. And as long as we’re sending you to that documentation, we may as well tell you that there are actually two built-in functions in Perl named select, and both are covered in the perlfunc documentation. The other select always has four arguments, so it’s sometimes called “four-argument select”. ‡ Don’t do this without a reason. It’s nearly always better to let the user set up redirection when launching your program, rather than have redirection hardcoded. But this is handy in cases where your program is being run automatically by another program (say, by a web server or a scheduling utility like cron or at). Another reason might be that your program is going to start another process (probably with system or exec, which you’ll see in Chapter 14), and you need that process to have different I/O connections. Reopening a Standard Filehandle | 101 After reopening STDERR, any error messages from Perl go into the new file. But what happens if the die is executed—where will that message go, if the new file couldn’t be opened to accept the messages? The answer is that if one of the three system filehandles—STDIN, STDOUT, or STDERR— fails to reopen, Perl kindly restores the original one.§ That is, Perl closes the original one (of those three) only when it sees that opening the new connection is successful. Thus, this technique could be used to redirect any (or all) of those three system file- handles from inside your program,‖ almost as if the program had been run with that I/O redirection from the shell in the first place. Output with say Perl 5.10 borrowed the say built-in from the ongoing development of Perl 6 (which may have borrowed its say from Pascal’s println). It’s the same as print, although it adds a newline to the end. These forms all output the same thing: use 5.010; print "Hello!\n"; print "Hello!", "\n"; say "Hello!"; To just print a variable’s value followed by a newline, I don’t need to create an extra string or print a list. I just say the variable. This is especially handy in the common case of simply wanting to put a newline after whatever I want to output: use 5.010; my $name = 'Fred'; print "$name\n"; print $name, "\n"; say $name; To interpolate an array, I still need to quote it, though. It’s the quoting that puts the spaces between the elements: use 5.010; my @array = qw( a b c d ); say @array; # "abcd\n" say "@array"; # "a b c d\n"; § At least, this is true if you haven’t changed Perl’s special $^F variable, which tells Perl that only those three are special like this. But you’d never change that. ‖ But don’t open STDIN for output or the others for input. Just thinking about that makes our heads hurt. 102 | Chapter 5: Input and Output Just like with print, I can specify a filehandle with say: use 5.010; say BEDROCK "Hello!"; Since this is a Perl 5.10 feature though, we’ll only use it when we are otherwise using a Perl 5.10 feature. The old, trusty print is still as good as it ever was, but we suspect that there will be some Perl programmers out there who want the immediate savings of not typing the four extra characters (two in the name and the \n). Filehandles in a Scalar Since Perl 5.6, you can create a filehandle in a scalar variable so you don’t have to use a bareword. This makes many things, such as passing a filehandle as a subroutine argument, storing them in arrays or hashes, or controlling its scope, much easier. Al- though, you still need to know how to use the barewords because you’ll still find them in Perl code and they are actually quite handy in short scripts where you don’t benefit that much from the filehandles in a variable. If you use a scalar variable without a value in place of the bareword in open, your filehandle ends up in the variable. People typically do this with a lexical variable since that ensures you get a variable without a value; some like to put a _fh on the end of these variable names to remind themselves that they are using it for a filehandle: my $rocks_fh; open $rocks_fh, '<', 'rocks.txt' or die "Could not open rocks.txt: $!"; You can even combine those two statements so you declare the lexical variable right in the open: open my $rocks_fh, '<', 'rocks.txt' or die "Could not open rocks.txt: $!"; Once you have the filehandle in your scalar variable, you use the variable, sigil and all, in the same place that you used the bareword version: while( <$rocks_fh> ) { chomp; ... } This works for output filehandles too. You open the filehandle with the appropriate mode then use the scalar variable in place of the bareword filehandle: open my $rocks_fh, '>>', 'rocks.txt' or die "Could not open rocks.txt: $!"; foreach my $rock ( qw( slate lava granite ) ) { say $rocks_fh $rock } Filehandles in a Scalar | 103 print $rocks_fh "limestone\n"; close $rocks_fh; Notice that you still don’t use a comma after the filehandle in these examples. Perl realizes that $fh is a filehandle because there’s no comma after the first thing following print. If you put a comma after the filehandle, your output looks odd. This probably isn’t what you want to do: print $rocks_fh, "limestone\n"; # WRONG That example produces something like this: GLOB(0xABCDEF12)limestone What happened? Since you used the comma after the first argument, Perl treated that first argument as a string to print instead of the filehandle. Although we don’t talk about references until the next book, Intermediate Perl, you’re seeing a stringification of the reference instead of using it as you probably intend. This also means that these two are subtly different: print STDOUT; print $rock_fh; # WRONG, probably In the first case, Perl knows that STDOUT is a filehandle because it is a bareword. Since there are no other arguments, it uses $_ by default. In the second one, Perl can’t tell what $rock_fh will have until it actually runs the statement. Since it doesn’t know that it’s a filehandle ahead of time, it always assumes that the $rock_fh has a value you want to output. To get around this, you can always surround anything that should be a filehandle in braces to make sure that Perl does the right thing, even if you are using a filehandle that you stored in an array or a hash: print { $rock_fh }; # uses $_ by default print { $rocks[0] } "sandstone\n"; Depending on the sort of programming that you actually do, you might go one way or the other choosing between bareword and scalar variable filehandles. For short pro- grams, such as in system administration, barewords don’t pose much of a problem. For big application development, you probably want to use the lexical variables to control the scope of your open filehandles. Exercises See “Answers to Exercises” on page 302 for answers to the following exercises: 1. [7] Write a program that acts like cat, but reverses the order of the output lines. (Some systems have a utility like this named tac.) If you run yours as ./tac fred barney betty, the output should be all of file betty from last line to first, then barney and then fred, also from last line to first. (Be sure to use the ./ in your program’s invocation if you call it tac so that you don’t get the system’s utility instead!) 104 | Chapter 5: Input and Output 2. [8] Write a program that asks the user to enter a list of strings on separate lines, printing each string in a right-justified, 20-character column. To be certain that the output is in the proper columns, print a “ruler line” of digits as well. (This is simply a debugging aid.) Make sure that you’re not using a 19-character column by mis- take! For example, entering hello, good-bye should give output something like this: 123456789012345678901234567890123456789012345678901234567890 hello good-bye 3. [8] Modify the previous program to let the user choose the column width, so that entering 30, hello, good-bye (on separate lines) would put the strings at the 30th column. (Hint: see “Interpolation of Scalar Variables into Strings” on page 32 in Chapter 2, about controlling variable interpolation.) For extra credit, make the ruler line longer when the selected width is larger. Exercises | 105 CHAPTER 6 Hashes In this chapter, you will see a feature that makes Perl one of the world’s truly great programming languages—hashes.* Although hashes are a powerful and useful feature, you may have used other powerful languages for years without ever hearing of hashes. But you’ll use hashes in nearly every Perl program you write from now on; they’re that important. What Is a Hash? A hash is a data structure, not unlike an array in that it can hold any number of values and retrieve them at will. But instead of indexing the values by number, as you did with arrays, you look up hash values by name. That is, the indices, called keys, aren’t num- bers, but instead they are arbitrary, unique strings (see Figure 6-1). Hash keys are strings, first of all, so instead of getting element number 3 from an array, you access the hash element named wilma, for instance. These keys are arbitrary strings—you can use any string expression for a hash key. And they are unique strings—just as there’s only one array element numbered 3, there’s only one hash element named wilma. Another way to think of a hash is that it’s like a barrel of data, where each piece of data has a tag attached. You can reach into the barrel and pull out any tag and see what piece of data is attached. But there’s no “first” item in the barrel; it’s just a jumble. In an array, you start with element 0, then element 1, then element 2, and so on. But in a hash there’s no fixed order, no first element. It’s just a collection of key-value pairs. * In the olden days, we called these “associative arrays.” But the Perl community decided around 1995 that this was too many letters to type and too many syllables to say, so we changed the name to “hashes.” 107 Figure 6-1. Hash keys and values The keys and values are both arbitrary scalars, but the keys are always converted to strings. So, if you used the numeric expression 50/20 as the key,† it would be turned into the three-character string "2.5", which is one of the keys shown in Figure 6-2. Figure 6-2. A hash as a barrel of data † That’s a numeric expression, not the five-character string "50/20". If you used that five-character string as a hash key, it would stay the same five-character string, of course. 108 | Chapter 6: Hashes As usual, Perl’s “no unnecessary limits” philosophy applies: a hash may be of any size, from an empty hash with zero key-value pairs, up to whatever fills up your memory. Some implementations of hashes (such as in the original awk language, where Larry borrowed the idea from) slow down as the hashes get larger and larger. This is not the case in Perl—it has a good, efficient, scalable algorithm.‡ So, if a hash has only three key-value pairs, it’s very quick to “reach into the barrel” and pull out any one of those. If the hash has three million key-value pairs, it should be just about as quick to pull out any one of those. A big hash is nothing to fear. It’s worth mentioning again that the keys are always unique, although you may use the same value more than once. The values of a hash may be all numbers, all strings, undef values, or a mixture.§ But the keys are all arbitrary, unique strings. Why Use a Hash? When you first hear about hashes, especially if you’ve lived a long and productive life as a programmer using languages that don’t have hashes, you may wonder why anyone would want one of these strange beasts. Well, the general idea is that you’ll have one set of data “related to” another set of data. For example, here are some hashes you might find in typical applications of Perl: Given name, family name The given name (first name) is the key, and the family name is the value. This requires unique given names, of course; if there were two people named randal, this wouldn’t work. With this hash, you can look up anyone’s given name, and find the corresponding family name. If you use the key tom, you get the value phoenix. Hostname, IP address You may know that each computer on the Internet has both a hostname (like http: //www.stonehenge.com) and an IP address number (like 123.45.67.89). That’s be- cause machines like working with the numbers, but we humans have an easier time remembering the names. The hostnames are unique strings, so they can be used to make this hash. With this hash, you could look up a hostname and find the corresponding IP address.‖ ‡ Technically, Perl rebuilds the hash table as needed for larger hashes. In fact, the term “hashes” comes from the fact that a hash table is used for implementing them. § Or, in fact, any scalar values, including other scalar types than the ones we’ll see in this book. ‖ This isn’t a great example because we know that some hosts may have multiple IP addresses, and some IP addresses might map to multiple hosts, but you get the idea. What Is a Hash? | 109 IP address, hostname Or you could go in the opposite direction. You might think of an IP address as a number, but it can also be a unique string (like any Perl number), so it’s suitable for use as a hash key. In this hash, we can use the IP address to look up the corre- sponding hostname. Note that this is not the same hash as the previous example: hashes are a one-way street, running from key to value; there’s no way to look up a value in a hash and find the corresponding key! So these two are a pair of hashes, one for storing IP addresses, one for hostnames. It’s easy enough to create one of these given the other, though, as you’ll see below. Word, count of number of times that word appears This is a very common use of a hash. It’s so common, in fact, that it just might turn up in the exercises at the end of this chapter! The idea here is that you want to know how often each word appears in a given document. Perhaps you’re building an index to a number of documents so that when a user searches for fred, you’ll know that a certain document mentions fred five times, another mentions fred seven times, and yet another doesn’t men- tion fred at all—so you’ll know which documents the user is likely to want. As the index-making program reads through a given document, each time it sees a men- tion of fred, it adds one to the value filed under the key of fred. That is, if you had seen fred twice already in this document, the value would be 2, but now you in- crement it to 3. If you had not yet seen fred, you change the value from undef (the implicit, default value) to 1. Username, number of disk blocks they are using [wasting] System administrators like this one: the usernames on a given system are all unique strings, so they can be used as keys in a hash to look up information about that user. Driver’s license number, name There may be many, many people named John Smith, but you hope that each one has a different driver’s license number. That number makes for a unique key, and the person’s name is the value. Yet another way to think of a hash is as a very simple database, in which just one piece of data may be filed under each key. In fact, if your task description includes phrases like “finding duplicates,” “unique,” “cross-reference,” or “lookup table,” it’s likely that a hash will be useful in the implementation. Hash Element Access To access an element of a hash, you use syntax that looks like this: $hash{$some_key} 110 | Chapter 6: Hashes This is similar to what you used for array access, but here you use curly braces instead of square brackets around the subscript (key).# And that key expression is now a string, rather than a number: $family_name{'fred'} = 'flintstone'; $family_name{'barney'} = 'rubble'; Figure 6-3 shows how the resulting hash keys are assigned. Figure 6-3. Assigned hash keys This lets you use code like this: foreach my $person (qw< barney fred >) { print "I've heard of $person $family_name{$person}.\n"; } The name of the hash is like any other Perl identifier. And it’s from a separate name- space; that is, there’s no connection between the hash element $family_name{"fred"} and a subroutine &family_name, for example. Of course, there’s no reason to confuse everyone by giving everything the same name. But Perl won’t mind if you also have a scalar called $family_name and array elements like $family_name[5]. We humans will have to do as Perl does; that is, you look to see what punctuation appears before and after the identifier to see what it means. When there is a dollar sign in front of the name and curly braces afterward, you’re accessing a hash element. When choosing the name of a hash, it’s often nice to think of the word “for” between the name of the hash and the key. As in, “the family_name for fred is flintstone”. So the hash is named family_name. Then the relationship between the keys and their values becomes clear. Of course, the hash key may be any expression, not just the literal strings and simple scalar variables that you’re showing here: $foo = 'bar'; print $family_name{ $foo . 'ney' }; # prints 'rubble' #Here’s a peek into the mind of Larry Wall: Larry says that you use curly braces instead of square brackets because you’re doing something fancier than ordinary array access, so you should use fancier punctuation. Hash Element Access | 111 When you store something into an existing hash element, it overwrites the previous value: $family_name{'fred'} = 'astaire'; # gives new value to existing element $bedrock = $family_name{'fred'}; # gets 'astaire'; old value is lost That’s analogous to what happens with arrays and scalars; if you store something new into $pebbles[17] or $dino, the old value is replaced. If you store something new into $family_name{'fred'}, the old value is replaced as well. Hash elements spring into existence when you first assign to them: $family_name{'wilma'} = 'flintstone'; # adds a new key (and value) $family_name{'betty'} .= $family_name{'barney'}; # creates the element if needed That’s also just like what happens with arrays and scalars;* if you didn’t have $pebbles[17] or $dino before, you will have it after you assign to it. If you didn’t have $family_name{'betty'} before, you do now. And accessing outside the hash gives undef: $granite = $family_name{'larry'}; # No larry here: undef Once again, this is just like what happens with arrays and scalars; if there’s nothing yet stored in $pebbles[17] or $dino, accessing them will yield undef. If there’s nothing yet stored in $family_name{'larry'}, accessing it will yield undef. The Hash As a Whole To refer to the entire hash, use the percent sign (%) as a prefix. So, the hash you’ve been using for the last few pages is actually called %family_name. For convenience, you can convert a hash into a list and back again. Assigning to a hash (in this case, the one from Figure 6-1) is a list-context assignment, where the list is key- value pairs:† %some_hash = ('foo', 35, 'bar', 12.4, 2.5, 'hello', 'wilma', 1.72e30, 'betty', "bye\n"); The value of the hash (in a list context) is a simple list of key-value pairs: @any_array = %some_hash; Perl calls this unwinding the hash; turning it back into a list of key-value pairs. Of course, the pairs won’t necessarily be in the same order as the original list: print "@any_array\n"; # might give something like this: # betty bye (and a newline) wilma 1.72e+30 foo 35 2.5 hello bar 12.4 * This is a feature called autovivification, which we talk about more in Intermediate Perl. † Although you can use any list expression, it must have an even number of elements, because the hash is made of key-value pairs. An odd element will likely do something unreliable, although it’s a warnable offense. 112 | Chapter 6: Hashes The order is jumbled because Perl keeps the key-value pairs in an order that’s conven- ient for Perl so that it can look up any item quickly.‡ You use a hash either when you don’t care what order the items are in, or when you have an easy way to put them into the order you want. Of course, even though the order of the key-value pairs is jumbled, each key “sticks” with its corresponding value in the resulting list. So, even though you don’t know where the key foo will appear in the list, you know that its value, 35, will be right after it. Hash Assignment It’s rare to do so, but you can copy a hash using the obvious syntax of simply assigning one hash to another: my %new_hash = %old_hash; This is actually more work for Perl than meets the eye. Unlike what happens in lan- guages like Pascal or C, where such an operation would be a simple matter of copying a block of memory, Perl’s data structures are more complex. So, that line of code tells Perl to unwind the %old_hash into a list of key-value pairs, then assign those to %new_hash, building it up one key-value pair at a time. It’s more common to transform the hash in some way, though. For example, you could make an inverse hash: my %inverse_hash = reverse %any_hash; This takes %any_hash and unwinds it into a list of key-value pairs, making a list like (key, value, key, value, key, value, …). Then reverse turns that list end-for-end, making a list like (value, key, value, key, value, key, …). Now the keys are where the values used to be, and the values are where the keys used to be. When you store that in %inverse_hash, you can look up a string that was a value in %any_hash—it’s now a key of %inverse_hash. And the value you find is one that was one of the keys from %any_hash. So, you have a way to look up a “value” (now a key), and find a “key” (now a value). Of course, you might guess (or determine from scientific principles, if you’re clever) that this will work properly only if the values in the original hash were unique— otherwise you’d have duplicate keys in the new hash, and keys are always unique. Here’s the rule that Perl uses: the last one in wins. That is, the later items in the list overwrite any earlier ones. ‡ Perl also jumbles the order so an attacker can’t predict how Perl will store the hash. Hash Element Access | 113 Of course, you don’t know what order the key-value pairs will have in this list, so there’s no telling which ones would win. You’d use this technique only if you know there are no duplicates among the original values.§ But that’s the case for the IP address and hostname examples given earlier: %ip_address = reverse %host_name; Now you can look up a hostname or IP address with equal ease to find the correspond- ing IP address or hostname. The Big Arrow When assigning a list to a hash, sometimes it’s not obvious which elements are keys and which are values. For example, in this assignment (which you saw earlier), we humans have to count through the list, saying, “key, value, key, value…,” in order to determine whether 2.5 is a key or a value: %some_hash = ('foo', 35, 'bar', 12.4, 2.5, 'hello', 'wilma', 1.72e30, 'betty', "bye\n"); Wouldn’t it be nice if Perl gave you a way to pair up keys and values in that kind of a list so it would be easy to see which ones were which? Larry thought so, too, which is why he invented the big arrow (=>).‖ To Perl, it’s just a different way to “spell” a comma, so it’s also sometimes called the “fat comma.” That is, in the Perl grammar, any time that you need a comma (,), you can use the big arrow instead; it’s all the same to Perl.# So here’s another way to set up the hash of last names: my %last_name = ( # a hash may be a lexical variable 'fred' => 'flintstone', 'dino' => undef, 'barney' => 'rubble', 'betty' => 'rubble', ); Here, it’s easy (or perhaps at least easier) to see whose name pairs with which value, even if we end up putting many pairs on one line. And notice that there’s an extra comma at the end of the list. As we saw earlier, this is harmless, but convenient; if we need to add additional people to this hash, we’ll simply make sure that each line has a § Or if you don’t care that there are duplicates. For example, you could invert the %family_name hash (in which the keys are people’s given names and values are their family names) to make it easy to determine whether there is or is not anyone with a given family name in the group. Thus, in the inverted hash, if there’s no key of slate, you’d know that there’s no one with that name in the original hash. ‖ Yes, there’s also a little arrow, (->). It’s used with references, which is an advanced topic; see the perlreftut and perlref documentation when you’re ready for that. #Well, there’s one technical difference: any bareword (a sequence of nothing but letters, digits, and underscores not starting with a digit, but optionally prefixed with plus or minus) to the left of the big arrow is implicitly quoted. So you can leave off the quote marks on a bareword to the left of the big arrow. You may also omit the quote marks if there’s nothing but a bareword as a key inside the curly braces of a hash. 114 | Chapter 6: Hashes key-value pair and a trailing comma. Perl will see that there is a comma between each item and the next, and one extra (harmless) comma at the end of the list. It gets better though. Perl offers many shortcuts that can help the programmer. Here’s a handy one: you may omit the quote marks on some hash keys when you use the fat comma, which automatically quotes the values to its left: my %last_name = ( fred => 'flintstone', dino => undef, barney => 'rubble', betty => 'rubble', ); Of course, you can’t omit the quote marks on just any key, since a hash key may be any arbitrary string. If that value on the left looks like a Perl operator, Perl can get confused. This won’t work because Perl thinks the + is the addition operator, not a string to quote: my %last_name = ( + => 'flintstone', # WRONG! Compilation error! ); But keys are often simple. If the hash key is made up of nothing but letters, digits, and underscores without starting with a digit, you may be able to omit the quote marks. This kind of simple string without quote marks is called a bareword, since it stands alone without quotes. Another place you are permitted to use this shortcut is the most common place a hash key appears: in the curly braces of a hash element reference. For example, instead of $score{'fred'}, you could write simply $score{fred}. Since many hash keys are simple like this, not using quotes is a real convenience. But beware; if there’s anything inside the curly braces besides a bareword, Perl will interpret it as an expression. For instance, if there is a ., Perl interprets it as a string concatenation: $hash{ bar.foo } = 1; # that's the key 'foobar' Hash Functions Naturally, there are some useful functions that can work on an entire hash at once. The keys and values Functions The keys function yields a list of all the keys in a hash, while the values function gives the corresponding values. If there are no elements to the hash, then either function returns an empty list: my %hash = ('a' => 1, 'b' => 2, 'c' => 3); my @k = keys %hash; my @v = values %hash; Hash Functions | 115 So, @k will contain 'a', 'b', and 'c', and @v will contain 1, 2, and 3—in some order. Remember, Perl doesn’t maintain the order of elements in a hash. But, whatever order the keys are in, the values are in the corresponding order: if 'b' is last in the keys, 2 will be last in the values; if 'c' is the first key, 3 will be the first value. That’s true as long as you don’t modify the hash between the request for the keys and the one for the values. If you add elements to the hash, Perl reserves the right to rearrange it as needed, to keep the access quick.* In a scalar context, these functions give the number of elements (key- value pairs) in the hash. They do this quite efficiently, without having to visit each element of the hash: my $count = keys %hash; # gets 3, meaning three key-value pairs Once in a long while, you’ll see that someone has used a hash as a Boolean (true/false) expression, something like this: if (%hash) { print "That was a true value!\n"; } That will be true if (and only if) the hash has at least one key-value pair.† So, it’s just saying, “If the hash is not empty….” But this is a pretty rare construct, as such things go. The each Function If you wish to iterate over (that is, examine every element of) an entire hash, one of the usual ways is to use the each function, which returns a key-value pair as a two-element list.‡ On each evaluation of this function for the same hash, the next successive key- value pair is returned, until you have accessed all the elements. When there are no more pairs, each returns an empty list. In practice, the only way to use each is in a while loop, something like this: while ( ($key, $value) = each %hash ) { print "$key => $value\n"; } There’s a lot going on here. First, each %hash returns a key-value pair from the hash, as a two-element list; let’s say that the key is "c" and the value is 3, so the list is ("c", 3). That list is assigned to the list ($key, $value), so $key becomes "c", and $value becomes 3. * Of course, if you started adding elements to the hash between keys and values, your list of values (or keys, whichever you did second) would have additional items, which would be tough to match up with the first list. So no normal programmer would do that. † The actual result is an internal debugging string useful to the people who maintain Perl. It looks something like “4/16”, but the value is guaranteed to be true when the hash is non-empty, and false when it’s empty, so the rest of us can still use it for that. ‡ The other usual way to iterate over an entire hash is to use foreach on a list of keys from the hash; you’ll see that by the end of this section. 116 | Chapter 6: Hashes But that list assignment is happening in the conditional expression of the while loop, which is a scalar context. (Specifically, it’s a Boolean context, looking for a true/false value; and a Boolean context is a particular kind of scalar context.) The value of a list assignment in a scalar context is the number of elements in the source list—2, in this case. Since 2 is a true value, you enter the body of the loop and print the message c => 3. The next time through the loop, each %hash gives a new key-value pair; say it’s ("a", 1) this time. (It knows to return a different pair than previously because it keeps track of where it is; in technical jargon, there’s an iterator stored in with each hash.§) Those two items are stored into ($key, $value). Since the number of elements in the source list was again 2, a true value, the while condition is true, and the loop body runs again, telling us a => 1. You go one more time through the loop, and by now you know what to expect, so it’s no surprise to see b => 2 appear in the output. But you knew it couldn’t go on forever. Now, when Perl evaluates each %hash, there are no more key-value pairs available so each has to return an empty list.‖ The empty list is assigned to ($key, $value), so $key gets undef, and $value also gets undef. But that hardly matters, because you’re evaluating the whole thing in the conditional expression of the while loop. The value of a list assignment in a scalar context is the number of elements in the source list—in this case, that’s 0. Since 0 is a false value, the while loop is done, and execution continues with the rest of the program. Of course, each returns the key-value pairs in a jumbled order. (It’s the same order as keys and values would give, incidentally; the “natural” order of the hash.) If you need to go through the hash in order, simply sort the keys, perhaps something like this: foreach $key (sort keys %hash) { $value = $hash{$key}; print "$key => $value\n"; # Or, we could have avoided the extra $value variable: # print "$key => $hash{$key}\n"; } We’ll see more about sorting hashes in Chapter 14. § Since each hash has its own private iterator, loops using each may be nested as long as they are iterating over different hashes. And, as long as we’re already in a footnote, we may as well tell you: it’s unlikely you’ll ever need to do so, but you may reset the iterator of a hash by using the keys or values function on the hash. Perl automatically resets the iterator if a new list is stored into the entire hash, or if each has iterated through all of the items to the “end” of the hash. On the other hand, adding new key-value pairs to the hash while iterating over it is generally a bad idea, since that won’t necessarily reset the iterator. That’s likely to confuse you, your maintenance programmer, and each as well. ‖ It’s being used in list context, so it can’t return undef to signal failure; that would be the one-element list (undef) instead of the empty (zero-element) list ( ). Hash Functions | 117 Typical Use of a Hash At this point, you may find it helpful to see a more concrete example. The Bedrock Library uses a Perl program in which a hash keeps track of how many books each person has checked out, among other information: $books{'fred'} = 3; $books{'wilma'} = 1; It’s easy to see whether an element of the hash is true or false; do this: if ($books{$someone}) { print "$someone has at least one book checked out.\n"; } But there are some elements of the hash that aren’t true: $books{"barney"} = 0; # no books currently checked out $books{"pebbles"} = undef; # no books EVER checked out; a new library card Since Pebbles has never checked out any books, her entry has the value of undef, rather than 0. There’s a key in the hash for everyone who has a library card. For each key (that is, for each library patron), there’s a value that is either a number of books checked out, or undef if that person’s library card has never been used. The exists Function To see whether a key exists in the hash (that is, whether someone has a library card or not), use the exists function, which returns a true value if the given key exists in the hash, whether the corresponding value is true or not: if (exists $books{"dino"}) { print "Hey, there's a library card for dino!\n"; } That is to say, exists $books{"dino"} will return a true value if (and only if) dino is found in the list of keys from keys %books. The delete Function The delete function removes the given key (and its corresponding value) from the hash (if there’s no such key, its work is done; there’s no warning or error in that case): my $person = "betty"; delete $books{$person}; # Revoke the library card for $person Note that this is not the same as storing undef into that hash element—in fact, it’s precisely the opposite! Checking exists($books{"betty"}) will give opposite results in these two cases; after a delete, the key can’t exist in the hash, but after storing undef, the key must exist. 118 | Chapter 6: Hashes In the example, delete versus storing undef is the difference between taking away Betty’s library card versus giving her a card that has never been used. Hash Element Interpolation You can interpolate a single hash element into a double-quoted string just as you’d expect: foreach $person (sort keys %books) { # each patron, in order if ($books{$person}) { print "$person has $books{$person} items\n"; # fred has 3 items } } But there’s no support for entire hash interpolation; "%books" is just the six characters of (literally) %books.# So you’ve seen all of the magical characters that need backslashing in double quotes: $ and @, because they introduce a variable that Perl will try to inter- polate; ", since that’s the quoting character that would otherwise end the double- quoted string; and \, the backslash itself. Any other characters in a double-quoted string are nonmagical and should simply stand for themselves.* The %ENV hash Your Perl program, like any other program, runs in a certain environment, and your program can look at the environment to get information about its surroundings. Perl stores this information in the %ENV hash. For instance, you’ll probably see a PATH key in %ENV: print "PATH is $ENV{PATH}\n"; Depending on your particular setup and operating system, you’ll see something like this: PATH is /usr/local/bin:/usr/bin:/sbin:/usr/sbin Most of these are set for you automatically, but you can add to the environment your- self. How you do this depends on your operating system and shell: Bourne shell $ CHARACTER=Fred; export CHARACTER $ export CHARACTER=Fred #Well, it couldn’t really be anything else; if you tried to print out the entire hash, as a series of key-value pairs, that would be nearly useless. And, as you saw in Chapter 5, the percent sign is frequently used in printf format strings; giving it another meaning here would be terribly inconvenient. * But do beware of the apostrophe ('), left square bracket ([), left curly brace ({), the small arrow (->), or double colon (::) following a variable name in a double-quoted string, as they could perhaps mean something you didn’t intend. The %ENV hash | 119 csh % setenv CHARACTER Fred DOS or Windows command C:> set CHARACTER=Fred Once you set these environment variables outside of your Perl program, you can access them inside your Perl program: print "CHARACTER is $ENV{CHARACTER}\n"; Exercises See “Answers to Exercises” on page 304 for answers to the following exercises: 1. [7] Write a program that will ask the user for a given name and report the corre- sponding family name. Use the names of people you know, or (if you spend so much time on the computer that you don’t know any actual people) use the fol- lowing table: Input Output fred flintstone barney rubble wilma flintstone 2. [15] Write a program that reads a series of words (with one word per line†) until end-of-input, then prints a summary of how many times each word was seen. (Hint: remember that when an undefined value is used as if it were a number, Perl auto- matically converts it to 0. It may help to look back at the earlier exercise that kept a running total.) So, if the input words were fred, barney, fred, dino, wilma, fred (all on separate lines), the output should tell us that fred was seen 3 times. For extra credit, sort the summary words in code point order in the output. 3. [15] Write a program to list all of the keys and values in %ENV. Print the results in two columns in ASCIIbetical order. For extra credit, arrange the output to vertically align both columns. The length function can help you figure out how wide to make the first column. Once you get the program running, try setting some new envi- ronment variables and ensuring that they show up in your output. † It has to be one word per line because we still haven’t shown you how to extract individual words from a line of input. 120 | Chapter 6: Hashes CHAPTER 7 In the World of Regular Expressions Perl has many features that set it apart from other languages. Of all those features, one of the most important is its strong support for regular expressions. These allow fast, flexible, and reliable string handling. But that power comes at a price. Regular expressions are actually tiny programs in their own special language, built inside Perl. (Yes, you’re about to learn another program- ming language!* Fortunately, it’s a simple one.) So in this chapter, you’ll visit the world of regular expressions, where (mostly) you can forget about the world of Perl. Then, in the next chapter, we’ll show you where this world fits into Perl’s world. Regular expressions aren’t merely part of Perl; they’re also found in sed and awk, procmail, grep, most programmers’ text editors such as vi and emacs, and even in more esoteric places. If you’ve seen some of these already, you’re ahead of the game. Keep watching, and you’ll see many more tools that use or support regular expressions, such as search engines on the Web, email clients, and others. The bad news is that every- body’s regular expressions have slightly different syntax, so you may need to learn to include or omit an occasional backslash. What Are Regular Expressions? A regular expression, often called a pattern in Perl, is a template that either matches or doesn’t match a given string.† That is, there are an infinite number of possible text strings; a given pattern divides that infinite set into two groups: the ones that match, and the ones that don’t. There’s never any kinda-sorta-almost-up-to-here wishy-washy matching: either it matches or it doesn’t. * Some might argue that regular expressions are not a complete programming language. We won’t argue, but Perl does have a way to embed more Perl code inside its regular expressions. † Purists would ask for a more rigorous definition. But then again, purists say that Perl’s patterns aren’t really regular expressions. If you’re serious about regular expressions, we highly recommend the book Mastering Regular Expressions by Jeffrey Friedl (O’Reilly). 121 A pattern may match just one possible string, or just two or three, or a dozen, or a hundred, or an infinite number. Or it may match all strings except for one, or except for some, or except for an infinite number.‡ We already referred to regular expressions as being little programs in their own simple programming language. It’s a simple lan- guage because the programs have just one task: to look at a string and say “it matches” or “it doesn’t match.”§ That’s all they do. One of the places you’re likely to have seen regular expressions is in the Unix grep command, which prints out text lines matching a given pattern. For example, if you wanted to see which lines in a given file mention flint and, somewhere later on the same line, stone, you might do something like this with the Unix grep command: $ grep 'flint.*stone' chapter*.txt chapter3.txt:a piece of flint, a stone which may be used to start a fire by striking chapter3.txt:found obsidian, flint, granite, and small stones of basaltic rock, which chapter9.txt:a flintlock rifle in poor condition. The sandstone mantle held several Don’t confuse regular expressions with shell filename-matching patterns, called globs, which is a different sort of pattern with its own rules. A typical glob is what you use when you type *.pm to the Unix shell to match all filenames that end in .pm. The previous example uses a glob of chapter*.txt. (You may have noticed that you had to quote the pattern to prevent the shell from treating it like a glob.) Although globs use a lot of the same characters that you use in regular expressions, those characters are used in totally different ways.‖ We’ll visit globs later, in Chapter 13, but for now try to put them totally out of your mind. Using Simple Patterns To match a pattern (regular expression) against the contents of $_, simply put the pat- tern between a pair of forward slashes (/). The simple sort of pattern is just a sequence of literal characters: $_ = "yabba dabba doo"; if (/abba/) { print "It matched!\n"; } The expression /abba/ looks for that four-letter string in $_; if it finds it, it returns a true value. In this case, it’s found more than one, but that doesn’t make any difference. If it’s found at all, it’s a match; if it’s not in there at all, it fails. ‡ And as you’ll see, you could have a pattern that always matches or that never does. In rare cases, even these may be useful. Generally, though, they’re mistakes. § The programs also pass back some information that Perl can use later. One such piece of information is the “regular expressions captures” that you’ll learn about in Chapter 8. ‖ Globs are also (alas) sometimes called patterns. What’s worse, though, is that some bad Unix books for beginners (and possibly written by beginners) have taken to calling globs “regular expressions,” which they certainly are not. This confuses many folks at the start of their work with Unix. 122 | Chapter 7: In the World of Regular Expressions Because you generally use a pattern match to return a true or false value, you almost always want to use it in the conditional expression of if or while. You’ll see more reasons for that in Chapter 8. All of the usual backslash escapes that you can put into double-quoted strings are available in patterns, so you could use the pattern /coke\tsprite/ to match the 11 characters of coke, a tab, and sprite. Unicode Properties Unicode characters know something about themselves; they aren’t just sequences of bits. Every character not only knows what it is, but it also knows what properties it has. Instead of matching on a particular character, you can match a type of character. Each property has a name, which you can read about in the perluniprops documenta- tion. To match a particular property, you put the name in \p{PROPERTY}. For instance, some characters are whitespace, corresponding to the property name Space. To match any sort of space, you use \p{Space}: if (/\p{Space}/) { # 26 different possible characters print "The string has some whitespace.\n"; } If you want to match a digit, you use the Digit property: if (/\p{Digit}/) { # 411 different possible characters print "The string has a digit.\n"; } Those are both much more expansive than the sets of characters you may have run into. Some properties are more specific, though. How about matching two hex digits, [0-9A-Fa-f], next to each other: if (/\p{Hex}\p{Hex}/) { print "The string has a pair of hex digits.\n"; } You can also match characters that don’t have a particular Unicode property. Instead of a lowercase p, you use an uppercase one to negate the property: if (/\P{Space}/) { # Not space (many many characters!) print "The string has one or more non-whitespace characters.\n"; } About Metacharacters Of course, if patterns matched only simple literal strings, they wouldn’t be very useful. That’s why there are a number of special characters, called metacharacters, that have special meanings in regular expressions. For example, the dot (.) is a wildcard character—it matches any single character except a newline (which is represented by "\n"). So, the pattern /bet.y/ would match betty. Using Simple Patterns | 123 Or it would match betsy, or bet=y, or bet.y, or any other string that has bet, followed by any one character (except a newline), followed by y. It wouldn’t match bety or betsey, though, since those don’t have exactly one character between the t and the y. The dot always matches exactly one character. So, if you want to match a period in the string, you could use the dot. But that would match any possible character (except a newline), which might be more than you wan- ted. If you wanted the dot to match just a period, you can simply backslash it. In fact, that rule goes for all of Perl’s regular expression metacharacters: a backslash in front of any metacharacter makes it nonspecial. So, the pattern /3\.14159/ doesn’t have a wildcard character. So the backslash is the second metacharacter. If you mean a real backslash, just use a pair of them—a rule that applies just as well everywhere else in Perl: $_ = 'a real \\ backslash'; if (/\\/) { print "It matched!\n"; } Simple Quantifiers It often happens that you need to repeat something in a pattern. The star (*) means to match the preceding item zero or more times. So, /fred\t*barney/ matches any number of tab characters between fred and barney. That is, it matches "fred\tbarney" with one tab, or "fred\t\tbarney" with two tabs, or "fred\t\t\tbarney" with three tabs, or even "fredbarney" with nothing in between at all. That’s because the star means “zero or more”—so you could even have hundreds of tab characters in between, but nothing other than tabs. You may find it helpful to think of the star as saying, “That previous thing, any number of times, even zero times” (because * is the “times” operator in multiplication).# What if you want to allow something besides tab characters? The dot matches any character,* so .* will match any character, any number of times. That means that the pattern /fred.*barney/ matches “any old junk” between fred and barney. Any line that mentions fred and (somewhere later) barney will match that pattern. People often call .* the “any old junk” pattern, because it can match any old junk in your strings. The star is a type of quantifier, meaning that it specifies a quantity of the preceding item. But it’s not the only quantifier; the plus (+) is another. The plus means to match the preceding item one or more times: /fred +barney/ matches if fred and barney are #In the math of regular expressions, it’s called the Kleene star. * Except newline. But we’re going to stop reminding you of that so often, because you know it by now. Most of the time it doesn’t matter, anyway, because your strings will more often not have newlines. But don’t forget this detail, because someday a newline will sneak into your string and you’ll need to remember that the dot doesn’t match newline. 124 | Chapter 7: In the World of Regular Expressions separated by spaces and only spaces. (The space is not a metacharacter.) This won’t match fredbarney, since the plus means that there must be one or more spaces between the two names, so at least one space is required. It may be helpful to think of the plus as saying, “That last thing, plus (optionally) more of the same thing.” There’s a third quantifier like the star and plus, but more limited. It’s the question mark (?), which means that the preceding item is optional. That is, the preceding item may occur once or not at all. Like the other two quantifiers, the question mark means that the preceding item appears a certain number of times. It’s just that in this case the item may match one time (if it’s there) or zero times (if it’s not). There aren’t any other possibilities. So, /bamm-?bamm/ matches either spelling: bamm-bamm or bammbamm. This is easy to remember, since it’s saying, “That last thing, maybe? Or maybe not?” All three of these quantifiers must follow something, since they tell how many times the previous item may repeat. Grouping in Patterns You can use parentheses (“( )”) to group parts of a pattern. So, parentheses are also metacharacters. As an example, the pattern /fred+/ matches strings like freddddddddd because the quantifier only applies to the thing right before it, but strings like that don’t show up often in real life. The pattern /(fred)+/ matches strings like fredfredfred, which is more likely to be what you wanted. And what about the pat- tern /(fred)*/? That matches strings like hello, world.† The parentheses also give you a way to reuse part of the string directly in the match. You can use back references to refer to text that you matched in the parentheses, called a capture group.‡ You denote a back reference as a backslash followed by a number, like \1, \2, and so on. The number denotes the capture group. When you use the parentheses around the dot, you match any non-newline character. You can match again whichever character you matched in those parentheses by using the back reference \1: $_ = "abba"; if (/(.)\1/) { # matches 'bb' print "It matched same character next to itself!\n"; } The (.)\1 says that you have to match a character right next to itself. At first try, the (.) matches an a, but when it looks at the back reference, which says the next thing it must match is a, that trial fails. Perl starts over, using the (.) to match the next character, † The star means to match zero or more repetitions of fred. When you’re willing to settle for zero, it’s hard to be disappointed! That pattern will match any string, even the empty string. ‡ You may also see “memories” or “capture buffers” in older documentation and earlier editions of this book, but the official name is “capture group.” Later you’ll see how to make a noncapturing group. Using Simple Patterns | 125 a b. The back reference \1 now says that the next character in the pattern is b, which Perl can match. The back reference doesn’t have to be right next to the capture group. The next pattern matches any four non-newline characters after a literal y, and you use the \1 back reference to denote that you want to match the same four characters after the d: $_ = "yabba dabba doo"; if (/y(....) d\1/) { print "It matched the same after y and d!\n"; } You can use multiple groups of parentheses, and each group gets its own back reference. You want to match a non-newline character in a capture group, followed by another non-newline character in a capture group. After those two groups, you use the back reference \2 followed by the back reference \1. In effect, you’re matching a palindrome such as abba: $_ = "yabba dabba doo"; if (/y(.)(.)\2\1/) { # matches 'abba' print "It matched after the y!\n"; } Now, this brings up the question, “How do I know which group gets which number?” Fortunately, Larry did the easiest thing for humans to understand: just count the order of the opening parenthesis and ignore nesting: $_ = "yabba dabba doo"; if (/y((.)(.)\3\2) d\1/) { print "It matched!\n"; } You might be able to see this if you write out the regular expression to see the different parts (although this isn’t a valid regular expression)§: ( # first open parenthesis (.) # second open parenthesis (.) # third open parenthesis \3 \2 ) Perl 5.10 introduced a new way to denote back references. Instead of using the back- slash and a number, you can use \g{N}, where N is the number of the back reference that you want to use. Consider the problem where you want to use a back reference next to a part of the pattern that is a number. In this regular expression, you want to use \1 to repeat the character you matched in the parentheses and follow that with the literal string 11: § You can expand regular expressions like this by using the /x modifier, but we’re not showing that to you until Chapter 8. 126 | Chapter 7: In the World of Regular Expressions $_ = "aa11bb"; if (/(.)\111/) { print "It matched!\n"; } Perl has to guess what you mean there. Is that \1, \11, or \111? Perl will create as many back references as it needs, so it assumes that you mean \111. Since you don’t have 111 (or 11) capture groups, Perl complains when it tries to compile the program. By using \g{1}, you disambiguate the back reference and the literal parts of the pattern:‖ use 5.010; $_ = "aa11bb"; if (/(.)\g{1}11/) { print "It matched!\n"; } With the \g{N} notation, you can also use negative numbers. Instead of specifying the absolute number of the capture group, you can specify a relative back reference. You can rewrite the last example to use –1 as the number to do the same thing: use 5.010; $_ = "aa11bb"; if (/(.)\g{–1}11/) { print "It matched!\n"; } If you decide to add more to that pattern later, you don’t have to remember to change the back reference. If you add another capture group, you change the absolute num- bering of all the back references. The relative back reference, however, just counts from its own position and refers to the group right before it no matter its absolute number, so it stays the same: use 5.010; $_ = "xaa11bb"; if (/(.)(.)\g{–1}11/) { print "It matched!\n"; } Alternatives The vertical bar (|), often called “or” in this usage, means that either the left side may match, or the right side. That is, if the part of the pattern on the left of the bar fails, the part on the right gets a chance to match. So, /fred|barney|betty/ will match any string that mentions fred, or barney, or betty. ‖ In general, you could leave the curly braces off the \g{1} and just use \g1, but in this case you need the braces. Instead of thinking about it, we recommend just using the braces all the time, at least until you’re more sure of yourself. Using Simple Patterns | 127 Now you can make patterns like /fred( |\t)+barney/, which matches if fred and barney are separated by spaces, tabs, or a mixture of the two. The plus means to repeat one or more times; each time it repeats, the ( |\t) has the chance to match either a space or a tab.# There must be at least one of those characters between the two names. If you want the characters between fred and barney to all be the same, you could rewrite that pattern as /fred( +|\t+)barney/. In this case, the separators must be all spaces or all tabs. The pattern /fred (and|or) barney/ matches any string containing either of the two possible strings: fred and barney, or fred or barney.* You could match the same two strings with the pattern /fred and barney|fred or barney/, but that would be too much typing. It would probably also be less efficient, depending upon what optimizations are built into the regular expression engine. Character Classes A character class, a list of possible characters inside square brackets ([]), matches any single character from within the class. It matches just one single character, but that one character may be any of the ones you list in the brackets. For example, the character class [abcwxyz] may match any one of those seven charac- ters. For convenience, you may specify a range of characters with a hyphen (-), so that class may also be written as [a-cw-z]. That didn’t save much typing, but it’s more usual to make a character class like [a-zA-Z] to match any one letter out of that set of 52. Those 52 don’t include letters like Ä and é and ø and Ü. Those are different characters, but we’ll show you how to match them later. You may use the same character shortcuts as in any double-quotish string to define a character, so the class [\000-\177] matches any seven-bit ASCII character.† Of course, a character class will be just part of a full pattern; it will never stand on its own in Perl. For example, you might see code that says something like this: $_ = "The HAL-9000 requires authorization to continue."; if (/HAL-[0-9]+/) { print "The string mentions some model of HAL computer.\n"; } Sometimes, it’s easier to specify the characters you want to leave out, rather than the ones within the character class. A caret (^) at the start of the character class negates it. That is, [^def] will match any single character except one of those three. And [^n\-z] #This particular match would normally be done more efficiently with a character class, as you’ll see in the next section. * Note that the words and and or are not operators in regular expressions! They are shown here in a fixed-width typeface because they’re part of the strings. † At least, if you use ASCII and not EBCDIC. 128 | Chapter 7: In the World of Regular Expressions Character Classes | 129 .effectiveperlprogramming.com/blog/991. Is this getting too weird yet? See “Know your character classes under different semantics” at http://www ‡ Even under Unicode semantics, the \s still doesn’t match the vertical tab, next line, or nonbreaking space. Unicode property \p{Space}.‡ Before Perl 5.6, \s only matched the five whitespace The \s shortcut is good for matching any whitespace, which is almost the same as the } say 'The string mentions some model of HAL computer.'; if (/HAL-[\d]+/a) { # old, ASCII semantics $_ = 'The HAL-9000 requires authorization to continue.'; use 5.014; tells Perl to use the old ASCII interpretation: modifier on the end of the match operator (we explain options further in Chapter 8) way for you to restore the old ASCII semantics if that’s what you really want. The /a Recognizing this problematic shift from ASCII to Unicode thinking, Perl 5.14 adds a counting in Arabic, Mongolian, or Thai. Still, \d matches those for all modern Perls. or ๒ is rare, unless you’re , ,٤ still mostly works because running into digits such as character class [0-9], and that’s how many people used it then and still use it today. It Before Perl 5.6, the \d shortcut was the same as the .٩٠٠٠-so that will also match HAL However, there are many more digits than the 0 to 9 that you may expect from ASCII, } say 'The string mentions some model of HAL computer.'; if (/HAL-[\d]+/) { $_ = 'The HAL-9000 requires authorization to continue.'; write the pattern from the example about HAL as /HAL-\d+/ instead: For example, you can abbreviate the character class for any digit as \d. Thus, you could though, the only thing that matters is working code. is actually quite a big deal in Perl, likely to draw some heated debates. In the end, use these character classes because they don’t realize it’s not the 1990s anymore. This people—whether its stuff they wrote a long time ago or even yesterday—and they may reality. And you can’t escape reality either. You’re going to see code from other bit sad for those of us who have used Perl for a long, long time, but we can’t escape many characters, but with Unicode, they may have outlived their usefulness. That’s a were much easier to use in Perl’s ASCII days when you didn’t have to worry about so Some character classes appear so frequently that they have shortcuts. These shortcuts Character Class Shortcuts need a backslash because hyphens aren’t special outside a character class. because it’s special inside a character class. But the first hyphen in /HAL-[0-9]+/ doesn’t matches any character except for n, hyphen, or z. Note that the hyphen is backslashed characters form-feed, tab, newline, carriage return, and the space character itself, which, taken together, is the character class [\f\t\n\r ]. You can still get back to the ASCII whitespace semantics just like you did with /d: use 5.014; if (/\s/a) { # old, ASCII semantics say 'The string matched ASCII whitespace.'; } Perl 5.10 added more restrictive character classes for whitespace. The \h shortcut only matches horizontal whitespace. The \v shortcut only matches vertical whitespace. Taken together, the \h and \v are the same as \p{Space}: use 5.010; if (/\h/) { say 'The string matched some horizontal whitespace.'; } if (/\v/) { say 'The string matched some vertical whitespace.'; } if (/[\v\h]/) { # same as \p{Space}, but not more than \s say 'The string matched some whitespace.'; } The \R shortcut, introduced in Perl 5.10, matches any sort of linebreak, meaning that you don’t have to think about which operating system you’re using and what it thinks a linebreak is since \R will figure it out. This means you don’t have to sweat the differ- ence between \r\n, \n, and the other sorts of line endings that Unicode allows. It doesn’t matter to you if there are DOS or Unix line endings. The shortcut \w is a so-called “word” character, although its idea of a word isn’t like a normal word at all. Although popular, this character class has always been a bit of a problem. It’s not a big problem, but it advertises itself oddly. The “word” was actually meant as an identifier character: the ones that you could use to name a Perl variable or subroutine.§ With ASCII semantics, the \w matches the set of characters [a-zA-Z0-9_], and even with just that, it caused problems because most of the time people wanted to match only letters, as you would expect in a real word. Sometimes they’d want to match letters and numbers, but forgot that underscore was in there too. The trick with a good pattern is to not match more than you ever mean to match, and there’s only really one good place for [a-zA-Z0-9_], and that’s matching variable names. How often do you want to do that? § It’s really a C identifier, but Perl used the same thing in its teenage years. 130 | Chapter 7: In the World of Regular Expressions The Unicode expansion of \w matches quite a bit more: over 100,000 different char- acters.‖ The modern definition of \w is more cosmopolitan and correct, but a lot less useful for real world applications for most people. That doesn’t mean you can ignore it; people still use \w quite a bit and you are probably going to see it in a lot of code. It’s your job to figure out which of those 100,000 characters the authors intended to match: in most cases it’s going to be [a-zA-Z]. You’ll see more on this in “Word An- chors” on page 140 when we talk about “word boundaries.” In many cases, you’re going to make better, more maintainable patterns by avoiding these character class shortcuts in new code. Negating the Shortcuts Sometimes you may want the opposite of one of these three shortcuts. That is, you may want [^\d], [^\w], or [^\s], meaning a nondigit character, a nonword character, or a nonwhitespace character, respectively. That’s easy enough to accomplish by using their uppercase counterparts: \D, \W, or \S. These match any character that their counterpart would not match. Any of these shortcuts will work either in place of a character class (standing on their own in a pattern), or inside the square brackets of a larger character class. That means that you could now use /[0-9A-F]+/i to match hexadecimal (base 16) numbers, which use letters ABCDEF (or the same letters in lowercase) as additional digits. Another compound character class is [\d\D], which means any digit, or any nondigit. That is to say, any character at all! This is a common way to match any character, even a newline. (As opposed to ., which matches any character except a newline.) And then there’s the totally useless [^\d\D], which matches anything that’s not either a digit or a nondigit. Right—nothing! Exercises See “Answers to Exercises” on page 306 for answers to the following exercises. Remember, it’s normal to be surprised by some of the things that regular expressions do; that’s one reason that the exercises in this chapter are even more important than the others. Expect the unexpected: 1. [10] Make a program that prints each line of its input that mentions fred. (It shouldn’t do anything for other lines of input.) Does it match if your input string is Fred, frederick, or Alfred? Make a small text file with a few lines mentioning “fred flintstone” and his friends, then use that file as input to this program and the ones later in this section. ‖ The perluniprops documentation lists all of the Unicode properties and how many characters match that property. Exercises | 131 2. [6] Modify the previous program to allow Fred to match as well. Does it match now if your input string is Fred, frederick, or Alfred? (Add lines with these names to the text file.) 3. [6] Make a program that prints each line of its input that contains a period (.), ignoring other lines of input. Try it on the small text file from the previous exercise: does it notice Mr. Slate? 4. [8] Make a program that prints each line that has a word that is capitalized but not ALL capitalized. Does it match Fred but neither fred nor FRED? 5. [8] Make a program that prints each line that has a two of the same nonwhitespace characters next to each other. It should match lines that contain words such as Mississippi, Bamm-Bamm, or llama. 6. [8] Extra credit exercise: write a program that prints out any input line that men- tions both wilma and fred. 132 | Chapter 7: In the World of Regular Expressions CHAPTER 8 Matching with Regular Expressions In Chapter 7, you visited the world of Regular Expressions. Now you’ll see how that world fits into the world of Perl. Matches with m// In Chapter 7, you put patterns in pairs of forward slashes, like /fred/. But this is actually a shortcut for the m// (pattern match operator), the pattern match operator. As you saw with the qw// operator, you may choose any pair of delimiters to quote the contents. So, you could write that same expression as m(fred), m, m{fred}, or m[fred] using those paired delimiters, or as m,fred,, m!fred!, m^fred^, or many other ways using nonpaired delimiters.* The shortcut is that if you choose the forward slash as the delimiter, you may omit the initial m. Since Perl folks love to avoid typing extra characters, you’ll see most pattern matches written using slashes, as in /fred/. Of course, you should wisely choose a delimiter that doesn’t appear in your pattern.† If you wanted to make a pattern to match the beginning of an ordinary web URL, you might start to write /http:\/\// to match the initial "http://". But that would be easier to read, write, maintain, and debug if you used a better choice of delimiter: m%http://%.‡ It’s common to use curly braces as the delimiter. If you use a programmer’s * Nonpaired delimiters are the ones that don’t have a different “left” and “right” variety; the same punctuation mark is used for both ends. † If you’re using paired delimiters, you shouldn’t generally have to worry about using the delimiter inside the pattern, since that delimiter will generally be paired inside your pattern. That is, m(fred(.*)barney) and m{\w{2,}} and m[wilma[\n \t]+betty] are all fine, even though the pattern contains the quoting character, since each “left” has a corresponding “right.” But the angle brackets (< and >) aren’t regular expression metacharacters, so they may not be paired; if the pattern were m{(\d+)\s*>=?\s*(\d+)}, quoting it with angle brackets would mean having to backslash the greater-than sign so that it wouldn’t prematurely end the pattern. ‡ Remember, the forward slash is not a metacharacter, so you don’t need to escape it when it’s not the delimiter. 133 text editor, it probably has the ability to jump from an opening curly brace to the corresponding closing one, which can be handy in maintaining code. Match Modifiers There are several modifier letters, sometimes called flags,§ which you can append as a group right after the ending delimiter of a match operator to change its behavior from the default. We showed you the /a in Chapter 7, but there are many more. Case-Insensitive Matching with /i To make a case-insensitive pattern match, so that you can match FRED as easily as fred or Fred, use the /i modifier: print "Would you like to play a game? "; chomp($_ = ); if (/yes/i) { # case-insensitive match print "In that case, I recommend that you go bowling.\n"; } Matching Any Character with /s By default, the dot (.) doesn’t match newline, and this makes sense for most “look within a single line” patterns. If you might have newlines in your strings, and you want the dot to be able to match them, the /s modifier will do the job. It changes every dot‖ in the pattern to act like the character class [\d\D] does, which is to match any character, even if it is a newline. Of course, you have to have a string with newlines for this to make a difference: $_ = "I saw Barney\ndown at the bowling alley\nwith Fred\nlast night.\n"; if (/Barney.*Fred/s) { print "That string mentions Fred after Barney!\n"; } Without the /s modifier, that match would fail, since the two names aren’t on the same line. This sometimes leaves you with a problem though. The /s applies to every . in the pattern. What if you wanted to still match any character except a newline? You could use the character class [^\n], but that’s a bit much to type, so Perl 5.12 added the shortcut \N to mean the complement of \n. § And, in the land of Perl 6, these sorts of things have the formal name adverbs, but that boat has already sailed for Perl 5. ‖ If you wish to change just some of them, and not all, you’ll probably want to replace just those few with [\d\D]. 134 | Chapter 8: Matching with Regular Expressions Adding Whitespace with /x The third modifier you’ll see allows you to add arbitrary whitespace to a pattern, in order to make it easier to read: /-?[0-9]+\.?[0-9]*/ # what is this doing? / -? [0-9]+ \.? [0-9]* /x # a little better Since the /x allows whitespace inside the pattern, Perl ignores literal space or tab char- acters within the pattern. You could use a backslashed space or \t (among many other ways) to match these, but it’s more common to use \s (or \s* or \s+) when you want to match whitespace. Remember that Perl considers comments a type of whitespace, so you can put com- ments into that pattern to tell other people what you are trying to do: / -? # an optional minus sign [0-9]+ # one or more digits before the decimal point \.? # an optional decimal point [0-9]* # some optional digits after the decimal point /x # end of string Since the pound sign indicates the start of a comment, you need to use the escaped character, \#, or the character class, [#], if you need to match a literal pound sign: / [0-9]+ # one or more digits before the decimal point [#] # literal pound sign /x # end of string Also, be careful not to include the closing delimiter inside the comments, or it will prematurely terminate the pattern. This pattern ends before you think it does: / -? # with / without - <--- OOPS! [0-9]+ # one or more digits before the decimal point \.? # an optional decimal point [0-9]* # some optional digits after the decimal point /x # end of string Combining Option Modifiers If you want to use more than one modifier on the same match, just put them both at the end (their order isn’t significant): if (/barney.*fred/is) { # both /i and /s print "That string mentions Fred after Barney!\n"; } Match Modifiers | 135 Or as a more expanded version with comments: if (m{ barney # the little guy .* # anything in between fred # the loud guy }six) { # all three of /s and /i and /x print "That string mentions Fred after Barney!\n"; } Note the shift to curly braces here for the delimiters, allowing programmer-style editors to easily bounce from the beginning to the ending of the regular expression. Choosing a Character Interpretation Perl 5.14 adds some modifiers that let you tell Perl how to interpret the characters in a match for two important topics: case-folding and character class shortcuts. Everything in this section applies only to Perl 5.14 or later. There are three interpretations for this: ASCII, Unicode, and locale. It’s only that last one that causes the problems, though. The /a tells Perl to use ASCII, the /u tells Perl to use Unicode, and the /l tells Perl to respect the locale. Without these modifiers, Perl does what it thinks is the right thing based on the situations described in the perlre documentation. You use these modifiers to tell Perl exactly what you want despite whatever else is going on in the program. First, the character class shortcuts. You’ve already seen the /a modifier. That tells Perl to include only the ASCII ranges in \w, \d, and \s character class shortcuts. The /u match modifier tells Perl to use the much more expansive Unicode ranges for those shortcuts. The /l tells Perl to respect the locale settings, so any character that the locale thinks is a word character shows up in \w, for instance.# If you are going to use the character class shortcuts and want one interpretation over another, use the right modi- fier for your situation: use 5.014; /\w+/a # A-Z, a-z, 0-9, _ /\w+/u # any Unicode word charcter /\w+/l # The ASCII version, and word chars from the locale, # perhaps characters like Œ from Latin-9 Which one is right for you? We can’t tell you because we don’t know what you’re trying to do. Each of them might be right for you at different times. Of course, you can always create your own character classes to get exactly what you want if the shortcuts don’t work for you. #There’s also a /d, which tells Perl to use “traditional” behavior, where Perl might guess what to do. 136 | Chapter 8: Matching with Regular Expressions Now on to the harder issue. Consider the case-folding issue, where you need to know which lowercase letter you should get from an uppercase letter.* If you want to match while ignoring case, Perl has to know how to produce lowercase characters. In ASCII, you know a K’s (0x4B) partner is a k (0x6B). In ASCII, you also know that a k’s up- percase partner is K (0x4B), which seems sensible but is actually not. In Unicode, things are not as simple, but it’s still easy to deal with because the mapping is well defined.† The Kelvin sign, K (U+212A), also has k (0x6B) as its lowercase partner. Even though K and K might look the same to you, they aren’t to the computer.‡ That is, lowercasing is not one-to-one. Once you get the lowercase k, you can’t go back to its uppercase partner because there is more than one uppercase character for it. Not only that, some characters, such as the ligature ff (U+FB00), have two characters as their lowercase equivalent, in this case ff. The letter ß is ss in lowercase, but maybe you don’t want to match that. A single /a modifier affects the character class shortcuts, but if you have two /a, it also tells Perl to use ASCII-only case-folding: /k/aai # only matches the ASCII K or k, not Kelvin sign /k/aia # the /a's don't need to be next to each other /ss/aai # only matches ASCII ss, SS, sS, Ss, not ß /ff/aai # only matches ASCII ff, FF, fF, Ff, not ff With locales it’s not so simple. You have to know which locale you are using to know what a character is. If you have the ordinal value 0xBC, is that Latin-9’s Œ or Latin-1’s ¼ or something else in some other locale? You can’t know how to lowercase it until you know what the locale thinks that value represents:§ $_ = ; my $OE = chr( 0xBC ); # get exactly what we intend if (/$OE/i) { # case-insensitive??? Maybe not. print "Found $OE\n"; } In this case, you might get different results depending on how Perl treats the string in $_ and the string in match operator. If your source code is in UTF-8 but your input is Latin-9, what happens? In Latin-9, the character Œ has ordinal value 0xBC and its lowercase partner œ has 0xBD. In Unicode, Œ is code point U+0152 and œ is code point U+0153. In Unicode, U+0OBC is ¼ and doesn’t have a lowercase version. If your input in $_ is 0xBD and Perl treats that regular expression as UTF-8, you won’t get the * This is part of the “Unicode bug” in Perl, where the internal representation decides what answer you get. See the perlunicode documentation for gory details. † See http://unicode.org/Public/UNIDATA/CaseFolding.txt. ‡ Unless the production process distorted our source, you should be able to copy those characters from the ebook and verify they are not the same thing even if they have the same appearance. § We make the character with chr() to ensure we get the right bit pattern regardless of the encoding issues. Match Modifiers | 137 answer you expect. You can, however, add the /l modifier to force Perl to interpret the regular expression using the locale’s rules: $_ = ; my $OE = chr( 0xBC ); # get exactly what we intend if (/$OE/li) { # that's better print "Found $OE\n"; } If you always want to use Unicode semantics (which is the same as Latin-1) for this part, you can use the /u modifier: $_ = ; if (/Œ/ui) { # now uses Unicode print "Found Œ\n"; } If you think this is a big headache, you’re right. No one likes this situation, but Perl does the best it can with the input and encodings it has to deal with. If only we could reset history and not make so many mistakes next time. Other Options There are many other modifiers available. We’ll cover those as we get to them, or you can read about them in the perlop documentation and in the descriptions of m// and the other regular expression operators that you’ll see later in this chapter. Anchors By default, if a pattern doesn’t match at the start of the string, it can “float” on down the string, trying to match somewhere else. But there are a number of anchors that may be used to hold the pattern at a particular point in a string. The \A anchor matches at the absolute beginning of a string, meaning that your pattern will not float down the string at all. This pattern looks for an https only at the start of the string: m{\Ahttps?://}i If you want to anchor something to the end of the string, you use \z. This pattern matches .png only at the absolute end of the string: m{\.png\z}i Why “absolute end of string”? We have to emphasize that nothing can come after the \z because there is a bit of history here. There’s another end-of-string anchor, the \Z, which allows an optional newline after it. That makes it easy to match something at the end of a single line without worrying about the trailing newline: 138 | Chapter 8: Matching with Regular Expressions while () { print if /\.png\Z/; } If you had to think about the newline, you’d have to remove it before the match and put it back on for the output: while () { chomp; print "$_\n" if /\.png\z/; } Sometimes, you’ll want to use both of these anchors to ensure that the pattern matches an entire string. A common example is /\A\s*\Z/, which matches a blank line. But this “blank” line may include some whitespace characters, like tabs and spaces, which are invisible to you and me. Any line that matches that pattern looks just like any other one on paper, so this pattern treats all blank lines as equivalent. Without the anchors, it would match nonblank lines as well. The \A, \Z, and \z are Perl 5 regular expression features, but not everyone uses them. In Perl 4, where many people picked up their programming habits, the beginning-of- string anchor was the caret‖ (^) and the end-of-string was $. Those still work in Perl 5, but they morphed into the beginning-of-line and end-of-line anchors. What’s the difference between the beginning-of-line and beginning-of-string? It comes down to the difference between how you think about lines and how the computer thinks about lines. When you match against the string in $_, Perl doesn’t care what’s in it. To Perl, it’s just one big string, even if it looks like multiple lines to you because you have newlines in the string. Lines matter to humans because we spatially separate parts of the string: $_ = 'This is a wilma line barney is on another line but this ends in fred and a final dino line'; Suppose your task, however, is to find strings that have fred at the end of any line instead of just at the end of the entire string. In Perl 5, you can do that with the $ anchor and the /m modifier to turn on multiline matching. This pattern matches because in the multiline string, fred is at the end of a line: /fred$/m The addition of the /m changes how the old Perl 4 anchor works. Now it matches fred anywhere as long as it’s either followed by a newline anywhere in the string, or is at the absolute end of the string. ‖ Yes, you’ve seen the caret used in another way in patterns. As the first character of a character class, it negates the class. But outside of a character class, it’s a metacharacter in a different way, being the start-of-string anchor. Anchors | 139 The /m does the same for the ^ anchor, which then matches either at the absolute beginning of the string or anywhere after a newline. This pattern matches because in the multiline string, barney is at the beginning of a line: /^barney/m Without the /m, the ^ and $ act just like \A and \z. However, since someone might come along later and add a /m switch, changing your anchors to something you didn’t intend, it’s safer to use only the anchors that mean exactly what you want and nothing more. But, as we said, many programmers have habits they carried over from Perl 4, so you’ll still see many ^ and $ anchors that really should be \A and \z. For the rest of the book, we’ll use \A and \z unless we specifically want multiline matching. Word Anchors Anchors aren’t just at the ends of the string. The word-boundary anchor \b matches at either end of a word.# So you can use /\bfred\b/ to match the word fred but not frederick or alfred or manfred mann. This is similar to the feature often called something like “match whole words only” in a word processor’s search command. Alas, these aren’t words as you and I are likely to think of them; they’re those \w-type words made up of ordinary letters, digits, and underscores. The \b anchor matches at the start or end of a group of \w characters. This is subject to the rules that \w is following, as we showed you earlier in this chapter. In Figure 8-1, there’s a gray underline under each “word,” and the arrows show the corresponding places where \b could match. There are always an even number of word boundaries in a given string, since there’s an end-of-word for every start-of-word. The “words” are sequences of letters, digits, and underscores; that is, a word in this sense is what’s matched by /\w+/. There are five words in that sentence: That, s, a, word, and boundary.* Notice that the quote marks around word don’t change the word boundaries; these words are made of \w characters. Each arrow points to the beginning or the end of one of the gray underlines, since the word-boundary anchor \b matches only at the beginning or the end of a group of word characters. #Some regular expression implementations have one anchor for start-of-word and another for end-of-word, but Perl uses \b for both. * You can see why we wish we could change the definition of “word”; That's should be one word, not two words with an apostrophe in between. And even in text that may be mostly ordinary English, it’s normal to find a soupçon of other characters spicing things up. 140 | Chapter 8: Matching with Regular Expressions Figure 8-1. Word-boundary matches with \b The word-boundary anchor is useful to ensure that you don’t accidentally find cat in delicatessen, dog in boondoggle, or fish in selfishness. Sometimes you’ll want just one word-boundary anchor, as when using /\bhunt/ to match words like hunt or hunting or hunter, but not shunt, or when using /stone\b/ to match words like sand stone or flintstone but not capstones. The nonword-boundary anchor is \B; it matches at any point where \b would not match. So the pattern /\bsearch\B/ will match searches, searching, and searched, but not search or researching. The Binding Operator =~ Matching against $_ is merely the default; the binding operator (=~) tells Perl to match the pattern on the right against the string on the left, instead of matching against $_.† For example: my $some_other = "I dream of betty rubble."; if ($some_other =~ /\brub/) { print "Aye, there's the rub.\n"; } The first time you see it, the binding operator looks like some kind of assignment operator. But it’s no such thing! It is simply saying, “This pattern match that would attach to $_ by default—make it work with this string on the left instead.” If there’s no binding operator, the expression uses $_ by default. In the (somewhat unusual) example below, $likes_perl is set to a Boolean value ac- cording to what the user typed at the prompt. This is a little on the quick-and-dirty side because you discard the line of input itself. This code reads the line of input, tests that string against the pattern, then discards the line of input.‡ It doesn’t use or change $_ at all: print "Do you like Perl? "; my $likes_perl = ( =~ /\byes\b/i); ... # Time passes... † The binding operator is also used with some other operations besides the pattern match, as you’ll see later. ‡ Remember, Perl doesn’t automatically store the line of input into $_ unless the line-input operator () is all alone in the conditional expression of a while loop. The Binding Operator =~ | 141 if ($likes_perl) { print "You said earlier that you like Perl, so...\n"; ... } Because the binding operator has fairly high precedence, the parentheses around the pattern test expression aren’t required, so the following line does the same thing as the one above—it stores the result of the test (and not the line of input) into the variable: my $likes_perl = =~ /\byes\b/i; Interpolating into Patterns The match operator acts just as if it were a double-quoted string, interpolating any variables it finds. This allows you to write a quick grep-like program like this: #!/usr/bin/perl -w my $what = "larry"; while (<>) { if (/\A($what)/) { # pattern is anchored at beginning of string print "We saw $what in beginning of $_"; } } The pattern is built up out of whatever’s in $what when you run the pattern match. In this case, it’s the same as if we wrote /\A(larry)/, looking for larry at the start of each line. But you didn’t have to get the value of $what from a literal string; you could get it instead from the command-line arguments in @ARGV: my $what = shift @ARGV; Now, if the first command-line argument were fred|barney, the pattern be- comes /\A(fred|barney)/, looking for fred or barney at the start of each line.§ The parentheses (which weren’t really necessary when searching for larry) are important, now, because without them you match fred at the start or barney anywhere in the string. With that line changed to get the pattern from @ARGV, this program resembles the Unix grep command. But you have to watch out for metacharacters in the string. If $what contains 'fred(barney', the pattern would look like /^(fred(barney)/, and you know that can’t work right—it’ll crash your program with an invalid regular expression error. With some advanced techniques,‖ you can trap this kind of error (or prevent the magic of the metacharacters in the first place) so that it won’t crash your program. But for § The astute reader will know that you can’t generally type fred|barney as an argument at the command line because the vertical bar is a shell metacharacter. See the documentation to your shell to learn about how to quote command-line arguments. ‖ In this case, you would use an eval block to trap the error, or you would quote the interpolated text using quotemeta (or its \Q equivalent form) so that it’s no longer treated as a regular expression. 142 | Chapter 8: Matching with Regular Expressions now, just know that if you give your users the power of regular expressions, they’ll also need the responsibility to use them correctly. The Match Variables Parentheses normally trigger the regular expression engine’s capturing features. The capture groups hold the part of the string matched by the part of the pattern inside parentheses. If there is more than one pair of parentheses, there will be more than one capture group. Each regular expression capture holds part of the original string, not part of the pattern. You could refer to these groups in your pattern using back references, but these groups also stick around after the match as the capture variables. Since these variables hold strings, they are scalar variables; in Perl, they have names like $1 and $2. There are as many of these variables as there are pairs of capturing parentheses in the pattern. As you’d expect, $4 means the string matched by the fourth set of parentheses. This is the same string that the back reference \4 would refer to during the pattern match. But these aren’t two different names for the same thing; \4 refers back to the capture during the pattern while it is trying to match, while $4 refers to the capture of an already completed pattern match. For more information on back references, see the perlre documentation. These match variables are a big part of the power of regular expressions, because they let you pull out the parts of a string: $_ = "Hello there, neighbor"; if (/\s(\[a-zA-Z]+),/) { # capture the word between space and comma print "the word was $1\n"; # the word was there } Or you could use more than one capture at once: $_ = "Hello there, neighbor"; if (/(\S+) (\S+), (\S+)/) { print "words were $1 $2 $3\n"; } That tells you that the words were Hello there neighbor. Notice that there’s no comma in the output. Because the comma is outside of the capture parentheses in the pattern, there is no comma in capture two. Using this technique, you can choose exactly what you want in the captures, as well as what you want to leave out. You could even have an empty match variable,# if that part of the pattern might be empty. That is, a match variable may contain the empty string: my $dino = "I fear that I'll be extinct after 1000 years."; if ($dino =~ /([0-9]*) years/) { print "That said '$1' years.\n"; # 1000 #An empty string is different than an undefined one. If you have three or fewer sets of parentheses in the pattern, $4 will be undef. The Match Variables | 143 } $dino = "I fear that I'll be extinct after a few million years."; if ($dino =~ /([0-9]*) years/) { print "That said '$1' years.\n"; # empty string } The Persistence of Captures These capture variables generally stay around until the next successful pattern match.* That is, an unsuccessful match leaves the previous capture values intact, but a successful one resets them all. This correctly implies that you shouldn’t use these match variables unless the match succeeded; otherwise, you could be seeing a capture from some previous pattern. The following (bad) example is supposed to print a word matched from $wilma. But if the match fails, it’s using whatever leftover string happens to be found in $1: my $wilma = '123'; $wilma =~ /([0-9]+)/; # Succeeds, $1 is 123 $wilma =~ /([a-zA-Z]+)/; # BAD! Untested match result print "Wilma's word was $1... or was it?\n"; # Still 123! This is another reason a pattern match is almost always found in the conditional ex- pression of an if or while: if ($wilma =~ /([a-zA-Z]+)/) { print "Wilma's word was $1.\n"; } else { print "Wilma doesn't have a word.\n"; } Since these captures don’t stay around forever, you shouldn’t use a match variable like $1 more than a few lines after its pattern match. If your maintenance programmer adds a new regular expression between your regular expression and your use of $1, you’ll be getting the value of $1 for the second match, rather than the first. For this reason, if you need a capture for more than a few lines, it’s generally best to copy it into an ordinary variable. Doing this helps make the code more readable at the same time: if ($wilma =~ /([a-zA-Z]+)/) { my $wilma_word = $1; ... } Later, in Chapter 9, you’ll see how to get the capture value directly into the variable at the same time as the pattern match happens, without having to use $1 explicitly. * The actual scoping rule is much more complex (see the documentation if you need it), but as long as you don’t expect the match variables to be untouched many lines after a pattern match, you shouldn’t have problems. 144 | Chapter 8: Matching with Regular Expressions Noncapturing Parentheses So far you’ve seen parentheses that capture parts of a matched string and store them in the capture variables, but what if you just want to use the parentheses to group things? Consider a regular expression where we want to make part of it optional, but only capture another part of it. In this example, you want “bronto” to be optional, but to make it optional you have to group that sequence of characters with parentheses. Later in the pattern, you use an alternation to get either “steak” or “burger,” and you want to know which one you found: if (/(bronto)?saurus (steak|burger)/) { print "Fred wants a $2\n"; } Even if “bronto” is not there, its part of the pattern goes into $1. Perl just counts the order of the opening parentheses to decide what the capture variables will be. The part that you want to remember ends up in $2. In more complicated patterns, this situation can get quite confusing. Fortunately, Perl’s regular expressions have a way to use parentheses to group things but not trigger the capture groups, called noncapturing parentheses, and you write them with a special sequence. You add a question mark and a colon after the opening pa- renthesis, (?:)†, and that tells Perl you only want to use these parentheses for grouping. You change your regular expression to use noncapturing parentheses around “bronto,” and the part that you want to remember now shows up in $1: if (/(?:bronto)?saurus (steak|burger)/) { print "Fred wants a $1\n"; } Later, when you change your regular expression, perhaps to include a possible barbecue version of the brontosaurus burger, you can make the added “BBQ ” (with a space!) optional and noncapturing, so the part you want to remember still shows up in $1. Otherwise, you’d potentially have to shift all of your capture variable names every time you add grouping parentheses to your regular expression: if (/(?:bronto)?saurus (?:BBQ )?(steak|burger)/) { print "Fred wants a $1\n"; } Perl’s regular expressions have several other special parentheses sequences that do fancy and complicated things, like look-ahead, look-behind, embed comments, or even run code right in the middle of a pattern. You’ll have to check out the perlre documen- tation for the details though. † This is the fourth type of ? you’ll see in regular expressions: a literal question mark (escaped), the 0 or 1 quantifier, the nongreedy modifier (Chapter 9), and now the start of an extended pattern. The Match Variables | 145 Named Captures You can capture parts of the string with parentheses and then look in the number variables $1, $2, and so on to get the parts of the string that matched. Keeping track of those number variables and what should be in them can be confusing even for simple patterns. Consider this regular expression that tries to match the two names in $names: use 5.010; my $names = 'Fred or Barney'; if ( $names =~ m/(\w+) and (\w+)/ ) { # won't match say "I saw $1 and $2"; } You don’t see the message from say because the string has an or where you were ex- pecting an and. Maybe you were supposed to have it both ways, so you change the regular expression to have an alternation to handle both and and or, adding another set of parentheses to group the alternation: use 5.010; my $names = 'Fred or Barney'; if ( $names =~ m/(\w+) (and|or) (\w+)/ ) { # matches now say "I saw $1 and $2"; } Oops! You see a message this time, but it doesn’t have the second name in it because you added another set of capture parentheses. The value in $2 is from the alternation and the second name is now in $3 (which we don’t output): I saw Fred and or You could have used the noncapturing parentheses to get around this, but the real problem is that you have to remember which numbered parentheses belong to which data you are trying to capture. Imagine how much tougher this gets with many captures. Instead of remembering numbers such as $1, Perl 5.10 lets you name the captures di- rectly in the regular expression. It saves the text it matches in the hash named %+: the key is the label you used and the value is the part of the string that it matched. To label a match variable, you use (?