ptg6882136 ptg6882136 JavaTM Performance ptg6882136 Publications in The Java™ Series are supported, endorsed, and written by the creators of Java at Sun Microsystems, Inc. This series is the official source for expert instruction in Java and provides the complete set of tools you’ll need to build effective, robust, and portable applications and applets. The Java™ Series is an indispensable resource for anyone looking for definitive information on Java technology. Visit Sun Microsystems Press at sun.com/books to view additional titles for developers, programmers, and system administrators working with Java and other Sun technologies. Visit informit.com/thejavaseries for a complete list of available publications. The Java™ Series ptg6882136 JavaTM Performance Charlie Hunt Binu John Upper Saddle River, NJ • Boston • Indianapolis • San Francisco New York • Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City ptg6882136 Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC Inter- national, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd. The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for inci- dental or consequential damages in connection with or arising out of the use of the information or programs contained herein. This document is provided for information purposes only and the contents hereof are subject to change with- out notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact U.S. Corporate and Government Sales (800) 382-3419 corpsales@pearsontechgroup.com For sales outside the United States, please contact: International Sales international@pearson.com Visit us on the Web: informit.com/aw Library of Congress Cataloging-in-Publication Data Hunt, Charlie, 1962- Java performance / Charlie Hunt, Binu John. p. cm. Includes bibliographical references and index. ISBN-13: 978-0-13-714252-1 ISBN-10: 0-13-714252-8 (pbk. : alk. paper) 1. Java (Computer program language) 2. Computer programming. I. John, Binu, 1967- II. Title. QA76.73.J38H845 2012 005.13’3—dc23 2011031889 Copyright © 2012 Oracle America, Inc. All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or like- wise. To obtain permission to use material from this work, please submit a written request to Pearson Educa- tion, Inc., Permissions Department, One Lake Street, Upper Saddle River, New Jersey 07458, or you may fax your request to (201) 236-3290. ISBN-13: 978-0-13-714252-1 ISBN-10: 0-13-714252-8 Text printed in the United States on recycled paper at Edwards Brothers in Ann Arbor, Michigan. First printing, September 2011 ptg6882136 v To the three B’s, Barb, Boyd, and Beau – C.H. To Rita, Rachael, and Kevin – B.J. ptg6882136 This page intentionally left blank ptg6882136 Contents Foreword by James Gosling xi Foreword by Steve Wilson xiii Preface xv Acknowledgments xix About the Authors xxi Chapter 1 Strategies, Approaches, and Methodologies 1 Forces at Play 2 Two Approaches, Top Down and Bottom Up 5 Choosing the Right Platform and Evaluating a System 8 Bibliography 11 Chapter 2 Operating System Performance Monitoring 13 Definitions 14 CPU Utilization 14 CPU Scheduler Run Queue 28 Memory Utilization 32 Network I/O Utilization 41 Disk I/O Utilization 46 vii ptg6882136 Additional Command Line Tools 49 Monitoring CPU Utilization on SPARC T-Series Systems 50 Bibliography 53 Chapter 3 JVM Overview 55 HotSpot VM High Level Architecture 56 HotSpot VM Runtime 58 HotSpot VM Garbage Collectors 80 HotSpot VM JIT Compilers 92 HotSpot VM Adaptive Tuning 100 References 106 Chapter 4 JVM Performance Monitoring 107 Definitions 108 Garbage Collection 108 JIT Compiler 146 Class Loading 147 Java Application Monitoring 150 Bibliography 153 Chapter 5 Java Application Profiling 155 Terminology 157 Oracle Solaris Studio Performance Analyzer 159 NetBeans Profiler 189 References 209 Chapter 6 Java Application Profiling Tips and Tricks 211 Performance Opportunities 211 System or Kernel CPU Usage 212 Lock Contention 222 Volatile Usage 234 Data Structure Resizing 235 Increasing Parallelism 243 High CPU Utilization 246 Other Useful Analyzer Tips 247 Bibliography 249 viii Contents ptg6882136 Contents ix Chapter 7 Tuning the JVM, Step by Step 251 Methodology 252 Application Systemic Requirements 255 Rank Systemic Requirements 257 Choose JVM Deployment Model 258 Choose JVM Runtime 259 GC Tuning Fundamentals 262 Determine Memory Footprint 268 Tune Latency/Responsiveness 278 Tune Application Throughput 307 Edge Cases 316 Additional Performance Command Line Options 316 Bibliography 321 Chapter 8 Benchmarking Java Applications 323 Challenges with Benchmarks 324 Design of Experiments 347 Use of Statistical Methods 348 Reference 355 Bibliography 355 Chapter 9 Benchmarking Multitiered Applications 357 Benchmarking Challenges 357 Enterprise Benchmark Considerations 360 Application Server Monitoring 382 Profiling Enterprise Applications 399 Bibliography 401 Chapter 10 Web Application Performance 403 Benchmarking Web Applications 404 Web Container Components 405 Web Container Monitoring and Performance Tunings 408 Best Practices 427 Bibliography 450 Chapter 11 Web Services Performance 453 XML Performance 454 ptg6882136 x Contents Validation 460 Resolving External Entities 462 Partial Processing of XML Documents 465 Selecting the Right API 468 JAX-WS Reference Implementation Stack 471 Web Services Benchmarking 473 Factors That Affect Web Service Performance 477 Performance Best Practices 486 Bibliography 503 Chapter 12 Java Persistence and Enterprise Java Beans Performance 505 EJB Programming Model 506 The Java Persistence API and Its Reference Implementation 507 Monitoring and Tuning the EJB Container 511 Transaction Isolation Level 521 Best Practices in Enterprise Java Beans 522 Best Practices in Java Persistence 540 Bibliography 551 Appendix A HotSpot VM Command Line Options of Interest 553 Appendix B Profiling Tips and Tricks Example Source Code 573 Lock Contention First Implementation 573 Lock Contention Second Implementation 583 Lock Contention Third Implementation 593 Lock Contention Fourth Implementation 603 Lock Contention Fifth Implementation 613 First Resizing Variant 624 Second Resizing Variant 636 Increasing Parallelism Single-Threaded Implementation 647 Increasing Parallelism Multithreaded Implementation 657 Index 669 ptg6882136 Foreword Tuning a Java application can be challenging in today’s large-scale mission-critical world. There are issues to be aware of in everything from the structure of your algo- rithms, to their memory allocation patterns, to the way they do disk and file I/O. Almost always, the hardest part is figuring out where the issues are. Even (perhaps especially) seasoned practitioners find that their intuitions are wrong. Performance- killing gremlins hide in the most unlikely places. As Wikipedia says, “Science (from Latin: scientia meaning ‘knowledge’) is a sys- tematic enterprise that builds and organizes knowledge in the form of testable expla- nations and predictions about the world.” Performance tuning must be approached as an experimental science: To do it properly, you have to construct experiments, perform them, and from the result construct hypotheses. Fortunately, the Java universe is awash in performance monitoring tools. From standalone applications to profilers built into development environments to tools provided by the operating system. They all need to be applied in a cohesive way to tease out the truth from a sea of noise. This book is the definitive masterclass in performance tuning Java applications. It readably covers a wide variety of tools to monitor and measure performance on a variety of hardware architectures and operating systems. And it covers how to con- struct experiments, interpret their results, and act on them. If you love all the gory details, this is the book for you. —James Gosling xi ptg6882136 This page intentionally left blank ptg6882136 Foreword xiii Today, Java is used at the heart of the world’s largest and most critical computing systems. However, when I joined the Java team in 1997 the platform was young and just gaining popularity. People loved the simplicity of the language, the portabil- ity of bytecodes, and the safety of garbage collection (versus traditional malloc/free memory management of other systems). However, there was a trade-off for these great features. Java was slow, and this limited the kinds of environments where you could use it. Over the next few years, we set about trying to fix this. We believed that just because Java applications were portable and safe they didn’t have to be slow. There were two major areas where we focused our attention. The first was to simply make the Java platform faster. Great strides were made in the core VM with advanced Just In Time compilation techniques, parallel garbage collection, and advanced lock management. At the same time the class libraries were tweaked and tuned to make them more efficient. All this led to substantial improvements in the ability to use Java for larger, more critical systems. The second area of focus for us was to teach people how to write fast software in Java. It turned out that although the syntax of the language looked similar to C, the techniques you needed to write efficient programs were quite different. To that end, Jeff Kessleman and I wrote one of the first books on Java performance, which was published back in 2000. Since then, many books have covered this topic, and experi- enced developers have learned to avoid some of the most common pitfalls that used to befall Java developers. ptg6882136 xiv Foreword After the platform began to get faster, and developers learned some of the tricks of writing faster applications, Java transformed into the enterprise-grade software powerhouse it is today. It began to be used for the largest, most important systems anywhere. However, as this started to happen, people began to realize one part was still missing. This missing piece was observability. When these systems get larger and larger, how do you know if you’re getting all the performance you can get? In the early days of Java we had primitive profiling tools. While these were useful, they had a huge impact on the runtime performance of the code. Now, modern JVMs come with built-in observability tools that allow you to understand key elements of your system’s performance with almost no performance penalty. This means these tools can be left enabled all the time, and you can check on aspects of your application while it’s running. This again changes the way people can approach performance. The authors of JavaTM Performance bring all these concepts together and update them to account for all the work that’s happened in the last decade since Jeff and I published our book. This book you are now reading is the most ambitious book on the topic of Java performance that has ever been written. Inside are a great many techniques for improving the performance of your Java applications. You’ll also come to understand the state of the art in JVM technology from the inside out. Curious about how the latest GC algorithms work? It’s in here! You’ll also learn how to use the latest and greatest observability tools, including those built into the JDK and other important tools bundled into popular operating systems. It’s exciting to see how all these recent advancements continue to push the plat- form forward, and I can’t wait to see what comes next. —Steve Wilson VP Engineering, Oracle Corporation Founding member of the Java Performance team Coauthor of JavaTM Platform Performance: Strategies and Tactics ptg6882136 Preface xv Welcome to the definitive reference on Java performance tuning! This book offers Java performance tuning advice for both Java SE and Java EE applications. More specifically, it offers advice in each of the following areas: perfor- mance monitoring, profiling, tuning the Java HotSpot VM (referred to as HotSpot VM hereafter), writing effective benchmarks, and Java EE application performance tuning. Although several Java performance books have been written over the years, few have packed the breadth of information found in this book. For example, the topics covered in this book include items such as an introduction into the inner work- ings of a modern Java Virtual Machine, garbage collection tuning, tuning Java EE applications, and writing effective benchmarks. This book can be read from cover to cover to gain an in-depth understanding of many Java performance topics. It can also be used as a task reference where you can pick up the text, go to a specific chapter on a given topic of interest, and find answers. Readers who are fairly new, or consider themselves a novice in the area of Java performance tuning, will likely benefit the most by reading the first four chapters and then proceeding to the topics or chapters that best address the particular Java performance tuning task they are undertaking. More experienced readers, those who have a fundamental understanding of performance tuning approaches and a basic understanding of the internals of the HotSpot VM along with an understanding of the tools to use for monitoring operating system performance and monitoring JVM performance, will find jumping to the chapters that focus on the performance tuning task at hand to be most useful. However, even those with advanced Java performance skills may find the information in the first four chapters useful. ptg6882136 Reading this book cover to cover is not intended to provide an exact formula to follow, or to provide the full and complete knowledge to turn you into an experienced Java performance tuning expert. Some Java performance issues will require special- ized expertise to resolve. Much of performance tuning is an art. The more you work on Java performance issues, the better versed you become. Java performance tuning also continues to evolve. For example, the most common Java performance issues observed five years ago were different from the ones observed today. Modern JVMs continue to evolve by integrating more sophisticated optimizations, runtimes, and garbage collectors. So too do underlying hardware platforms and operating systems evolve. This book provides up-to-date information as of the time of its writing. Read- ing and understanding the material presented in this book should greatly enhance your Java performance skills. It may also allow you to build a foundation of funda- mentals needed to become fluent in the art of Java performance tuning. And once you have a solid foundation of the fundamentals you will be able to evolve your per- formance tuning skills as hardware platforms, operating systems, and JVMs evolve. Here’s what you can expect to find in each chapter. Chapter 1, “Strategies, Approaches, and Methodologies,” presents various different approaches, strategies, and methodologies often used in Java performance tuning efforts. It also proposes a proactive approach to meeting performance and scalability goals for a software application under development through an enhancement to the traditional software development process. Chapter 2, “Operating System Performance Monitoring,” discusses performance monitoring at the operating system level. It presents which operating system statistics are of interest to monitor along with the tools to use to monitor those statistics. The operating systems of Windows, Linux, and Oracle Solaris are covered in this chapter. The performance statistics to monitor on other Unix-based systems, such as Mac OS X, use similar commands, if not the same commands as Linux or Oracle Solaris. Chapter 3, “JVM Overview,” provides a high level overview of the HotSpot VM. It provides some of the fundamental concepts of the architecture and workings of a modern Java Virtual Machine. It establishes a foundation for many of the chapters that follow in the book. Not all the information presented in this chapter is required to resolve every Java performance tuning task. Nor is it exhaustive in providing all the necessary background to solve any Java performance issue. However, it does provide sufficient background to address a large majority of Java performance issues that may require some of the concepts of the internal workings and capabilities of a modern Java Virtual Machine. The information in this chapter is applicable to understanding how to tune the HotSpot VM along with understanding the subject matter of Chapter 7 and how to write effective benchmarks, the topics covered in Chapters 8 and 9. Chapter 4, “JVM Performance Monitoring,” as the title suggests, covers JVM per- formance monitoring. It presents which JVM statistics are of interest to monitor xvi Preface ptg6882136 Preface xvii along with showing tools that can be used to monitor those statistics. It concludes with suggesting tools that can be extended to integrate both JVM level monitoring statistics along with Java application statistics of interest within the same monitor- ing tool. Chapter 5, “Java Application Profiling,” and Chapter 6, “Java Application Profiling Tips and Tricks,” cover profiling. These two chapters can be seen as complementary material to Chapter 2 and Chapter 4, which cover performance monitoring. Perfor- mance monitoring is typically used to identify whether a performance issue exists, or provides clues as to where the performance issue exists, that is, in the operating system, JVM, Java application, and so on. Once a performance issue is identified and further isolated with performance monitoring, a profiling activity usually follows. Chapter 5 presents the basics of Java method profiling and Java heap (memory) pro- filing. This profiling chapter presents free tools for illustrating the concepts behind these types of profiling. The tools shown in this chapter are not intended to suggest they are the only tools that can be used for profiling. Many profiling tools are avail- able both commercially and for free that offer similar capabilities, and some tools offer capabilities beyond what’s covered in Chapter 5. Chapter 6 offers several tips and tricks to resolving some of the more commonly observed patterns in profiles that tend to be indicative of particular types of performance problems. The tips and tricks identified in this chapter are not necessarily an exhaustive list but are ones that have been observed frequently by the authors over the course of years of Java performance tuning activities. The source code in many of the examples illustrated in this chapter can be found in Appendix B. Chapter 7, “Tuning the JVM, Step by Step,” covers tuning the HotSpot VM. The topics of tuning the HotSpot VM for startup, memory footprint, response time/ latency, and throughput are covered in the chapter. Chapter 7 presents a step-by- step approach to tuning the HotSpot VM covering choices such as which JIT compiler to use, which garbage collector to use, and how to size Java heaps, and also provides an indication when the Java application itself may require some rework to meet the performance goals set forth by application stakeholders. Most readers will likely find Chapter 7 to be the most useful and most referenced chapter in this book. Chapter 8, “Benchmarking Java Applications,” and Chapter 9, “Benchmarking Multi-tiered Applications,” present information on how to write effective benchmarks. Often benchmarks are used to help qualify the performance of a Java application by implementing a smaller subset of a larger application’s functionality. These two chap- ters also discuss the art of creating effective Java benchmarks. Chapter 8 covers the more general topics associated with writing effective benchmarks such as exploring some of the optimizations performed by a modern JVM. Chapter 8 also includes infor- mation on how to incorporate the use of statistical methods to gain confidence in your benchmarking experiments. Chapter 9 focuses more specifically on writing effective Java EE benchmarks. ptg6882136 For readers who have a specific interest in tuning Java EE applications, Chapter 10, “Web Application Performance,” Chapter 11, “Web Services Performance,” and Chap- ter 12, “Java Persistence and Enterprise Java Beans Performance,” focus specifically on the areas of Web applications, Web services, persistence, and Enterprise Java Bean performance, respectively. These three chapters present in-depth coverage of the performance issues often observed in Java EE applications and provide suggested advice and/or solutions to common Java EE performance issues. This book also includes two appendixes. Appendix A, “HotSpot VM Command Line Options of Interest,” lists HotSpot VM command line options that are referenced in the book and additional ones that may be of interest when tuning the HotSpot VM. For each command line option, a description of what the command line option does is given along with suggestions on when it is applicable to use them. Appendix B, “Profiling Tips and Tricks Example Source Code,” contains the source code used in Chapter 6’s examples for reducing lock contention, resizing Java collections, and increasing parallelism. xviii Preface ptg6882136 Acknowledgments Charlie Hunt Without the help of so many people this book would not have been possible. First I have to thank my coauthor, Binu John, for his many contributions to this book. Binu wrote all the Java EE material in this book. He is a talented Java performance engi- neer and a great friend. I also want to thank Greg Doech, our editor, for his patience. It took almost three years to go from a first draft of the book’s chapter outline until we handed over a manuscript. Thank you to Paul Hohensee and Dave Keenan for their insight, encouragement, support, and thorough reviews. To Tony Printezis and Tom Rodriguez, thanks for your contributions on the details of the inner workings of the Java HotSpot VM garbage collectors and JIT compilers. And thanks to all the engineers on the Java HotSpot VM runtime team for having detailed documentation on how various pieces of the HotSpot VM fit together. To both James Gosling and Steve Wilson, thanks for making time to write a foreword. Thanks to Peter Kessler for his thorough review of Chapter 7, “Tuning the JVM, Step by Step.” Thanks to others who contributed to the quality of this book through their insight and reviews: Dar- ryl Gove, Marty Itzkowitz, Geertjan Wielenga, Monica Beckwith, Alejandro Murillo, Jon Masamitsu, Y. Srinivas Ramkakrishna (aka Ramki), Chuck Rasbold, Kirk Pep- perdine, Peter Gratzer, Jeanfrancois Arcand, Joe Bologna, Anders Åstrand, Henrik Löf, and Staffan Friberg. Thanks to Paul Ciciora for stating the obvious, “losing the race” (when the CMS garbage collector can’t free enough space to keep up with the young generation promotion rate). Also, thanks to Kirill Soshalskiy, Jerry Driscoll, xix ptg6882136 xx Acknowledgments both of whom I have worked under during the time of writing this book, and to John Pampuch (Director of VM Technologies at Oracle) for their support. A very special thanks to my wife, Barb, and sons, Beau and Boyd, for putting up with a grumpy writer, especially during those times of “writer’s cramp.” Binu John This book has been possible only because of the vision, determination, and persever- ance of my coauthor, Charlie Hunt. Not only did he write the sections relating to Java SE but also completed all the additional work necessary to get it ready for publication. I really enjoyed working with him and learned a great deal along the way. Thank you, Charlie. A special thanks goes to Rahul Biswas for providing content relating to EJB and Java persistence and also for his willingness to review multiple drafts and provide valuable feedback. I would like to thank several people who helped improve the qual- ity of the content. Thank you to Scott Oaks and Kim Lichong for their encouragement and valuable insights into various aspects of Java EE performance; Bharath Mundla- pudi, Jitendra Kotamraju, and Rama Pulavarthi for their in-depth knowledge of XML and Web services; Mitesh Meswani, Marina Vatkina, and Mahesh Kannan for their help with EJB and Java persistence; and Jeanfrancois Arcand for his explanations, blogs, and comments relating to Web container. I was fortunate to work for managers who were supportive of this work. Thanks to Madhu Konda, Senior Manager during my days at Sun Microsystems; Sef Kloninger, VP of Engineering, Infrastructure, and Operations; and Sridatta Viswanath, Senior VP of Engineering and Operations at Ning, Inc. A special thank you to my children, Rachael and Kevin, and my wonderful wife, Rita, for their support and encouragement during this process. ptg6882136 About the Authors xxi Charlie Hunt is the JVM Performance Lead Engineer at Oracle. He is responsible for improving the performance of the HotSpot Java Virtual Machine and Java SE class librar- ies. He has also been involved in improving the performance of both GlassFish Server Open Source Edition and Oracle WebLogic application servers. He wrote his first Java program in 1998 and joined Sun Microsystems, Inc., in 1999 as a Senior Java Architect. He has been working on improving the performance of Java and Java applications ever since. He is a regular speaker on the subject of Java performance at many worldwide conferences including the JavaOne Conference. Charlie holds a Master of Science in Computer Science from the Illinois Institute of Technology and a Bachelor of Science in Computer Science from Iowa State University. Binu John is a Senior Performance Engineer at Ning, Inc., the world’s largest plat- form for creating social web sites. In his current role, he is focused on improving the performance and scalability of the Ning platform to support millions of page views per month. Before joining Ning, Binu spent more than a decade working on Java perfor- mance at Sun Microsystems, Inc. As a member of the Enterprise Java Performance team, he worked on several open source projects including the GlassFish Server Open Source Edition application server, the Open Source Enterprise Service Bus (Open ESB), and Open MQ JMS product. He has been an active contributor in the development of the vari- ous industry standard benchmarks such as SPECjms2007 and SPECjEnterprise2010, has published several performance white papers and has previously contributed to the XMLTest and WSTest benchmark projects at java.net. Binu holds Master of Science degrees in Biomedical Engineering and Computer Science from The University of Iowa. ptg6882136 This page intentionally left blank ptg6882136 1 1 Strategies, Approaches, and Methodologies With Java performance tuning, as with many other activities, you need a plan of action, an approach, or strategy. And, like many other activities, a set of information or background is required in a given domain to be successful. To be successful in a Java performance tuning effort, you need to be beyond the stage of “I don’t know what I don’t know” and into the “I know what I don’t know” stage or already be in the “I already know what I need to know” stage. If you find yourself a little lost in the definition of these three stages, they are further clarified here: I don’t know what I don’t know. Sometimes you are given a task that involves understanding a new problem domain. The first challenge in under- standing a new problem domain is to learn as much about the problem as you can because you may know little if anything about the problem domain. In this new problem domain there are many artifacts about the problem domain you do not know, or do not know what is important to know. In other words, you do not know what you need to know about the problem domain. Hence, the phrase, “I don’t know what I don’t know.” I know what I don’t know. Normally when you enter a new problem domain, one that you know little about, you eventually reach a point where you have discovered many different things about the problem domain that are important to know. But you do not know the specific details about those things that are important to know. When you have reached this stage it is called the “I know what I don’t know” stage. ptg6882136 2 Chapter 1 Strategies, Approaches, and Methodologies I already know what I need to know. At other times you are given a task in a problem domain in which you are familiar or you have developed the nec- essary skills and knowledge in the area to the point where you are considered a subject matter expert. Or as you learn a new problem domain you reach a point where you feel comfortable working within it, i.e., you have learned the information necessary to be effective in the problem domain. When you have reached this point, you are at the stage of “I already know what I need to know.” Given you have either bought this book or are considering buying this book, you probably are not in the “I already know what I need know” stage, unless you have a need to keep a good reference close by. If you are in the “I don’t know what I don’t know” stage, this chapter will likely help you identify what you don’t know and help you with an approach or strategy to tackle your Java performance issue. Those in the “I know what I don’t know” stage may also find the information in this chapter useful. This chapter begins by looking at the traditional forces at play that typically result in a performance tuning effort and suggests a high level process for integrating per- formance tuning into the software development process. This chapter then looks at two different performance tuning approaches, top down and bottom up. Forces at Play It is generally accepted at a high level that the traditional software development process consists of four major phases: analysis, design, coding, and testing. How these phases flow to together is illustrated in Figure 1-1. Analysis is the first phase of the process where requirements are evaluated, archi- tectural choices are weighed against their advantages and challenges, and high level abstractions are conceived. Design is the phase where, given the high level archi- tecture choices made in the analysis phase along with its high level abstractions, finer grained abstractions are realized and concrete implementations begin their conception. Coding, of course, is the phase where implementation of the design occurs. Following coding is the testing phase where the implementation is tested against the application requirements. It is worth noting that often the testing phase encompasses only functional testing, i.e, does the application do what it is specified to do, does it execute the actions it is specified to execute. Once the testing phase is completed the application is shipped or released to its customer(s). Many applications developed through these traditional software development phases tend to give little attention to performance or scalability until the applica- tion is released, or at the earliest in the testing phase. Wilson and Kesselman in their popular Java Platform Performance book [Wilson & Kesselman 2000] introduced an ptg6882136 Forces at Play 3 additional performance phase to complement the traditional software development process. The proposed performance testing phase was added after the testing phase and contained a decision branch of “performance acceptable.” If the performance and scalability criteria are met in this phase, the application is deemed ready to be shipped. Otherwise, the work flow results in profiling the application and branches back into one or more of the previous phases. Which particular phase the work flow branches back into depends on the results of the profiling activity. In other words, the output of the profiling activity identifies where the performance issue was intro- duced. A diagram illustrating Wilson and Kesselman’s additional performance phase is shown in Figure 1-2. To aid in the development of performance criteria to be evaluated in the perfor- mance testing phase, Wilson and Kesselman proposed the notion of specifying use cases to meet or address requirements specifically targeting performance in the anal- ysis phase. However, it is often the case an application’s requirements document fails to specify performance or scalability requirements. If an application you are work- ing with, or developing, does not specify performance and scalability requirements explicitly you should ask for specific performance and scalability requirements. Analysis Design Code Deploy Yes No Start Quality OK? Test Figure 1-1 Traditional Software Development Process ptg6882136 4 Chapter 1 Strategies, Approaches, and Methodologies For example, you should ask for throughput and latency requirements. The follow- ing list is an example of the types of questions these requirements should answer: What is the expected throughput of the application? What is the expected latency between a stimulus and a response to that stimulus? How many concurrent users or concurrent tasks shall the application support? What is the accepted throughput and latency at the maximum number of con- current users or concurrent tasks? What is the maximum worst case latency? What is the frequency of garbage collection induced latencies that will be tolerated? The requirements and corresponding use cases documented to answer questions such as those listed above should be used to drive the development of benchmarks Analysis Design Code Profile Deploy Yes No Start Performance Acceptable? Performance Testing Figure 1-2 Wilson & Kesselman’s Performance Process ptg6882136 Two Approaches, Top Down and Bottom Up 5 and performance tests to ensure the application meets the expected performance and scalability. These benchmarks and performance tests should be executed as part of the performance testing phase. As you evaluate use cases, some may be considered high risk, i.e., those that may be difficult to meet. High risk cases should be mitigated well before completion of the analysis phase by implementing prototypes, bench- marks, and micro-benchmarks. This approach allows you to catch painful decisions that are expensive to change once the software development leaves the analysis phase. It has been well documented that the later in the software development cycle a defect, poor design, or poor implementation choice is detected, the more expensive it is to fix it. Mitigating high risk use cases helps avoid those costly mistakes. Today many applications under development utilize automated build and test procedures. As a result, the enhanced software development process proposed by Wilson and Kesselman can be further improved by integrating automated perfor- mance testing as part of the automated build and test activity. The output of an automated performance testing activity could emit notifications, such as sending e-mail to the application stakeholders notifying them of performance results, such as identified performance regressions, identified performance improvements, or status on how well performance criteria is being met. The automated procedures could also file defects in a tracking system and automatically include pertinent performance statistics from the performance tests that fail to meet the application’s performance criteria. Integrating performance testing into automated build processes allows perfor- mance regressions to be identified earlier in the software development process by more easily tracking performance at each coding change committed to the source code base. Another practice worth consideration of integrating into the automated perfor- mance testing system is the use of statistical methods and automated statistical analysis. The use of statistical methods improves confidence in your performance testing results. Guidance and advice on the use of statistical methods, which can be challenging for many software developers (and mere mortals for that matter), is presented in the latter part of Chapter 8, “Benchmarking Java Applications.” Two Approaches, Top Down and Bottom Up There are two commonly accepted approaches to performance analysis: top down or bottom up. Top down, as the term implies, focuses at the top level of the application and drills down the software stack looking for problem areas and optimization oppor- tunities. In contrast, bottom up begins at the lowest level of the software stack, at the CPU level looking at statistics such as CPU cache misses, inefficient use of CPU instructions, and then working up the software stack at what constructs or idioms are ptg6882136 6 Chapter 1 Strategies, Approaches, and Methodologies used by the application. The top down approach is most commonly used by application developers. The bottom up approach is commonly used by performance specialists in situations where the performance task involves identifying performance differ- ences in an application on differing platform architectures, operating systems, or in the case of Java differing implementations of Java Virtual Machines. As you might expect, each approach finds different types of performance issues. In the following two subsections, these two approaches are looked at more closely by presenting more specifics about the activities performed within each approach. Top Down Approach The top down approach, as mentioned earlier, is likely the most common approach utilized for performance tuning. This approach is also commonly used when you have the ability to change the code at the highest level of the application software stack. In this approach, you begin by monitoring the application of interest under a load at which a stakeholder observes a performance issue. There are also situations in which an application is continuously monitored and as a result of a change in the application’s configuration or a change in the typical load the application experiences a degradation in performance. There may also be situations in which performance and scalability requirements for the application change and the application in its current state cannot meet those new requirements. Whatever the cause that stimulates the performance tuning activity, monitoring the application while it is running under a load of particular interest is the first step in a top down approach. This monitoring activity may include observing operating system level statistics, Java Virtual Machine (JVM) statistics, Java EE container statistics, and/or application performance instrumentation statistics. Then based on what the monitoring information suggests you begin the next step such as tuning the JVM’s garbage collectors, tuning the JVM’s command line options, tuning the operating system or profiling the application. Profiling the application may result in making implementation changes to the application, identifying an inefficient imple- mentation of a third-party library, or identifying an inefficient implementation of a class or method in the Java SE class library. For assistance in knowing what to monitor in a top down approach you can turn to Chapters 2, “Operating System Performance Monitoring,” and Chapter 4, “JVM Performance Monitoring.” These two chapters document the statistics of interest to monitor and suggest clues as to what values of a given statistic should be cause for further investigation. Then based on what the monitored statistics indicate as some- thing worthy of further investigation, you can turn to other chapters for suggestions on corrective actions. For example, if monitoring operating system statistics suggests high sys CPU utilization, you should profile the application to determine what methods are consuming the highest sys CPU cycles. Instructions on how to use the NetBeans ptg6882136 Two Approaches, Top Down and Bottom Up 7 Profiler and Oracle Solaris Studio Performance Analyzer (formerly known as Sun Studio Performance Analyzer) can be found in Chapter 5, “Java Application Profiling,” and Chapter 6, “Java Application Profiling Tips and Tricks.” If the monitoring activity and monitored statistic suggests the JVM’s garbage collectors require tuning, turn to Chapter 7, “Tuning the JVM, Step by Step.” If you are familiar with the general operation and basic workings of the Java HotSpot VM’s garbage collectors, consider reading the section on garbage collectors in Chapter 3, “JVM Overview,” before reading the chapter on tuning the JVM. If you are monitoring application level statistics, such as those provided by a Java EE container, read the chapters on Java EE performance tuning: Chapter 10, “Web Application Performance”; Chapter 11, “Web Services Perfor- mance”; and Chapter 12, “Java Persistence and Enterprise Java Beans Performance,” to learn how to resolve performance issues in an enterprise application. Bottom Up Approach The bottom up approach is most commonly used by performance specialists when wanting to improve the performance of an application on one platform relative to another where differences exists in the underlying CPU, CPU architecture, or number of CPUs. The bottom up approach is also often used when wanting to improve the per- formance of an application when it is migrated to support a different operating system. This approach is also frequently used when it is not possible to make a change to the application’s source code such as when an application is currently deployed in produc- tion environments or in competitive situations where computer systems vendors are vying for the business opportunity to run an application at peak performance. In the bottom up approach, the gathering of performance statistics and the moni- toring activity begin at the lowest level, the CPU. Statistics that are monitored at the CPU level may include the number of CPU instructions required to execute a given workload on the CPU, often referred to as path length and the number of CPU cache misses that occur while running the application under load. Other CPU statistics may be of interest, but the number of CPU instructions and the number CPU caches misses tend to be the most commonly observed statistics in a bottom up approach. If an application can perform and scale well under load by executing a fewer number of CPU instructions it will likely execute the application faster. Reducing CPU cache misses also improves an application’s performance since a CPU cache miss results in wasted CPU cycles waiting for requested data to be fetched from memory. By reduc- ing CPU cache misses, the application performs better since the CPU spends less time waiting for data to be fetched from memory. The focus of the bottom up approach is usually to improve the utilization of the CPU without making changes to the application. In cases where the application can be modified, the bottom up approach may result in making changes to the applica- tion. These modifications may include a change to the application source code such as ptg6882136 8 Chapter 1 Strategies, Approaches, and Methodologies moving frequently accessed data near each other so they can be accessed on the same CPU cache line and thus not having to wait to fetch the data from memory. Such a change could reduce CPU cache misses and thereby reduce the amount of time the CPU waits for data to be fetched from memory. In the context of a Java application executing in a modern Java Virtual Machine that has a sophisticated JIT compiler there may be cause to implement optimiza- tions that would, for example, emit more efficient generated machine code based on memory access patterns exhibited by the application or the specific code paths taken by the application. There may also be settings at the operating system level that may be tuned or modified to allow for improved performance such as chang- ing a CPU scheduling algorithm or the amount time the operating system waits before it migrates an executing application thread to a different CPU hardware thread. If you find yourself in a situation where a bottom up approach would be useful, you should begin by collecting operating system statistics and JVM statistics. Moni- toring these statistics provides hints as to where to focus in the next step. Chapter 2 and Chapter 4 provide information as to what statistics to monitor. From there you decide whether it makes sense to profile the application and the JVM. To profile both the application and the JVM, use a profiling tool that can provide that information. The Oracle Solaris Studio Performance Analyzer tool does this for Oracle Solaris SPARC, Oracle Solaris x86/x64, and Linux x86/x64 operating systems. Other popular tools such as Intel VTune or AMD’s CodeAnalyst Performance Analyzer can provide similar information on Windows and Linux. All three tools also have the capability to collect specific CPU counter information such as the number of CPU instruc- tions executed and CPU cache misses along with being able to associate them with specific methods or functions in a Java application of Java Virtual Machine. Using a profiler with these capabilities is essential in a bottom up approach. You can find additional information on how to use the Oracle Solaris Studio Performance Analyzer in C hapter 5 and Chapter 6. Choosing the Right Platform and Evaluating a System At times a performance specialist is called upon to improve the performance of an application only to find that the application is being run on an inappropriate CPU architecture or system. CPU architectures and systems have evolved substantially with the introduction of multiple cores per CPU and multiple hardware threads per core (also known as CMT, chip multithreading). As a result, choosing the right platform and CPU architecture for a given application has become more impor- tant. In addition, the way in which the performance of a system is evaluated must also be updated or revised as a result of the evolution of CPU architectures. This ptg6882136 Choosing the Right Platform and Evaluating a System 9 section looks at some of the differences in CPU architectures available on modern systems and presents some considerations to keep in mind when choosing a system on which to run an application. This section also describes why traditional methods of evaluating a system’s performance are invalid when it comes to modern multiple hardware thread per core CPU architectures such as the SPARC T-series family of processors. Choosing the Right CPU Architecture The introduction of the SPARC T-series processor brought chip multiprocessing and chip multithreading to Oracle’s offering of processors. One of the major design points behind the SPARC T-series processors is to address CPU cache misses by introducing multiple hardware threads per core. The first generation SPARC T-series, UltraSPARC T1, has four hardware threads per core and comes in four, six, or eight cores per CPU. An UltraSPARC T1 processor with eight cores looks like a 32-processor system from an operating system viewpoint. That is, the operating system views each of the four hardware threads per core as a processor. Hence, a system configured with an UltraSPARC T1 having eight cores would be seen as having 32 processors from an operating system. An important distinction between an UltraSPARC T1 is it has four hardware threads per core. Of the four hardware threads per core, only one of the four threads per core executes on a given clock cycle. However, when a long latency event occurs, such as a CPU cache miss, if there is another runnable hardware thread in the same UltraSPARC T1 core, that hardware thread executes on the next clock cycle. In contrast, other modern CPUs with a single hardware thread per core, or even hyperthreaded cores, will block on long latency events such as CPU cache misses and may waste clock cycles while waiting for a long latency event to be satisfied. In other modern CPUs, if another runnable application thread is ready to run and no other hardware threads are available, a thread context switch must occur before another runnable application thread can execute. Thread context switches generally take hundreds of clock cycles to complete. Hence, on a highly threaded application with many threads ready to execute, the SPARC T-series processors have the capability to execute the application faster as a result of their capability to switch to another runnable thread within a core on the next clock cycle. The capability to have multiple hardware threads per core and switch to a different runnable hardware thread in the same core on the next clock cycle comes at the expense of a CPU with a slower clock rate. In other words, CPUs such as the SPARC T-series processor that have multiple hardware threads tend to execute at a slower clock rate than other modern CPUs that have a single hardware thread per core or do not offer the capability to switch to another runnable hardware thread on a subsequent clock cycle. ptg6882136 10 Chapter 1 Strategies, Approaches, and Methodologies When it comes to choosing a computing system, if the target application is expected to have a large number of simultaneous application threads executing concurrently, it is likely this type of application will perform and scale better on a SPARC T-series processor than a smaller number of hardware threads per core type of processor. In contrast, an application that is expected to have a small number of application threads, especially if the number of simultaneous application threads is expected to be less than the total number of hardware threads on a SPARC T-series processor, this application will likely perform better on a higher clock rate, smaller number of hardware threads per core type of processor than a slower clock rate SPARC T-series processor. In short, for a SPARC T-series processor to perform well, it needs a large number of simultaneous application threads to keep the larger number of hardware threads busy to leverage its capability to switch to a different hardware thread on subsequent clock cycles when events such as CPU cache misses occur. In the absence of a large number of simultaneous application threads, the SPARC T-series gener- ally performs like slower clock rate traditional processors. The artifact of requiring a large number of simultaneous application threads to keep the many SPARC T-series hardware threads busy also suggests the traditional manner in which a system’s performance is qualified may not represent a system’s true performance. This is the topic of the next subsection. Evaluating a System’s Performance To evaluate the performance of a SPARC T-series, since it has the capability to switch to another runnable hardware thread within a core on the next clock cycle, it must be loaded with a workload having a large number of simultaneous application threads. A common approach used to qualify or evaluate the performance of a new system has been to place a portion of the expected target load on the system, or execute one or more micro-benchmarks and observe how the system performs or observe the amount of work the application does per some unit of time. However, to evaluate the performance of a SPARC T-series processor, it must be loaded with enough concurrent application threads to keep the large number of hardware threads busy. The work- load needs to be large enough for the SPARC T-series to reap the benefit of switching Tip Sun Microsystems evolved the first generation SPARC T-series processor from the UltraSPARC T1 to the UltraSPARC T2 and T3 by adding additional hardware threads per core and the capability for multiple hardware threads per core to execute in a clock cycle. However, for the purposes of this discussion, it is easier to talk about and understand how the UltraSPARC T1 processor differs from other modern CPUs. Once the difference in CPU architecture is understood, it becomes easier to extend the design points behind the UltraSPARC T1 to the UltraSPARC T2 and T3 processors. ptg6882136 Bibliography 11 to a different runnable thread on the next clock cycle when long latency events such as CPU cache misses occur. Blocking and waiting for a CPU cache miss to be satisfied takes many CPU cycles, on the order of hundreds of clock cycles. Therefore, to take full advantage of a SPARC T-series processor, the system needs to be loaded with enough concurrent work to where its design point of switching to another runnable hardware thread on the next clock cycle can be realized. In situations where a subset of a targeted workload is executed by a SPARC T-series processor, it may appear as though the system does not perform very well since all its hardware threads may not be busy. Remember that one of the major design points for the SPARC T-series processors is to address long latency CPU events by allowing other runnable hardware threads to execute on the next clock cycle. In a single hardware thread per core family of processors, long latency events such as a CPU cache miss mean many CPU clock cycles are wasted waiting for data to be fetched from memory. To switch to another runnable application thread, that other runnable application thread and its state information must replace the exist- ing thread and its state information. This not only requires clock cycles to make this context switch, it may also require the CPU cache to fetch different state information for the new runnable application thread. Hence, when evaluating the performance of a SPARC T-series processor it is impor- tant to put enough load on the system to take advantage of the additional hardware threads and its capability to switch to another runnable hardware thread within the same CPU core on the next clock cycle. Bibliography Dagastine, David, and Brian Doherty. Java Platform Performance presentation. Java- One Conference, San Francisco, CA, 2005. Wilson, Steve, and Jeff Kesselman. Java Platform Performance: Strategies and Tac- tics, Addison-Wesley, Reading, MA, 2000. ISBN 0-201-70969-4. ptg6882136 This page intentionally left blank ptg6882136 13 2 Operating System Performance Monitoring Knowing when an application is not performing as desired or expected is important to an application’s capability to meet service level agreement(s) set forth by the appli- cation’s stakeholders. Hence, knowing what to monitor, where in the software stack to monitor, and what tools to use are critical. This chapter describes what should be monitored at the operating system level and presents operating system tools that can be used to observe an application’s performance. Additionally, general guidelines are given to help identify potential performance issues. The operating systems covered in this chapter are Windows, Linux, and Oracle Solaris, also referred to as Solaris hereafter. The monitoring tools presented are not intended to be an exhaustive list of tools or the only means to monitor an application’s or a system’s performance. Rather, the principles of why and what attributes of a system are important to monitor is the intention. Readers who are running a Java application on an operating system other than those covered should be able to identify the operating system performance statistics to monitor and be able to identify appropriate monitoring tools. Tip The first step in isolating a performance issue is to monitor the application’s behavior. Monitoring offers clues as to the type or general category of performance issue. This chapter begins by presenting definitions for performance monitoring, per- formance profiling, and performance tuning. Then sections that describe operating system statistics of interest to monitor are presented. Both command line and GUI ptg6882136 14 Chapter 2 Operating System Performance Monitoring tools that can be used to monitor the performance statistics are included. In addi- tion, guidelines are offered as to what performance statistic values are indicators of potential root causes, or a next step to take in your performance analysis. Definitions Three distinct activities are involved when engaging in performance improvement activities: performance monitoring, performance profiling, and performance tuning. Performance monitoring is an act of nonintrusively collecting or observing per- formance data from an operating or running application. Monitoring is usu- ally a preventative or proactive type of action and is usually performed in a production environment, qualification environment, or development environ- ment. Monitoring is also usually the first step in a reactive situation where an application stakeholder has reported a performance issue but has not provided sufficient information or clues as to a potential root cause. In this situation, performance profiling likely follows performance monitoring. Performance profiling in contrast to performance monitoring is an act of col- lecting performance data from an operating or running application that may be intrusive on application responsiveness or throughput. Performance profiling tends to be a reactive type of activity, or an activity in response to a stakeholder reporting a performance issue, and usually has a more narrow focus than per- formance monitoring. Profiling is rarely done in production environments. It is typically done in qualification, testing, or development environments and is often an act that follows a monitoring activity that indicates some kind of performance issue. Performance tuning, in contrast to performance monitoring and perfor- mance profiling, is an act of changing tune-ables, source code, or configura- tion attribute(s) for the purposes of improving application responsiveness or throughput. Performance tuning often follows performance monitoring or performance profiling activities. CPU Utilization For an application to reach its highest performance or scalability it needs to not only take full advantage of the CPU cycles available to it but also to utilize them in a manner that is not wasteful. Being able to make efficient use of CPU cycles can be challenging for multithreaded applications running on multiprocessor and multicore ptg6882136 CPU Utilization 15 systems. Additionally, it is important to note that an application that can saturate CPU resources does not necessarily imply it has reached its maximum performance or scalability. To identify how an application is utilizing CPU cycles, you monitor CPU utilization at the operating system level. CPU utilization on most operating systems is reported in both user CPU utilization and kernel or system (sys) CPU utilization. User CPU utilization is the percent of time the application spends in application code. In contrast, kernel or system CPU utilization is the percent of time the applica- tion spends executing operating system kernel code on behalf of the application. High kernel or system CPU utilization can be an indication of shared resource contention or a large number of interactions between I/O devices. The ideal situation for maxi- mum application performance and scalability is to have 0% kernel or system CPU utilization since CPU cycles spent executing in operating system kernel code are CPU cycles that could be utilized by application code. Hence, one of the objectives to achieving maximum application performance and scalability is to reduce kernel or system CPU utilization as much as possible. For applications that are compute intensive, performance monitoring may go much deeper than observing user CPU utilization and kernel or system utilization. On compute-intensive systems, further monitoring of the number of CPU instructions per CPU clock cycle (also known as IPC, instructions per clock) or the number of CPU clock cycles per CPU instruction (also known as CPI, cycles per instruction) may be required. These two additional metrics are of interest to compute intensive appli- cations because CPU utilization monitoring tools bundled with modern operating systems report CPU utilization and do not report the percentage of CPU clock cycles the CPU has been executing instructions. This means that the operating system tools report a CPU as being utilized even though the CPU may be waiting for data to be fetched from memory. This scenario is commonly referred to as a stall. Stalls occur any time the CPU executes an instruction and the data being operated on by the instruction is not readily available in a CPU register or cache. When this occurs, the CPU wastes clock cycles because it must wait for the data to be loaded from memory into a CPU register before the CPU instruction can execute on it. It is common for a CPU to wait (waste) several hundred clock cycles during a stall. Thus the strategy for increasing the performance of a compute intensive application is to reduce the number of stalls or improve the CPU’s cache utilization so fewer CPU clock cycles are wasted waiting for data to be fetched from memory. Performance monitoring activi- ties of this kind go beyond the scope of this book and may require the assistance of a performance expert. However, the profiler covered in Chapter 5, “Java Application Profiling,” Oracle Solaris Studio Performance Analyzer, has the capability to capture a profile of a Java application including this kind of data. Each operating system presents user CPU utilization and kernel or system CPU utilization differently. The next several sections describe tools to monitor CPU utili- zation on Microsoft Windows, Linux, and Solaris operating systems. ptg6882136 16 Chapter 2 Operating System Performance Monitoring Monitoring CPU Utilization on Windows The commonly used CPU utilization monitoring tool on Windows is Task Manager and Performance Monitor. Both Task Manager and Performance Monitor use a color- ing scheme to distinguish between user CPU and kernel or system CPU utilization. Figure 2-1 shows the Windows Task Manager performance monitoring window. CPU utilization is shown in the upper half of the Windows Task Manager. CPU utilization across all processors is shown in the CPU Usage panel on the upper left. A running history of CPU utilization for each processor is displayed in the CPU Usage History panel on the upper right. The upper line, a green colored line, indicates the combined user and system or kernel CPU utilization. The lower line, a red colored line, indicates the percentage of system or kernel CPU usage. The space between the lower line and upper line represents the percentage of user CPU utilization. Note that to view system or kernel CPU utilization in Window’s Task Manager, the Show Kernel Utilization option must be enabled in the View 7 Show Kernel Utilization menu. On Windows systems that include the Performance Monitor (perfmon), the default view of the Performance Monitor varies depending on the Windows operating system. Figure 2-1 Windows Task Manager. The graph lines in the two CPU usage history windows shows both user and kernel/system CPU utilization ptg6882136 CPU Utilization 17 This chapter describes the Performance Monitor view in Windows 7. Note that to run the Windows Performance Monitor you must have membership in either the Administrators, Performance Log Users, or equivalent group. The Windows Performance Monitor uses a concept of performance objects. Per- formance objects are categorized into areas such as network, memory, processor, thread, process, network interface, logical disk, and many others. Within each of these categories are specific performance attributes, or counters, that can be selected as performance statistics to monitor. Covering all the performance counters available to monitor is beyond the scope of this chapter. The focus in this chapter is to identify the performance statistics of most interest to monitor and the tools to monitor them. User CPU utilization and kernel or system CPU utilization can be added to the Performance Monitor by right-clicking in the Performance Monitor’s display area and selecting the Add Counters option from the context sensitive menu. User and kernel or system CPU utilization can be monitored by selecting the Processor performance object, and then selecting both % User Time and % Privileged Time counters and click- ing the Add button. Windows uses the term “Privileged Time” to represent kernel or system CPU utilization. See Figure 2-2 for an example of the Add Counters screen. Figure 2-2 Performance Monitor’s user time and privileged time ptg6882136 18 Chapter 2 Operating System Performance Monitoring Figure 2-3 Monitoring CPU utilization. The upper line represents % processor time. The middle line is % user time. The bottom line is % privileged time The Performance Monitor display is updated with the new counters after they have been added. At the bottom of the Performance Monitor, you can see the coun- ters that are currently being monitored (see Figure 2-3). Right-clicking on the list of performance counters allows you to change the performance counters’ properties. For example, you can change the color associated with a performance counter. This is useful when the performance counters you have selected to monitor use the same default color. You can also add and remove performance counters from the same context sensitive menu. By default the Performance Monitor uses a scrolling style window to show the last 60 seconds of performance statistics. The scrolling part of the window is identified by a vertical bar. The values to the immediate left of the vertical bar are the most recent performance statistics, see Figure 2-3. You can choose a different type of data presentation by selecting the Properties option from the context sensitive menu in the Performance Monitor and clicking the Graph tab. In Figure 2-3, the upper line is the % user processor time, the total of % user time, and % privileged time. In this example, the monitored application has higher % user ptg6882136 CPU Utilization 19 time than % privileged time. That relationship is a desired relationship to observe. In other words, it is desirable for an application to spend more time executing applica- tion code than executing in operating system kernel code. Many additional capabilities in the Performance Monitor can be leveraged such as the ability to create a Data Collector Set and generate performance reports. Creat- ing Data Collector Sets, generating performance reports, and other capabilities are beyond the scope of this chapter but may be of interest to further explore as part of your performance monitoring efforts. Monitoring CPU Utilization with Windows typeperf Windows typeperf is a command line tool that can be used to collect operating sys- tem performance statistics. typeperf can be run in a Windows Command Prompt window, or it can be scripted and run from a bat or cmd file. You specify the perfor- mance statistics you want to collect using the Microsoft performance counter names. The Microsoft performance counter names are the same as those used in the Perfor- mance Monitor. For example, to collect user and kernel or system CPU utilization you specify the User Time and Privileged Time performance counters. In a Command Prompt window, or in a cmd file, the command looks like typeperf "\Processor(_Total)\% Privileged Time" "\Processor(_Total)\% User Time" Each performance counter should be enclosed in quotation marks, and the syntax of the performance counter follows the name as you would find it in the Performance Monitor. You can also assemble a list of performance counters in a file and pass the name of the file to the typeperf command. For example, you can enter the following performance counters in a file named cpu-util.txt: \Processor(_Total)\% Privileged Time \Processor(_Total)\% User Time Then, invoke the typeperf command with the option -cf followed by the file name. typeperf -cf cpu-util.txt The following output shows the result of executing the typeperf command using three performance counters to capture the total, kernel, or system and user CPU utilization. ptg6882136 20 Chapter 2 Operating System Performance Monitoring In the preceding output, the first row is a header describing the data to be col- lected. That is followed by rows of reported data. In each row, there is a date and time stamp indicating when the data was collected along with the values of the performance counters. By default, the typeperf reporting interval is one second. The reporting interval can be changed using the -si option. The -si option accepts a form of [mm:]ss where mm: is optional minutes and ss is the number of seconds. You may consider specifying a larger interval than the default if you intend to moni- tor over an extended period of time to reduce the amount of data you need to process. Additional details on the typeperf command and its options can be found at http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ nt_command_typeperf.mspx?mfr=true. Monitoring CPU Utilization on Linux On Linux, CPU utilization can be monitored graphically with the GNOME System Monitor tool, which is launched with the gnome-system-monitor command. The GNOME System Monitor tool displays CPU utilization is the upper portion of the display of the Resource tab, as shown in Figure 2-4. The GNOME System Monitor shown in Figure 2-4 is running on a system with two virtual processors. The number of virtual processors matches the number returned by the Java API Runtime.availableProcessors(). A system with a single CPU socket with a quad core processor with hyperthreading disabled will show four CPUs in the GNOME System Monitor and report four virtual processors using the Java API Runtime.availableProcessors(). In the GNOME System Monitor, there is a CPU history area where a line for each virtual processor’s CPU utilization is drawn illustrating its CPU utilization over a period of time. The GNOME System Monitor also displays the current CPU utilization for each virtual processor found on the system in a table below the CPU history. typeperf "\Processor(_Total)\% User Time" "\Processor(_Total)% Privileged Time" "\Processor(_Total)% Processor Time” "(PDH-CSV 4.0)","\\PICCOLO\Processor(_Total)% User Time","\\PICCOLO\Processor(_Total)% Privileged Time","\\PICCOLO\Processor(_Total)% Processor Time" "02/15/2011 11:33:54.079","77.343750","21.875000","99.218750" "02/15/2011 11:33:55.079","75.000000","21.875000","96.875000" "02/15/2011 11:33:56.079","58.593750","21.875000","80.468750" "02/15/2011 11:33:57.079","62.500000","21.093750","83.593750" "02/15/2011 11:33:58.079","64.062500","15.625000","79.687500" ptg6882136 CPU Utilization 21 Figure 2-4 GNOME System Monitor on Linux Another popular graphical tool to monitor CPU utilization on Linux is xosview. Some Linux distributions may not include xosview in their default distribution. But a search of their distribution’s software package management facility for xosview will likely find it. One of the additional features of xosview in CPU utilization is further broken down into user CPU, kernel or system CPU, and idle CPU. Monitoring CPU Utilization on Solaris On Solaris, CPU utilization can be monitored graphically with the GNOME System Monitor tool. It is launched with the gnome-system-monitor command. An exam- ple of the GNOME System Monitor monitoring a system with 32 virtual processors is shown in Figure 2-5. Another way to graphically observe CPU utilization on Solaris is using an optional tool called cpubar found on the Solaris Performance Tools CD 3.0 (also download- able at http://www.schneider4me.de/ToolsCD-v3.0.iso.zip). In addition to monitor- ing CPU utilization, other system attributes can be monitored with cpubar such as kernel thread queue depths, memory paging, and memory scan rate. Figure 2-6 shows cpubar. ptg6882136 22 Figure 2-5 GNOME System Monitor on Solaris Figure 2-6 Solaris cpubar uses color to indicate system status. In 0 bar, 1 bar, and avg bar, green represents user CPU utilization, red represents system or kernel CPU utilization, and a blue color is idle. For the r bar, b bar, and w bar, red indicates occupancy, and blue represents emptiness. For the p/s bar, red represents activity; blue represents idle. For the ram bar, red represents the amount of memory committed, yellow represents allocated, and blue represents free/available memory. The sr bar is similar to the p/s bar: red indicates activity; blue represents idle. In the vm bar, red represents committed virtual memory, green represents allocated memory, and blue represents free/available virtual memory. ptg6882136 CPU Utilization 23 On multicore and multiprocessor systems the bar with the avg label shows the overall CPU utilization. To the left of the overall CPU utilization bar there is an indi- vidual bar for each virtual processor’s CPU utilization. The combined colors of green and red show overall CPU utilization. The blue color indicates idle CPU utilization. The green color shows the percentage of user CPU utilization, and a red color shows the percentage of system or kernel CPU utilization. The hyphenated/dashed horizon- tal bar embedded within the CPU utilization bars represents a running historical average CPU utilization since the system was last booted. Also shown in Solaris cpubar are additional performance statistics such as kernel thread queue depths, memory paging, amount of memory utilized, page scan rate, and amount of memory utilized by the Solaris VM. The kernel threads’ queue depths are found to the right of the CPU utilization bars and have an “r” label, “b” label, and “w” label. Each of the vertical bars above those three labels represents a queue depth. The vertical bar for the “r” label represents the run queue depth. Entries show up in the run queue when there are kernel threads that are ready to run but do not have an available processor to run. In Figure 2-6, the vertical bar above the “r” label indicates there are two kernel threads ready to run and waiting for a CPU to execute. Monitoring the kernel thread’s run queue is an important statistic to monitor. How you monitor the kernel thread’s run queue is presented in the “CPU Scheduler Run Queue” section later in this chapter. The vertical bar for the “b” label represents the blocked queue. Entries show up in the blocked queue when kernel threads are wait- ing on resources such as I/O, memory pages, and so on. The vertical bar for the “w” label represents the waiting queue. Entries show up in the waiting queue when a swapped out lightweight process is waiting for resources to finish. The number above the three kernel thread queue vertical bars is the number of kernel threads currently running on the system. In Figure 2-6, 93 kernel threads were running at the time the screenshot was taken. To the right of the kernel thread queue depths is a vertical bar illustrating the page in/page out activity, that is, the number of memory pages paged in or paged out. This vertical bar has the “p/s” label below it. In Figure 2-6, there was little paging activity at the time the screenshot was taken. Monitoring paging activity is covered in the “Memory Utilization” section later in this chapter. To the right of the memory paging activity (p/s) vertical bar is a vertical bar illustrating the amount of physical RAM currently being utilized by the system. This vertical bar has the “ram” label below it. A red color shows the amount of memory utilized by the kernel. A yellow color shows the amount of memory utilized by user processes, and blue is the amount of free or available memory. Figure 2-6 shows there was little free memory available at the time the screenshot was taken. To the right of the physical memory utilization (ram) vertical bar is a vertical bar illustrating the page scanning rate. This vertical bar has an “sr” label below it. As the amount of free physical memory reduces, the system attempts to free up memory ptg6882136 24 Chapter 2 Operating System Performance Monitoring by locating pages that have not been used in a long time. It then pages these out to disk. This page scanning activity is reported as scan rate. A high scan rate is an indicator of low physical memory. Monitoring the page scan rate is essential to iden- tifying when a system is swapping. This is presented in more detail in the “Memory Utilization” section of this chapter. To the right of the page scanning (sr) vertical bar is a bar representing vir- tual memory usage, or swap usage. This vertical bar has the “vm” label below it. The amount of virtual memory used is colored red. The amount of virtual memory reserved is colored yellow, and the amount of free memory is colored blue. The total amount of virtual memory is displayed at the top of the vertical bar. Figure 2-6 shows 1.33 gigabytes of virtual memory on the system. Monitoring CPU Utilization on Linux and Solaris with Command Line Tools Linux and Solaris also provide command line tools to monitor CPU utilization. These command line tools are useful when you want to keep a running textual history of CPU utilization or keep a log of CPU utilization. Linux and Solaris have vmstat, which shows combined CPU utilization across all virtual processors. Both versions of vmstat optionally take a reporting interval, in seconds, as a command line argu- ment. If no reporting interval is given to vmstat, the reported output is a summary of all CPU utilization data collected since the system has last been booted. When a reporting interval is specified, the first row of statistics is a summary of all data col- lected since the system was last booted. As a result, the first row of data from vmstat is most often ignored. The display format of vmstat for Linux and Solaris is similar. For example, the following shows vmstat from Linux. The columns of interest for monitoring CPU utilization are shown in bold. procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 4 0 0 959476 340784 1387176 0 0 0 0 1030 8977 63 35 1 0 3 0 0 959444 340784 1387176 0 0 0 0 1014 7981 62 36 2 0 6 0 0 959460 340784 1387176 0 0 0 16 1019 9380 63 36 1 0 1 0 0 958820 340784 1387176 0 0 0 0 1036 9157 63 35 2 0 4 0 0 958500 340784 1387176 0 0 0 29 1012 8582 62 37 1 0 The “us” column shows the percentage of user CPU utilization. The “sy” column shows the percentage of kernel or system CPU utilization. The “id” column shows the percentage of idle or available CPU. The sum of the “us” column and “sy” col- umn should be equal to 100 minus the value in the “id” column, that is, 100 – (“id” column value). ptg6882136 CPU Utilization 25 The vmstat output from Solaris, shown in the following example, has three col- umns of CPU utilization interest and has column headings of “us,” “sy,” and “id” that show user, kernel or system, and idle CPU utilization, respectively. kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr f0 s0 s1 s2 in sy cs us sy id 0 0 0 672604 141500 10 40 36 6 10 0 20 0 3 0 2 425 1043 491 4 3 93 1 1 0 888460 632992 7 32 97 0 0 0 0 0 21 0 12 462 1099 429 32 19 49 0 1 0 887848 631772 4 35 128 0 0 0 0 0 30 0 13 325 575 314 38 13 49 0 1 0 887592 630844 6 26 79 0 0 0 0 0 40 0 11 324 501 287 36 10 54 1 0 0 887304 630160 5 33 112 0 0 0 0 0 50 0 16 369 899 367 37 11 52 0 1 0 886920 629092 4 30 101 0 0 0 0 0 26 0 18 354 707 260 39 14 46 Solaris and Linux also offer a tabular view of CPU utilization for each virtual processor using the command line tool mpstat. Tip Most Linux distributions require an installation of the sysstat package to use mpstat. Using mpstat to observe per virtual processor CPU utilization can be useful in identifying whether an application has threads that tend to consume larger percent- ages of CPU cycles than other threads or whether application threads tend to utilize the same percentage of CPU cycles. The latter observed behavior usually suggests an application that may scale better. CPU utilization in Solaris mpstat, as shown in the following example, is reported in the columns “usr,” “sys,” “wt,” and “idl,” where usr is the percentage of CPU time spent executing user code, sys is the percentage of CPU time spent executing kernel code, wt is the percentage of I/O wait time (no lon- ger calculated and always reports 0), and idl is percentage of time the CPU was idle. CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 28 2 0 192 83 92 32 14 2 0 185 78 15 0 7 1 49 1 0 37 1 80 28 16 2 0 139 80 16 0 4 2 28 1 0 20 7 94 34 17 1 0 283 83 12 0 5 3 39 1 2 52 1 99 36 16 3 0 219 74 19 0 7 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 34 0 2 171 75 78 32 12 1 0 173 90 9 0 2 1 38 1 0 39 1 84 29 13 2 0 153 66 12 0 23 2 28 8 0 21 9 97 31 20 2 0 167 67 13 0 20 3 35 3 1 43 1 98 29 20 3 0 190 52 25 0 23 If no reporting interval is given to mpstat, the reported output is a summary of all mpstat data collected since the system was last booted. When a reporting interval ptg6882136 26 Chapter 2 Operating System Performance Monitoring is given, the first row of statistics is a summary of all data collected since the system was last booted. Other popular alternatives to vmstat on Solaris and Linux can be used to moni- tor CPU utilization. A couple of the more common ones are prstat for Solaris and top for Linux. Linux top reports not only CPU utilization but also process statistics and memory utilization. Its display, shown in the following example, has two major parts. The upper section of the display reports overall system statistics, while the lower section reports process level statistics that, by default, are ordered in highest to lowest CPU utilization. top - 14:43:56 up 194 days, 2:53, 4 users, load average: 8.96, 6.23, 3.96 Tasks: 127 total, 2 running, 125 sleeping, 0 stopped, 0 zombie Cpu(s): 62.1% us, 26.2% sy, 0.8% ni, 1.7% id, 0.0% wa, 0.0% hi, 9.1% si Mem: 4090648k total, 3141940k used, 948708k free, 340816k buffers Swap: 4192956k total, 0k used, 4192956k free, 1387144k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30156 root 25 10 32168 18m 10m R 2.3 0.5 20:41.96 rhn-applet-gui 30072 root 15 0 16344 12m 2964 S 0.7 0.3 13:08.52 Xvnc 5830 huntch 16 0 3652 1084 840 R 0.7 0.0 0:00.16 top 1 root 16 0 3516 560 480 S 0.0 0.0 0:01.62 init 2 root RT 0 0 0 0 S 0.0 0.0 0:07.38 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.27 ksoftirqd/0 4 root RT 0 0 0 0 S 0.0 0.0 0:08.03 migration/1 ... Solaris prstat shows similar information to Linux top. The following example is the default output for prstat. PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 1807 huntch 356M 269M cpu1 45 0 0:00:37 46% java/40 1254 huntch 375M 161M run 29 0 0:06:51 2.9% firefox-bin/13 987 huntch 151M 123M sleep 59 0 0:06:25 2.7% Xorg/1 1234 huntch 257M 132M sleep 49 0 0:03:52 0.5% soffice.bin/7 ... Solaris prstat does not show an overall system summary section like top. But, like top, it does report per process level statistics that are ordered, by default, from highest to lowest CPU utilization. Both prstat and top are good tools for providing a high level view of CPU utili- zation at a per process level. But as the need arises to focus more on per process and per lightweight process CPU utilization, Solaris prstat has additional capabilities such as reporting both user and kernel or system CPU utilization along with other ptg6882136 CPU Utilization 27 microstate information using the prstat -m and -L options. The -m option prints microstate information, and -L prints statistics on per lightweight process. Using the -m and -L options can be useful when you want to isolate CPU utili- zation per lightweight process and Java thread. A Java process showing high CPU utilization with prstat -mL can be mapped to a Java process and Java thread(s) on Solaris through a sequence of steps using prstat, pstack, and Java 6’s jstack command line tool. The following example illustrates how to do this. The output in the following example, gathered with prstat -mL 5, shows process id 3897 has three lightweight process ids consuming about 5% of kernel or system CPU. LWPID 2 is consuming the most at 5.7%. PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROC/LWPID 3897 huntch 6.0 5.7 0.1 0.0 0.0 2.6 8.2 78 9K 8K 64K 0 java/2 3897 huntch 4.9 4.8 0.0 0.0 0.0 59 0.0 31 6K 6K 76K 0 java/13 3897 huntch 4.7 4.6 0.0 0.0 0.0 56 0.0 35 5K 6K 72K 0 java/14 3917 huntch 7.4 1.5 0.0 0.0 0.0 3.8 53 34 5K 887 16K 0 java/28 ... In the absence of using a profiler, which is covered in detail in Chapter 5, there is a quick way to isolate which Java thread, along with which Java method, is consuming large amounts of CPU as reported by prstat, either USR or SYS. A Java, or JVM, process thread stack dump at the Solaris level can be generated using the Solaris command line tool pstack and the process id 3897. The pstack output in the follow- ing example, produced using the command pstack 3897/2, shows the lightweight process (lwp) id and thread id that matches LWPID 2 from prstat. ----------------- lwp# 2 / thread# 2 -------------------- fef085c7 _lwp_cond_signal (81f4200) + 7 feb45f04 __1cNObjectMonitorKExitEpilog6MpnGThread_pnMObjectWaiter__v_ (829f2d4, 806f800, e990d710) + 64 fe6e7e26 __1cNObjectMonitorEexit6MpnGThread__v_ (829f2d4, 806f800) + 4fe fe6cabcb __1cSObjectSynchronizerJfast_exit6FpnHoopDesc_pnJBasicLock_ pnGThread__v_ (ee802108, fe45bb10, 806f800) + 6b If you convert the thread id value to hexadecimal and use the JDK’s jstack command you can find the Java thread that corresponds to Solaris thread# 2 by searching for a “nid” label. The thread number, 2 in decimal, is also 2 in hexadeci- mal. The following output from the JDK’s jstack command is trimmed but shows that a Java thread with a 0x2 is the “main” Java thread. According to the stack trace produced by jstack, the Java thread corresponding to Solaris pstack’s LWPID 2 and prstat’s LWPID 2 is executing a Java NIO Selector.select() method. ptg6882136 28 Chapter 2 Operating System Performance Monitoring Once a Java thread has been identified and with the stack trace readily available, you can begin to investigate in more detail the methods shown in the stack trace for possible candidates of high kernel or system CPU utilization through a more thor- ough profiling activity. CPU Scheduler Run Queue In addition to CPU utilization, monitoring the CPU scheduler’s run queue is impor- tant to tell if the system is being saturated with work. The run queue is where light- weight processes are held that are ready to run but are waiting for a CPU where it can execute. When there are more lightweight processes ready to execute than the system’s processors can handle, the run queue builds up with entries. A high run queue depth can be an indication a system is saturated with work. A system operat- ing at a run queue depth equal to the number of virtual processors may not experi- ence much user visible performance degradation. The number of virtual processors is the number of hardware threads on the system. It is also the value returned by the Java API, Runtime.availableProcessors(). In the event the run queue depth reaches four times the number of virtual processors or greater, the system will have observable sluggish responsiveness. A general guideline to follow is observing run queue depths over an extended period of time greater than 1 times the number of virtual processors is something to be concerned about but may not require urgent action. Run queue depths at 3 to 4 times, or greater, than the number of virtual processors over an extended time period should be considered an observation that requires immediate attention or action. There are generally two alternative resolutions to observing high run queue depth. One is to acquire additional CPUs and spread the load across those additional CPUs, or reduce the amount of load put on the processors available. This approach essen- tially reduces the number of active threads per virtual processor and as a result fewer lightweight processes build up in the run queue. ”main” prio=3 tid=0x0806f800 nid=0x2 runnable [0xfe45b000..0xfe45bd38] java.lang.Thread.State: RUNNABLE at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0xee809778> (a sun.nio.ch.Util$1) - locked <0xee809768> (a java.util.Collections$UnmodifiableSet) - locked <0xee802440> (a sun.nio.ch.DevPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at com.sun.grizzly.SelectorThread.doSelect(SelectorThread.java:1276) ptg6882136 CPU Scheduler Run Queue 29 The other alternative is to profile the applications being run on the system and improve the CPU utilization of those applications. In other words, explore alterna- tive approaches that will result in fewer CPU cycles necessary to run the applica- tion such as reducing garbage collection frequency or alternative algorithms that result in fewer CPU instructions to execute the same work. Performance experts often refer to this latter alternative as reducing code path length and better CPU instruction selection. A Java programmer can realize better performance through choosing more efficient algorithms and data structures. The JVM, through a mod- ern JIT compiler, can improve an application’s performance by generating code that includes sophisticated optimizations. Since there is little a Java application pro- grammer can do to manipulate a JVM’s JIT compiler, the focus for Java developers should be on more efficient alternative algorithms and data structures. Where to focus with alternative algorithms and data structures is identified through profil- ing activities. Monitoring Windows CPU Scheduler Run Queue The run queue depth on Windows is monitored using the \System\Processor Queue Length performance counter. This performance counter can be added to the Per- formance Monitor by selecting the System 7 Processor Queue Length performance counter from the Add Counters dialog. Recall from the “Monitoring CPU Utilization on Windows” section earlier in the chapter, the Add Counters dialog is displayed by right-clicking in the Performance Monitor’s main window and selecting the Add Counters option from the context sensitive menu. Figure 2-7 shows the Performance Monitor monitoring a system’s run queue depth. It is important to notice the scale factor in Performance Monitor. In Figure 2-7, the scale factor is 10. This means a run queue depth of 1 is displayed on the chart as 10, 2 as 20, 3 as 30, and so on. Based on a scale factor of 10, the actual run queue depth in Figure 2-7 ranges from 3 to at least 10. The reported run queue depth should be evaluated against the number of virtual processors on the system to determine whether further action is required such as monitoring over a longer period of time or initiating profiling activities. Windows typeperf can also be used to monitor run queue depth. As mentioned in earlier sections, the typeperf command accepts Windows performance counter names and prints the collected performance data in a tabular form. The following typeperf command monitors run queue depth: typeperf ”\System\Processor Queue Length” ptg6882136 30 Chapter 2 Operating System Performance Monitoring Figure 2-7 Processor queue length What follows is example output using typeperf and the \System\Processor Queue Length performance counter reporting at a 5 second interval rather than a default 1 second. typeperf -si 5 ”\System\Processor Queue Length” ”(PDH-CSV 4.0)”,”\\PICCOLO\System\Processor Queue Length” ”02/26/2011 18:20:53.329”,”3.000000” ”02/26/2011 18:20:58.344”,”7.000000” ”02/26/2011 18:21:03.391”,”9.000000” ”02/26/2011 18:21:08.485”,”6.000000” ”02/26/2011 18:21:13.516”,”3.000000” ”02/26/2011 18:21:18.563”,”3.000000” ”02/26/2011 18:21:23.547”,”3.000000” ”02/26/2011 18:22:28.610”,”3.000000” The run queue depth reported by typeperf is its actual value. There is no scale factor involved as there is with the Performance Monitor. In the above data, the run ptg6882136 CPU Scheduler Run Queue 31 queue depth over the reported 35 second interval ranges from 3 to 9. The run queue data suggests the peak of 9 may be short lived. If further monitoring confirms this is the case, no corrective action is needed since this data is from a system that has four virtual processors. Monitoring Solaris CPU Scheduler Run Queue On Solaris, a system’s run queue depth can be monitored graphically using cpubar and via command line using vmstat. Solaris cpubar, shown in Figure 2-8, shows run queue depth to the right of the CPU utilization bars with the vertical bar above the “r” label. The height of the bar is scaled based on the actual number of entries in the run queue, not a percentage of queue fullness. The run queue can also be monitored with the vmstat command. The first column in vmstat reports the run queue depth. The value reported is the number of light- weight processes in the run queue. The following is an example with the run queue column in bold. kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr cd s0 - - n sy cs us sy id 2 0 0 333273 177562 99 265 0 0 0 0 0 97 0 0 0 1737 14347 1225 28 4 68 4 0 0 330234 174274 69 977 0 0 0 0 0 70 0 0 0 1487 13715 1293 68 3 29 2 0 0 326140 169259 48 303 0 0 0 0 0 85 0 0 0 1746 29014 2394 48 5 47 6 0 0 323751 164876 92 730 0 0 0 0 0 58 0 0 0 1662 48860 3029 67 5 28 5 0 0 321284 160069 38 206 0 0 0 0 0 48 0 0 0 1635 50938 2714 83 5 12 Monitoring Linux CPU Scheduler Run Queue On Linux a system’s run queue depth can be monitored using the vmstat command. The first column in vmstat reports the run queue depth. The number reported is the Figure 2-8 Solaris cpubar showing run queue depth ptg6882136 32 Chapter 2 Operating System Performance Monitoring actual number of lightweight processes in the run queue. The following is an example with the run queue column in bold. procs ----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 4 0 0 959476 340784 1387176 0 0 0 0 1030 8977 63 35 1 0 3 0 0 959444 340784 1387176 0 0 0 0 1014 7981 62 36 2 0 6 0 0 959460 340784 1387176 0 0 0 16 1019 9380 63 36 1 0 1 0 0 958820 340784 1387176 0 0 0 0 1036 9157 63 35 2 0 4 0 0 958500 340784 1387176 0 0 0 29 1012 8582 62 37 1 0 Memory Utilization In addition to CPU utilization there are attributes of a system’s memory that should be monitored, such as paging or swapping activity, locking, and voluntary and invol- untary context switching along with thread migration activity. A Java application or JVM that is swapping or utilizing virtual memory experi- ences pronounced performance issues. Swapping occurs when there is more memory being consumed by applications running on the system than there is physical memory available. To deal with this potential situation, a system is usually configured with an area called swap space. Swap space is typically allocated on a disk in a distinct disk partition. When the amount of physical memory is exhausted by the applications running on the system, the operating system swaps out a portion of an application to swap space on disk. Usually the operating system swaps out a portion of an appli- cation that is executing the least frequently so as to not impact the applications or the portions of applications that are the busiest. When a portion of an application is accessed that has been swapped out, that portion of the application must be paged in from the swap space on disk to memory. Swapping in from disk to memory can have a significant impact on an application’s responsiveness and throughput. A JVM’s garbage collector performs poorly on systems that are swapping because a large portion of memory is traversed by the garbage collector to reclaim space from objects that are unreachable. If part of the Java heap has been swapped out it must be paged into memory so its contents can be scanned for live objects by the garbage collector. The time it takes to page in any portion of the Java heap into memory can dramatically increase the duration of a garbage collection. If the garbage collection is a “stop the world” type of operation, one that stops all application threads from executing, a system that is swapping during a garbage collection is likely to experi- ence lengthy JVM induced pause times. If you observe lengthy garbage collections, it is a possibility that the system is swapping. To prove whether the lengthy garbage collection pauses are caused by swapping, you must monitor the system for swapping activity. ptg6882136 Memory Utilization 33 Monitoring Memory Utilization on Windows On Windows systems that include the Performance Monitor, monitoring memory pages per second (\Memory\Pages / second) and available memory bytes (\Memory\ Available MBytes), can identify whether the system is swapping. When the avail- able memory, as reported by the \Memory\Available MBytes counter, is low and you observe paging activity, as reported by the \Memory\Pages / Second counter, the system is likely swapping. It is easiest to illustrate a Windows system that is swapping using the output from the typeperf command. The following is a typeperf command to report available memory and paging activity at 5 second intervals (the -si specifies the reporting interval). typeperf -si 5 ”\Memory\Available Mbytes” ”\Memory\Pages/sec” The following output from typeperf is taken from a system that is swapping. The first column of data is the date and time stamp. The second column is the available memory, and the third column is the pages per second. ”02/15/2011 15:28:11.737”,”150.000000”,”0.941208” ”02/15/2011 15:28:16.799”,”149.000000”,”1.857361” ”02/15/2011 15:28:21.815”,”149.000000”,”2.996049” ”02/15/2011 15:28:26.831”,”149.000000”,”17.687691” ”02/15/2011 15:28:31.909”,”149.000000”,”0.929074” ”02/15/2011 15:28:36.940”,”149.000000”,”1.919541” ”02/15/2011 15:28:41.956”,”149.000000”,”0.991037” ”02/15/2011 15:28:46.971”,”149.000000”,”1.977258” ”02/15/2011 15:28:51.002”,”149.000000”,”0.969558” ”02/15/2011 15:28:56.065”,”149.000000”,”14.120284” ”02/15/2011 15:29:01.127”,”150.000000”,”8.470692” ”02/15/2011 15:29:06.174”,”152.000000”,”9.552139” ”02/15/2011 15:29:11.174”,”151.000000”,”2.000104” ”02/15/2011 15:29:16.174”,”152.000000”,”1.999969” ”02/15/2011 15:29:21.174”,”153.000000”,”0.999945” Notice the amount of memory available is staying fairly constant around 150 megabytes yet there is consistent paging activity. Since the amount of available memory is staying fairly constant, it is reasonable to assume no new applications are being launched. When an application launches, the amount of available memory is expected to drop, and it is expected to see paging activity since the application must be paged into memory. Therefore, if the system is using a fairly consistent amount ptg6882136 34 Chapter 2 Operating System Performance Monitoring of memory and no new applications are launching, yet there is paging activity, it is likely the system is swapping. It is important to note that a system can report little available memory and report no paging activity. In such a situation, the system is not swapping. It just simply is utilizing most of the physical RAM available on the system. Likewise, a system may be experiencing paging activity, yet have sufficient memory available and as a result not be swapping. The paging activity could be the result of an application being launched. Monitoring Memory Utilization on Solaris On Solaris, when available memory becomes low, the kernel’s page scanner begins looking for memory pages no longer in use by an application so they can be made available for other applications and processes. If the page scanner is unable to find the memory demanded by the applications and no additional physical memory is available, it begins to swap out the least recently used memory pages to a swap space on disk. The lower the amount of available memory, the higher the page scan rate. In other words, as lower memory is available, the page scanner gets more aggressive with trying to find available memory pages it can reclaim. Since the page scanner becomes more aggressive as available memory becomes low, identifying a Solaris system that is experiencing swapping requires monitoring a combination of the amount of free memory and page scanner activity. Both avail- able free memory and page scanner activity are reported in Solaris vmstat columns labeled “free” and “sr.” When vmstat, cpubar, or any other Solaris monitoring tool reports a scan rate of 0, regardless of the reported available free memory, no swapping is occurring. How- ever, if the scan rate is nonzero and the trend of reported free memory is decreasing, then swapping is likely occurring. The following example output from Solaris vmstat illustrates a system currently using most of its available physical memory; about 100 megabytes are free, as shown in the “free” column, but it is not swapping since its scan rate, the “sr” column, is 0. kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr cd f0 s0 - in sy cs us sy id 0 0 0 1641936 861222 106 2591 0 3 3 0 0 0 0 0 0 4930 24959 10371 60 10 30 0 0 0 1594944 116940 37 1718 8 0 0 0 0 8 0 0 0 4169 17820 10111 52 5 43 0 0 0 1579952 103208 24 521 0 0 0 0 0 1 0 0 0 2948 14274 6814 67 4 29 0 0 0 1556244 107408 97 1116 3 0 0 0 0 11 0 0 0 1336 7662 1576 45 3 52 In contrast, the following example illustrates a system that is experiencing a short- age of available physical memory, dropping pretty rapidly from about 150 Mbytes ptg6882136 Memory Utilization 35 to 44 Mbytes, and by the time it reaches 17 Mbytes, the scan rate, the “sr” column, is reporting significant activity. Observing this kind of pattern with vmstat is an indication the system may be swapping and its performance will begin to become sluggish if it is not already. kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr cd f0 s0 -- in sy cs us sy id 1 0 0 499792 154720 1 1697 0 0 0 0 0 0 0 0 12 811 612 1761 90 7 4 1 0 0 498856 44052 1 3214 0 0 0 0 0 0 0 0 12 1290 2185 3078 66 18 15 3 0 0 501188 17212 1 1400 2 2092 4911 0 37694 0 53 0 12 5262 3387 1485 52 27 21 1 0 0 500696 20344 26 2562 13 4265 7553 0 9220 0 66 0 12 1192 3007 2733 71 17 12 1 0 0 499976 20108 3 3146 24 3032 10009 0 10971 0 63 0 6 1346 1317 3358 78 15 7 1 0 0 743664 25908 61 1706 70 8882 10017 0 19866 0 78 0 52 1213 595 688 70 12 18 Notice in the example, paying attention only to either the “free” or “swap” col- umns can be misleading and alone do not provide obvious clues that a system may be swapping. Monitoring Memory Utilization on Linux On Linux, monitoring for swapping activity can be done using vmstat and observing the free column. There are other ways to monitor for swap activity on Linux such as using the top command or observing the contents of the file /proc/meminfo. Monitoring for swapping activity using Linux vmstat is shown here. The columns in Linux vmstat to monitor are the “si” and “so” columns, which represent the amount of memory paged-in and the amount of memory paged-out. In addition, the “free” column reports the amount of available free memory. The actual units are not as important as observing whether the amount of free memory is low and high paging activity is occurring at the same time. Observing the pattern just described in these statistics is an indication that the system maybe experiencing swapping activity. The following is an example of a system that is experiencing no swapping activity; since there is no paging activity as shown in the “si” and “so” columns and the amount of free memory is not very low. procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 9383948 265684 1879740 0 0 0 0 1 1 0 0 100 0 0 3 0 0 9383948 265684 1879740 0 0 0 11 1012 529 14 0 86 0 0 3 0 0 9383916 265684 1879740 0 0 0 0 1021 5105 20 0 80 0 0 3 0 0 9383932 265684 1879740 0 0 0 13 1014 259 19 0 81 0 0 3 0 0 9383932 265684 1879740 0 0 0 7 1018 4952 20 0 80 0 0 ptg6882136 36 Chapter 2 Operating System Performance Monitoring However, the following vmstat output from a Linux system illustrates a system that is experiencing swapping. procs ------------memory----------- ----swap--- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 9500784 265744 1879752 0 0 0 0 1015 228 0 6 94 0 0 1 0 0 8750540 265744 1879752 0 0 0 2 1011 216 0 6 94 0 0 1 0 0 2999792 265744 1879752 0 0 0 2 1012 218 0 6 94 0 0 2 0 0 155964 185204 1370300 0 0 0 0 1009 215 0 9 90 0 0 2 0 9816 155636 24160 815332 0 1963 0 2000 1040 238 0 13 87 0 0 0 2 218420 165152 384 18964 0 41490 0 41498 1247 286 0 6 88 5 0 0 6 494504 157028 396 18280 45 55217 67 55219 1363 278 0 1 79 21 0 0 7 799972 159508 408 18356 70 61094 145 61095 1585 337 0 1 72 27 0 0 8 1084136 155592 416 18512 65 56833 90 56836 1359 292 0 1 75 24 0 0 3 1248428 174292 500 23420 563 32858 1689 32869 1391 550 0 0 83 17 0 1 1 1287616 163312 624 28800 13901 7838 15010 7838 2710 6765 1 0 93 6 0 1 0 1407744 163508 648 29688 18218 24026 18358 24054 3154 2465 1 1 92 6 0 0 2 1467764 159484 648 28380 19386 12053 19395 12118 2893 2746 2 1 91 5 0 Notice the pattern in this example. Where free memory initially decreases, there is little paging activity shown in either the “si” column or “so” column. But as free memory reaches values in the 155,000 – 175,000 range, page-out activity picks up as shown in the “so” column. Once the page-out activity begins to plateau, the page-in activity begins and increases rather quickly as shown in the “si” column. In general what is happening is the system has an application, or set of applications, that placed significant memory allocation and/or memory access pressure on the system. As the amount of physical memory started to become exhausted, the system began to page-out to virtual memory the least recently used pages in memory. As the applications on the system began to demand pages from memory, page-in activity began to occur. As the paging activity increased, the amount of free memory remained about the same. In other words, the system is swapping in pages nearly as quickly as it is paging them out while the amount of free memory remained rather small. This is a typical pattern that can be observed in Linux vmstat when a Linux system is experiencing swapping. Monitoring Lock Contention on Solaris Many Java applications that do not scale suffer from lock contention. Identifying that lock contention in Java applications can be difficult and the tools to identify lock contention are limited. In addition, optimizations have been made in modern JVMs to improve the per- formance of applications that experience lock contention. For example, in Java 5, ptg6882136 Memory Utilization 37 optimizations were integrated into the Java HotSpot VM (also referred to HotSpot VM hereafter) to implement much of the locking logic, the artifact resulting from Java synchronized methods and synchronized blocks, in user code rather than rely- ing immediately on operating system lock primitives. Prior to Java 5, the HotSpot VM delegated almost all of the locking logic to operating system locking primitives. This allowed for operating system tools such as Solaris mpstat to easily monitor a Java application for lock contention by observing the “smtx” (spin on mutex) column along with observing system or kernel CPU utilization. As a result of the Java 5 HotSpot VM optimizations to implement much of locking logic in user code, using Solaris mpstat and observing the “smtx” column and “sys” CPU utilization columns no longer work as well. Instead, an alternative approach is needed. A high level simplistic description of the lock optimization added to Java 5 Hot- Spot VMs and later is given as follows; spin in a tight loop trying to acquire a lock, if not successful after a number of tight loop spins, park the thread and wait to be notified when to try acquiring the lock again. The act of parking a thread along with awaking a thread results in an operating system voluntary context switch. Hence, an application experiencing heavy lock contention exhibits a high number of voluntary context switches. The cost of a voluntary context switch at a proces- sor clock cycle level is an expensive operation, generally upwards of about 80,000 clock cycles. Context switching can be monitored on Solaris with mpstat by observing the “csw” column. The value reported by the “csw” column in mpstat is the total num- ber of context switches including involuntary context switches. Involuntary context switching is also reported in mpstat in the “icsw” column. Hence, the number of voluntary context switches is the “csw” minus “icsw.” A general rule to follow is that any Java application experiencing 5% or more of its available clock cycles in voluntary context switches is likely to be suffering from lock contention. Even a 3% to 5% level is worthy of further investigation. An estimate of the number of clock cycles spent in voluntary context switching can be calculated by taking the number of thread context switches (csw) observed in an mpstat interval, minus the involuntary context switches observed in an mpstat interval, (icsw), multiplying that number by 80,000 (the general cost of a context switch in number clock cycles), and dividing it by the total number of clock cycles available in the mpstat interval. To illustrate with an example, the following Solaris mpstat output captured at a 5 second interval from a 3.0GHz dual core Intel Xeon CPU executing a Java application shows context switches (csw) at about 8100 per 5 second interval and involuntary context switches (icsw) at about 100 per 5 second interval. ptg6882136 38 Chapter 2 Operating System Performance Monitoring An estimate of the number of clock cycles wasted due to voluntary context switches is roughly (8100 - 100) * 80,000 = 640,000,000 clock cycles. The number of clock cycles available in a 5 second interval is 3,000,000,0001 * 5 = 15,000,000,000. Hence, 640,000,000 / 15,000,000,000 = 4.27%. About 4.27% of the available clock cycles are consumed in voluntary context switches. Based on the general rule of a Java appli- cation spending 3% to 5% or more of available clock cycles in voluntary clock cycles implies this Java application is suffering from lock contention. This lock contention is likely coming from areas where multiple threads are trying to access the same synchronized method or synchronized block of code, or a block of code that is guarded by a Java locking construct such as a java.util.concurrent.locks.Lock. $ mpstat 5 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 4 0 1 479 357 8201 87 658 304 0 6376 86 4 0 10 1 3 0 1 107 3 8258 97 768 294 0 5526 85 4 0 10 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 551 379 8179 91 717 284 0 6225 85 5 0 10 1 2 0 0 2292 2 8247 120 715 428 0 7062 84 5 0 10 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 562 377 8007 98 700 276 0 6493 85 5 0 10 1 0 0 0 2550 4 8133 137 689 417 0 6627 86 4 0 11 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 544 378 7931 90 707 258 0 6609 87 5 0 8 1 0 0 0 2428 1 8061 125 704 409 0 6045 88 3 0 9 1. A 3.0 GHz processor executes 3 billion clock cycles per second. Tip Profiling a Java application with Oracle Solaris Studio Performance Analyzer is a strategy to employ when more concrete information on lock contention and whether lock contention may be a performance concern is required. Profiling with Oracle Solaris Studio Performance Analyzer is covered in detail in Chapter 5 of this book. Monitoring Lock Contention on Linux It is possible to monitor lock contention by observing thread context switches in Linux with the pidstat command from the sysstat package. However, for pidstat to report context switching activity, a Linux kernel version of 2.6.23 or later is required. The use of pidstat -w reports voluntary context switches in a “cswch/s” column. It is important to notice that Linux pidstat -w reports voluntary con- text switches, not a sum of all context switches that Solaris mpstat reports. Addi- tionally, Linux pidstat -w reports the number of voluntary context switches per ptg6882136 Memory Utilization 39 second, not per measurement interval like Solaris mpstat. Therefore, the estimate of the percentage of clock cycles wasted on voluntary context switching is calculated as the number of pidstat -w voluntary context switches divided by the number of virtual processors. Remember that pidstat -w reports voluntary context switches for all virtual processors. As a result, the number of voluntary context switches times 80,000 divided by the number of clock cycles per second of the CPU provides the percentage of CPU clock cycles spent in voluntary context switches. The follow- ing is an example from pidstat -w monitoring a Java application having a process id of 9391 reporting results every 5 seconds. $ pidstat -w -I -p 9391 5 Linux 2.6.24-server (payton) 07/10/2008 08:57:19 AM PID cswch/s nvcswch/s Command 08:57:26 AM 9391 3645 322 java 08:57:31 AM 9391 3512 292 java 08:57:36 AM 9391 3499 310 java To estimate the percentage of clock cycles wasted on context switching, there are about 3500 context switches per second occurring on the system being monitored with pidstat -w, a 3.0GHz dual core Intel CPU. Hence, 3500 divided by 2, the num- ber of virtual processors = 1750. 1750 * 80,000 = 140,000,000. The number of clock cycles in 1 second on a 3.0GHz processor is 3,000,000,000. Thus, the percentage of clock cycles wasted on context switches is 140,000,000/3,000,000,000 = 4.7%. Again applying the general guideline of 3% to 5% of clock cycles spent in voluntary context switches implies a Java application that may be suffering from lock contention. Monitoring Lock Contention on Windows On Windows, in contrast to Solaris and Linux, observing Java lock contention using built-in operating system tools is more difficult. Windows operating systems that include the Performance Monitor and typeperf have the capability to monitor context switches. But the capability to distinguish between voluntary and involuntary context switching is not available via a performance counter. To monitor Java lock contention on Windows, tools outside the operating system are often used, such as Intel VTune or AMD CodeAnalyst. Both of these tools have Java lock profiling capabilities along with capabilities to monitor other performance statistics and CPU performance counters. Isolating Hot Locks Tracing down the location in Java source code of contended locks has historically been a challenge. A common practice to find contended locks in a Java application ptg6882136 40 Chapter 2 Operating System Performance Monitoring has been to periodically take thread dumps and look for threads that tend to be blocked on the same lock across several thread dumps. An example of this procedure is presented in Chapter 4, “JVM Performance Monitoring.” Oracle Solaris Studio Performance Analyzer, which is available for Linux and Solaris, is one of the best tools the authors have used to isolate and report on Java lock contention. Using Performance Analyzer to find contended locks in a Java appli- cation is covered in detail in Chapter 5, and an example is presented in Chapter 6, “Java Application Profiling Tips and Tricks.” Other profilers can identify contended locks on Windows. Profilers that are similar in functionality to the Oracle Solaris Studio Performance Analyzer are Intel VTune and AMD CodeAnalyst. Monitoring Involuntary Context Switches Involuntary context switching was mentioned earlier but not explained in any detail, or how it differs from voluntary context switching. In contrast to voluntary context switching where an executing thread voluntarily takes itself off the CPU, involun- tary thread context switches occur when a thread is taken off the CPU as a result of an expiring time quantum or has been preempted by a higher priority thread. Involuntary context switches can be monitored with Solaris mpstat by observing the “icsw” column. CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 11 13 558 760 212 265 1 3 1 0 525 9 1 0 90 1 9 11 479 467 0 251 1 3 1 0 474 9 1 0 89 2 7 4 226 884 383 147 0 4 2 0 192 4 1 0 96 3 7 4 234 495 0 146 0 3 0 0 215 5 1 0 95 Involuntary context switching can also be observed using Solaris prstat -m. High involuntary context switches are an indication there are more threads ready to run than there are virtual processors available to run them. As a result it is com- mon to observe a high run queue depth in vmstat, high CPU utilization, and a high number of migrations (migrations are the next topic in this section) in conjunction with a large number of involuntary context switches. Strategies to reduce involun- tary context switches include using the Solaris command psrset to create processor sets for systems running multiple applications and assigning applications to specific processor sets, or reducing the number of application threads being run on the sys- tem. An alternative strategy, and usually less effective, is profiling the application to identify areas of the application where you can reduce CPU usage by using improved algorithms so they consume fewer CPU cycles. ptg6882136 Network I/O Utilization 41 Involuntary context switches can also be monitored on Linux using pidstat -w. But as mentioned earlier, pidstat -w reporting of involuntary context switch- ing requires Linux kernel 2.6.23 or later. On Linux, creation of processor sets and assigning applications to those processor sets can be accomplished using the Linux taskset command. See your Linux distribution’s documentation for details on how to use Linux taskset. On Windows systems, applications can be assigned to a processor or set of proces- sors by using Task Manager’s Process tab. Select a target process, right-click, and select Set Affinity. Then choose the processors the selected process should execute on. On Windows Server operating systems, Windows Vista and Windows 7, an applica- tion can be launched from the command line with start /affinity , where is the processor affinity mask in hexadecimal. See the Windows operating system’s documentation for the use of start command and affinity mask. Monitoring Thread Migrations Migration of ready-to-run threads between processors can also be a source of observed performance degradation. Most operating systems’ CPU schedulers attempt to keep a ready-to-run thread on the same virtual processor it last executed. If that same virtual processor is busy, the scheduler may migrate that ready-to-run thread to some other available virtual processor. Migration of threads can impact an applica- tion’s performance since data, or state information, used by a ready-to-run thread may not be readily available in a virtual processor’s cache. On Solaris you can use mpstat and observe the “migr” column to monitor whether thread migrations are an issue to a Java application’s performance. If you are running a Java application on a multicore system and observing a high number of migrations, a strategy to reduce thread migrations is creating processor sets and assigning Java applications to those processor sets. As a general guideline, Java applications scaling across multiple cores or virtual processors and observing migrations greater than 500 per second could benefit from binding Java applications to processor sets. In extreme cases, the Solaris kernel tunable rechoose_interval can be increased as a means to reduce thread migrations. The former, creating processor sets, is the preferred strategy, and the latter, tuning the kernel, should be considered only as a secondary approach. Network I/O Utilization Distributed Java applications may find performance and scalability limited to either network bandwidth or network I/O performance. For instance, if a system’s network interface hardware is sent more traffic than it can handle, messages can be queued ptg6882136 42 Chapter 2 Operating System Performance Monitoring in operating system buffers, which may cause application delays. Additionally, other things may be occurring on the network that cause delays as well. Identifying and monitoring a single network utilization statistic can be hard to find in bundled operating system utilities. For example, even though Linux has netstat with its optional sysstat package and Solaris bundles netstat, neither the Linux nor Solaris implementation of netstat reports network utilization. Both provide statistics such as packets sent and packets received per second along with errors and collisions. Collisions in a small amount are a normal occurrence of Ethernet. Large numbers of errors usually are the result of a faulty network interface card, poor wiring or auto- negotiation problems. Also, for a given number of packets received or transmitted per interval as reported by netstat, it is difficult to know whether the interface is being fully utilized. For example, if a netstat -i command reports 2500 packets per second passing through the network interface card, you do not know whether the network is at 100% utilization or 1% utilization. One conclusion you can make is network traffic is occurring. But that is about the only conclusion you can make without knowing the rated throughput of the underlying network cards and the packet sizes being trans- mitted. In short, it is difficult to tell from the output of netstat on Linux or Solaris to determine whether network utilization is limiting an application’s performance. Regardless of the operating system running your Java application, there is a need for a tool that can show network utilization on the network interfaces your application is using. The next two subsections present tools that can be used on Solaris, Linux, and Windows to monitor network utilization. Monitoring Network I/O Utilization on Solaris On Solaris, a tool called nicstat from the freeware K9Toolkit reports network uti- lization and saturation by network interface. The K9Toolkit is also included in the Solaris Performance Tools CD 3.0 package mentioned earlier in the “Monitoring CPU Utilization on Solaris” section of this chapter. The K9Toolkit can also be downloaded from http://www.brendangregg.com/k9toolkit.html. nicstat has the following command line syntax: nicstat [-hnsz] [-i interface[,...]] | [interval [count]] where -h displays a help message, -n shows nonlocal interfaces only, -s shows a summary output, -z skips reporting of zero values, -i interface is the network inter- face device name, interval is the frequency at which output is to be reported in sec- onds, and count is the number of samples to report. The following is example output from nicstat -i yukonx0 1, which samples the network interface device yukonx0 at a 1 second interval. ptg6882136 Network I/O Utilization 43 The column headings are Int is the network interface device name. rKb/s is the number of kilobytes read per second. wKb/s is the number of kilobytes written per second. rPk/s is the number of packets read per second. wPk/s is the number of packets written per second. rAvs is average bytes read per read. wAvs is the average bytes written per write. %Util is the network interface utilization. Sat is the saturation value. As you can see a wealth of meaningful data is presented with nicstat to help you identify whether your distributed Java application is saturating the network. You can see there is activity occurring at the yukonx0 network interface as shown in the number of bytes read and written yet the network utilization never reaches much above 4% utilization. As a result, you can conclude the applications running on this system are not experiencing a performance issue as a result of a saturated network. Monitoring Network I/O Utilization on Linux A port of the Solaris nicstat monitoring tool for Linux is available. The source code can be downloaded from http://blogs.sun.com/roller/resources/timc/nicstat/ nicstat-1.22.tar.gz. It requires compilation before being able to use it. It reports net- work utilization in the same way as described in the previous section on monitoring network utilization on Solaris. Time Int rKB/s wKB/s rPk/s wPk/s rAvs wAvs %Util Sat 19:24:16 yukonx0 0.75 4.68 2.72 3.80 281.3 1261.9 0.00 0.00 19:24:17 yukonx0 54.14 1924.9 724.1 1377.2 76.56 1431.2 1.58 0.00 19:24:18 yukonx0 44.64 1588.4 598.0 1138.0 76.45 1429.3 1.30 0.00 19:24:19 yukonx0 98.89 3501.8 1320.0 2502.0 76.72 1433.2 2.87 0.00 19:24:20 yukonx0 0.43 0.27 2.00 3.00 222.0 91.33 0.00 0.00 19:24:21 yukonx0 44.53 1587.2 598.0 1134.0 76.26 1433.2 1.30 0.00 19:24:22 yukonx0 101.9 3610.1 1362.0 2580.0 76.64 1432.8 2.96 0.00 19:24:23 yukonx0 139.9 4958.1 1866.7 3541.4 76.73 1433.6 4.06 0.00 19:24:24 yukonx0 77.23 2736.4 1035.1 1956.2 76.40 1432.4 2.24 0.00 19:24:25 yukonx0 48.12 1704.1 642.0 1220.0 76.75 1430.3 1.40 0.00 19:24:26 yukonx0 59.80 2110.8 800.0 1517.0 76.54 1424.8 1.73 0.00 ptg6882136 44 Chapter 2 Operating System Performance Monitoring Monitoring Network I/O Utilization on Windows Monitoring network utilization on Windows is not as simple as adding performance counters to Performance Monitor and observing their values. It requires knowing the possible bandwidth of the network interface you are interested in monitoring and some measure of the amount of data passing through the network interface. The number of bytes transmitted across a network interface can be obtained using the “\Network Interface(*)\Bytes Total/sec” performance counter. The “*” wild- card reports the bandwidth for all network interfaces on the system. You can use the typeperf \Network Interface(*)\Bytes Total/sec command to see the names of the network interfaces. Then, you can replace the wildcard “*” with the network interface you are interested in monitoring. For example, suppose the output from typeperf \Network Interface(*)\Bytes Total/sec shows the network interfaces as Intel[R] 82566DM-2 Gigabit Network Connection, isatap. gateway.2wire.net, Local Area Connection* 11 and you know the network interface card installed in your system is an Intel network card. You can substitute “Intel[R] 82566DM-2 Gigabit Network Connection” for the “*” wildcard when adding the per- formance counter to Performance Monitor or when using the typeperf command. In addition to the bytes transmitted across the interface, the bandwidth of the net- work interface must also be obtained. It can be obtained using the “\Network Inter- face(*)\Current Bandwidth” performance counter. Again, the “*” wildcard should be replaced with the network interface you are interested in monitoring. It is important to note that the Current Bandwidth performance counter reports bandwidth in bits per second. In contrast, the Bytes Total/sec reports in units of bytes per second. Therefore, the formula to calculate network utilization must compen- sate for the proper units, bits per second, or bytes per second. The following are two formulas that compute network utilization: the first one by adjusting the Current Bandwidth into bytes per second by dividing the Current Bandwidth by 8, and the second one by adjusting the Bytes Total/sec into bits per second by multiplying it by 8 (8 bits per byte). network utilization % = Bytes Total/sec/(Current Bandwidth / 8) x 100 Or, alternatively as network utilization % = (Bytes Total/sec * 8) / Current Bandwidth x 100 Network utilization can also be monitored in Windows using Task Manager and clicking on the Networking tab. An example is shown in Figure 2-9. ptg6882136 Network I/O Utilization 45 Figure 2-9 Task Manager showing network utilization Application Performance Improvement Considerations An application executing a large number of reads and writes to a network with small amounts of data in each individual read or write call consumes large amounts of system or kernel CPU and may also report a high number of system calls. A strat- egy to reduce system or kernel CPU in such an application is to reduce the number network read or write system calls. Additionally, the use of nonblocking Java NIO instead of blocking java.net.Socket may also improve an application’s perfor- mance by reducing the number of threads required to process incoming requests or send outbound replies. A strategy to follow when reading from a nonblocking socket is to design and implement your application to read as much data as there is available per read call. Also, when writing data to a socket, write as much data as possible per write call. There are Java NIO frameworks that incorporate such practices, such as Project Grizzly (https://grizzly.dev.java.net). Java NIO frameworks also tend to simplify the programming of client-server type applications. Java NIO, as offered in the JDK, tends to be a “bare metal” type of implementation, and there is plenty of room to make poor use of its Java APIs that can lead to disappointing application perfor- mance, and hence the suggestion of using a Java NIO framework. ptg6882136 46 Chapter 2 Operating System Performance Monitoring Disk I/O Utilization If an application performs disk operations, disk I/O should be monitored for possible performance issues. Some applications make heavy use of disk as a major part of its core functionality such as databases, and almost all applications utilize an application log to write important information about the state or behavior of the application as events occur. Disk I/O utilization is the most useful monitoring statistic for under- standing application disk usage since it is a measure of active disk I/O time. Disk I/O utilization along with system or kernel CPU utilization can be monitored using iostat on Linux and Solaris. To use iostat on Linux, the optional sysstat package must be installed. To monitor disk utilization on Windows Server systems, the Performance Monitor has several performance counters available under its Logical Disk performance object. On Solaris, iostat -xc shows disk utilization for each disk device on the system along with reporting CPU utilization. This command is useful for showing both disk utilization and system or kernel CPU utilization together. The following example shows a system that has three disks, sd0, sd2, and sd4, with disk I/O utilization of 22%, 13%, and 36%, respectively, along with 73% system or kernel CPU utilization. The other statistics from iostat are not as important for application performance monitoring since they do not report a “busy-ness” indicator. $ iostat -xc 5 extended disk statistics cpu disk r/s w/s Kr/s Kw/s wait actv svc_t %w %b us sy wt id sd0 3.4 1.1 17.1 9.8 0.1 0.2 16.2 1 22 3 73 8 16 sd2 2.1 0.5 16.7 4.0 0.0 0.1 23.6 1 13 sd4 5.2 6.0 41.4 45.2 0.2 0.4 59.2 8 36 To monitor disk I/O utilization and system or kernel CPU utilization on Linux you can use iostat -xm. The following is an example of iostat -xm from a Linux system showing 97% and 69% for disks hda and hdb, respectively, along with 16% system or kernel CPU utilization. Columns reporting 0 values were removed from the output for ease of reading. $ iostat -xm 5 avg-cpu: %user %nice %system %iowait 0.20 0.40 16.37 83.03 Device: rrqm/s r/s rsec/s rMB/s avgqu-sz await svctm %util hda 9662.87 305.59 87798.80 42.87 1.64 5.39 3.17 97.01 hdb 7751.30 225.15 63861.08 31.18 1.18 5.24 3.11 69.94 ptg6882136 Disk I/O Utilization 47 One of the challenges with monitoring disk I/O utilization is identifying which files are being read or written to and which application is the source of the disk activity. Recent versions of Solaris 10 and Solaris 11 Express include several DTrace scripts in the /usr/demo/dtrace directory that can help monitor disk activity. The iosnoop.d DTrace script provides details such as which user id is accessing the disk, what process is accessing the disk, the size of the disk access, and the name of the file being accessed. The iosnoop.d script is also included in the Solaris DTraceToolKit downloadable at http://www.solarisinternals.com/wiki/index.php/ DTraceToolkit. The following is example output from executing iosnoop.d while launching NetBeans IDE. The entire output is not displayed since there are many files accessed during a NetBeans IDE launch. Hence, for brevity the output is trimmed. Tip The Solaris Performance Tools CD 3.0, presented in the “CPU Utilization” section earlier in this chapter contains a graphical tool called iobar that displays disk I/O in a cpubar like manner. The Solaris Performance Tools CD 3.0, also contains a command line tool called iotop that displays Solaris iostat -x information in a prstat or top manner. $ iosnoop.d UID PID D BLOCK SIZE COMM PATHNAME 97734 1617 R 4140430 1024 netbeans /huntch/tmp/netbeans 97734 1617 R 4141518 1024 bash /huntch/tmp/netbeans/modules 97734 1617 R 4150956 1024 bash /huntch/tmp/netbeans/update 97734 1697 R 4143242 1024 java /huntch/tmp/netbeans/var 97734 1697 R 4141516 1024 java /huntch/tmp/netbeans/config 97734 1697 R 4143244 1024 java /huntch/tmp/netbeans/var/log 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 4153884 1024 java /huntch/tmp/netbeans/docs 97734 1697 R 12830464 8192 java /usr/jdk1.6.0/jre/lib/rt.jar 97734 1697 R 12830480 20480 java /usr/jdk1.6.0/jre/lib/rt.jar 97734 1697 R 12830448 8192 java /usr/jdk1.6.0/jre/lib/rt.jar 97734 1697 R 12830416 8192 java /usr/jdk1.6.0/jre/lib/rt.jar 97734 1697 R 12830432 4096 java /usr/jdk1.6.0/jre/lib/rt.jar 97734 1697 R 12828264 8192 java /usr/jdk1.6.0/jre/lib/rt.jar [... additional output removed ...] ptg6882136 48 Chapter 2 Operating System Performance Monitoring The “UID” column reports the user id responsible for performing the disk access. The “PID” column is the process id of the process performing the disk access. The “D” column indicates whether the disk access is the result of a read or write, “R” = read, “W” = write. The “BLOCK” column is the disk block. The “SIZE” column is the amount of data accessed in bytes. The “COMM” column is the name of the command performing the disk access, and the “PATHNAME” column is the name of the file being accessed. Patterns to look for in the output of iosnoop.d is repeated accesses to the same file, same disk block, by the same command, process id, and user id. For example, in the preceding output there are many disk accesses of 1024 bytes on the same disk block 4153884, which may indicate a possible optimization opportunity. It may be that the same information is being accessed multiple times. Rather than re-reading the data from disk each time, the application may be able to keep the data in memory, reuse it, and avoid re- reading and experiencing an expensive disk read. If the same data is not being accessed, it may be possible to read a larger block of data and reduce the number of disk accesses. At a larger scale, if high disk I/O utilization is observed with an application, it may be worthwhile to further analyze the performance of your system’s disk I/O subsys- tem by looking more closely at its expected workload, disk service times, seek times, and the time spent servicing I/O events. If improved disk utilization is required, several strategies may help. At the hardware and operating system level any of the following may improve disk I/O utilization: A faster storage device Spreading file systems across multiple disks Tuning the operating system to cache larger amounts of file system data structures At the application level any strategy to minimize disk activity will help such as reducing the number of read and write operations using buffered input and output streams or integrating a caching data structure into the application to reduce or eliminate disk interaction. The use of buffered streams reduces the number of sys- tem calls to the operating system and consequently reduces system or kernel CPU utilization. It may not improve disk I/O performance, but it will make more CPU cycles available for other parts of the application or other applications running on the system. Buffered data structures are available in the JDK that can easily be utilized, such as java.io.BufferedOutputStream and java.io.BufferedInputStream. An often overlooked item with disk performance is checking whether the disk cache is enabled. Some systems are configured and installed with the disk cache disabled. An enabled disk cache improves an application’s performance that heavily relies on disk I/O. However, you should use caution if you discover the default set- ting of a system has the disk cache disabled. Enabling the disk cache may result in corrupted data in the event of an unexpected power failure. ptg6882136 Additional Command Line Tools 49 Additional Command Line Tools When monitoring applications for an extended period of time such as several hours or several days, or in a production environment, many performance engineers and system administrators of Solaris or Linux systems use sar to collect performance statistics. With sar, you can select which data to collect such as user CPU utilization, system or kernel CPU utilization, number of system calls, memory paging, and disk I/O statistics. Data collected from sar is usually looked at after-the-fact, as opposed to while it is being collected. Observing data collected over a longer period of time can help identify trends that may provide early indications of pending performance concerns. Additional information on what performance data can be collected and reported with sar can be found in the Solaris and Linux sar man pages. Another tool that can be useful on Solaris is kstat, which reports kernel statistics. Its use can be powerful for applications in need of every bit of performance they can get. There are many kernel statistics kstat can report on. A kstat -l command lists all the possible kernel statistics that can be monitored with kstat. The most important thing to understand about using kstat is that it reports the number of events since the system was last powered on. So, to monitor an application with kstat, running kstat before and after some interval of interest and then taking the difference between reported values is required. In addition, the application of monitoring interest should be the only application running when using kstat since kstat does not report on which application is correlated to the statistics, or the values reported. If more than one application is running on the system when using kstat, you will have no way of identifying which application is producing the values reported by kstat. On Solaris, processor specific CPU performance counters can be monitored using Solaris bundled commands cpustat or cputrack. Use of these specific CPU perfor- mance counters is usually left to performance specialists looking for specific tuning optimizations but are mentioned in this section since there may be some performance specialists among the readers of this chapter. Both cpustat and cputrack commands require a set of event counters that are specific to a processor such as AMD, Intel, or SPARC. The set of CPU performance Tip On Solaris and Solaris 11 Express, the disk cache can be enabled when the disk is formatted using the format -e command. However, do not run the format -e command on a disk or partition where it is desirable to preserve the data. The format -e command destroys all data on the disk or partition where the format command is executed. Disk performance on Solaris can also be improved by configuring and using Oracle Solaris ZFS file systems. See the Solaris man pages for tips on how to configure and use Oracle Solaris ZFS file systems. ptg6882136 50 Chapter 2 Operating System Performance Monitoring counters may also vary within a processor family. To obtain a list of available perfor- mance counters, you can use the -h option. Additionally, CPU performance counters can also be found in the processor manufacturer’s documentation. In contrast to cpustat, which gathers information from CPU performance counters for all appli- cations on the system and tends to be more intrusive, cputrack collects CPU per- formance counter statistics for individual applications with little or no interference to other activities on the system. Additional details on the usage of cpustat and cputrack can be found in the Solaris man pages. Monitoring CPU Utilization on SPARC T-Series Systems The SPARC T-series processor from Oracle combines both chip multiprocessing and chip multithreading. Its architecture differs enough from traditional chip architec- tures that monitoring its CPU utilization deserves its own section. To understand CPU utilization of a SPARC T-series based system it is important to understand some of the basics of the SPARC T-series chip architecture, how it differs from traditional processor architectures, and why conventional Unix monitoring tools such as vmstat and mpstat do not truly show SPARC T-series CPU utilization. The SPARC T-series processors have not only multiple cores, but also multiple hard- ware threads per core. It is easiest to explain the first generation SPARC T-series first and then extend it to its later generations. The UltraSPARC T1 is the first generation SPARC T-series processor. It has eight cores with four hardware threads per core and one pipeline per core. The UltraSPARC T2, the second generation SPARC T-series pro- cessor, consists of eight cores with eight hardware threads per core and two pipelines per core. On an UltraSPARC T1, only one hardware thread per core executes in a given clock cycle. On an UltraSPARC T2, since there are two pipelines per core, two hardware threads per core execute per clock cycle. However, what makes the SPARC T-series processors unique is the capability to switch to a different hardware thread within a core when the one that had been executing becomes stalled. Stalled is defined as a CPU state such as a CPU cache miss where the processor must wait for a memory data fetch. Applications with a large number of concurrent software threads that tend to experience stalls tend to perform very well on a SPARC T-series processor since the amount of time spent on CPU stalls tends to be much longer than the time it takes for a SPARC T-series processor core to switch to a different runnable hardware thread. In contrast, applications with a small number of concurrent threads, especially ones that do not experience many CPU stalls, tend not to perform as well as they would on a faster clock rate traditional processor. For example, consider an application that has eight concurrent software threads that are runnable at all times with few or a very small number of CPU stalls. Such an application would utilize one hardware thread per core on an UltraSPARC T1 since there are eight cores on an UltraSPARC ptg6882136 Monitoring CPU Utilization on SPARC T-Series Systems 51 T1 and only one of those four hardware threads per core can execute per clock cycle. Additionally, since only one of those four hardware threads per core can execute per clock cycle, those eight concurrent software threads will execute at a clock rate of one-fourth the clock frequency. For example, a 1.2GHz UltraSPARC T1 on such a workload would be executing each of those eight concurrent software threads at an effective clock rate of 300MHz, 1.2GHz/4 = 300MHz. In contrast, a dual CPU socket quad core Intel or AMD based system, a system with eight cores, which has a clock rate of 2.33GHz, for example, would execute each of those eight concurrent software threads at 2.33GHz since each concurrent software thread can execute on a single core and each core is a single hardware thread executing at 2.33GHz. However, in practice, few workloads operate with few memory stalls. On workloads with a much larger number of runnable threads, especially threads that experience CPU stalls, the SPARC T-series will likely perform better than an x86/x64 quad core processor since the time it takes to switch between hardware threads on a SPARC T-series is faster than the time it takes for a thread context switch on a single hardware thread per core architecture because the thread context switch may require CPU caches to be primed with data, which means the switched-to-thread will waste clock cycles waiting for data to be loaded from memory. With a better understanding of SPARC T-series architecture and its differences from traditional single hardware thread per core processor architecture, it becomes easier to understand how to monitor a SPARC T-series based system. It is also impor- tant to realize that the Solaris operating system treats each hardware thread of a SPARC T-series as a virtual processor. This means monitoring tools such as mpstat report 32 virtual processors for an UltraSPARC T1 (8 cores * 4 hardware threads per core) and 64 processors for an UltraSPARC T2 (8 cores * 8 hardware threads per core). Remember that not all virtual processors in a SPARC T-series can execute on the same clock cycle. When reporting CPU utilization for a virtual processor, both mpstat and vmstat commands assume a virtual processor that is not idle, is a busy virtual processor that is making progress on processing a workload. In other words, both mpstat and vmstat will report a virtual processor as busy, or as being utilized, even when that virtual processor is stalled. Recall that on a SPARC T-series proces- sor, a stalled software thread, which is running on a virtual processor (hardware thread), does not necessarily mean the pipeline is stalled or the entire processor core is stalled. Since the SPARC T-series processors have hardware threads reported as virtual processors, vmstat and mpstat actually report the percentage of pipeline occupancy of software threads. Tip More detailed information about the SPARC T-series processors can be found on the Solaris Internals wiki at http://www.solarisinternals.com/wiki/index.php/CMT_Utilization. ptg6882136 52 Chapter 2 Operating System Performance Monitoring On systems running processors that do not have multiple hardware threads per core, idle time reported by mpstat or vmstat can be used to decide whether the sys- tem can take on additional load. On a SPARC T-series, a hardware thread being idle, which is reported as a virtual processor by mpstat, and a SPARC T-series processor core being idle are two different things. Remember that mpstat reports statistics on each hardware thread since each hardware thread is seen as a virtual processor. To understand CPU utilization of a SPARC T-series processor, both processor core utili- zation and core hardware thread utilization need to be observed. Processor core utili- zation of a SPARC T-series can be observed by monitoring the number of instructions executed by a given processor core. The Solaris cpustat command can monitor the number of instructions executed per hardware thread within a core. But it does not have the capability to report on the number of instructions executed per core. However, the cpustat data reporting the number of instructions executed per hardware thread could be aggregated to show the number of instructions executed per core. A utility called corestat aggregates the instruction count per hardware thread reported by cpustat to derive a SPARC T-series core CPU utilization. The corestat command is not included in Solaris distributions. But, corestat can be downloaded from Oracle’s cool tools Web site, http://cooltools.sunsource.net/corestat/ index.html. Additional information and instructions on how to use corestat can also be found on Oracle’s cool tools Web site. Looking at vmstat, mpstat, and corestat data collected on a SPARC T-series based system provides information about how the system is performing. For example, suppose vmstat or mpstat is reporting the system is 35% busy and corestat is reporting core utilization is 50%. Since core utilization is higher than the CPU utili- zation reported by vmstat or mpstat, if the system continues to take on additional similar load by adding more application threads, the system may reach core satura- tion before it reaches CPU saturation. As a result, this application may reach peak scalability prior to vmstat or mpstat reporting the system is 100% busy. Consider a different scenario: vmstat and mpstat are reporting the system is 100% busy and corestat is reporting core utilization is 40%. This indicates the system will not be able to take on additional work unless you are able to improve core utiliza- tion. Improving core utilization requires improving pipeline performance. To realize improved pipeline performance you have to focus on reducing CPU stalls. Reducing CPU stalls can be difficult and requires an in-depth understanding of the application being run on the system so that the application can better utilize the CPU caches. This usually means improving memory locality for the application. The skill nec- essary to reduce CPU stalls usually requires special assistance from performance engineers. These two example scenarios illustrate how important it is to monitor both CPU utilization with vmstat or mpstat and also monitor core utilization on SPARC T-series systems. ptg6882136 Bibliography 53 Bibliography Linux nicstat source code download. http://blogs.sun.com/roller/resources/timc/ nicstat/nicstat-1.22.tar.gz. Microsoft Windows typeperf description. http://www.microsoft.com/resources/ documentation/windows/xp/all/proddocs/en-us/nt_command_typeperf.mspx?mfr=true. Oracle’s cool tools Web site. http://cooltools.sunsource.net/corestat/index.html. Project Grizzly Web site. http://grizzly.java.net. Solaris Internals wiki. http://www.solarisinternals.com/wiki/index.php/ CMT_Utilization. Solaris K9 Toolkit and nicstat Web site and download. http://www.brendangregg. com/k9toolkit.html. Solaris Performance Tools CD 3.0 Web site. http://www.scalingbits.com/solaris/ performancetoolcd. Tim Cook blog Web site. http://blogs.sun.com/timc/entry/nicstat_the_solaris_and_linux. ptg6882136 This page intentionally left blank ptg6882136 55 3 JVM Overview Since its introduction in 1995, Java has evolved substantially. So too have Java Virtual Machines, (JVMs). In Java’s early days, Java performance was a challenge for many applications despite its advantages of developer productivity and memory management. The integration of JIT compilers, more sophisticated garbage collectors, and improvements in the JVM runtime environment have allowed many Java appli- cations to meet their performance requirements. Even with the many enhancements added to modern JVMs, performance and scalability remain important to applica- tion stakeholders. For example, many applications have increased their performance requirements and performance service level agreements. Additionally, new families or classes of applications are able to utilize Java technologies as a result of the per- formance and scalability improvements available in modern JVMs. One of challenges introduced by modern JVMs is many users of Java technology see a JVM as a black box, which can make it a difficult task to improve the performance or scalability of a Java application. Thus, having a basic, fundamental understanding of a modern JVM is essential to the ability to improve a Java application’s performance. This chapter provides an overview of the HotSpot Java Virtual Machine, (also referred to as the HotSpot VM hereafter), architecture. Not all the information in this chapter is required to tackle the task of improving all Java application performance issues running in a HotSpot VM. Nor is this chapter an exhaustive description of the Java HotSpot VM (also referred to as the HotSpot VM hereafter). But it does present its major components and its architecture. There are three major components of the HotSpot VM: VM Runtime, JIT compiler, and a memory manager. This chapter begins with a high level architecture view of ptg6882136 56 Chapter 3 JVM Overview the HotSpot VM followed by an overview of each of the three major components. In addition, information on ergonomic decisions the HotSpot VM makes automatically is included at the end of the chapter. HotSpot VM High Level Architecture The HotSpot VM possesses an architecture that supports a strong foundation of fea- tures and capabilities. Its architecture supports the ability to realize high performance and massive scalability. For example, the HotSpot VM JIT compilers generate dynamic optimizations; in other words, it makes optimization decisions while the Java applica- tion is running and generates high performing native machine instructions targeted for the underlying system architecture. In addition, through its maturing evolution and continuous engineering of its runtime environment and multithreaded garbage collector, the HotSpot VM yields high scalability on even the largest computer systems available. A high level view of the HotSpot VM architecture is shown in Figure 3-1. Figure 3-1 HotSpot VM high level architecture. Garbage Collector [ Serial | Throughput | Concurrent | G1 ] JIT Compiler [ Client | Server ] HotSpot VM Runtime HotSpot Java Virtual Machine ptg6882136 HotSpot VM High Level Architecture 57 As shown in Figure 3-1, the JIT compiler, client or server, is pluggable as is the choice of garbage collector: Serial GC, Throughput, Concurrent, or G1. At the time of this writing, the G1 garbage collector is under development and expected to be available in Java 7 HotSpot VMs. The HotSpot VM Runtime provides services and common APIs to the HotSpot JIT compilers and HotSpot garbage collector. In addi- tion the HotSpot VM Runtime provides basic functionality to the VM such as a launcher, thread management, Java Native Interface, and so on. Further details on VM Runtime’s components and their responsibilities are described in the next sec- tion, “HotSpot VM Runtime.” Early releases of the HotSpot VM were limited to 32-bit JVMs, which have a memory address limitation of four gigabytes. It is important to note that the actual Java heap space available for a 32-bit HotSpot VM may be further limited depending on the underlying operating system. For instance, on Microsoft Windows operating systems the maximum Java heap available to a HotSpot VM is around 1.5 gigabytes. For Linux operating systems, the maximum Java heap available to the HotSpot VM is around 2.5 to 3.0 gigabytes for very recent Linux kernels and about 2 gigabytes for less recent Linux kernels. On Oracle Solaris, also referred to as Solaris hereafter, operating systems the maximum Java heap available to the HotSpot VM is around 3.3 gigabytes. The actual maximums vary due to the memory address space con- sumed by both a given Java application and a JVM version. As server systems were introduced with much larger amounts of memory, a 64-bit version of the HotSpot VM was introduced. A 64-bit HotSpot VM allows these sys- tems to utilize additional memory through the use of increased Java heaps. There are several classes of applications where using 64-bit addressing can be useful. How- ever, with 64-bit VMs come a performance penalty due to an increase in size of the internal HotSpot VM’s representation of Java objects, called ordinary object point- ers, or oops, which have an increase in width from 32 bits to 64 bits. This increase in width results in fewer oops being available on a CPU cache line and as a result decreases CPU cache efficiency. The decrease in CPU cache efficiency on 64-bit JVMs often results in about a 8% to 15% performance degradation compared to a 32-bit JVM. However, beginning with more recent Java 6 HotSpot VMs, along with those found in OpenJDK, a new feature called compressed oops, which is enabled with the -XX:+UseCompressedOops VM command line option, can yield 32-bit JVM perfor- mance with the benefit of larger 64-bit Java heaps. In fact, some Java applications realize better performance with a 64-bit HotSpot VM using compressed oops than they achieve with a 32-bit VM. The performance improvement realized from com- pressed oops arises from being able to pack a 64-bit pointer into 32 bits by relying on alignment and possibly having an offset. In other words, the increase in performance comes from using smaller, more space efficient compressed pointers rather than full width 64-bit pointers, which improves CPU cache utilization. An application expe- riencing improved CPU cache utilization is one that executes faster. In addition, on ptg6882136 58 Chapter 3 JVM Overview some platforms such as Intel or AMD x64, 64-bit JVMs can make use of additional CPU registers, which can also improve application performance. Having additional registers available helps avoid what is known as register spilling. Register spilling occurs where there is more live state (i.e. variables) in the application than the CPU has registers. When register spilling occurs, some of the live state must be “spilled” from CPU registers to memory. Therefore, avoiding register spilling can result in a faster executing application. Today 32-bit and 64-bit HotSpot VMs are available for the following hardware plat- forms and operating systems: Solaris SPARC, Solaris x86, Linux x86, and Windows x86 for both Intel Xeon and AMD along with Solaris x64, Linux x64, and Windows x64 for both Intel Xeon and AMD. Various ports of the HotSpot VM also exist for other platforms, such as Apple x64, Apple PPC, Intel Itanium, HP-UX, MIPS, and ARM. HotSpot VM Runtime The VM Runtime is an often overlooked part of the HotSpot VM. The VM’s garbage collectors and JIT compilers tend to get more attention than the VM Runtime. How- ever, the VM Runtime provides the core functionality of the HotSpot VM. This section provides an introduction to the HotSpot VM Runtime environment. The objective of this section is to provide a better understanding of the responsibilities and roles the Runtime plays in the VM. Having this understanding allows readers to take full per- formance advantage of the services provided by the VM Runtime. Not all the details presented in this section are necessary to realize a high performance Java applica- tion. However, it can be beneficial to have a basic understanding of the HotSpot VM Runtime since there may be cases where tuning a property of service provided by the VM Runtime may yield significant improvement in Java application performance. The HotSpot VM Runtime encompasses many responsibilities, including parsing of command line arguments, VM life cycle, class loading, byte code interpreter, excep- tion handling, synchronization, thread management, Java Native Interface, VM fatal error handling, and C++ (non-Java) heap management. In the following subsections, each of these areas of the VM Runtime is described in more detail. Command Line Options The HotSpot VM Runtime parses the many command line options and configures the HotSpot VM based on those options. A number of command line options and environ- ment variables can affect the performance characteristics of the HotSpot VM. Some of these options are consumed by the HotSpot VM launcher such as the choice of JIT compiler and choice of garbage collector; some are processed by the launcher and passed to the launched HotSpot VM where they are consumed such as Java heap sizes. ptg6882136 HotSpot VM Runtime 59 There are three main categories of command line options: standard options, nonstandard options, and developer options. Standard command line options are expected to be accepted by all Java Virtual Machine implementations as required by the Java Virtual Machine Specification. [1] Standard command line options are sta- ble between releases. However, it is possible for standard command line options to be deprecated in subsequent releases after the release in which it was first introduced. Nonstandard command line options begin with a -X prefix. Nonstandard command line options are not guaranteed to be supported in all JVM implementations, nor are they required to be supported in all JVM implementations. Nonstandard command line options are also subject to change without notice between subsequent releases of the Java SDK. Developer command line options in the HotSpot VM begin with a -XX prefix. Developer command line options often have specific system requirements for correct operation and may require privileged access to system configuration param- eters. Like nonstandard command line options, developer command line options are also subject to change between releases without notice. Command line options control the values of internal variables in the HotSpot VM, all of which have a type and a default value. For boolean values, the mere presence or lack of presence of an option on the HotSpot VM command line can con- trol the value of these variables. For developer command line options (-XX options) with boolean flags, a + or - before the name of the options indicates a true or false value, respectively, to enable or disable a given HotSpot VM feature or option. For example, -XX:+AggressiveOpts sets a HotSpot internal boolean variable to true to enable additional performance optimizations. In contrast, -XX:-AggressiveOpts sets the same internal variable to false to disable additional performance optimiza- tions. Developer command line options (the -XX options) that take an additional argument, those that are nonboolean, tend to be of the form, -XX:OptionName= where is some numeric value. Almost all developer command line options that take an additional argument, accept an integer value along with a suffix of k, m, or g, which are used as kilo-, mega-, or giga- multipliers for the integer value specified. There is also a small set of developer command line options that accept data passed in directly after the name of the flag without any delineation. The approach depends on the particular command line option and its parsing mechanism. VM Life Cycle The HotSpot VM Runtime is responsible for launching the HotSpot VM and the shut- down of the HotSpot VM. This section provides an overview of what occurs within the HotSpot VM prior to it executing a Java program and what it does when a Java pro- gram terminates or exits. A large amount of detail is presented in this section, perhaps more than necessary for purposes of performance tuning. But it is included to give you sense of the complexity involved in the starting and stopping of a Java application. ptg6882136 60 Chapter 3 JVM Overview The component that starts the HotSpot VM is called the launcher. There are several HotSpot VM launchers. The most commonly used launcher is the java command on Unix/Linux and on Windows the java and javaw commands. It is also possible to launch an embedded JVM through the JNI interface, JNI_Cre- ateJavaVM. In addition, there is also a network-based launcher called javaws, which is used by Web browsers to launch applets. The trailing “ws” on the javaws is often referred to as “web start.” Hence the term “Java web start” for the javaws launcher. The launcher executes a sequence of operations to start the HotSpot VM. These steps are summarized here: 1. Parse command line options. Some of the command line options are consumed immediately by the launcher such as -client or – server, which determines the JIT compiler to load. Other command line options are passed to the launched HotSpot VM. 2. Establish the Java heap sizes and the JIT compiler type (client or server) if these options are not explicitly specified on the command line. If Java heap sizes and JIT compiler are not explicitly specified as a com- mand line option, these are ergonomically established by the launcher. Ergo- nomic defaults vary depending on the underlying system configuration and operating system. Ergonomic choices made by the HotSpot VM are described in more detail in the “HotSpot VM Adaptive Tuning” section later in this chapter. 3. Establish environment variables such as LD_LIBRARY_PATH and CLASSPATH. 4. If the Java Main-Class is not specified on the command line, the launcher fetches the Main-Class name from the JAR’s manifest. 5. Create the HotSpot VM using the standard Java Native Interface method JNI_ CreateJavaVM in a newly created nonprimordial thread. In contrast to a nonprimordial thread, a primordial thread is the first thread allocated by an operating system kernel when a new process is launched. Hence, when a HotSpot VM is launched, the primordial thread is the first thread allocated by the operating system kernel running in the newly created HotSpot VM process. Creating the HotSpot VM in a nonprimordial thread provides the ability to cus- tomize the HotSpot VM such as changing the stack size on Windows. More details of what happens in the HotSpot VM’s implementation of JNI_CreateJavaVM are provided in the “JNI_CreateJavaVM Details” sidebar. 6. Once the HotSpot VM is created and initialized, the Java Main-Class is loaded and the launcher gets the Java main method’s attributes from the Java Main-Class. ptg6882136 HotSpot VM Runtime 61 7. The Java main method is invoked in the HotSpot VM using the Java Native Interface method CallStaticVoidMethod passing it the marshaled arguments from the command line. At this point the HotSpot VM is executing the Java program specified on the com- mand line. Once a Java program, or Java main method completes its execution, the HotSpot VM must check and clear any pending exceptions that may have occurred during the program’s or method’s execution. Additionally, both the method’s exit status and program’s exit status must be passed back to their caller’s. The Java main method is detached from the HotSpot VM using the Java Native Interface method DetachCurrentThread. When the HotSpot VM calls DetachCurrentThread, it decrements the thread count so the Java Native Interface knows when to safely shut down the HotSpot VM and to ensure a thread is not performing operations in the HotSpot VM along with there being no active Java frames on its stack. Specific details of the operations performed by the HotSpot VM’s Java Native Interface method implementation of DestroyJavaVM is described in the “DestroyJavaVM Details” sidebar. JNI_CreateJavaVM Details The HotSpot VM’s implementation of the JNI_CreateJavaVM method performs the following sequence of operations when it is called during the launch of the HotSpot VM. 1. Ensure no two threads call this method at the same time and only one HotSpot VM instance is created in the process. Because the HotSpot VM creates static data structures that cannot be reinitialized, only one HotSpot VM can be created in a process space once a certain point in initialization is reached. To the engineers who develop the HotSpot VM this stage of launching a HotSpot VM is referred to as the “point of no return.” 2. Check to make sure the Java Native Interface version is supported, and the output stream is initialized for garbage collection logging. 3. The OS modules are initialized such as the random number generator, the current process id, high-resolution timer, memory page sizes, and guard pages. Guard pages are no-access memory pages used to bound memory region accesses. For example, often operating systems put a guard page at the top of each thread stack to ensure references off the end of the stack region are trapped. 4. The command line arguments and properties passed in to the JNI_CreateJavaVM method are parsed and stored for later use. 5. The standard Java system properties are initialized, such as java.version, java.vendor, os.name, and so on. ptg6882136 62 Chapter 3 JVM Overview 6. The modules for supporting synchronization, stack, memory, and safepoint pages are initialized. 7. Libraries such as libzip, libhpi, libjava, and libthread are loaded. 8. Signal handlers are initialized and set. 9. The thread library is initialized. 10. The output stream logger is initialized. 11. Agent libraries (hprof, jdi), if any are being used, are initialized and started. 12. The thread states and the thread local storage, which holds thread specific data required for the operation of threads, are initialized. 13. A portion of the HotSpot VM global data is initialized such as the event log, OS synchronization primitives, perfMemory (performance statistics memory), and chunkPool (memory allocator). 14. At this point, the HotSpot VM can create threads. The Java version of the main thread is created and attached to the current operating system thread. However, this thread is not yet added to the known list of threads. 15. Java level synchronization is initialized and enabled. 16. bootclassloader, code cache, interpreter, JIT compiler, Java Native Interface, system dictionary, and universe are initialized. 17. The Java main thread is now added to the known list of threads. The universe, a set of required global data structures, is sanity checked. The HotSpot VMThread, which performs all the HotSpot VM’s critical functions, is created. At this point the appropriate JVMTI events are posted to notify the current state of the HotSpot VM. 18. The following Java classes java.lang.String, java.lang.System, java. lang.Thread, java.lang.ThreadGroup, java.lang.reflect.Method, java.lang.ref.Finalizer, java.lang.Class, and the rest of the Java System classes are loaded and initialized. At this point, the HotSpot VM is initialized and operational, but not quite fully functional. 19. The HotSpot VM’s signal handler thread is started, the JIT compiler is initialized, and the HotSpot’s compile broker thread is started. Other HotSpot VM helper threads such as watcher threads and stat sampler are started. At this time the HotSpot VM is fully functional. 20. Finally, the JNIEnv is populated and returned to the caller and the HotSpot VM is ready to service new JNI requests. DestroyJavaVM Details The DestroyJavaVM method can be called from the HotSpot launcher to shut down the HotSpot VM when errors occur during the HotSpot VM launch sequence. The DestroyJavaVM method can also be called by the HotSpot VM during execution, after the HotSpot VM has been launched, when a very serious error occurs. ptg6882136 HotSpot VM Runtime 63 The shutdown of the HotSpot VM takes the following steps through the DestroyJavaVM method: 1. Wait until there is only one nondaemon thread executing noting that the HotSpot VM is still functional. 2. Call the Java method java.lang.Shutdown.shutdown(), which invokes the Java level shutdown hooks and runs Java object finalizers if finalization-on-exit is true. 3. Prepare for HotSpot VM exit by running HotSpot VM level shutdown hooks (those that were registered through JVM_OnExit()), stop the following HotSpot VM threads: profiler, stat sampler, watcher, and garbage collector threads. Post status events to JVMTI, disable JVMTI, and stop the Signal thread. 4. Call the HotSpot method JavaThread::exit() to release Java Native Interface handle blocks, remove guard pages, and remove the current thread from known threads list. From this point on the HotSpot VM cannot execute any Java code. 5. Stop the HotSpot VM thread. This causes the HotSpot VM to bring the remaining HotSpot VM threads to a safepoint and stop the JIT compiler threads. 6. Disable tracing at the Java Native Interface, HotSpot VM, and JVMTI barriers. 7. Set HotSpot “vm exited” flag for threads that may be running in native code. 8. Delete the current thread. 9. Delete or remove any input/output streams and release PerfMemory (performance statistics memory) resources. 10. Finally return to the caller. VM Class Loading The Hotspot VM supports class loading as defined by the Java Language Speci- fication, Third Edition, [2] the Java Virtual Machine Specification, Second Edi- tion, [1] and as amended by the updated Java Virtual Machine Specification, Chapter 5, Loading, Linking and Initializing. [3] The HotSpot VM and Java SE class loading libraries share the responsibility for class loading. The HotSpot VM is responsible for resolving constant pool symbols, that require loading, linking, and then initializing Java classes and Java interfaces. The term class loading is used to describe the overall process of mapping a class or interface name to a class object, and the more specific terms loading, linking, and initializing for the phases of class loading as defined by the Java Virtual Machine Specification. The most common reason for class loading is during bytecode resolution, when a con- stant pool symbol in a Java classfile requires resolution. Java APIs such as Class. forName(), ClassLoader.loadClass(), reflection APIs, and JNI_FindClass can initiate class loading. The HotSpot VM itself can also initiate class loading. ptg6882136 64 Chapter 3 JVM Overview The HotSpot VM loads core classes such as java.lang.Object and java.lang. Thread along with many others at HotSpot VM startup time. Loading a class requires loading all Java superclasses and all Java superinterfaces. And classfile verification, which is part of the linking phase, can require loading additional classes. The loading phase is a cooperative effort between the HotSpot VM and specific class loaders such as java.lang.ClassLoader. Class Loading Phases For a given Java class or Java interface, the load class phase takes its name, finds the binary in Java classfile format, defines the Java class, and creates a java.lang. Class object to represent that given Java class or Java interface. The load class phase can throw a NoClassDefFound error if a binary representation of a Java class or Java interface cannot be found. In addition, the load class phase does format checking on the syntax of the classfile, which can throw a ClassFormatError or UnsupportedClassVersionError. Before completing the load of a Java class, the HotSpot VM must load all its superclasses and superinterfaces. If the class hierarchy has a problem such that a Java class is its own superclass or superinterface (recur- sively), then the HotSpot VM throws a ClassCircularityError. The HotSpot VM also throws an IncompatibleClassChangeError if the direct superinterface is not an interface, or the direct superclass is an interface. The link phase first does verification, which checks the classfile semantics, checks the constant pool symbols, and does type checking. These checks can throw a VerifyError. Linking then does what is called preparation, which creates and initializes static fields to standard defaults and allocates method tables. It is worth noting at this point of execution no Java code has yet been run. The link class phase then optionally does resolution of symbolic references. Next, class initialization runs the class static initializers, and initializers for static fields. This is the first Java code that runs for this class. It is important to note that class initialization requires superclass initialization, although not superinterface initialization. The Java Virtual Machine Specification specifies that class initialization occurs on the first active use of a class. However, the Java Language Specification allows flex- ibility in when the symbolic resolution step of linking occurs as long as the seman- tics of the language are held; the JVM finishes each step of loading, linking, and initializing before performing the next step; and throws errors when Java programs would expect them to be thrown. As a performance optimization, the HotSpot VM generally waits until class initialization to load and link a class. This means if class A references class B, loading class A will not necessarily cause loading of class B (unless class B is required for verification). Execution of the first instruction that references class B causes the class initialization of B, which requires loading and linking of class B. ptg6882136 HotSpot VM Runtime 65 Class Loader Delegation When a class loader is asked to find and load a class, it can ask another class loader to do the loading. This is called class loader delegation. The first class loader is an initiating class loader, and the class loading that ultimately defines the class is called the defining class loader. In the case of bytecode resolution, the initiating class loader is the class loader for the class whose constant pool symbol is being resolved. Class loaders are defined hierarchically and each class loader has a delegation parent. The delegation defines a search order for binary class representations. The Java SE class loader hierarchy searches the bootstrap class loader, the extension class loader, and the system class loader in that order. The system class loader is the default application class loader, which loads the main Java method and loads classes from the classpath. The application class loader can be a class loader from the Java SE class loader libraries, or it can be provided by an applica- tion developer. The Java SE class loader libraries implement the extension class loader, which loads classes from the lib/ext directory of the JRE (Java Runtime Environment). Bootstrap Class Loader The HotSpot VM implements the bootstrap class loader. The bootstrap class loader loads classes from the HotSpot VM’s BOOTCLASSPATH, including for example rt.jar, which contains the Java SE class libraries. For faster startup, the Client HotSpot VM can also process preloaded classes via a feature called class data shar- ing, which is enabled by default. It can be explicitly enabled with the -Xshare:on HotSpot VM command line switch. Likewise, it can be explicitly disabled with -Xshare:off. As of this writing, the Server HotSpot VM, does not support the class data sharing feature, and class data sharing is also not supported on the Cli- ent HotSpot VM when a garbage collector other than the serial garbage collector is in use. Class data sharing is described in more detail in the “Class Data Sharing” section later in this chapter. Type Safety A Java class or Java interface name is defined as a fully qualified name, which includes the package name. A Java class type is uniquely determined by that fully qualified name and the class loader. In other words, a class loader defines a namespace. This means the same fully qualified class name loaded by two distinctly defined class loaders results in two distinct class types. Given the existence of custom class loaders, the HotSpot VM is responsible for ensuring that non-well-behaved class loaders cannot violate type safety. See Dynamic Class Loading in the Java Virtual Machine, [4] and the Java Virtual Machine Specification 5.3.4 [3] for additional infor- mation. The HotSpot VM ensures that when class A calls B.someMethodName(), ptg6882136 66 Chapter 3 JVM Overview A’s class loader and B’s class loader agree on someMethodName()’s parameters and return type by tracking and checking class loader constraints. Class Metadata in HotSpot Class loading in the HotSpot VM creates an internal representation of a class in either an instanceKlass or an arrayKlass in the HotSpot VM’s permanent generation space. The HotSpot VM’s permanent generation space is described in more detailed in the “HotSpot VM Garbage Collectors” section later in this chapter. The instanceKlass refers to a Java mirror, which is the instance of java.lang.Class mirroring this class. The HotSpot VM internally accesses the instanceKlass using an internal data structure called a klassOop. An “Oop” is an ordinary object pointer. Hence, a klassOop is an internal HotSpot abstraction for a reference, an ordinary object pointer, to a Klass representing or mirroring a Java class. Internal Class Loading Data The HotSpot VM maintains three hash tables to track class loading. The System- Dictionary contains loaded classes, which maps a class name/class loader pair to a klassOop. The SystemDictionary contains both class name/initiating loader pairs and class name/defining loader pairs. Entries are currently only removed at a safepoint. Safepoints are described in more detail in the “VM Operations and Safe- points” section later in the chapter. The PlaceholderTable contains classes that are currently being loaded. It is used for ClassCircularityError checking and for parallel class loading for class loaders that support multithreaded class loading. The LoaderConstraintTable tracks constraints for type safety checking. These hash tables are all guarded by a lock; in the HotSpot VM it is called the System- Dictionary_lock. In general, the load class phase in the HotSpot VM is serialized using the Class loader object lock. Byte Code Verification The Java language is a type-safe language, and standard Java compilers (javac) produce valid classfiles and type-safe code; but a Java Virtual Machine cannot guar- antee that the code was produced by a trustworthy javac compiler. It must reestab- lish type-safety through a process at link time called bytecode verification. Bytecode verification is specified in section 4.8 of the Java Virtual Machine Specification. The specification prescribes both static and dynamic constraints on the code that a Java Virtual Machine verifies. If any violations are found, the Java Virtual Machine throws a VerifyError and prevents the class from being linked. Many of the constraints on bytecodes can be checked statically, such as the operand of an “ldc” bytecode must be a valid constant pool index whose type ptg6882136 HotSpot VM Runtime 67 is CONSTANT_Integer, CONSTANT_String, or CONSTANT_Float. Other con- straints that check the type and number of arguments for other instructions requires dynamic analysis of the code to determine which operands will be pres- ent on the expression stack during execution. There are currently two methods of analyzing bytecodes to determine the types and number of operands present for each instruction. The traditional method is called type inference. It operates by performing an abstract interpretation of each bytecode and merging type states at branch targets or exception handles. The analysis iterates over the bytecode until a steady state for the types is found. If a steady state cannot be found, or if the resulting types violate some bytecode constraint, then a VerifyError is thrown. The code for this verification step is present in the HotSpot VM’s libverify. so external library, and uses JNI to gather whatever information is needed for classes and types. New in the Java 6 HotSpot VMs is a second method for verification called type verification. In this approach the Java compiler provides the steady-state type infor- mation for each branch target or exception target, via the code attribute, Stack- MapTable. The StackMapTable consists of a number of stack map frames; each indicates the types of the items on the expression stack and in the local variables at some offset in the method. The Java Virtual Machine needs to then only perform one pass through the bytecode to verify the correctness of the types to verify the bytecode. This verification approach is faster and smaller than the traditional type inference approach for bytecode verification approach. For all classfiles with a version number less than 50, such as those created prior to Java 6, the HotSpot VM uses the traditional type inference method to verify the classfiles. For classfiles greater than or equal to 50, the StackMapTable attri- butes are present and the new “type verification” verifier is used. Because of the possibility of older external tools that might instrument the bytecode but neglect to update the StackMapTable attribute, certain verification errors that occur dur- ing type-checking verification may failover to the type inference method. Should this type inference verification pass fail, only then will the HotSpot VM throw a VerifyError. Class Data Sharing Class data sharing is a feature introduced in Java 5 that was intended to reduce the startup time for Java applications, in particular small Java applications, as well as reduce their memory footprint. When the Java Runtime Environment (JRE) is installed on 32-bit platforms using the Java HotSpot JRE provided installer, the installer loads a set of classes from the system jar file into a private internal rep- resentation, and dumps that representation to a file, called a shared archive. If the ptg6882136 68 Chapter 3 JVM Overview Java HotSpot JRE installer is not being used, this can be done manually. During subsequent Java Virtual Machine invocations, the shared archive is memory-mapped into the JVM, which saves the cost of loading those classes and allowing much of the JVM’s metadata for these classes to be shared among multiple JVM processes. Tip As of the writing of this chapter (Java 6 Update 21), class data sharing is supported only with the HotSpot Client VM, and only with the serial garbage collector. The primary motivation for the class data sharing feature is the decrease in startup time it provides. Class data sharing produces better results for smaller appli- cations because it eliminates a fixed cost of loading certain Java SE core classes. The smaller the application relative to the number of Java SE core classes it uses, the larger the saved fraction of startup time. With class data sharing, the memory footprint cost of new JVM instances is reduced in two ways. First, a portion of the shared archive, currently between five and six megabytes of space, is memory mapped read-only and therefore shared among multiple JVM processes. Previously this data was replicated in each JVM instance. Second, since the shared archive contains class data in the form in which the Hotspot VM uses it, the memory that would otherwise be required to access the original class information in the Java SE core libraries jar file, rt.jar, is not needed. These savings allow more applications to be run concurrently on the same machine. On Microsoft Windows, the footprint of a process, as measured by various tools, may appear to increase, because a larger number of pages are being mapped into the pro- cess address space. This is offset by the reduction in the amount of memory (inside Windows) that is needed to hold portions of the Java SE library jar file rt.jar. Reduc- ing memory footprint in the HotSpot VM remains a high priority. In the HotSpot VM, the class data sharing implementation introduces new Java subspaces into the permanent generation space that contains the shared data. The classes.jsa shared archive is memory mapped into these spaces in permanent gen- eration at HotSpot VM startup time. Subsequently, the shared region is managed by the existing HotSpot VM memory management subsystem. Read-only shared data, which is one of the new subspaces in permanent generation includes constant method objects, symbol objects and arrays of primitives, mostly character arrays. Read-write shared data, the other new Java heap space introduced in permanent generation, consists of mutable method objects, constant pool objects, HotSpot VM internal rep- resentation of Java classes and arrays, and various String, Class, and Exception objects. ptg6882136 HotSpot VM Runtime 69 Interpreter The HotSpot VM interpreter is a template based interpreter. The HotSpot VM Runtime generates the interpreter in memory at JVM startup using information stored internally in a data structure called a TemplateTable. The TemplateTable contains machine dependent code corresponding to each bytecode. A template is a description of each bytecode. The HotSpot VM’s TemplateTable defines all the tem- plates and provides accessor functions to get the template for a given bytecode. The template table generated in memory can be viewed using what is called a HotSpot “debug” VM and the nonproduct flag -XX:+PrintInterpreter. Tip A HotSpot debug VM is a version of the HotSpot VM that contains additional debugging information and additional HotSpot VM command line options that can be used together to debug, or further instrument the HotSpot VM. Its use is not recommended for production environments. The template design of the HotSpot VM interpreter performs better than a classic switch statement loop approach. For example, a switch statement approach must per- form repeated compare operations. In the worst case, the switch statement approach may be required to compare a given command with all but one bytecodes to locate the required one. Additionally, the switch statement approach must use a separate soft- ware stack to pass Java arguments. The HotSpot VM uses the native C stack to pass Java arguments. A number of HotSpot VM internal variables, such as the program counter or the stack pointer for a Java thread, are stored in C variables, that are not guaranteed to be always kept in underlying hardware registers. As a result, the man- agement of these software interpreter data structures consumes a considerable share of total execution time. [5] Overall, the performance gap between the HotSpot VM and the real machine is significantly narrowed by the HotSpot interpreter, which makes the interpretation speed considerably higher. However, this comes at a price of large amounts of machine-specific code. For example, approximately 10,000 lines of code are dedicated to Intel x86 platforms, and about 14,000 lines of code are dedicated to SPARC platforms. The overall code size and complexity are also significantly higher, since the code supporting dynamic code generation (JIT compilation) is needed. Obvi- ously, debugging dynamically generated machine code (JIT compiled code) is much more difficult than debugging static code. These properties certainly do not facilitate implementation of runtime evolution, but they do not make it infeasible either. [5] There are interpreter calls out to the HotSpot VM Runtime for complex opera- tions, which are essentially anything too complex or complicated to do in assembly language such as constant pool lookup. ptg6882136 70 Chapter 3 JVM Overview The HotSpot VM interpreter is also a critical part of the overall HotSpot VM adap- tive optimization story. Adaptive optimization solves the problems of JIT compila- tion by taking advantage of an interesting program property. Virtually all programs spend the vast majority of their time executing a minority of their code. Rather than compiling method by method, “just in time” or “ahead of time,” the HotSpot VM immediately runs the program using an interpreter, and analyzes the code as it runs to detect the critical hot spots in the program. Then it focuses the attention of a global machine code optimizer on those hot spots. By avoiding compilation of infrequently executed code the HotSpot VM JIT compiler can devote more attention to the performance-critical parts of the program, without necessarily increasing the overall compilation time. Tip The term JIT compiler is not very descriptive for how the HotSpot VM utilizes a compiler to generate optimized machine dependent code. The HotSpot VM actually generates machine code dynamically as it observes a program’s behavior rather than compiling it “just in time” or “ahead of time.” This hot spot monitoring is continued dynamically as the program runs, so that it lit- erally adapts its performance on the fly to the program’s execution and the user’s needs. Exception Handling Java Virtual Machines use exceptions to signal that a program has violated the semantic constraints of the Java language. For example, an attempt to index outside the bounds of an array causes an exception. An exception causes a nonlocal transfer of control from the point where the exception occurred, or was thrown, to a point specified by the programmer, or where the exception is caught. [6] The HotSpot VM interpreter, its JIT compilers, and other HotSpot VM components all cooperate to implement exception handling. There are two general cases of exception handling; either the exception is thrown or caught in the same method, or it is caught by a caller. The latter case is more complicated and requires stack unwinding to find the appropriate handler. Exceptions can be initiated by the thrown bytecode, a return from a VM-internal call, a return from a JNI call, or a return from a Java call. The last case is simply just a later stage of the first three. When the VM recognizes that an exception has been thrown, the HotSpot VM Runtime system is invoked to find the nearest handler for that exception. Three pieces of information are used to find the handler: the current method, the current bytecode, and the exception object. If a handler is not found in the current method, as mentioned previously, the current ptg6882136 HotSpot VM Runtime 71 activation stack frame is popped and the process is iteratively repeated for previous frames. Once the correct handler is found, the HotSpot VM execution state is updated, and the HotSpot VM jumps to the handler as Java code execution is resumed. Synchronization Broadly, synchronization is described as a mechanism that prevents, avoids, or recovers from the inopportune interleavings, commonly called races, of concurrent operations. In Java, concurrency is expressed through the thread construct. Mutual exclusion is a special case of synchronization where at most a single thread is per- mitted access to protected code or data. The HotSpot VM provides Java monitors by which threads running application code can participate in a mutual exclusion pro- tocol. A Java monitor is either locked or unlocked, and only one thread may own the monitor at any one time. Only after acquiring ownership of a monitor may a thread enter a critical section protected by the monitor. In Java, critical sections are referred to as synchronized blocks and are delineated in code by the synchronized statement. If a thread attempts to lock a monitor and the monitor is in an unlocked state, the thread immediately gains ownership of the monitor. If a subsequent second thread attempts to gain ownership of the monitor while the monitor is locked that second thread will not be permitted to proceed into the critical section until the owner releases the lock and the second thread manages to gain (or is granted) exclusive ownership of the lock. For clarification, to enter a monitor means to acquire exclusive ownership of the monitor and enter the associated critical section. Likewise, to exit a monitor means to release ownership of the monitor and exit the critical section. Additionally, a thread that has locked a monitor, owns that moni- tor. Uncontended refers to synchronization operations on an otherwise unowned monitor by only a single thread. The HotSpot VM incorporates leading-edge techniques for both uncontended and contended synchronization operations, which boost synchronization performance by a large factor. Uncontended synchronization operations, which comprise the majority of synchronizations, are implemented with constant time techniques. With biased lock- ing, a feature introduced in Java 5 HotSpot VMs with the -XX:+UseBiasedLocking command line option, in the best case these operations are essentially free of cost. Since most objects are locked by at most one thread during their lifetime, enabling -XX:+UseBiasedLocking allows that thread to bias the lock toward itself. Once biased, that thread can subsequently lock and unlock the object without resorting to expensive atomic instructions. [7] Contended synchronization operations use advanced adaptive spinning techniques to improve throughput even for applications with significant amounts of lock con- tention. As a result, synchronization performance becomes so fast that it is not a significant performance issue for the vast majority of real-world programs. ptg6882136 72 Chapter 3 JVM Overview In the HotSpot VM, most synchronization is handled through what is called fast- path code. The HotSpot VM has two JIT compilers and an interpreter, all of which will emit fast-path code. To HotSpot engineers, the JIT compilers are known as “C1” —the -client JIT compiler—and “C2”—the -server JIT compiler. C1 and C2 both emit fast-path code directly at the synchronization site. In the normal case when there is no contention, the synchronization operation will be completed entirely in fast-path code. If, however, there is a need to block or wake a thread (in monitor-enter or monitor-exit state, respectively), the fast-path code will call into the slow-path code. The slow-path implementation is C++ code, while fast-path code is machine dependent code emitted by the JIT compilers. Java object synchronization state is encoded for every Java object internally within the HotSpot VM’s object representation of that Java object in the first word, often referred to as the mark word. For several states, the mark word is multiplexed to point to additional synchronization metadata. The possible Java object synchroniza- tion states stored in HotSpot VM’s mark word are Neutral. Unlocked. Biased. Locked/Unlocked + Unshared. Stack-Locked. Locked + Shared but uncontended. Shared means the mark points to a displaced mark word on the owner thread’s stack. Inflated. Locked/Unlocked + Shared and contended. Threads are blocked in monitorenter or wait(). The mark points to a heavyweight “objectmonitor” structure. As a side note, the mark word is also multiplexed to contain the garbage collector’s object age data, and the object’s identity hash code value. Thread Management Thread management covers all aspects of the thread life cycle, from creation through termination along with the coordination of threads within the HotSpot VM. This involves management of threads created from Java code, regardless of whether they are created from application code or library code, native threads that attach directly to the HotSpot VM, or internal HotSpot VM threads created for other purposes. While the broader aspects of thread management are platform independent, the details vary depending on the underlying operating system. Threading Model The threading model in the Hotspot VM is a one-to-one mapping between Java threads, an instance of java.lang.Thread, and native operating system threads. A native operating system thread is created when a Java thread is started and is ptg6882136 HotSpot VM Runtime 73 reclaimed once it terminates. The operating system is responsible for scheduling all threads and dispatching them to an available CPU. The relationship between Java thread priorities and operating system thread priorities is a complex one that varies across systems. Thread Creation and Destruction There are two ways for a thread to be introduced in the HotSpot VM; either by exe- cuting Java code that calls the start() method on a java.lang.Thread object, or by attaching an existing native thread to the HotSpot VM using JNI. Other threads created by the HotSpot VM for internal use are discussed later. Internally to the HotSpot VM there are a number of objects, both C++ and Java, associated with a given thread in the HotSpot VM. These objects, both Java and C++, are as follows: A java.lang.Thread instance that represents a thread in Java code. A C++ JavaThread instance that represents the java.lang.Thread instance internally within the HotSpot VM. It contains additional information to track the state of the thread. A JavaThread holds a reference to its associated java. lang.Thread object, as an ordinary object pointer, and the java.lang.Thread object also stores a reference to its JavaThread as a raw int. A JavaThread also holds a reference to its associated OSThread instance. An OSThread instance represents an operating system thread and contains additional operating-system-level information needed to track thread state. The OSThread also contains a platform specific “handle” to identify the actual thread to the operating system. When a java.lang.Thread is started the HotSpot VM creates the associated JavaThread and OSThread objects, and ultimately the native thread. After prepar- ing all the HotSpot VM state, such as thread-local storage and allocation buffers, synchronization objects and so forth, the native thread is started. The native thread completes initialization and then executes a startup method that leads to the execu- tion of the java.lang.Thread object’s run() method, and then, upon its return, terminates the thread after dealing with any uncaught exceptions, and interacting with the HotSpot VM to check whether termination of this thread requires termina- tion of the entire HotSpot VM. Thread termination releases all allocated resources, removes the JavaThread from the set of known threads, invokes destructors for the OSThread and JavaThread, and ultimately ceases execution when its initial startup method completes. A native thread attaches to the HotSpot VM using the JNI call AttachCurrent- Thread. In response to this an associated OSThread and JavaThread instance is ptg6882136 74 Chapter 3 JVM Overview created, and basic initialization is performed. Next a java.lang.Thread object must be created for the attached thread; this is done by reflectively invoking the Java code for the Thread class constructor, based on the arguments supplied when the thread attached. Once attached, a thread can invoke whatever Java code it needs via other JNI methods. Finally, when the native thread no longer wishes to be involved with the HotSpot VM it can call the JNI DetachCurrentThread method to disas- sociate it from the HotSpot VM by releasing resources, dropping the reference to the java.lang.Thread instance, destructing the JavaThread and OSThread objects, and so on. A special case of attaching a native thread is the initial creation of the HotSpot VM via the JNI CreateJavaVM call, which can be done by a native application or by the HotSpot VM launcher. This causes a range of initialization operations to take place and then acts effectively as if a call to AttachCurrentThread was made. The thread can then invoke Java code as needed, such as reflective invocation of the Java main method of an application. See the “Java Native Interface” section later in the chapter for further details. Thread States The HotSpot VM uses a number of different internal thread states to characterize what each thread is doing. This is necessary both for coordinating the interactions of threads and for providing useful debugging information if things go wrong. A thread’s state transitions as different actions are performed, and these transition points are used to check that it is appropriate for a thread to proceed with the requested action at that point in time; see the discussion of safepoints for details. From the HotSpot VM perspective the possible states of the main thread are New thread. A new thread in the process of being initialized Thread in Java. A thread that is executing Java code Thread in vm. A thread that is executing inside the HotSpot VM Blocked thread. The thread is blocked for some reason (acquiring a lock, waiting for a condition, sleeping, performing a blocking I/O operation, and so on) For debugging purposes additional state information is also maintained for report- ing by tools, in thread dumps, stack traces, and so on. This is maintained in the internal HotSpot C++ object OSThread. Thread states reported by tools, in thread dumps, stack traces, and so on, include MONITOR_WAIT. A thread is waiting to acquire a contended monitor lock. CONDVAR_WAIT. A thread is waiting on an internal condition variable used by the HotSpot VM (not associated with any Java object). ptg6882136 HotSpot VM Runtime 75 OBJECT_WAIT. A Java thread is performing a java.lang.Object.wait() call. Other HotSpot VM subsystems and libraries impose their own thread state informa- tion, such as the JVMTI system and the thread state exposed by the java.lang. Thread class itself. Such information is generally not accessible to, nor relevant to, the management of threads inside the HotSpot VM. Internal VM Threads Much to the surprise of many, the executing of a trivial “Hello World” Java program can result in the creation of a dozen or more threads in the HotSpot VM. These arise from a combination of internal HotSpot VM threads and HotSpot VM library related threads such as the reference handler and finalizer threads. The internal HotSpot VM threads are VM thread. A singleton C++ object instance that is responsible for executing VM operations. VM operations are further discussed in the next subsection. Periodic task thread. A singleton C++ object instance, also called the Watch- erThread, simulates timer interrupts for executing periodic operations within the HotSpot VM. Garbage collection threads. These threads, of different types, support the serial, parallel, and concurrent garbage collection. JIT compiler threads. These threads perform runtime compilation of byte- code to machine code. Signal dispatcher thread. This thread waits for process directed signals and dispatches them to a Java level signal handling method. All these threads are instances of the internal HotSpot C++ Thread class, and all threads that execute Java code are internal HotSpot C++ JavaThread instances. The HotSpot VM internally keeps track of all threads in a linked-list known as the Threads_list and is protected by the Threads_lock—one of the key synchroni- zation locks used within the HotSpot VM. VM Operations and Safepoints The internal HotSpot VM VMThread spends its time waiting for operations to appear in a C++ object called VMOperationQueue and executing those operations. Typi- cally these operations are passed on to the VMThread because they require that the HotSpot VM reach what is called a safepoint before they can be executed. In simple terms, when the HotSpot VM is at a safepoint all Java executing threads are blocked, and any threads executing in native code are prevented from returning to Java code while the safepoint is in progress. This means that a HotSpot VM operation can be ptg6882136 76 Chapter 3 JVM Overview executed knowing that no thread can be in the middle of modifying the Java heap, and all threads are in a state where their Java stacks are not changing and can be examined. The most familiar HotSpot VM safepoint operation is to support garbage collec- tion, or more specifically stop-the-world phases of garbage collection. Tip “Stop-the-world” in the context of garbage collection means that all Java executing threads are blocked or stopped from executing in Java code while the garbage collector frees up memory as a result of finding Java objects no longer in use by the application. If an application thread is executing in native code (i.e., JNI), it is allowed to continue, but will block if it attempts to cross the native boundary into Java code. There many other safepoints, such as biased locking revocation, thread stack dumps, thread suspension or stopping (i.e., java.lang.Thread.stop() method), and numerous inspection and modification operations requested through JVMTI. Many HotSpot VM operations are synchronous, that is, the requester blocks until the operation has completed, but some are asynchronous or concurrent, meaning that the requester can proceed in parallel with the VMThread (assuming no safepoint is initiated). Safepoints are initiated using a cooperative, polling-based mechanism. In simplis- tic terms, every so often a thread asks “should I block for a safepoint?” Asking this question efficiently is not so simple. One place where the question is often asked is during a thread state transition. Not all state transitions do this, for example, a thread leaving the HotSpot VM to go to native code, but many do. Another place where a thread asks, “should I block for a safepoint?” is when JIT compiled code is returning from a method or at certain stages during loop iteration. Threads executing interpreted code do not usually ask whether they should block for a safepoint. Instead the safepoint is requested when the interpreter switches to a different dispatch table. Included as part of the switching operation is code that asks when the safepoint is over. When the safepoint is over, the dispatch table is switched back again. Once a safepoint has been requested, the VMThread must wait until all threads are known to be in a safepoint-safe state before proceeding to execute a VM operation. During a safepoint the Threads_lock is used to block any threads that are running. The VMThread releases the Threads_lock after the VM operation has been performed. C++ Heap Management In addition to HotSpot VM’s Java heap, which is maintained by the HotSpot VM’s memory manager and garbage collectors, the HotSpot VM also uses a C/C++ heap for storage of HotSpot VM internal objects and data. Within the HotSpot VM and ptg6882136 HotSpot VM Runtime 77 not exposed to a user of the HotSpot VM, a set of C++ classes derived from a base class called Arena is used to manage the HotSpot VM C++ heap operations. The Arena base class and its subclasses provide a rapid C/C++ allocation layer that sits on top of the C/C++ malloc/free memory management routines. Each Arena allocates memory blocks (internally the HotSpot VM refers to them as Chunks) from three global ChunkPools. Each ChunkPool satisfies allocation requests for a dis- tinct range of allocation sizes. For example, an allocation request for 1K of memory is allocated from the “small” ChunkPool, while a 10K allocation request is made from the “medium” ChunkPool. This is done to avoid wasteful memory fragmenta- tion. The Arena approach for allocating memory provides better performance than directly using the C/C++ malloc/free memory management routines. The latter operations may require acquisition of global OS locks, which can affect scalability and impact performance. Arenas are thread-local objects that cache a certain amount of memory storage. This allows for fast-path allocation where a global shared lock is not required. Likewise, Arena free operations do not require a lock in the common uses of releasing memory back to the Chunks. Arenas are also used for thread-local resource management implemented internally within the HotSpot VM as a C++ object called ResourceArea. Arenas are additionally used for handle manage- ment implemented internally within the HotSpot VM as a C++ HandleArea object. Both the HotSpot client and server JIT compilers use Arenas during JIT compilation. Java Native Interface The Java Native Interface, referred to as JNI hereafter, is a native programming interface. It allows Java code that runs inside a Java Virtual Machine to interoper- ate with applications and libraries written in other programming languages, such as C, C++, and assembly language. Although applications can be written entirely in Java, there are circumstances where Java alone does not meet the requirements of an application. Programmers can use JNI to write native methods to handle those situations when an application cannot be written entirely in Java. JNI native methods can be used to create, inspect, and update Java objects, call Java methods, catch and throw exceptions, load classes and obtain class information, and perform runtime type checking. JNI may also be used with the Invocation API to enable an arbitrary native application to embed the Java VM. This allows program- mers to easily make their existing applications Java-enabled without having to link with the VM source code. [8] It is important to remember that once an application uses JNI, it risks losing two benefits of the Java platform. First, Java applications that depend on JNI can no longer readily run on multiple heterogeneous hardware platforms. Even though ptg6882136 78 Chapter 3 JVM Overview the part of an application written in the Java programming language is portable to multiple heterogeneous hardware platforms, it is necessary to recompile the part of the application written in native programming languages. In other words, using JNI loses one of the Java promises, “write once, run anywhere.” Second, the Java programming language is type-safe and secure; native languages such as C or C++ are not. As a result, Java developers must use extra care when writing applications using JNI. A misbehaving native method can corrupt an entire application. For this reason, Java applications are subject to security checks before invoking JNI meth- ods. The additional security checks and the copying of data between the Java layer and JNI layer within the HotSpot VM can infringe on an application’s performance. Tip As a general rule, developers should architect their application so that native methods are defined in as few classes as possible. This entails a cleaner isolation between native code and the rest of the application. [9] The HotSpot VM provides a command line option to aid in debugging problems with native methods using JNI called -Xcheck:jni. Specifying -Xcheck:jni causes an alternate set of debugging interfaces to be used by an application’s JNI calls. The alternate interface verifies arguments to JNI calls more stringently, as well as performing additional internal consistency checks. Internally to the HotSpot VM, the implementation of JNI functions is straightfor- ward. It uses various HotSpot VM internal primitives to perform activities such as object creation, method invocation, and so on. In general, these are the same runtime primitives used by other HotSpot VM subsystems such as the interpreter described earlier in this chapter. The HotSpot VM must take special care to keep track of which threads are cur- rently executing in native methods. During some HotSpot VM activities, most notably some phases of garbage collection, one or more threads must be halted at a safepoint to guarantee that the Java memory heap is not modified to ensure garbage collection accuracy. When the HotSpot VM wants to bring a thread executing in native code to a safepoint, that thread is allowed to continue executing in native code until it attempts to either return into Java code or makes a JNI call. VM Fatal Error Handling The designers of the HotSpot VM believe it is important to provide sufficient informa- tion to its users and developers to diagnose and fix VM fatal errors. A common VM fatal error is an OutOfMemoryError. Another common fatal error on Solaris and Linux platforms is a segmentation fault. The equivalent error on Windows is called ptg6882136 HotSpot VM Runtime 79 Access Violation error. When these fatal errors occur, it is critical to understand the root cause to fix them. Sometimes the resolution to the root cause requires a change in a Java application, and sometimes the root cause is within the HotSpot VM. When the HotSpot VM crashes on a fatal error, it dumps a HotSpot error log file called hs_err_pid.log where is replaced with the process id of the crashed HotSpot VM. The hs_err_pid.log file is created in the directory where HotSpot VM was launched. Since this feature’s initial introduction in HotSpot VM 1.4.2 version, many enhancements have been made to improve the diagnosability of the root cause of a fatal error. These additional enhancements include A memory map is included in the hs_err_pid.log error log file to make it is easy to see how memory is laid out during the VM crash. A -XX:ErrorFile command line option is provided so you can set the path name of the hs_err_pid.log error log file. An OutOfMemoryError also triggers the hs_err_pid.log file to be generated. An additional popular feature often used to diagnose the root cause of a VM fatal error is using the HotSpot VM command line option -XX:OnError=cmd1 args...; com2 ... . This HotSpot VM command line option executes the list of commands given to -XX:OnError whenever the HotSpot VM crashes. A common use of this fea- ture is invoking a debugger such as Linux/Solaris dbx or Windows Winddbg to imme- diately examine the crash. For releases that do not have support for -XX:OnError, an alternative HotSpot VM command line option can be used called -XX:+Show MessageBoxOnError. This option stops the VM before it exits by displaying a dia- log box saying the VM has experienced a fatal error. This provides an opportunity to attach to the HotSpot VM with a debugger prior to it exiting. When the HotSpot VM experiences a fatal error, it internally uses a class called VMError to aggregate and dump the hs_err_pid.log file. The VMError class is invoked by operating specific code when an unrecognized signal or exception is observed. Tip The HotSpot VM uses signals internally for communication. The fatal error handler in the HotSpot VM is invoked when a signal is not recognized. This unrecognized case may originate from a fault in application JNI code, OS native libraries, JRE native libraries, or the HotSpot VM itself. The HotSpot VM’s fatal error handler had to be carefully written to avoid causing faults itself as a result of fatal errors such as StackOverflow or fatal errors when critical locks are held such as a malloc lock. ptg6882136 80 Chapter 3 JVM Overview Since an OutOfMemoryError is possible to experience, especially on some large scale applications, it is critical to provide useful diagnostic information to users so a resolution can be quickly identified. Often it can be resolved by just simply specifying a larger Java heap size. When an OutOfMemoryError happens, the error message indicates which type of memory is problematic. For example, it could be a result of a Java heap space or permanent generation space being specified as too small. Begin- ning with Java 6, a stack trace is included in the error message produced by the HotSpot VM. Also, the -XX:OnOutOfMemoryError= option was introduced so a command can be run when the first OutOfMemoryError is thrown. An additional useful feature worth mentioning is being able to generate a heap dump on an Out OfMemoryError. This can be enabled by specifying -XX:+HeapDumpOnOutOfMemory Error HotSpot VM command line option. There is an additional HotSpot VM com- mand line option that allows a user to specify a path where the heap dump will be placed, -XX:HeapDumpPath=. Although applications are written with the intent to avoid thread deadlocks, devel- opers sometimes make mistakes and deadlocks occur. When a deadlock occurs, doing a Ctrl + Break on Windows forces a Java level thread stack trace to print to standard output. On Solaris and Linux, sending a SIGQUIT signal to the Java process id does the same. With a thread stack trace, the source of the deadlock can be analyzed. Beginning with Java 6, the bundled JConsole tool added the capability to attach to a hung Java process and analyze the root cause of the deadlock. Most of the time, a deadlock is caused by acquiring locks in the wrong order. Tip The “Trouble-Shooting and Diagnostic Guide” [10] for Java 5 contains a lot of information that may be useful to diagnosing fatal errors. HotSpot VM Garbage Collectors “Heap storage for objects is reclaimed by an automatic storage management system (typically a garbage collector); objects are never explicitly de-allocated.” —Java Virtual Machine Specification [1] The Java Virtual Machine (JVM) specification dictates that any JVM implementa- tion must include a garbage collector to reclaim unused memory (i.e., unreachable objects).[1] The behavior and efficiency of the garbage collector used can heavily influ- ence the performance and responsiveness of an application that’s taking advantage of ptg6882136 HotSpot VM Garbage Collectors 81 it. This section gives an introduction to the garbage collectors included in the HotSpot VM. The aim is to gain a better understanding of how garbage collection works in the HotSpot VM and, as a result, be able to take full advantage of it when designing, developing, and deploying applications. Generational Garbage Collection The HotSpot VM uses a generational garbage collector, [11] a well-known garbage collection approach that relies on the following two observations: Most allocated objects become unreachable quickly. Few references from older to younger objects exist. These two observations are collectively known as the weak generational hypoth- esis, which generally holds true for Java applications. To take advantage of this hypothesis, the HotSpot VM splits the heap into two physical areas (also called spaces), which are referred to as generations: The young generation. Most newly allocated objects are allocated in the young generation (see Figure 3-2), which, relatively to the Java heap, is typi- cally small and collected frequently. Since most objects in it are expected to become unreachable quickly, the number of objects that survive a young gen- eration collection (also referred to as a minor garbage collection) is expected to be low. In general, minor garbage collections are efficient because they concen- trate on a space that is usually small and is likely to contain a lot of garbage objects. The old generation. Objects that are longer-lived are eventually promoted, or tenured, to the old generation (see Figure 3-2). This generation is typically larger than the young generation, and its occupancy grows more slowly. As a result, old generation collections (also referred to as major garbage collections, or full garbage collections) are infrequent, but when they do occur they can be quite lengthy. The permanent generation. This is a third area in the HotSpot VM’s mem- ory layout, and it is also shown in Figure 3-2. Even though it is also referred to as a generation, it should not be seen as part of the generation hierarchy (i.e., user-allocated objects do not eventually move from the old generation to the permanent generation). Instead, it is only used by the HotSpot VM itself to hold metadata, such as class data structures, interned strings, and so on. ptg6882136 82 Chapter 3 JVM Overview Figure 3-2 HotSpot VM generational spaces Young Generation Old Generation Permanent Generation Allocation Promotion To keep minor garbage collections short, the garbage collector must be able to identify live objects in the young generation without having to scan the entire (and potentially larger) old generation. To achieve this, the garbage collectors in the Hot- Spot VM use a data structure called a card table. [11] The old generation is split into 512-byte chunks called cards. The card table is an array with one byte entry per card in the heap. Every update to a reference field of an object must also ensure that the card containing the updated reference field is marked dirty by setting its entry in the card table to the appropriate value. During a minor garbage collection, only the areas that correspond to dirty cards are scanned to potentially discover old-to-young generation references (see Figure 3-3). Figure 3-3 Garbage collector interaction with the card table Young Generation Old Generation Card Table ptg6882136 HotSpot VM Garbage Collectors 83 In cooperation with the bytecode interpreter and the JIT compiler, the HotSpot VM uses a write barrier [11] to maintain the card table. This barrier is a small fragment of code that sets an entry of the card table to the dirty value. The interpreter executes a write barrier every time it executes a bytecode that updates a reference field. Addi- tionally, the JIT compiler emits the write barrier after emitting the code that updates a reference field. Although write barriers do impose a small performance overhead on the application threads, their use allows for much faster minor garbage collections, and much higher overall garbage collector efficiency, which typically improves the throughput of an application. Tip The bytecode interpreter is considered part of the HotSpot VM Runtime. Additional information on the HotSpot VM Runtime can be found in the “HotSpot VM Runtime” section earlier in this chapter. Likewise, additional information on the HotSpot JIT compiler can be found in the “HotSpot VM JIT Compilers” section later in this chapter. A big advantage of generational garbage collection is that each generation can be managed by the garbage collection algorithm most appropriate for its characteristics. A fast garbage collector usually manages the young generation, as minor garbage collections are frequent. This garbage collector might be a little space wasteful, but since the young generation typically is a small portion of the Java heap, this is not a big problem. On the other hand, a garbage collector that is space efficient usually manages the old generation, as the old generation takes up most of the Java heap. This garbage collector might not be quite as fast, but because full garbage collections are infrequent, it doesn’t have a big performance impact. To take full advantage of generational garbage collection, applications should conform to the weak generational hypothesis, as it is what generational garbage col- lection exploits. For the Java applications that do not do so, a generational garbage collector might add more overhead. In practice, however, such applications are rare. The Young Generation Figure 3-4 illustrates the layout of the young generation of the HotSpot VM (the spaces are not drawn to proportion). It is split into three separate areas (or spaces): The eden. This is where most new objects are allocated (not all, as large objects may be allocated directly into the old generation). The eden is almost always empty after a minor garbage collection. A case where it may not be empty is described in Chapter 7, “Tuning the JVM, Step By Step.” ptg6882136 84 Chapter 3 JVM Overview Figure 3-4 Eden and survivor spaces of young generation Eden Survivor Spaces Young Generation Old Generation From To Unused The two survivor spaces. These hold objects that have survived at least one minor garbage collection but have been given another chance to become unreachable before being promoted to the old generation. As illustrated in Figure 3-4, only one of them holds objects, while the other is most of the time unused. Figure 3-5 illustrates the operation of a minor garbage collection. Objects that have been found to be garbage are marked with a gray X. As seen in Figure 3-5a, live objects in the eden that survive the collection are copied to the unused survivor space. Live objects in the survivor space that is in use, which will be given another chance to be reclaimed in the young generation, are also copied to the unused survi- vor space. Finally, live objects in the survivor space that is in use, that are deemed “old enough,” are promoted to the old generation. Figure 3-5 Minor garbage collection illustration Eden (a) (b) Survivor Spaces Young Generation Old Generation From To Unused EdenYoung Generation Old Generation To From Survivor Spaces Unused Empty ptg6882136 HotSpot VM Garbage Collectors 85 At the end of the minor garbage collection, the two survivor spaces swap roles (see Figure 3-5b). The eden is entirely empty; only one survivor space is in use; and the occupancy of the old generation has grown slightly. Because live objects are copied dur- ing its operation, this type of garbage collector is called a copying garbage collector. [11] It should be pointed out that, during a minor garbage collection, there is no guar- antee that the allocating survivor space will always be large enough to accommodate the surviving objects from both the eden and the other survivor space. If it overflows, the rest of the objects that need to be evacuated will be moved to the old generation. This is referred to as premature promotion. It causes the old generation to grow with potentially short-lived objects, and it can potentially be a serious performance issue. Further, if during a minor garbage collection the old generation becomes full and it is not possible to copy more objects into it, that minor garbage collection is typically fol- lowed by a full garbage collection, which collects the entire Java heap. This is referred to as promotion failure. Careful user-tuning, as well as some self-tuning done by the garbage collectors, typically makes the likelihood of either those two undesirable events very low. Tuning the HotSpot VM is the subject matter found in Chapter 7. Fast Allocation The operation of the object allocator is tightly coupled with the operation of the gar- bage collector. The garbage collector has to record where in the heap the free space it reclaims is located. In turn, the allocator needs to discover where the free space in the heap is before it can reuse it to satisfy allocation requests. The copying garbage collector that collects the young generation of the HotSpot VM has the advantage of always leaving the eden empty. That allows allocations into the eden to be efficient by using what’s referred to as the bump-the-pointer technique. According to this technique, the end of the last allocated object is tracked (this is usually referred to as top), and when a new allocation request needs to be satisfied, the allocator needs only to check whether it will fit between top and the end of the eden. If it does, top is bumped to the end of the newly allocated object. Additionally, most interesting Java applications are multithreaded, and their alloca- tion operations need to be multithreaded safe. If they simply used global locks to ensure this, then allocation into eden would become a bottleneck and degrade performance. Instead, the HotSpot VM has adopted a technique called Thread-Local Allocation Buf- fers (TLABs), which improves multithreaded allocation throughput by giving each thread its own buffer (i.e., a small chunk of the eden) from which to allocate. Since only one thread can be allocating into each TLAB, allocation can take place quickly with the bump-the-pointer technique and without any locking. However, when a thread fills up its TLAB and needs to get a new one (an infrequent operation), it needs to do so in a multithreaded safe way. In the HotSpot VM, the new Object() operation is, most of the time, around ten assembly code instructions. It is the operation of the garbage collector, which empties the eden space, that enables this fast allocation. ptg6882136 86 Chapter 3 JVM Overview Garbage Collectors: Spoiled for Choice “The Java Virtual Machine assumes no particular type of automatic storage management system, and the storage management technique may be chosen according to the implementor’s system requirements.” —Java Virtual Machine Specification [1] The HotSpot VM has three different garbage collectors, as well as a fourth one that at the time of this writing is under development. Each garbage collector is targeted to a different set of applications. The next four sections describe them. The Serial GC The configuration of the Serial GC is a young generation that operates as described earlier, over an old generation managed by a sliding compacting mark-sweep, also known as a mark-compact garbage collector. [11] Both minor and full garbage col- lections take place in a stop-the-world fashion (i.e., the application is stopped while a collection is taking place). Only after the garbage collection has finished is the application restarted (see Figure 3-6a). The mark-compact garbage collector first identifies which objects are still live in the old generation. It then slides them toward the beginning of the heap, leaving any free space in a single contiguous chunk at the end of the heap. This allows any future allocations into the old generation, which will most likely take place as objects are being promoted from the young generation, to use the fast bump-the-pointer technique. Figure 3-7a illustrates the operation of such a garbage collector. Objects marked with a gray X are assumed to be garbage. The shaded area at the end of the compacted space denotes reclaimed (e.g., free) space. Figure 3-6 Stop-the-world garbage collection GC (a) Serial GC Parallel GC (b) Application ptg6882136 HotSpot VM Garbage Collectors 87 Figure 3-7 Garbage collection sequences (a) (b) Start of Compaction End of Compaction Start of Sweeping End of Sweeping The Serial GC is the garbage collector of choice for most applications that do not have low pause time requirements and run on client-style machines. It takes advan- tage of only a single virtual processor for garbage collection work (hence, its name). Still, on today’s hardware, the Serial GC can efficiently manage a lot of nontrivial applications with a few 100MBs of Java heap, with relatively short worst-case pauses (around a couple of seconds for full garbage collections). Another popular use for the Serial GC is in environments where a high number of JVMs are run on the same machine (in some cases, more JVMs than available processors!). In such environ- ments when a JVM does a garbage collection it is better to use only one processor to minimize the interference on the remaining JVMs, even if the garbage collection might last longer. And the Serial GC fits this trade-off nicely. The Parallel GC: Throughput Matters! These days, a lot of important Java applications run on (sometimes dedicated) servers with a lot of physical memory and multiple processors. Ideally, the garbage collector should take advantage of all available processing resources and not leave most of them idle while it is doing garbage collection work. To decrease garbage collection overhead and hence increase application through- put on server-style machines, the HotSpot VM includes the Parallel GC, also called the Throughput GC. Its operation is similar to that of the Serial GC (i.e., it is a stop- the-world GC with a copying young generation over a mark-compact old generation). However, both the minor and full garbage collections take place in parallel, using all available processing resources, as illustrated in Figure 3-6b. Note that earlier version of this garbage collector actually performed old collections serially. This has been rectified since the introduction of the Parallel Old GC. Applications that can benefit from the Parallel GC are those that require high throughput and have pause time requirements that can be met by the worst-case stop-the-world induced full garbage collection durations along with being run on machines with more than one processor. Applications such as batch processing ptg6882136 88 Chapter 3 JVM Overview engines, scientific computing, and so on are well suited for Parallel GC. The Parallel GC, compared to the Serial GC, improves overall garbage collection efficiency, and as a result also improves application throughput. The Mostly-Concurrent GC: Latency Matters! For a number of applications, end-to-end throughput is not as important as rapid response time. In the stop-the-world garbage collection model, when a garbage collec- tion is taking place, the application threads are not running, and external requests will not be satisfied until the application threads are restarted at the end of a garbage collection. Minor garbage collections do not typically cause long pauses. However, full garbage collections or compacting garbage collections, even though infrequent, can impose long pauses, especially when large Java heaps are involved. To deal with this, the HotSpot VM includes the Mostly-Concurrent GC, also known as the Concurrent Mark-Sweep GC (CMS). It manages its young generation the same way the Parallel and Serial GCs do. Its old generation, however, is managed by an algorithm that performs most of its work concurrently, imposing only two short pauses per garbage collection cycle. Figure 3-8a illustrates how a garbage collection cycle works in CMS. It starts with a short pause, called the initial mark, that identifies the set of objects that are immediately reachable from outside the old generation. Then, during the concurrent marking phase, it marks all live objects that are transitively reachable from this set. Because the application is running and it might be updating reference fields (hence, modifying the object graph) while the marking phase is taking place, not all live objects are guaranteed to be marked at the end of the concurrent marking phase. To deal with this, the application is stopped again for a second pause, called the Figure 3-8 Comparison of CMS GC versus garbage first GC Marking / Pre-cleaning Initial Mark Remark GC GC GC (a) Mostly-Concurrent GC Garbage-First GC (b) Sweeping Marking ptg6882136 HotSpot VM Garbage Collectors 89 remark pause, which finalizes the marking information by revisiting any objects that were modified during the concurrent marking phase. The card table data structure is reused to also keep track of modified objects. Because the remark pause is more substantial than the initial mark, it is parallelized to increase its efficiency. To reduce further the amount of work the remark pause has to do, the concurrent pre-cleaning phase was introduced. As Figure 3-8a shows, it takes place after the concurrent marking phase and before the remark pause and does some of the work that would have been done during the remark pause, i.e., revisiting objects that were modified concurrently with the marking phase. Even though there is still a need for the remark pause to finalize marking (given that the application might update more objects during the pre-cleaning phase), the use of pre-cleaning can reduce, sometimes dramatically, the number of objects that need to be visited during the remark pause, and, as a result, it is very effective in reducing the duration of the remark pause. At the end of the remark pause, all live objects in the Java heap are guaranteed to have been marked. Since revisiting objects during the pre-cleaning and remark phases increases the amount of work the garbage collector has to do (as compared to, say, the Parallel GC that only visits objects once during marking), the overall over- head of CMS also increases accordingly. This is a typical trade-off for most garbage collectors that attempt to reduce pause times. Having identified all live objects in the old generation, the final phase of the garbage collection cycle is the concurrent sweeping phase, which sweeps over the Java heap, deallocating garbage objects without relocating the live ones. Figure 3-7b illustrates the operation of the sweeping phase. Again, objects marked with a gray X are assumed to be garbage, and the shaded areas in the post-sweep space denote free space. In this case, free space is not contiguous (unlike in the previous two garbage collectors, as illustrated in Figure 3-7a), and the garbage collector needs to employ a data structure (free lists, in the case of the HotSpot VM) that records which parts of the heap contain free space. As a result, allocation into the old generation is more expensive, as allocation from free lists is not as efficient as the bump-the-pointer approach. This imposes extra overhead to minor garbage collections, as most allo- cations in the old generation take place when objects are promoted during minor garbage collections. Another disadvantage that CMS has, that the previous two don’t, is that it typi- cally has larger Java heap requirements. There are a few reasons for this. First, a concurrent marking cycle lasts much longer than that of a stop-the-world garbage collection. And it is only during the sweeping phase that space is actually reclaimed. Given that the application is allowed to run during the marking phase, it is also allowed to allocate memory, hence the occupancy of the old generation potentially increases during the marking phase and decreases only during the sweeping phase. Additionally, despite the garbage collector’s guarantee to identify all live objects dur- ing the marking phase, it doesn’t actually guarantee that it will identify all objects ptg6882136 90 Chapter 3 JVM Overview that are garbage. Objects that become garbage during the marking phase may or may not be reclaimed during the cycle. If they are not, then they will be reclaimed during the next cycle. Garbage objects that are not identified during a garbage collection are usually referred to as floating garbage. Finally, fragmentation issues [11] due to the lack of compaction might also prevent the garbage collector from using all the available free space as efficiently as pos- sible. If the old generation is full before the collection cycle in progress has actually reclaimed sufficient space, CMS reverts to an expensive stop-the-world compacting phase, similar to that of the Parallel and Serial GCs. It should be noted that, in the latest versions of the HotSpot VM, both the concur- rent phases of CMS (marking and sweeping) are parallelized, as demonstrated in Figure 3-8a. This is a useful feature when running on machines with high hardware parallelism (which are becoming more and more common). Otherwise, one concurrent CMS thread would not have been able to keep up with the work the many application threads would generate. Compared to the Parallel GC, CMS decreases old-generation pauses—sometimes dramatically—at the expense of slightly longer young generation pauses, some reduc- tion in throughput, and extra heap size requirements. Due to its concurrency, it also takes CPU cycles away from the application during a garbage collection cycle. Appli- cations that can benefit from it are ones that require rapid response times (such as data-tracking servers, Web servers, and so on), and it is in fact widely used in this context. The Garbage-First GC: CMS Replacement The Garbage-First GC (aka G1) is a parallel, concurrent, and incrementally com- pacting low-pause garbage collector intended to be the long-term replacement of CMS. G1 uses a drastically different Java heap layout to the other garbage collectors in the HotSpot VM. It splits the Java heap into equal-sized chunks called regions. Even though G1 is generational, it does not have physically separate spaces for the young and old generations. Instead, each generation is a set of (maybe noncontigu- ous) regions. This allows G1 to resize the young generation in a flexible way. All space reclamation in G1 takes place by evacuating the surviving objects from one set of regions to another and then reclaiming the initial (and typically larger) set of regions. Most of the time such garbage collections collect only young regions (which make up G1’s young generation), and they are the equivalent of minor garbage col- lections. G1 also periodically performs concurrent marking cycles that identify which non-young regions are either empty or mostly empty. These are the regions that are the most efficient to collect (i.e., G1 gets back the most free space for the least amount of work), and they are scheduled for garbage collection in favor to the rest of ptg6882136 HotSpot VM Garbage Collectors 91 the regions. This is where G1 gets its name from: It goes after regions with the most garbage objects in them. Figure 3-8b shows the parallelism and concurrency in G1. Note that, apart from the concurrent marking phase, G1 also has additional short concurrent tasks. For more information on G1, please listen to the talk located at http://developers.sun.com/ learning/javaoneonline/j1sessn.jsp?sessn=TS-5419&yr=2008&track=javase]. [12] Comparisons Table 3-1 summarizes the trade-offs between the garbage collectors that are covered in this section. Creating Work for the Garbage Collector This section includes a brief overview of how an application can create work for the garbage collector. Generally, there are three ways of doing so: Allocation. Garbage collections are triggered when a generation occupancy reaches a certain limit (e.g., a minor garbage collection takes place when the eden is full, a CMS cycle starts when the old generation occupancy goes over the CMS initiating limit). As a result, the higher the allocation rate of an applica- tion, the more often garbage collections are triggered. Live data size. All garbage collectors in the HotSpot VM do work propor- tional to the amount of live data that exists in each generation (a minor gar- bage collection copies all live objects as shown in Figure 3-5, a mark-compact garbage collector first needs to mark all live objects before moving them, etc.). As a result, the more live objects there are in the Java heap, the more work the garbage collector needs to do. Reference updates in the old generation. An update of a reference field in the old generation might create an old-to-young reference (which, as shown in Serial GC Parallel GC CMS GC G1 GC Parallelism No Yes Yes Yes Concurrency No No Yes Yes Young GCs Serial Parallel Parallel Parallel Old GCs Serial Parallel Parallel & Conc Parallel & Conc Table 3-1 Comparison of Garbage Collectors ptg6882136 92 Chapter 3 JVM Overview Figure 3-3, will have to be processed during the next minor garbage collection) or might cause an object to be revisited at the pre-cleaning or the remark phase (if it takes place during a CMS marking cycle). Typically garbage collection overhead can be reduced by reducing one or more of the preceding metrics. However, sometimes this is either impossible (e.g., it might not be possible to compress further the data that needs to be loaded into the Java heap; or it is difficult to write a useful application that does not update references at all), or even undesirable (reusing objects can reduce the allocation rate, but it is also more time-consuming to implement and maybe more error-prone too). But by avoiding some bad programming practices, it is possible to find a good balance between having low garbage collection overhead, as well as well-written, easily maintained code. Bad programming practices to avoid include object pooling (pooled objects are long-lived, hence they increase the live data size of the old generation and initializing writes to them can also increase the number of reference updates in the old generation), sloppy sizing of array-based data structures (e.g., if an ArrayList is initially sized too small, its backing array might subsequently need to be resized several times, causing unnecessary allocation), and so on. Expanding on this goes beyond the scope of this book, but you can find some more information in this talk. [13] A Historical Perspective The Serial GC was the first garbage collector included in the HotSpot VM (introduced in Java 1.3), as well as another incremental garbage collector called the Train GC. The latter, however, was not used very widely and was end-of-lifed in Java 6. Java 1.4.2 saw the introduction of both the Parallel GC (which only had a parallel young generation garbage collector, but a serial old generation garbage collector), as well as CMS (which also had a parallel young generation garbage collector, whereas its con- current phases were serial). The Parallel Old GC, which parallelized the old genera- tion of the Parallel GC, was introduced in Java 5 Update 6. The concurrent marking and sweeping phases of CMS were parallelized in Java 5 Update 6 and Java 5 Update 7, respectively. Finally, at the time of this writing, G1 GC was included in Java 6 Update 20 (and available in later Java 6 releases). Java 7 will also have G1 GC. HotSpot VM JIT Compilers Before diving into the details of the JITs used in the HotSpot VM it is useful to digress a bit and talk about code generation in general and the particular trade-offs of JIT compilation. This will help frame the differences between the HotSpot VM Client and Server JIT compilers when they are discussed. ptg6882136 HotSpot VM JIT Compilers 93 Compilation is the generation of machine-dependent code from some high level language. Traditionally compilers have started from a source language like C or C++, compiling each of the individual source files into object files and then finally linking those objects into a library or executable that can then be run. Since this is a relatively infrequent task, compilation time isn’t a huge constraint on static compilers, though obviously developers won’t wait forever. Java on the other hand uses a compiler, javac, which takes the high level sources and converts them into class files. These class files are then collected into jar files for use by a Java Virtual Machine. So the Java Virtual Machine always starts with the bytecode representa- tion of the original program and is required to convert that dynamically into machine dependent code. All compilers have a roughly similar structure, and it is useful to describe this first. They must have a front end to take the source representation and convert it into an intermediate representation or IR. There are many different kinds of inter- mediate representations used in compilers, and a compiler might in fact use several since different representations can be useful for different stages of compilation. One common style of IR is called SSA, which stands for static single assignment. This is a representation that has the property that a variable is only assigned to once, and instructions directly use those values. This has the advantage that the values used by an instruction are directly visible to it. The other common style is a named form that is conceptually similar to a source language in that values are assigned to variables, or names, and instructions use the names. This gives a certain amount of flexibility and can simplify some operations such as cloning of code, but there’s a less direct relationship between an instruction and the values it uses. The IR produced by the front end is generally the focus of most optimizations in a compiler. What optimizations are supported can cover a large range and will often be driven by the time required for the optimization. The most basic classes of optimiza- tions are simple identity transformations, constant folding, common subexpression elimination, and inlining of functions. More complicated optimizations are commonly focused around improving the execution of loops and include range check elimination, unrolling, and loop invariant code motion. Illustrating how the HotSpot VM performs each of these optimizations is outside the scope of this book and is a topic worthy to expanding upon in an entire book. Once these high level optimizations are performed there’s a back end that takes the IR and converts it into some machine representation. This stage includes instruc- tion selection and assignment of values to machine registers. Instruction selection can be done in many ways. It can be handled in an explicit manner where the com- piler writer manages all the cases directly or by using a machine description with associated rules to drive automatic instruction selection. The automated approach can be somewhat complicated to build and maintain but can often take better advan- tage of the details of a machine. ptg6882136 94 Chapter 3 JVM Overview Once the instructions are selected registers must be assigned to all the values in the program, and code must be emitted to deal with the calling conventions of the machine. For most functions the number of values live will be greater than the num- ber of registers on the machine. The generated code will deal with this by assigning some values to registers and moving values between the registers and the stack to free them up for other values. Moving values to the stack is referred to as spilling the value or register spilling. Again there are several approaches to this problem. For simple code generators, a round robin style local allocator will execute quickly but is only suitable for the most simple code generators. The classic strategy for register allocation is called graph coloring and generally results in the best usage of the machine registers and the fewest spills of extra values onto the stack. A graph is built that represents which values are in use simultane- ously and which registers those values can live in. If there are more values live simul- taneously than there are registers available, then the least important of those values are moved to the stack so the other values can use registers. Assigning every value to a register commonly requires several rounds of graph construction and coloring. This leads to the downside of graph coloring, which is that it can be expensive both in terms of time spent and the space required for the data structures. A simpler strategy is called linear scan register allocation. The goal in linear scan is to assign registers in a single pass over all the instructions while still producing a good register assignment. It constructs lists of ranges where a value must be in a register and then in a single pass walks over that list assigning registers to values or spilling them to the stack. This can operate quickly but isn’t as good at keeping values in the same register for their whole lifetime. Class Hierarchy Analysis In an object-oriented language, intelligent inlining can be critical to getting good performance since code is often broken up into small methods. Java presents some interesting difficulties in this regard since by default any instance method could be overridden by a subclass, so just seeing the local type often isn’t enough to know what method to inline. One way the HotSpot VM addresses this is by something called Class Hierarchy Analysis. This is an on-demand analysis that can be used by the compiler to determine whether any loaded subclass has overridden a particular method. The important part of this trick is that the HotSpot VM is only considering the subclasses that are loaded and does not worry about any other subclasses that it hasn’t seen yet. When the compiler takes advantage of Class Hierarchy Analysis, often referred to as CHA, it records that fact in the compiled code. If later on in the execution of the program a subclass that overrides that method is requested to be loaded, then as part of the loading process the compiled code that assumed there was only one implementor is thrown out. If that compiled code is currently being executed ptg6882136 HotSpot VM JIT Compilers 95 somewhere, then a process called deoptimization is used to convert that compiled frame into an equivalent set of interpreter frames. This allows complete recovery from the assumptions of the CHA result. CHA is also used to identify cases where an interface or abstract class only has a single loaded implementation. Compilation Policy Since the JIT does not have time to compile every single method in an application, all code starts out initially running in the interpreter, and once it becomes hot enough it gets scheduled for compilation. In the HotSpot VM this is controlled through the use of counters associated with each method. Every method has two counters: the invoca- tion counter that is incremented every time a method is entered and the backedge counter that is incremented every time control flow moves from a higher bytecode index to a lower one. The backedge counter is used to detect methods that contain loops and to cause them to get compiled earlier than they would with just an invoca- tion counter. Whenever either counter is incremented by the interpreter it checks them against a threshold, and if they cross this threshold the interpreter requests a compile of that method. The threshold used for the invocation count is called the CompileThreshold, and the backedge counter uses a more complex formula of Com- pileThreshold * OnStackReplacePercentage / 100. When a compilation is requested it is enqueued in a list that is monitored by one or more compiler threads. If a compiler thread is not busy it removes the compilation request from the queue and begins to compile it. Normally the interpreter doesn’t wait for the compilation to complete. Instead it resets the invocation counter and resumes executing the method in the interpreter. Once the compile completes and the compiled code is associated with the method, then the next caller of the method begins using the compiled code. Normally this behavior of not waiting for the compile to complete is a good idea since the execution and compilation can continue in paral- lel. If you want the interpreter to wait for the compile to complete, then the HotSpot VM command line option -Xbatch or -XX:-BackgroundCompilation can be used to make it block waiting for the compile. The HotSpot VM can also perform special compiles called On Stack Replacement compiles, or OSRs as they are commonly known. These are used when Java code contains a long-running loop that started executing in the interpreter. Normally the way Java code ends up in compiled code is that when invoking a method the inter- preter detects that there’s compiled code for it, and it dispatches to that instead of staying in the interpreter. This does not help long-running loops that started in the interpreter since they are not being invoked again. When the backedge counter overflows, the interpreter requests a compile that starts its execution at the bytecode of backedge instead of starting at the first byte- code in the method. The resulting generated code takes an interpreter frame as its ptg6882136 96 Chapter 3 JVM Overview input and uses that state to begin its execution. In this way long-running loops are able to take advantage of compiled code. The act of the generated code taking an interpreter frame as its input to be its execution is called On Stack Replacement. Deoptimization Deoptimization is the term used in the HotSpot VM for the process of taking a com- piled frame, which may be the result of several levels of inlining, and converting that compiled frame into an equivalent set of interpreter frames. This is used to recover from various kinds of optimistic optimizations that compiled code can perform. In particular it is used to recover from the assumptions of class hierarchy analysis. The server compiler also uses it for something it refers to as uncommon traps. These are special points in the generated code where the compiler has chosen to use the interpreter to deal with some execution path. Most commonly this is either because at compilation time some class was unloaded, or a path appeared to have never been executed. Some kinds of exceptions are handled in this way as well. The HotSpot VM’s JIT compilers support deoptimization by recording some meta- data at every potential safepoint that describes what the state of the bytecode execu- tion was at that point. Every safepoint already has to include the chain of methods and bytecode indexes that describe the current execution state so that things like exception stack traces and the stack walking required by security checks can be implemented. For deoptimizations the compiler additionally records the location of every value referenced by the locals and expression stack of the method, along with which lock(s) are held. This is an abstract representation of the state of the inter- preter frame at that point and is sufficient to build a set of interpreter frames that resume execution in the interpreter. At first glance it may appear as though a lot of extra values are kept alive to sup- port this, but there are a few tricks used to reduce this. The HotSpot VM’s JIT com- pilers use a bytecode analysis called Method Liveness that computes for every Java local field whether there’s a bytecode later in the method that might use its value. These locals are considered live, and only locals that are live need to have values in the debug info state. In practice this means that the JIT compilers are not keeping many values alive solely for the purposes of deoptimization. Once compiled code has been generated it may be invalidated for several reasons, such as class loading that invalidates a CHA optimization or because classes refer- enced by the code have been unloaded. In this case the space for the compiled code is returned to the code cache for use by later compiles. In the absence of explicit invalidation of compiled code it is normally never freed. JIT compiled code has several kinds of metadata associated with it that’s required to support various features of the runtime. In particular because the HotSpot VM uses precise garbage collection, compiled code has to be able to describe which ptg6882136 HotSpot VM JIT Compilers 97 locations in a compiled frame contain references to Java objects. This is accomplished using OopMaps, which are tables listing registers and stack locations that must be visited by the garbage collector. These are required at any location in compiled code where the system might have to stop for a safepoint. This includes all call sites and places where allocation might occur. Additionally because there are VM operations such as garbage collection, biased lock revocation, and so on that require code be able to come to a halt in a reasonable amount of time for a safepoint, every loop that does not contain calls also requires an explicit safepoint check inside it. Otherwise a long-running loop could stop the entire system from performing a garbage collec- tion and cause it to hang. Each of these safepoints also contains all the information describing the chain of methods that were inlined and the description of the Java frame required for support of deoptimization. Client JIT Compiler Overview The HotSpot VM’s Client JIT compiler targets applications desiring rapid startup time and quick compilation so as to not introduce jitter in responsiveness such as client GUI applications. The Client JIT compiler started life as a fast, simple code generator intended to give Java reasonable startup performance without a lot of complexity. It was conceptually similar to the interpreter in that it generated a kind of template for each kind of bytecode and maintained a stack layout that was similar to an interpreter frame. It also only inlined field accessors. In Java 1.4, the HotSpot VM’s Client JIT compiler was upgraded to support full method inlining and added support for Class Hierarchy Analysis and deoptimization both of which provided a substantial improvement. The Java 5 Client JIT compiler saw few changes because a more substantial set of changes was being worked on for the Java 6 Client JIT compiler. Java 6’s Client JIT Compiler included many changes intended to improve per- formance across the board. The Client compiler’s intermediate representation was changed to an SSA style representation, and the simple local register allocator was replaced by a linear scan register allocator. Additionally value numbering was improved by extending it across multiple blocks, and some minor improvements to memory optimizations were made. On x86 platforms, support for using SSE for floating point operations was added, which significantly improved floating point performance. Server JIT Compiler Overview The HotSpot VM Server JIT compiler targets peak performance and high throughput for Java applications, so its design tends to focus on using the most powerful opti- mizations it can. This often means that compiles can require much more space or ptg6882136 98 Chapter 3 JVM Overview time than an equivalent compile by the Client JIT compiler. It tends to aggressively inline as well, which often leads to large methods, and larger methods take longer to compile. It also has an extensive set of optimizations covering a large number of corner cases, which is needed to generate optimal code for any bytecodes it might see. SSA—Program Dependence Graph The Server JIT compiler’s intermediate representation (IR) is internally called “ideal” and is an SSA style IR, but it uses a different way of representing control flow called the program dependence graph. The representation tries to capture the minimal set of constraints on the execution of each operation, which allows for aggressive reorder- ing of operations and global value numbering, which reduces redundant computa- tions. It has a rich type system that captures all the details of the Java type system and feeds that knowledge back into the optimizations. The Server JIT compiler also takes advantage of profile information collected by execution in the interpreter. During execution of bytecodes, if a method is executed enough times, the interpreter created an object known as a methodDataOop, which is a container for profile information about an individual method. It has entries for recording information about the types seen at call sites along with counts of how often they are seen. All the control flow bytecodes also record how often they are taken and which direction they go. All this information is used by the Server JIT compiler to find opportunities to inline based on common types and to compute fre- quencies for the control flow, which drives the block layout and register allocation. All JIT compilers of Java bytecodes have to deal with the possibility of unloaded or uninitialized classes, and the Server JIT compiler handles this by treating the path as unreached when it contains unresolved constant pool entries. In this case it emits what is called an uncommon trap for that bytecode and stops parsing that path through the method. An uncommon trap is a request to the HotSpot VM Runtime to deoptimize the current compiled method and resume execution in the interpreter where the constant pool entry that was unresolved can be processed and properly resolved. The compiled code for that method is thrown out, and executions continue in the interpreter until a new compile is triggered. Since that path has been properly resolved the new compile will compile that path normally, and future execution will use the compiled version of that path. Uncommon traps are also used to deal with unreached paths so that the compiler does not generate code for parts of the method that are never used, resulting in smaller code and more straight-line sections of code that are generally more optimiz- able. The Server JIT compiler additionally uses uncommon traps to implement some kinds of optimistic optimizations. These are cases where the Server JIT compiler has decided that some behavior is likely so it proceeds as if that was the only behavior but puts in a dynamic check that it is true. If the dynamic check fails the code heads to ptg6882136 HotSpot VM JIT Compilers 99 an uncommon trap, which handles that case in the interpreter. If the uncommon trap happens often enough the HotSpot VM Runtime decides that it is really not uncom- mon so the code should be thrown out and regenerated without the assumption that it is uncommon. This is done for some things like predicated call sites where it appears from profile information that call site only ever sees one receiver type, so the Server JIT compiler inlines assuming that it will see this type but puts in a guard checking that the type is really the expected one. If a call site mostly sees one type but some- times sees others, instead of emitting an uncommon trap for the other case the Server JIT compiler emits a regular call. The advantage of emitting that uncommon trap is that later code will see just the effects of the inlined version, which can result in bet- ter final generated code since a call has unknown side effects on the state of memory. The Server JIT compiler performs a large set of optimizations on loops in the generated code, including loop unswitching, loop unrolling, and range check elimi- nation through iteration splitting. Iteration splitting is the process of taking a loop and converting it into three loops: the preloop, the main loop, and the post loop. The idea is to compute bounds on each of the loops such that it is provable that the main loop does not need any range checks. The preloop and the post loop deal with the boundary conditions of the iteration where range checks are needed. In most cases the preloop and the post loop run a small number of times, and in many cases the post loop can be eliminated completely. This allows the main loop to run without any range checks at all. Once a loop has had its range checks removed it is possible that it can be unrolled. Loop unrolling takes relatively simple loop bodies and creates multiple copies of the body inside the loop, while reducing the number of iterations that the loop runs. This helps amortize the cost of the loop control flow and often allows the loop body to simplify more, allowing the loop to do more work in less time. In some cases repeated unrolling can cause a loop to go away completely. Loop unrolling enables another optimization called superword, which is a form of vectorization. Unrolling creates a parallel set of operations in the body, and if those operations are on sequential memory locations they can be collected into operations on a vector such that a single instruction performs multiple operations in the same amount of time. As of Java 6, HotSpot VMs, this is mainly focused on copying or ini- tialization patterns, but eventually it will fully support all available SIMD (single instruction, multiple data) arithmetic operations. Once all the high level optimizations are performed the IR (intermediate repre- sentation) is converted into a machine dependent form that is able to take advan- tage of all the special instructions and address modes available on the processor. The machine dependent nodes are scheduled into basic blocks based on the require- ments of their inputs and the expected frequency of the blocks. The graph coloring register allocator then assigns registers to all the instructions and inserts any needed register spills. Finally the code is turned into an nmethod, which is the ptg6882136 100 Chapter 3 JVM Overview HotSpot VM’s internal representation of compiled bytecodes and contains all the code along with the metadata required to use the code within the HotSpot VM Runtime. Future Enhancements The HotSpot VM currently supports two JIT compilers, Client and Server. At the time of this writing development is underway to introduce a hybrid HotSpot JIT compiler that combines the major attributes of the Client JIT compiler and the Server JIT compiler called tiered compilation. The promise offered by tiered com- pilation is the rapid startup features of the Client JIT compiler and continuing to improve the performance of an application through the use of the Server JIT compiler’s more advanced optimization techniques. For the adventurous or curi- ous, tiered compilation can be enabled on recent Java 6 HotSpot VMs using the -server-XX: +TieredCompilation command line options. However, tiered com- pilation as of this writing, is not recommended as the HotSpot JIT compiler of choice for production or critical systems if using Java 6 Update 24 or earlier. If you are using Java 6 Update 25, Java 7, or later, using -server -XX: +TieredCompila- tion may be an alternative for applications typically using the Client JIT compiler. As tiered compilation improves in its optimization capabilities and matures, it is likely to be the recommended JIT compiler for both client and server families of Java applications. HotSpot VM Adaptive Tuning The Java 5 HotSpot VMs introduced a new feature that evaluates the underlying platform and system configuration at JVM launch time and then automatically selects the garbage collector, configures Java heap size, and chooses a runtime JIT compiler to use. In addition, this feature also introduced an adaptive means for tuning the Java heap for the throughput garbage collector. This new adaptive Java heap tuning allowed the garbage collector to dynamically tune the sizes of the Java heap to meet application behavior and object allocation rates. This com- bination of automatic platform dependent selection of default values and adaptive Java heap sizing to lessen the burden of manual garbage collection tuning is called ergonomics. The ergonomics feature has been further enhanced in Java 6 Update 18 to improve the performance of rich client applications. In this section, the initial default values for heap sizes, garbage collector, and JIT compilers found in Java 1.4.2 HotSpot VMs are presented, followed by the default values chosen via the ergonomics feature and the Java 6 Update 18 ergonomics enhancements. ptg6882136 HotSpot VM Adaptive Tuning 101 Java 1.4.2 Defaults In the Java 1.4.2 HotSpot VM the following defaults were chosen for garbage collec- tor, JIT compiler, and Java heap sizes: Serial garbage collector, i.e., -XX:+UseSerialGC Client JIT compiler, i.e., -client 4 megabyte initial and minimum Java heap size along with a 64 megabyte maximum Java heap size, i.e., -Xms4m and -Xmx64m Java 5 Ergonomic Defaults In the Java 5 HotSpot VMs, a category called “server-class machine” was introduced that allowed the HotSpot VM to choose a different set of default values for garbage collector, JIT compiler, and Java heap sizes. A server-class machine in the HotSpot VM is defined as a system with an underlying configuration that has two or more gigabytes of physical memory and two or more virtual processors. The number of virtual processors identified by the HotSpot VM when determining whether a system is a server-class machine is also the same value returned by the Java API Runtime.availableProcessors(), and generally is the same number of processors reported by operating system tools such as mpstat for Linux and Solaris. Also note when running a HotSpot VM in an operating system configured with a processor set, the value returned by the Java API Runtime. availableProcessors() is the number of virtual processors observed within the processor set, not the number of virtual processors observed system wide. Tip The definition of server-class machine does not apply to systems running a 32-bit version of the Windows operating system. These systems default to using the Serial garbage collector (-XX: +UseSerialGC), Client JIT compiler (-client), and 4 megabyte initial and minimum heap size (-Xms4m) along with a 64-megabyte maximum Java heap size (-Xmx64m). 1. On recent Java 6 HotSpot VMs, or where the following switch is available, ergo- nomics may also automatically select -XX:+UseParallelOldGC, which also enables -XX:+UseParallelGC. When the HotSpot VM identifies a system as a server class machine, it selects the following defaults for garbage collector, JIT compiler, and Java heap sizes: Throughput garbage collector, also known as Parallel GC, i.e., -XX:+UseParallelGC1 Server JIT compiler, i.e., -server ptg6882136 102 Chapter 3 JVM Overview 1/64 of the physical memory up to a maximum of 1GB as the initial and minimum Java heap size along with a 1/4 the total physical memory up to a maximum of 1GB as the maximum Java heap size Table 3-2 summarizes the choices made by a Java 5 and later HotSpot VM. Serial GC means the Serial garbage collector is chosen. Parallel GC means the Throughput garbage collector is chosen. Client means the client JIT compiler is cho- sen. Server means the server JIT compiler is chosen. Under (If Server Class) Default GC, JIT, and Java Heap Sizes, Client means the Client JIT compiler is chosen for a 32-bit Windows platform where other criteria for a server-class machine matched. This choice is deliberately made on 32-bit Windows platforms because historically client applications (i.e., interactive applications) are run more often on this combi- nation of platform and operating system. Where Server is indicated, the Server JIT compiler is the only JIT compiler available in the HotSpot VM. To print the ergonomic choices the HotSpot VM has made, the -XX: +PrintCom- mandLineFlags command line option can be used. For instance, doing a simple java Platform Operating System (If Not Server Class) Default GC, JIT and Heap Sizes -Xms & -Xmx (If Server Class) Default GC, JIT and Java Heap Sizes -Xms & -Xmx SPARC (32-bit) Solaris Serial GC, Client, 4MB, 64MB Parallel GC,Server, 1/64 RAM, max of 1/4 RAM or 1GB i586 Solaris Serial GC, Client, 4MB, 64MB Parallel GC,Server, 1/64 RAM, max of 1/4 RAM or 1GB i586 Linux Serial GC, Client, 4MB, 64MB Parallel GC,Server, 1/64 RAM, max of 1/4 RAM or 1GB i586 Windows Serial GC, Client, 4MB, 64MB Serial GC, Client, 1/64 RAM, max of 1/4 RAM or 1GB SPARC (64-bit) Solaris Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64 RAM, max of 1/4 RAM or 1GB x64 (64-bit) Linux Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64th RAM, max of 1/4 RAM or 1GB x64 (64-bit) Windows Parallel GC, Server, 1/64 RAM, 1/4t RAM or 1GB max Parallel GC, Server, 1/64 RAM, max of 1/4 RAM or 1GB IA-64 Linux Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64 RAM, max 1/4 RAM or 1GB IA-64 Windows Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64 RAM, max of 1/4 RAM or 1GB Table 3-2 Summary of Choices Made by a Java 5 and Later HotSpot VM ptg6882136 HotSpot VM Adaptive Tuning 103 -XX:+PrintCommandLineFlags -version on any system with a Java 5 or Java 6 HotSpot VM prints the default ergonomic values. The following is an example of the output produced from a Java 5 HotSpot VM on a Sun UltraSPARC 5440 system configured with 128GB of RAM and 256 virtual processors running the Oracle 11 Express 2010.11 operating system: $ java -XX: +PrintCommandLineFlags -version -XX:MaxHeapSize=1073741824 -XX:ParallelGCThreads=85 -XX: +PrintCommandLineFlags -XX: +UseParallelGC java version "1.6.0_14" Java(TM) SE Runtime Environment (build 1.6.0_14-b07) Java HotSpot(TM) Server VM (build 14.0-b15, mixed mode) From the preceding output, the Java 6 HotSpot VM’s launcher chose the Server JIT com- piler, as shown in the last line of the output, a maximum Java heap size of 1073741824 bytes, or 1024 megabytes or 1 gigabyte along with selecting the throughput collector (-XX:+UseParallelGC) with 85 parallel gc threads (-XX:ParallelGCThreads=85). Note, -XX:MaxHeapSize is the same as the command line option -Xmx. Java 6 Update 18 Updated Ergonomic Defaults Java 6 Update 18 further updated the ergonomics feature to better adapt to rich client applications. The enhancements apply to when a system is identified as a non-server class machine. Remember that a server class machine is defined as a system with an underlying configuration that has 2 or more gigabytes of physical memory and two or more virtual processors. Hence these are enhancements made to systems identified as having less than 2 gigabytes of physical memory and less than two virtual processors. For systems identified as non-server class machines, the client JIT compiler remains as the automatically selected JIT compiler. However, its Java heap sizing defaults have changed, and the settings for garbage collection are better tuned. The maximum heap size for Java 6 Update 18 is now one-half of physical memory up to a physical memory size of 192MB. Otherwise, the maximum heap size is one-fourth of physical memory up to a physical memory size of 1GB. For systems with 1GB or more of physical memory, the default maximum heap size is 256m. The initial heap size for non-server class machines is 8MB up to a physical memory size of 512MB. Otherwise, the initial and minimum heap size is 1/64 of the physical memory size between 512MB and 1GB of physical memory. At 1GB and larger physical memory, the default initial and minimum heap size is explicitly 16MB. In addition, Java 6 Update 18 sizes the young generation space at one-third of the Java heap size. However, if the concurrent collector happens to be speci- fied explicitly with no additional Java heap sizing, initial, minimum, maximum, or young generation space sizing, Java 6 Update 18 reverts to the Java 5 ergonomic defaults. ptg6882136 104 Chapter 3 JVM Overview Platform Operating System (If Not Server Class) Default GC, JIT, and Heap Sizes -Xms & -Xmx (If Server Class) Default GC, JIT, and Java Heap Sizes -Xms & -Xmx SPARC (32-bit) Solaris Serial GC, Client, 8MB or 1/64 RAM or 16MB, 1/2 RAM or 1/4 RAM or 256MB Parallel GC,Server, 1/64 RAM, max of 1/4 RAM or 1GB i586 Solaris Serial GC, Client, 8MB or 1/64 RAM or 16MB, 1/2 RAM or 1/4 RAM or 256MB Parallel GC,Server, 1/64 RAM, max of 1/4 RAM or 1GB i586 Linux Serial GC, Client, 8MB or 1/64 RAM or 16MB, 1/2 RAM or 1/4 RAM or 256MB Parallel GC,Server, 1/64 RAM, max of 1/4 RAM or 1GB i586 Windows Serial GC, Client, 8MB or 1/64 RAM or 16MB, 1/2 RAM or 1/4 RAM or 256MB Serial GC, Client, 1/64 RAM, max of 1/4 RAM or 1GB SPARC (64-bit) Solaris Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64 RAM, max of 1/4 RAM or 1GB x64 (64-bit) Linux Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64 RAM, max of 1/4 RAM or 1GB x64 (64-bit) Windows Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64 RAM, max of 1/4 RAM or 1GB IA-64 Linux Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/4 RAM or 1GB max 4MB, 64MB IA-64 Windows Parallel GC, Server, 1/64 RAM, 1/4 RAM or 1GB max Parallel GC, Server, 1/64 RAM, max of 1/4 RAM or 1GB Table 3-3 Summary of Choices Made by Java 6 Update 18 and Later Table 3-3 summarizes the updated ergonomic choices made by Java 6 Update 18 when no command line options are specified. The values in the cells within Table 3-3 that have changed from the Java 5 ergonomics in Table 3-2 are in italic. Young generation space size is also sized at 1/3 of the Java heap size for those configurations listed in the table that are in italic. Adaptive Java Heap Sizing An artifact of the ergonomics feature enabling the throughput collector is the enabling of an additional feature called adaptive heap sizing. Adaptive heap sizing attempts to opti- mally size the young generation and old generation spaces of the HotSpot VM by evaluat- ing application object allocation rates and their lifetimes. The HotSpot VM monitors the ptg6882136 HotSpot VM Adaptive Tuning 105 Java application’s object allocation rate and their object lifetimes and then makes sizing decisions that attempt to size the young generation space such that short-lived objects are collected prior to getting promoted to old generation along with allowing longer lived objects to be promoted in a timely manner to avoid them unnecessarily being copied between survivor spaces. The HotSpot VM initially uses explicit young generation sizing such as those specified with -Xmn, -XX:NewSize, -XX:MaxNewSize, -XX:NewRatio, and -XX:SurvivorRatio as a starting point for young generation sizing. Adaptive siz- ing automatically adjusts young generation space sizes from those initial settings. Tip Adaptive heap sizing is available only with the throughput collectors -XX:+UseParallelGC or -XX:+UseParallelOldGC. It is not available with the concurrent collector or serial collector. Although there exists HotSpot VM command line options that can fine-tune the policies adaptive heap sizing uses in making its dynamic heap sizing decisions, these options are rarely used outside the guidance of HotSpot VM engineers. It is much more common to disable adaptive heap sizing and explicitly size the young generation space including eden and survivor spaces. On most Java applications using the through- put collector, enabled via -XX: +UseParallelGC or -XX: +UseParallelOldGC, adap- tive sizing does a good job at optimally sizing the young generation space. The family of applications that adaptive heap sizing finds the most challenging are those that have frequent fluctuations, or rapidly changing periods of object allocation rates and experi- ence frequent phases where object lifetimes vary dramatically. Applications that fall into this category may realize better performance by disabling adaptive heap sizing using the -XX:-UseAdaptiveSizePolicy HotSpot VM command line option. Note, the “-” character after the “-XX:”. The “-” character tells the HotSpot VM to disable the adaptive sizing policy. In contrast, a “+” character following the “-XX:” tells the HotSpot VM to enable the feature. Beyond Ergonomics Performance demanding applications often find tuning the HotSpot VM beyond its ergonomic defaults results in improved performance. The one exception to this is adaptive sizing, which is enabled automatically when using the throughput collector. Adaptive sizing tends to do well at automatically sizing the young generation space for most Java applications. More information on tuning the HotSpot VM can be found in Chapter 7 of this book. Ergonomics is a feature that continues to evolve with each release of the ptg6882136 106 Chapter 3 JVM Overview HotSpot VM with the goal of being able to meet or exceed the performance realized by specialized command line option tuning. References [1] Lindholm, Tim, and Frank Yellin. Java Virtual Machine Specification, Second Edition. Addison-Wesley, Reading, MA, 1999. [2] Gosling, James, Bill Joy, Guy Steele, and Gilad Bracha. Java Language Specifica- tion, Third Edition. Chapter 12.2: Loading of Classes and Interfaces. Addison-Wesley, Boston, MA, 2005. [3] Amendment to Java Virtual Machine Specification, Second Edition. Chapter 5: Linking and Initializing. http://java.sun.com/docs/books/vmspec/2nd-edition/Con- stantPool.pdf. [4] Liang, Shen, and Gilad Bracha. Dynamic Class Loading in the Java Virtual Machine. Proc. of the ACM Conf. on Object-Oriented Programming, Systems, Lan- guages and Applications. 1998. [5] Dmitriev, Mikhail. Safe Class and Data Evolution in Large and Long-Lived Java Applications. SML Technical Report Series, Palo Alto, CA, 2001. [6] Gosling, James, Bill Joy, Guy Steele, and Gilad Bracha. Java Language Specifica- tion, Third Edition. Addison-Wesley, Reading, MA, 2005, [7] Dice, Dave. Biased Locking in HotSpot. blog. http://blogs.sun.com/dave/entry/ biased_locking_in_hotspot, 2006. [8] Java Native Interface Specification. http://java.sun.com/javase/6/docs/technotes/ guides/jni/spec. [9] Liang, Sheng. The Java Native Interface. Addison-Wesley, Reading, MA, 1999. [10] Trouble-Shooting and Diagnostic Guide. http://java.sun.com/j2se/1.5/pdf/jdk50_ ts_guide.pdf, 2007. [11] Jones, Richard, and Rafael Lins. Garbage Collection. John Wiley & Sons, Ltd., West Sussex, PO19 IUD, England, 1996. [12] Printezis, Tony, and Paul Ciciora. The Garbage First Garbage Collector presenta- tion. JavaOne Conference. San Francisco, CA, 2008. http://www.oracle.com/technet- work/java/j1sessn-jsp-155531.html [13] Printezis, Tony, and John Coomes. GC Friendly Programming presentation. Java- One Conference. San Francisco, CA, 2007. http://www.oracle.com/technetwork/java/ index-jsp-156726.html ptg6882136 107 4 JVM Performance Monitoring This chapter describes the information to monitor at the Java Virtual Machine (JVM) level of the software stack. In addition, it shows tools that can be used to monitor a JVM and what to watch for as common patterns. The details of how to make JVM tuning decisions based on the information observed can be found in Chapter 7, “Tun- ing the JVM, Step By Step.” There is also a small section at the end of the chapter covering application monitoring. Monitoring a JVM is an activity that should be done all the time with a production application. Since the JVM is a critical component in the software stack, it should be monitored as much as the application itself and the operating system. Analysis of JVM monitoring information indicates when JVM tuning is needed. JVM tuning should be expected anytime there is a JVM version change, operating system change (configuration or version), application version or update, or a major change in appli- cation input. A change in application input is something that can occur frequently with many Java applications that can alter the performance of a JVM. Hence, moni- toring a JVM is an important activity. There are several areas of the JVM to monitor including garbage collection, JIT compiler activity, and class loading. Many tools are available to monitor a JVM. Some monitoring tools are distributed with a JDK, some tools are free, and others are commercial. The monitoring tools covered in this chapter are either distributed with the Oracle JDK, free, or open source. Additionally, all the tools presented in this chapter are available for Windows, Linux, and Oracle Solaris (also referred to as Solaris hereafter) operating systems. ptg6882136 108 Chapter 4 JVM Performance Monitoring To understand the material presented in this chapter, it is helpful to understand the major components and the general operations of a modern JVM. An overview of the Java HotSpot VM and its major components is given in Chapter 3, “JVM Overview.” Definitions Before delving into the details of what to monitor, a revisit of the definitions of performance monitoring and performance profiling presented at the beginning of Chapter 2, “Operating System Performance Monitoring,” is useful. Performance monitoring is an act of nonintrusively collecting or observing performance data from an operating or running application. Performance monitoring is usually a preventa- tive or proactive type of action and can be performed in a production environment, qualification environment, or development environment. Performance monitoring can also be a first step in a reactive situation where an application stakeholder has reported a performance issue but has not provided sufficient information or clues to a potential root cause. In this case, performance profiling likely follows the act of per- formance monitoring. Performance monitoring also helps identify or isolate potential issues without having a severe impact on application responsiveness or throughput. In contrast, performance profiling is an act of collecting performance data from an operating or running application that may be intrusive on application throughput or responsiveness. Performance profiling tends to be a reactive type of activity, or an activity in response to a stakeholder reporting a performance issue. It usually has a narrower focus than performance monitoring. Profiling is rarely done in production environments. It is typically done in qualification, testing, or development environ- ments and is often an act that follows a monitoring activity. Performance tuning, in contrast to performance monitoring and performance pro- filing, is an act of changing tunables, source code, or configuration attribute(s) for the purposes of improving throughput or responsiveness. Performance tuning often follows monitoring or performance profiling activities. Garbage Collection Monitoring JVM garbage collection is important since it can have a profound effect on an application’s throughput and latency. Modern JVMs, such as the Java HotSpot VM (also referred to as HotSpot VM hereafter), provide the ability to observe garbage collection statistics per garbage collection in either a textual form, directed to a log file, or by publishing the garbage collection statistics to a monitoring GUI. This section begins by listing the garbage collection data of interest. Then a list- ing of HotSpot VM command line options to report garbage collection statistics is presented along with an explanation of the reported data. In addition, graphical ptg6882136 Garbage Collection 109 tools that can be used to analyze garbage collection data is presented. And, most importantly, patterns to look for are given along with suggestions as to when JVM garbage collection tuning is advisable. Garbage Collection Data of Interest The data of interest in garbage collection statistics are The garbage collector in use The size of the Java heap The size of the young generation and old generation spaces The size of the permanent generation space The duration of minor garbage collections The frequency of minor garbage collections The amount of space reclaimed in minor garbage collections The duration of full garbage collections The frequency of full garbage collections The amount of space reclaimed in a concurrent garbage collection cycle The occupancy of the Java heap before and after garbage collections The occupancy of the young generation and old generation spaces before and after garbage collections The occupancy of the permanent generation space before and after garbage collections Whether it is the occupancy of the old generation space or the occupancy of the permanent generation space that triggers a full garbage collection Whether the application is making use of explicit calls to System.gc() Garbage Collection Reporting There is little additional overhead in the HotSpot VM to report garbage collection data. In fact, the overhead is so small it is recommended to collect garbage collection data in production environments. This section describes several different HotSpot VM command line options that produce garbage collection statistics along with an explanation of the statistics. There are generally two different types of garbage collections: a minor garbage collection, also called a young generation garbage collection, and a full garbage col- lection, also called a major garbage collection. A minor garbage collection collects ptg6882136 110 Chapter 4 JVM Performance Monitoring the young generation space. A full garbage collection generally expresses the notion of garbage collecting and compacting the old generation and permanent generation spaces. There are some exceptions to this. In the HotSpot VM, the default behavior on a full garbage collection is to garbage collect the young generation, old generation, and permanent generation spaces. In addition, the old generation and permanent generation spaces are compacted along with any live objects in young generation space being promoted to the old generation space. Hence, at the end of a full garbage collection, young generation space is empty, and old generation and permanent gen- eration spaces are compacted and hold only live objects. The behavior of each of the HotSpot garbage collectors is described in detail in Chapter 3. As mentioned earlier, a minor garbage collection frees memory occupied by unreachable objects in the young generation space. In contrast, the default behav- ior for the HotSpot VM on a full garbage collection is to free memory occupied by unreachable objects in the young generation, old generation, and permanent gen- eration spaces. It is possible to configure the HotSpot VM to not garbage collect the young generation space on a full garbage collection prior to garbage collecting the old generation space using the command line option -XX:-ScavengeBeforeFullGC. The “–” character preceding the ScavengeBeforeFullGC disables the garbage col- lection of the young generation space on a full garbage collection. In contrast, a “+” character in front of ScavengeBeforeFullGC enables the garbage collection of the young generation space on a full garbage collection. As just mentioned, the default behavior for the HotSpot VM is to enable garbage collection of the young generation space on full garbage collections. It is advisable to use the default behavior and not disable garbage collection of young generation on a full garbage collection. Garbage collecting the young generation space prior to garbage collecting the old generation space usually results in less work for the garbage collector and more objects being garbage collected since objects in the old generation space may be holding object references to objects in the young generation space. If the young generation space is not garbage collected, any object in old generation space that holds a reference to an object in young generation space cannot be garbage collected. Although -verbose:gc is probably the most commonly used garbage collection reporting command line option, -XX:+PrintGCDetails prints additional and more valuable garbage collection information. This subsection presents example output from -XX:+PrintGCDetails for the throughput and concurrent garbage collectors along with providing an explanation of data. Also, patterns to watch for in the output are also presented. It is important to note the additional information produced with -XX:+ PrintGCDetails can change between versions of the HotSpot VM. -XX:+PrintGCDetails ptg6882136 Garbage Collection 111 An example of -XX:+PrintGCDetails output from Java 6 Update 25’s throughput garbage collector, enabled via -XX:+UseParallelGC or -XX:+UseParallelOldGC, is shown in the following example. The output is spread across several lines for easier reading. [GC [PSYoungGen: 99952K->14688K(109312K)] 422212K->341136K(764672K), 0.0631991 secs] [Times: user=0.83 sys=0.00, real=0.06 secs] The GC label indicates this is minor garbage collection. [PSYoungGen: 99952K->14688K(109312K)] provides information about the young generation space. PSYoungGen indicates the young generation garbage collector in use is the multithreaded young generation garbage collector used with the throughput collec- tor, enabled with the command line option –XX:+UseParallelGC, or auto enabled with –XX:+UseParallelOldGC. Other possible young generation garbage collec- tors are ParNew, which is the multithreaded young generation garbage collector used with the concurrent old generation garbage collector known as CMS, and Def- New which is the single-threaded young generation garbage collector used with the serial garbage collector, enabled with the command line option –XX:+UseSerialGC. -XX:+UseSerialGC, (DefNew), can also be used in combination with the old genera- tion concurrent garbage collector, CMS, to indicate the use of a single-threaded young generation collector. At the time of this writing the G1 garbage collector, currently under development, does not use an identifier in the same way as the other three garbage collectors to identify the output as G1 GC. The value to the left of the ->, 99952K, is the occupancy of the young genera- tion space prior to the garbage collection. The value to the right of the ->, 14688K, is the occupancy of the young generation space after the garbage collection. Young generation space is further divided into an eden space and two survivor spaces. Since the eden space is empty after a minor garbage collection, the value to the right of the ->, 14688K, is the survivor space occupancy. The value inside the parentheses, (109312K), is the size, not the occupancy, of the young generation space, that is, the total size of eden and the two survivor spaces. On the next line of output, 422212K->341136K(764672K) provides the Java heap utilization (the total occupancy of both young generation and old generation spaces), before and after the garbage collection. In addition, it provides the Java heap size, which is the total size of young generation and old generation spaces. The value to the left of the ->, 422212K, is the occupancy of the Java heap before the garbage collection. The value to the right of the ->, 341136K, is the occupancy of the Java ptg6882136 112 Chapter 4 JVM Performance Monitoring heap after the garbage collection. The value inside the parentheses, (764672K), is the total size of the Java heap. Using the reported young generation size and the reported Java heap size, you can calculate the size of the old generation space. For example, the Java heap size is 764672K, and the young generation size is 109312K. Hence, the old generation size is 764672K - 109312K = 655360K. 0.0631991 secs indicates the elapsed time for the garbage collection. [Times: user=0.06 sys=0.00, real=0.06 secs] provides CPU usage and elapsed time information. The value to the right of user is the CPU time used by the garbage collection executing instructions outside the operating system. In this example, the garbage collector used 0.06 seconds of user CPU time. The value to the right of sys is the CPU time used by the operating system on behalf of the garbage collector. In this example, the garbage collector did not use any CPU time execut- ing operating system instructions on behalf of the garbage collection. The value to the right of real is the elapsed wall clock time in seconds of the garbage collection. In this example, it took 0.06 seconds to complete the garbage collection. The times reported for user, sys, and real are rounded to the nearest 100th of a second. An example of a full garbage collection with -XX:+PrintGCDetails follows. (The output is spread across several lines for easier reading.) [Full GC [PSYoungGen: 11456K->0K(110400K)] [PSOldGen: 651536K->58466K(655360K)] 662992K->58466K(765760K) [PSPermGen: 10191K->10191K(22528K)], 1.1178951 secs] [Times: user=1.01 sys=0.00, real=1.12 secs] The Full GC label indicates it is a full garbage collection. [PSYoungGen: 11456K->0K(110400K)] has the same meaning as in a minor garbage collection (explained previously). [PSOldGen: 651536K->58466K(655360K)] provides information about the old generation space. PSOldGen indicates the old generation garbage collector in use is the multithreaded old generation garbage collector used with the throughput collector enabled via the XX:+UseParallelOldGC command line option. In the PSOldGen row of output, the value to the left of the ->, 651536K, is the occupancy of the old generation space prior to the garbage collection. The value to the right of the ->, 58466K, is the occupancy of the old generation space after the garbage collection. The value inside the parentheses, (655360K), is the size of the old generation space. 662992K->58466K(765760K) provides the Java heap utilization. It is the cumu- lative occupancy of both young generation and old generation spaces before and after ptg6882136 Garbage Collection 113 the garbage collection. The value to the right of the -> can also be thought of as the amount of live data in the application at the time of the full garbage collection. Know- ing the amount of live data in the application, especially while the application is in steady state is important information to have when sizing the JVM’s Java heap and fine-tuning the JVM’s garbage collector. [PSPermGen: 10191K->10191K(22528K)] provides information about the permanent generation space. PSPermGen indicates the permanent generation garbage collector in use is the multithreaded permanent generation garbage col- lector used with the throughput collector enabled via the -XX:+UseParallelGC or -XX:+UseParallelOldGC command line options. In the PSPermGen row of data, the value to the left of the ->, 10191K, is the occupancy of the permanent generation space prior to the garbage collection. The value to the right of the ->, 10191K, is the occupancy of the permanent generation space after the garbage collection. The value inside the parentheses (22528K), is the size of the perma- nent generation space. An important observation to take notice of in a full garbage collection is the heap occupancies of the old generation and permanent generation spaces before the gar- bage collection. This is because a full garbage collection may be triggered by either the occupancy of the old generation or permanent generation space nearing its capac- ity. In the output, the occupancy of the old generation space before the garbage collection (651536K), is very near the size of the old generation space (655360K). In contrast, the occupancy of the permanent generation space before the garbage collection (10191K), is nowhere near the size of the permanent generation space (22528K). Therefore, this full garbage collection was caused by the old generation space filling up. 1.1178951 secs indicates the elapsed time for the garbage collection. [Times: user=1.01 sys=0.00, real=1.12 secs] provides CPU and elapsed time information. Its meaning is the same as described earlier for minor garbage collections. When using the concurrent garbage collector, CMS, the output produced by -XX:+PrintGCDetails is different, especially the data reporting what is happen- ing during a mostly concurrent garbage collection of the old generation space. The concurrent garbage collector, CMS, is enabled with the -XX:+UseConcMarkSweepGC command line option. It also auto-enables -XX:+UseParNewGC, a multithreaded young generation garbage collector. An example of a minor garbage collection using the concurrent garbage collector, CMS, follows: [GC [ParNew: 2112K->64K(2112K), 0.0837052 secs] 16103K->15476K(773376K), 0.0838519 secs] [Times: user=0.02 sys=0.00, real=0.08 secs] ptg6882136 114 Chapter 4 JVM Performance Monitoring The minor garbage collection output from the concurrent garbage collector is simi- lar to the minor garbage collection output for the throughput garbage collector. It is explained here for completeness. The GC label indicates this is minor garbage collection. [ParNew: 2112K->64K(2112K)] provides information about the young generation space. ParNew indicates the young generation garbage collector in use is the multithreaded young generation garbage collector used with the CMS concurrent garbage collector. If the serial young generation garbage collector is specified to be used with CMS, the label here will be DefNew. The value to the left of the ->, 2112K, and to the right of the ParNew label, is the occupancy of the young generation space prior to the garbage collection. The value to the right of the ->, 64K, is the occupancy of the young generation space after the garbage collection. The young generation space is further divided into an eden space and two survivor spaces. Since the eden space is empty after a minor garbage collec- tion, the value to the right of the ->, 64K, is the survivor space occupancy. The value inside the parentheses (2112K), is the size, not the occupancy, of the young genera- tion space, that is, the total size of eden and the two survivor spaces. The 0.0837052 secs output is the amount of time it took to garbage collect unreachable objects in the young generation space. On the next line of output, 16103K->15476K(773376K) provides the Java heap utilization (the total occupancy of both young generation and old generation spaces), before and after the garbage collection. In addition, it provides the Java heap size, which is the total size of young generation and old generation spaces. The value to the left of the ->, 16103K, is the occupancy of the Java heap before the garbage col- lection. The value to the right of the ->, 15476K, is the occupancy of the Java heap after the garbage collection. The value inside the parentheses (773376K), is the total size of the Java heap. Using the reported young generation size and the reported Java heap size, you can calculate the size of the old generation space. For example, the Java heap size is 773376K and the young generation size is 2112K. Hence, the old generation size is 773376K - 2112K = 771264K. 0.0838519 secs indicates the elapsed time for the minor garbage collection including the time it took to garbage collect the young generation space and promote any objects to old generation along with any remaining final cleanup work. [Times: user=0.02 sys=0.00, real=0.08 secs] provides CPU usage and elapsed time information. The value to the right of user is the CPU time used by the garbage collection executing instructions outside the operating system. In this example, the garbage collector used 0.02 seconds of user CPU time. The value to the right of sys is the CPU time used by the operating system on behalf of the garbage collector. In this example, the garbage collector did not use any CPU time executing operating system instructions on behalf of the garbage collection. The value to the ptg6882136 Garbage Collection 115 right of real is the elapsed wall clock time in seconds of the garbage collection. In this example, it took 0.08 seconds to complete the garbage collection. The times reported for user, sys, and real are rounded to the nearest 100th of a second. Recall from the description of CMS in Chapter 3 there is a mostly concurrent garbage collection cycle that can execute in the old generation space. -XX:+PrintGCDetails also reports garbage collection activity on each concurrent garbage collection cycle. The following example shows garbage collection output that reports an entire con- current garbage collection cycle. The concurrent garbage collection activity is inter- spersed with minor garbage collections to illustrate that minor garbage collections can occur during a concurrent garbage collection cycle. The output is reformatted for easier reading, and the concurrent garbage collection data is in bold. It should also be noted that the output reported from -XX:+PrintGCDetails when using CMS is subject to change between releases. [GC [1 CMS-initial-mark: 13991K(773376K)] 14103K(773376K), 0.0023781 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [CMS-concurrent-mark-start] [GC [ParNew: 2077K->63K(2112K), 0.0126205 secs] 17552K->15855K(773376K), 0.0127482 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] [CMS-concurrent-mark: 0.267/0.374 secs] [Times: user=4.72 sys=0.01, real=0.37 secs] [GC [ParNew: 2111K->64K(2112K), 0.0190851 secs] 17903K->16154K(773376K), 0.0191903 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] [CMS-concurrent-preclean-start] [CMS-concurrent-preclean: 0.044/0.064 secs] [Times: user=0.11 sys=0.00, real=0.06 secs] [CMS-concurrent-abortable-preclean-start] [CMS-concurrent-abortable-clean] 0.031/0.044 secs] [Times: user=0.09 sys=0.00, real=0.04 secs] [GC [YG occupancy: 1515 K (2112K) [Rescan (parallel) , 0.0108373 secs] [weak refs processing, 0.0000186 secs] [1 CMS-remark: 16090K(20288K)] 17242K(773376K), 0.0210460 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] [GC [ParNew: 2112K->63K(2112K), 0.0716116 secs] 18177K->17382K(773376K), 0.0718204 secs] [Times: user=0.02 sys=0.00, real=0.07 secs] [CMS-concurrent-sweep-start] Continued ptg6882136 116 Chapter 4 JVM Performance Monitoring A CMS cycle begins with the initial mark pause and ends at the completion of the concurrent reset phase. Each of the CMS cycle phases is in bold in the preced- ing output beginning with the CMS-initial-mark and ending with the CMS- concurrent-reset. The CMS-concurrent-mark entry indicates the end of the concurrent marking phase. The CMS-concurrent-sweep label marks the end of the concurrent sweeping phase. The CMS-concurrent-preclean and CMS- concurrent-abortable-preclean entries identify work that can be done concurrently and is in preparation for the remark phase, denoted with the CMS- remark label. The sweeping phase, noted with the CMS-concurrent-sweep entry, is the phase that frees the space consumed by objects marked as unreachable. The final phase is indicated by the CMS-concurrent-reset, which prepares for the next concurrent gar bage collection cycle. The initial mark is usually a short pause relative to the time it takes for a minor garbage collection. The time it takes to execute the concurrent phases (concurrent mark, concurrent precleaning, and concurrent sweep) may be relatively long (as in the preceding example) when compared to a minor garbage collection pause, but Java application threads are not stopped for the duration of the concurrent phases. The remark pause is affected by the specifics of the application (e.g., a higher rate of modifying objects can increase this pause time) and the time since the last minor garbage collection (i.e., a larger number of objects in the young generation space may increase the duration of this pause). A pattern to pay particular attention to in the output is the amount in the reduction of old generation space occupancy during the CMS cycle. In particular, how much the Java heap occupancy drops between the start and end of the CMS [GC [ParNew: 2111K->63K(2112K), 0.0830392 secs] 19363K->18757K(773376K), 0.0832943 secs] [Times: user=0.02 sys=0.00, real=0.08 secs] [GC [ParNew: 2111K->0K(2112K), 0.0035190 secs] 17527K->15479K(773376K), 0.0036052 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [CMS-concurrent-sweep: 0.291/0.662 secs] [Times: user=0.28 sys=0.01, real=0.66 secs] [GC [ParNew: 2048K->0K(2112K), 0.0013347 secs] 17527K->15479K(773376K), 0.0014231 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [CMS-concurrent-reset-start] [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] [GC [ParNew: 2048K->1K(2112K), 0.0013936 secs] 17527K->15479K(773376K), 0.0014814 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] ptg6882136 Garbage Collection 117 concurrent sweep denoted in the output as CMS-concurrent-sweep-start and CMS-concurrent-sweep. The Java heap occupancy can be observed by looking at the minor garbage collections. Hence, pay attention to the minor garbage collections between the start and end of the CMS concurrent sweep phase. If there is little drop in the Java heap occupancy between the start and end of the CMS concurrent sweep phase, then either few objects are being garbage collected, meaning the CMS garbage collection cycles are finding few unreachable objects to garbage collect and as a result are wasting CPU, or objects are being promoted into the old generation space at a rate that is equal to or greater than the rate at which the CMS concurrent sweep phase is able to garbage collect them. Either of these two observations is a strong indicator the JVM is in need of tuning. See Chapter 7 for information on tuning the CMS garbage collector. Another artifact to monitor when using the CMS garbage collector is the tenur- ing distribution enabled via the -XX:+PrintTenuringDistribution command line option. The tenuring distribution is a histogram showing the ages of objects in the young generation’s survivor spaces. When an object’s age exceeds the tenuring threshold it is promoted from the young generation space to the old generation space. The tenuring threshold and how to monitor the tenuring distribution along with why it is important to monitor is explained in the “Tenuring Threshold Explained” and “Monitoring the Tenuring Threshold” sections of Chapter 7. If objects are promoted too quickly to the old generation space and the CMS gar- bage collector cannot keep free enough available space to meet the rate that objects are promoted from young generation to old generation, it leads to the old genera- tion running out of available space, a situation known as a concurrent mode fail- ure. A concurrent mode failure can also occur if the old generation space becomes fragmented to a point where there is no longer a hole in the old generation space large enough to handle an object promotion from the young generation space. -XX:+PrintGCDetails reports a concurrent mode failure in the garbage collec- tion output with the text (concurrent mode failure). When a concurrent mode failure occurs, the old generation space is garbage collected to free available space, and it is compacted to eliminate fragmentation. This operation requires all Java application threads be stopped, and it can take a noticeably lengthy duration of time to execute. Therefore, if you observe concurrent mode failures, you should tune the JVM using the guidance in Chapter 7, especially the section on fine-tuning the application for low latency. Including Date and Time Stamps The HotSpot VM includes command line options to include a date or time stamp on each reported garbage collection. The -XX:+PrintGCTimeStamps command line option prints a time stamp that is the number of elapsed seconds since the JVM started. It is printed at each garbage collection. The following is example minor ptg6882136 118 Chapter 4 JVM Performance Monitoring garbage collection output from -XX:+PrintGCTimeStamps being used in combina- tion with -XX:+PrintGCDetails and the throughput garbage collector. (The output is spread across several lines for easier reading.) 77.233: [GC [PSYoungGen: 99952K->14688K(109312K)] 422212K->341136K(764672K), 0.0631991 secs] [Times: user=0.83 sys=0.00, real=0.06 secs] Notice the -XX:+PrintGCDetails output is prefixed with a time stamp represent- ing the number of seconds since the JVM started. The output for full garbage collec- tions also prefixes the output with a time stamp. In addition, a time stamp is also printed when using the concurrent garbage collector. Java 6 Update 4 and later include a -XX:+PrintGCDateStamps command line option. It produces an ISO 8601 date and time stamp. The date and time stamp have the following form; YYYY-MM-DD-T-HH-MM-SS.mmm-TZ, where: YYYY is the four-digit year. MM is the two-digit month; single-digit months are prefixed with 0. DD is the two-digit day of the month; single-digit days are prefixed with 0. T is a literal that denotes a date to the left of the literal and a time of day to the right. HH is the two-digit hour; single-digit hours are prefixed with 0. MM is the two-digit minute; single-digit minutes are prefixed with 0. SS is the two-digit second; single-digit seconds are prefixed with 0. mmm is the three-digit milliseconds; single- and two-digit milliseconds are pre- fixed with 00 and 0, respectively. TZ is the time zone offset from GMT. Although the time zone offset from GMT is included in the output, the date and time of day are not printed as GMT time. The date and time of day are adjusted to local time. The following example output uses -XX:+PrintGCDateStamps together with -XX:+PrintGCDetails when using the throughput garbage collector. The out- put is spread across several lines for easier reading. 2010-11-21T09:57:10.518-0500:[GC [PSYoungGen: 99952K->14688K(109312K)] 422212K->341136K(764672K), 0.0631991 secs] [Times: user=0.83 sys=0.00, real=0.06 secs] ptg6882136 Garbage Collection 119 The full garbage collections with the throughput garbage collector also prefixes a date and time stamp when using -XX:+PrintGCDateStamps. In addition, a date and time stamp are printed when using the concurrent garbage collector. The use of date and/or time stamps allows you to measure both the duration of minor and full garbage collections along with the frequency of minor and full gar- bage collections. With the inclusion of date and/or time stamps, you can calculate an expected frequency that minor and full garbage collections occur. If the garbage col- lection durations or frequency exceed the application’s requirements, consider tuning the JVM as described in Chapter 7. 77.233: [GC [PSYoungGen: 99952K->14688K(109312K)] 422212K->341136K(764672K), 0.0631991 secs] [Times: user=0.83 sys=0.00, real=0.06 secs] Since -Xloggc includes a time stamp automatically in its output, it is easy to deter- mine when minor and full garbage collections occur. In addition, you can also calcu- late the frequency of minor and full garbage collections. With the time stamps, you can calculate the expected frequency that minor and full garbage collections occur. If the garbage collection durations or frequency exceed the application’s requirements, consider tuning the JVM as described in Chapter 7. Application Stopped Time and Application Concurrent Time The HotSpot VM can report the amount of time an application runs between safepoint operations and the amount of time the HotSpot VM blocks executing Java threads To facilitate offline analysis of garbage collection statistics and to direct garbage collection output to a file, the -Xloggc: HotSpot VM command line option can be used. is the name of the file where you want the garbage collection data to be stored. Offline analysis of garbage collection data can represent a wider span of time and the ability to identify patterns without having to observe the data as the application is running. When -XX:+PrintGCDetails is used in combination with -Xloggc:, the output is automatically prefixed with a time stamp even without specify- ing -XX:+PrintGCTimeStamps. The time stamp is printed in the same way as -XX:+PrintGCTimeStamps is printed. Following is an example of -Xloggc: being used in combination with -XX:+PrintGCDetails with the throughput garbage collector. (The output is spread across several lines for easier reading.) -Xloggc ptg6882136 120 Chapter 4 JVM Performance Monitoring using the command line options -XX:+PrintGCApplicationConcurrentTime and -XX:+PrintGCApplicationStoppedTime. Observing safepoint operations using these two command line options can provide useful information in understand- ing and quantifying the impact of JVM induced latency events. It can also be used to identify whether a latency event of interest is the result of a JVM induced latency from a safepoint operation, or if the latency event occurred as a result of something in the application. Tip Chapter 3, “JVM Overview,” describes safepoint operations in more detail. An example using -XX:+PrintGCApplicationConcurrentTime and -XX:+PrintGCApplicationStoppedTime in addition to -XX:+PrintGCDetails is shown in the following: Application time: 0.5291524 seconds [GC [ParNew: 3968K->64K(4032K), 0.0460948 secs] 7451K->6186K(32704K), 0.0462350 secs] [Times: user=0.01 sys=0.00, real=0.05 secs] Total time for which application threads were stopped: 0.0468229 seconds Application time: 0.5279058 seconds [GC [ParNew: 4032K->64K(4032K), 0.0447854 secs] 10154K->8648K(32704K), 0.0449156 secs] [Times: user=0.01 sys=0.00, real=0.04 secs] Total time for which application threads were stopped: 0.0453124 seconds Application time: 0.9063706 seconds [GC [ParNew: 4032K->64K(4032K), 0.0464574 secs] 12616K->11187K(32704K), 0.0465921 secs] [Times: user=0.01 sys=0.00, real=0.05 secs] Total time for which application threads were stopped: 0.0470484 seconds The output shows the application ran for approximately .53 to .91 seconds with minor garbage collection pauses of approximately .045 to .047 seconds. That equates to about 5% to 8% overhead for minor garbage collections. Also notice there are no additional safepoints between each of the minor garbage collections. If there happens to be additional safepoints between garbage collections, the output will show Application time: and Total time for which appli- cation threads were stopped: messages for each safepoint that occurs between garbage collections. ptg6882136 Garbage Collection 121 Explicit Garbage Collections Explicit garbage collections can be identified in garbage collection output easily. The garbage collection output contains text indicating the full garbage collection is the result of an explicit call to System.gc(). Following is an example of a full garbage col- lection initiated with a call to System.gc() using the -XX:+PrintGCDetails com- mand line option. Again, the output is spread across several lines for easier reading. [Full GC (System) [PSYoungGen: 99608K->0K(114688K)] [PSOldGen: 317110K->191711K(655360K)] 416718K->191711K(770048K) [PSPermGen: 15639K->15639K(22528K)], 0.0279619 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] Notice the (System) label following the Full GC text. This indicates this is a System.gc() induced full garbage collection. If you observe an explicit full garbage collection in the garbage collection logs, investigate the reason why it is being used and then decide whether the call to System.gc() should be removed from the source code, or whether it should be disabled. Recommended Command Line Options for Monitoring Garbage Collection A minimum set of HotSpot VM garbage collection command line options to monitor garbage collection are -XX:+PrintGCDetails along with either -XX:+PrintGCTimeStamps or -XX:+PrintGCDateStamps. It may also be useful to use -Xloggc: to save the data to a file so the data can be further analyzed offline. Offline Analysis of Garbage Collection Data The purpose of doing offline analysis is to summarize garbage collection data and look for patterns of interest in the data. Offline analysis of garbage collection data can be done in a variety of different ways such as loading the data into a spreadsheet or using a charting tool to plot the data. GCHisto is a tool designed to do offline analysis. GCHisto is a free tool that can be downloaded at http://gchisto.dev.java.net. GCHisto reads garbage collection data saved in a file and presents both a tabular and graphical view of the data. Figure 4-1 shows a tabular summary from its GC Pause Stats tab. ptg6882136 122 Chapter 4 JVM Performance Monitoring The GC Pause Stats subtab provides information such as the number of, the over- head of, and duration of garbage collections. The additional GC Pause Stats subtabs narrow the focus to one of the aforementioned categories. All garbage collections or phases of garbage collections that induce stop-the- world pauses have a row in the table in addition to a total on the top row. Figure 4-1 shows data from the concurrent garbage collector. Recall from Chapter 3 that the concurrent garbage collector, in addition to minor (young) and major garbage col- lections, also has two stop-the-world garbage collection pauses: the CMS initial mark and CMS remark. If you observe initial mark or remark pauses greater than minor garbage collection pauses, it suggests the JVM requires tuning. Initial mark and remark phases are expected to be shorter in duration than minor garbage collections. When viewing statistics from the throughput garbage collector, since it has only two stop-the-world garbage collection pauses, only minor and full garbage collections are shown in the GC Pause Stats tab of GCHisto. The number of minor versus full garbage collections provides a sense of the fre- quency of full garbage collections. This information along with the full garbage col- lection pause times can be evaluated against the application’s requirements for frequency and duration of full garbage collections. The garbage collection overhead (the Overhead % column) is an indicator of how well the garbage collector is tuned. As a general guideline, concurrent garbage collec- tion overhead should be less than 10%. It may be possible to achieve 1% to 3%. For the throughput garbage collector, garbage collection overhead near 1% is considered as having a well-tuned garbage collector. 3% or higher can be an indication that tun- ing the garbage collector may improve the application’s performance. It is important to understand there is a relationship between garbage collection overhead and the size of the Java heap. The larger the Java heap, the better the opportunity for lower Figure 4-1 GC Pause Stats in GCHisto ptg6882136 Garbage Collection 123 garbage collection overhead. Achieving the lowest overhead for a given Java heap size requires JVM tuning. In Figure 4-1, the garbage collection overhead is a little over 14%. Applying the general guidelines just mentioned, JVM tuning will likely reduce its overhead. The maximum pause times, the far right column, can be evaluated against the application’s worst case garbage collection induced latency requirements. If any of the maximum pause times exceed the application’s requirements, tuning the JVM may be a consideration. The degree and how many pause times exceed the applica- tion’s requirements dictate whether JVM tuning is a necessity. The minimum, maximum, average, and standard deviation provide information about the distribution of pause times. The distribution of pause times can be viewed by clicking on the GC Pause Distribution tab as shown in Figure 4-2. The default view for the GC Pause Distribution shows the distribution of all garbage collection pauses. Which pause type is included in the view is controlled by selecting or deselecting the appropriate check box. The y-axis is the count of pauses and the x-axis is the pause time duration of the garbage collection event. It is generally more useful to look at full garbage collections separately since they usually have the longest duration. Looking at only minor garbage collections offers the possibility to see wide variations in pause times. A wide distribution in pause times can be an indication of wide swings in object allocation rates or promotion rates. If you observe a wide distribution of pause times, you should look at the GC Timeline tab to identify peaks in garbage collection activity. An example is shown in Figure 4-3. The default view for the GC Timeline shows all garbage collection pauses through the entire time line. To see time stamps at the bottom of the graph (the x-axis), you must have garbage collection statistics that include either -XX:+PrintGCTimeStamps, -XX:+PrintGCDateStamps, or used -Xloggc. For Figure 4-2 GC Pause Distribution ptg6882136 124 Chapter 4 JVM Performance Monitoring every garbage collection pause that occurred, a tick is put on the graph illustrating the duration of the pause (y-axis) and when the pause occurred relative to the start of the JVM (the x-axis). There are several patterns to look for in a time line. For example, you should take notice of when full garbage collections occur and how frequently. Selecting only full garbage collections as the pause type is useful for this analysis. With the time line you can observe when the full garbage collections occur relative to the start of the JVM to get a sense of when they occurred. Selecting only minor garbage collections as the pause type to show allows you to observe peaks, or possibly repeating peaks, in garbage collection duration over time. Any observed peaks or repeating patterns can be mapped back to application logs to get a sense of what is happening in the system at that time when the peaks occur. The use cases being executed at those time periods can be candidates to further explore for object allocation and object retention reduction opportunities. Reducing object allocation and object retention during these busiest garbage collection activity time periods reduces the frequency of minor garbage collections and potentially reduces the frequency of full garbage collections. An area of the time line can be zoomed in on by selecting an area of interest with the mouse, as illustrated in Figure 4-4. Zooming in allows you to narrow the focus of the time line to a specific area to see each garbage collection pause. You can zoom back out by pressing the right mouse button anywhere in the graph and selecting Auto Range > Both Axes from the context sensitive menu. GCHisto also provides the capability to load more than one garbage collection log at a time via the Trace Management tab. When multiple garbage collection logs are loaded, there is a separate tab for each garbage collection log, which allows you to Figure 4-3 GC Timeline tab ptg6882136 Garbage Collection 125 easily switch between logs. This can be useful when you want to compare garbage collection logs between different Java heap configurations or between different appli- cation loads. Graphical Tools Garbage collection can also be monitored with graphical tools, which can make the identification of trends or patterns a little easier than traversing textual output. The following graphical tools can be used to monitor the HotSpot VM: JConsole, Visu- alGC, and VisualVM. JConsole is distributed with Java 5 and later JDKs. VisualGC was originally developed and packaged with jvmstat. It is available as a free download at http://java.sun.com/performance/jvmstat. VisualVM is an open source project that brings together several existing light- weight Java application monitoring and profiling capabilities into a single tool. VisualVM is included in Java 6 Update 6 and later JDKs. It is also available as a free download from http://visualvm.dev.java.net. JConsole JConsole is a JMX (Java Management Extensions) compliant GUI tool that con- nects to a running Java 5 or later JVM. Java applications launched with a Java 5 JVM must add the -Dcom.sun.management.jmxremote property to allow the JConsole to connect. Java applications launched using Java 6 and later JVMs do not require this property. The following examples illustrate how to connect JCon- sole to an example demo application shipped with the JDK called Java2Demo. Using a Java 5 JDK, the Java2Demo application can be started using the following command line. Figure 4-4 GC Timeline zooming ptg6882136 126 Chapter 4 JVM Performance Monitoring On Solaris or Linux: $ /bin/java -Dcom.sun.management.jmxremote -jar /demo/jfc/Java2D/Java2Demo.jar is the path and directory where a Java 5 JDK is installed. On Windows: \bin\java -Dcom.sun.management.jmxremote -jar \demo\jfc\Java2D\Java2Demo.jar is the path and directory where a Java 5 JDK is installed. To start JConsole on either with a Java 6 or later JVM the -Dcom.sun.management. jmxremote property is not required as an argument to JConsole. On Solaris or Linux: $ /bin/jconsole is the path and directory where Java 5 JDK is installed. On Windows: \bin\jconsole is the path and directory where a Java 5 JDK is installed. When JConsole is launched it automatically discovers and provides the oppor- tunity to connect to Java applications running locally or remotely. The connection dialogs differ slightly between the JConsole version shipped in Java 5 versus Java 6 as shown in Figure 4-5 and Figure 4-6, respectively. Figure 4-5 Java 5 JConsole connection dialog ptg6882136 Garbage Collection 127 In Java 5 JConsoles, the applications listed in the connection dialog that can be monitored are applications that have been started with the -Dcom.sun.management. jmxremote property and applications that share the same user credentials as those of the user who has started JConsole. With Java 6 JConsole, the applications listed in the connection dialog that can be monitored are applications that are Java 6 applications and Java 5 applications that have been started with the -Dcom.sun.management.jmxremote property, which both share the same user credentials as those of the user who has started JConsole. Java 5 applications that have not been started with the -Dcom.sun.management. jmxremote property that share the same user credentials as those of JConsole are listed but grayed out. To monitor an application on a local system you select the Name and PID of the application from the list and click the Connect button. Remote monitoring is advanta- geous when you want to isolate the system resource consumption from the JConsole application from the system being monitored. To monitor an application on a remote system, the application to be monitored must be started with remote management enabled. Enabling remote management involves identifying a port number to com- municate with the monitored application and establishing password authentication along with optionally using SSL for security. Information on how to enable remote management can be found in the Java SE 5 and Java SE 6 monitoring and manage- ment guides: Figure 4-6 Java 6 JConsole connection dialog ptg6882136 128 Chapter 4 JVM Performance Monitoring Java SE 5 — http://java.sun.com/j2se/1.5.0/docs/guide/management/index.html Java SE 6 — http://java.sun.com/javase/6/docs/technotes/guides/management/toc. html Tip More than one Java application can be monitored with JConsole at any time by selecting the Connection > New Connection menu and selecting a different Name and PID pair. Once a JConsole is connected to an application it will load six tabs. The default JConsole display between Java 5 and Java 6 differs. Java 6’s JConsole displays a graphical representation of heap memory, thread, classes, and CPU usage. In con- trast, Java 5’s JConsole displays the same information but in a textual form. For the purposes of monitoring JVM garbage collection, the Memory tab is the most useful. The Memory tab in both Java 5 and Java 6 JConsole are the same. Figure 4-7 shows the JConsole Memory tab. Figure 4-7 JConsole Memory tab ptg6882136 Garbage Collection 129 The Memory tab uses charts to graphically show the JVM’s use of memory con- sumption over a period of time. Depending on the JVM being monitored and the gar- bage collector being used, the spaces that make up the Java heap, or memory pools as they are called in JConsole, may vary. But from their names it is straightforward to map them to following HotSpot VM space names: Eden space. The memory pool where almost all Java objects are allocated. Survivor space. The memory pool containing objects that have survived at least one garbage collection of the eden space. Old or tenured space. The memory pool containing objects that have sur- vived some garbage collection age threshold. Permanent generation space. The memory pool containing all the reflec- tive data of the JVM such as class and method objects. If the monitored JVM supports class data sharing, this space will be divided into read-only and read- write areas. Code cache. Applies to the HotSpot VM and contains memory that is used by the JIT compiler and for the storage of JIT compiled code. JConsole defines heap memory as the combination of eden space, survivor space, and old or tenured space. Non-heap memory is defined as the combination of perma- nent generation space and code cache. You can display charts of heap memory usage or non-heap memory usage by choosing one of the options in the Chart drop-down menu. You can also view charts of specific spaces. Additionally, clicking on any of the Heap or Non-Heap bar charts in the bottom right corner switches the chart to display the selected Heap or Non-Heap space. If you hover the mouse over any of the Heap or Non-Heap bar charts in the lower right corner, a tool tip displays text indicating the memory pool or space name. A pattern to watch for is whether the survivor space remains full for an extended period of time. This is an indication that survivor spaces are overflowing and objects are getting promoted into the old generation space before they have an opportunity to age. Tuning the young generation space can address survivor spaces overflowing. You can also change the time range over which memory usage is displayed by selecting an option in the Time Range drop-down menu. In the left-hand portion of the Details panel (the bottom left panel), several current JVM memory metrics are displayed including Used. The amount of memory currently used, including the memory occupied by all Java objects, both reachable and unreachable. Committed. The amount of memory guaranteed to be available for use by the JVM. The amount of committed memory may change over time. The JVM ptg6882136 130 Chapter 4 JVM Performance Monitoring may release memory to the system, and the amount of committed memory could be less than the amount of memory initially allocated at startup. The amount of committed memory will always be greater than or equal to the amount of used memory. Max. The maximum amount of memory that can be used for memory manage- ment. Its value may change or be undefined. A memory allocation may fail if the JVM attempts to increase the used memory to be greater than committed memory, even if the amount used is less than or equal to max (for example, when the system is low on virtual memory). GC time. The cumulative time spent in stop-the-world garbage collections and the total number of garbage collection invocations including concurrent garbage collection cycles. Multiple rows may be shown, each of which represents the garbage collector used in the JVM. Additional garbage collection monitoring capabilities are possible with JCon- sole. Many of these capabilities are described in the JConsole documentation found at Java SE 5 — http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole. html Java SE 6 — http://java.sun.com/javase/6/docs/technotes/guides/management/ jconsole.html VisualVM VisualVM is an open source graphical tool that began development in 2007. VisualVM was introduced in the Java 6 Update 7 JDK and is considered the second generation of the JConsole tool. VisualVM integrates several existing JDK software tools and lightweight memory monitoring tools such as JConsole along with adding profiling capabilities found in the popular NetBeans Profiler. VisualVM is designed for both production and development environments and further enhances the capabilities of monitoring and performance analysis for the Java SE platform. It also utilizes the NetBeans plug-in architecture, which allows the ability to easily add components, add plug-ins, or extend VisualVM’s existing components or plug-ins to performance monitor or profile any application. VisualVM requires a Java 6 version to run, but it can monitor Java 1.4.2, Java 5, or Java 6 applications locally or remotely. However, there are some limitations to VisualVM’s capabilities depending on the Java version used by the Java application being monitored and whether the Java application is running locally or remotely to VisualVM. Table 4-1 illustrates the VisualVM features available for a given Java application running with a given version of a JDK. ptg6882136 Garbage Collection 131 VisualVM also includes profiling capabilities. Although profiling is covered in detail in Chapter 5, “Java Application Profiling,” VisualVM’s remote profiling capa- bilities are covered in this chapter since VisualVM’s remote profiling is lightweight and fits well with monitoring activities. VisualVM can be launched from Windows, Linux, or Solaris using the following command line. (Note the command name is jvisualvm, not just visualvm.) Table 4.1 VisualVM Feature Table Feature JDK 1.4.2 Local and Remote JDK 5.0 Local and Remote JDK 6.0 (Remote) JDK 6.0 (Local) Overview • • • • System Properties (in Overview) • Monitor • • • • Threads • • • Profiler • Thread Dump • Heap Dump • Enable Heap Dump on OOME • MBean Browser (plug-in) • Wrapper for JConsole plug-ins (plug-in) ••• VisualGC (plug-in) • • • • \bin\jvisualvm is the path and directory where JDK 6 Update 6 or later is installed. If you have downloaded the standalone VisualVM from java.net, VisualVM can be launched from Windows, Linux, or Solaris using the following command line. (Note the open source version of VisualVM available on java.net is launched using visualvm rather than jvisualvm as is done when VisualVM is launched from a JDK distribution.) ptg6882136 132 Chapter 4 JVM Performance Monitoring Alternatively, you can launch VisualVM from a directory window display such as Windows Explorer by traversing to the VisualVM installation directory and double- clicking on the VisualVM executable icon. The initial VisualVM display shows an Applications window on the left and an empty monitoring window on the right, as shown in Figure 4-8. The Applications panel of VisualVM has three major nodes in an expandable tree. The first major node, Local, contains a list of local Java applications VisualVM can monitor. The second node, Remote, contains a list of remote hosts and Java applications on each remote host VisualVM can monitor. The third node, Snapshots, contains a list of snapshot files. With VisualVM you can take a snapshot of a Java application’s state. When a snapshot is taken, the Java application’s state is saved to a file and listed under the Snapshots node. Snapshots can be useful when you want to capture some important state about the application or to compare it against a different snapshot. Local Java applications are automatically identified by VisualVM at Java applica- tion launch time and at VisualVM launch time. For example, as shown in Figure 4-8, \bin\visualvm is the path and directory where VisualVM is installed. Figure 4-8 VisualVM ptg6882136 Garbage Collection 133 VisualVM automatically discovered the Java applications shown on the Local node. As additional Java applications are launched, VisualVM automatically detects them and adds them to the local node’s list. As Java applications are shut down, VisualVM automatically removes them. To monitor remote Java applications, configuration must be done on the remote system where you want to monitor the Java application. The remote system must be configured to run the jstatd daemon. The jstatd daemon is shipped in Java 5 and Java 6 JDKs. It is not included with Java 5 or Java 6 JREs. You can find the jstatd daemon in the same directory as jvisualvm and the java launcher. The jstatd daemon launches a Java RMI server application that watches for the creation and termination of HotSpot VMs and provides an interface to allow remote monitoring tools such as VisualVM to attach and monitor Java applications remotely. The jstatd daemon must be run with the same user credentials as those of the Java applications to be monitored. Since jstatd can expose instrumentation of JVMs, it must employ a security manager and requires a security policy file. Consideration should be given to the level of access granted via the security policy so that the monitored JVM’s security is not compromised. The policy file used by jstatd must conform to Java’s security policy specification. The following is an example policy file that can be used with jstatd: grant codebase ”file:${java.home}/../lib/tools.jar” { permission java.security.AllPermission; }; To use the preceding example policy and start the jstatd daemon, assuming the preceding policy is saved in a file called jstatd.policy, at the command line you would execute the following command: Tip Note that the preceding example policy file allows jstatd to run without any security exceptions. This policy is less liberal than granting all permissions to all codebases but is more liberal than a policy that grants the minimal permissions to run the jstatd server. More restrictive security than this example can be specified in a policy to further limit access. However, if security concerns cannot be addressed with a policy file, the safest approach is to not run jstatd and use the monitoring tools locally rather than connecting remotely. jstatd -J-Djava.security.policy=/jstatd.policy ptg6882136 134 Chapter 4 JVM Performance Monitoring Once the jstatd daemon is running on the remote system, you can verify the local system can attach to the remote jstatd daemon by running the jps command and providing the hostname of the remote system. jps is a command that lists the Java applications that can be monitored. When jps is supplied a hostname, it attempts to connect to the remote system’s jstatd daemon to discover which Java applications can be monitored remotely. When no optional hostname is supplied to jps, it returns a list of Java applications that can be monitored locally. Suppose the remote system where you have configured and have the jstatd dae- mon running is called halas. On the local system, you would execute the following jps command to verify the connectivity to the remote system. Tip Additional details on how to configure jstatd can be found at http://java.sun.com/javase/6/ docs/technotes/tools/share/jstatd.html. $ jps halas 2622 Jstatd If the jps command returns a Jstatd in its output, you have successfully config- ured the remote system’s jstatd daemon. The number preceding the Jstatd in the output is the process id of the jstatd daemon process. For the purposes of verifying remote connectivity, the process id is not important. To use VisualVM to monitor a remote Java application, it needs to be configured with the remote host’s name or IP address. This is done by right-clicking on the Remote node in VisualVM’s Applications panel and adding the remote host informa- tion. If you want to monitor Java applications on multiple remote hosts, you must configure each remote host with a jstatd daemon, using the procedure described earlier. Then, add each remote host’s information in VisualVM. VisualVM automati- cally discovers and provides a list of Java applications that can be monitored. Again, recall that the remote Java applications must match the user credentials of the user running VisualVM and jstatd along with those that meet the permissions specified in the jstatd policy file. Figure 4-9 shows VisualVM with a remote system configured and the Java applications it can monitor. To monitor an application, you can either double-click on an application name or icon under the Local or Remote node. You can also right-click on the application name or icon and select Open. Any of these actions opens a window tab in the right panel of VisualVM. Local applications running with Java 6 or later have additional subtabs. ptg6882136 Garbage Collection 135 The number of subtab windows displayed in the right panel of VisualVM depends on the application’s Java version, whether it is running locally or remotely, and whether any additional plug-ins have been added to VisualVM. The minimum set of window subtabs in the right panel is the Overview and Monitor subtabs. The Overview window provides a high level overview of the monitored application by showing the process id, host name where the application is running, main Java class name, any arguments passed to the application, the JVM name, the path to the JVM, any JVM flags used, whether heap dump on out of memory error is enabled or disabled, number of thread dumps or heap dumps have been taken, and, if available, the monitored application’s system properties. The Monitor window displays heap usage, permanent generation space usage, classes loaded information, and number of threads. An example of the Monitor window monitoring an application running remotely under Java 6 is shown in Figure 4-10. If a JMX connection is configured for use with the remote application, you can also request a garbage collection or heap dump from the Monitor window. To configure a JMX on the remote application, it must be started with at least the following system properties: com.sun.management.jmxremote.port= com.sun.management.jmxremote.ssl= com.sun.management.jmxremote.authenticate= Figure 4-9 VisualVM configured to monitor remote applications ptg6882136 136 Chapter 4 JVM Performance Monitoring To configure VisualVM to connect to the remote application via JMX, use the File > Add JMX Connection menu item. In the Add JMX Connection form, add the following information for each of the fields: hostname: for the Connection field. For example, if the remote application is running on a host named halas, and you configured the remote appli- cation with com.sun.management.jmxremote.port=4433, you enter halas:4433 in the Connection field. Optionally enter a display name to be displayed in VisualVM to identify the remote application via the JMX connection. By default, VisualVM uses what you entered in the Connection field as the display name. If you set com.sun.management.jmxremote.authenticate=true, enter the username and password in the Username and Password fields who are authenticated to connect remotely. Figure 4-10 VisualVM Monitor subtab ptg6882136 Garbage Collection 137 See VisualVM JMX connection documentation for additional information on con- figuring JMX for remote monitoring with VisualVM, http://download.oracle.com/ javase/6/docs/technotes/guides/visualvm/jmx_connections.html. After a JMX connection is configured, an additional icon is displayed in VisualVM’s Application’s panel representing that a remote JMX connection has been configured to the remote application. Configuring a JMX connection for remote applications in VisualVM increases the monitoring capabilities. For example, the Monitor window also shows CPU usage by the application and the ability to induce a full garbage collection or heap dump, as shown in Figure 4-11. In addition to more capabilities in the Monitor window, an additional Threads window is also available. The Threads window shows a view of threads in the applica- tion along with a color indicating whether the thread is currently running, sleeping, blocked, waiting, or contending on a monitor lock. The Threads window is available as a view for all locally monitored applications. The Threads window offers insight into which threads are most active, those that are involved in acquiring and releasing locks. The Threads window can be useful Figure 4-11 Remote monitoring via JMX ptg6882136 138 Chapter 4 JVM Performance Monitoring to observing specific thread behavior in an application, especially when operating system monitoring suggests the application may be experiencing lock contention. An additional option available in the Threads window is the ability to create a thread dump by clicking the Thread Dump button. When a thread dump is requested, VisualVM adds a window tab displaying the thread dump and also appends a thread dump entry to the monitored application entry in the Application’s window below the application being monitored. It is important to note that thread dumps are not persisted or available once VisualVM has been closed unless they are saved. Thread dumps can be saved by right-clicking on the thread dump icon or label below the application listed in the Applications panel. Thread dumps can be reloaded in VisualVM at a later time by selecting the File > Load menu item and traversing to the directory where the thread dump was saved. VisualVM also offers profiling capabilities to both local and remote applications. Local profiling capabilities include both CPU and memory profiling for Java 6 appli- cations. For monitoring purposes, the feature of monitoring CPU utilization or moni- toring memory utilization as the application is running can be useful. However, care should be taken when invoking either of these features on a production system as they may heavily tax the running application. Being able to monitor CPU utiliza- tion while an application is running can provide information as to which methods are the busiest during times when specific events are occurring. For example, a GUI application may exhibit performance issues only in a specific view. Hence, being able to monitor the GUI application when it is in that view can be helpful in isolating the root cause. Remote profiling requires a JMX connection to be configured and is limited to CPU profiling. It does not include memory profiling. But heap dumps can be generated from the Sampler window. They can also be generated from the Threads window. Heap dumps can be loaded into VisualVM to analyze memory usage. To initiate remote profiling in VisualVM, first select and open a remote application from the Application panel that is configured with a JMX connection. Then select the Sampler window in the right panel. In the Sampler window click the CPU button to initiate remote profiling. Figure 4-12 shows the Sampler window after clicking the CPU button. The view of CPU utilization is presented with the method name consuming the most time at the top. The second column, Self Time %, provides a his- togram view of the method time spent per method relative to the time spent in other methods. The Self Time column represents the amount of wall clock time the method has consumed. The remaining column, Self Time (CPU), reports the amount of CPU time the method has consumed. Any of the columns can be sorted in ascending or descending order by clicking on the column name. A second click on a column causes the ordering to toggle back and forth between ascending or descending. Profiling can be stopped and resumed by clicking the Pause button. And a snap- shot can be captured by clicking the Snapshot button. After a taking a snapshot, ptg6882136 Garbage Collection 139 VisualVM displays the snapshot. The snapshot can be saved to disk. To save the snapshot to share it with another developer, to be able to load the snapshot at a later time or to compare it with another snapshot, you can export the snapshot to a file as shown in Figure 4-13. A saved snapshot can be loaded by VisualVM or the NetBeans Profiler. To load a saved snapshot with VisualVM, you select the File > Load from the main menu; filter the files by Profiler Snapshots (*.nps) to find the saved profile and have it loaded. In the snapshot window, the call tree showing the call stacks for all threads in the captured snapshot are displayed. Each call tree can be expanded to observe the call stack and method consuming the most time and CPU. At the bottom of the snapshot window you can also view Hot Spots, which is a listing of methods with the method consuming the most Self Time at the top of the table. A combined view of the Call Tree and Hot Spots is also available. In the combined view, as you click on a call stack in the Call Tree, the table of Hot Spot methods is updated to show only the methods in the selected call stack. Additional details on profiling Java applications can be found in Chapter 5. Figure 4-12 Remote CPU profiling ptg6882136 140 Chapter 4 JVM Performance Monitoring VisualVM also has the capability to load binary heap dumps generated using jmap, JConsole, or upon reaching an OutOfMemoryError and using the -XX:+Heap DumpOnOutOfMemoryError HotSpot VM command line option. A binary heap dump is a snapshot of all the objects in the JVM heap at the time when the heap dump is taken. To generate a binary heap dump using the Java 6 jmap command, you use jmap -dump:format=b,file= where is the path and filename of the binary heap dump file and is the process id of the JVM running the application. For Java 5, you use jmap -heap:format=b where is the process id of the Java application. Java 5’s jmap command places the heap dump in a file named heap.bin in the directory where the jmap command was executed. Java 6’s JConsole can also generate a heap dump using its HotSpotDiagnostics MBean. Once a binary heap dump has been generated, it can be loaded in VisualVM using the File > Load menu where analysis can be performed. Figure 4-13 Saving a snapshot ptg6882136 Garbage Collection 141 VisualGC VisualGC is a plug-in for VisualVM. VisualGC can monitor garbage collection, class loader, and JIT compilation activities. It was originally developed as a standalone GUI program. It can be used as both a standalone GUI or as a plug-in for VisualVM to monitor 1.4.2, Java 5, and Java 6 JVMs. When VisualGC was ported to a VisualVM plug-in some additional enhancements were made to make it easier to discover and connect to JVMs. The advantage of using the VisualGC plug-in over the standalone GUI is JVMs that are monitor-able are automatically discovered and displayed in VisualVM. With the standalone GUI, you have to identify the process id of the Java application you want to monitor and pass that as an argument to program launcher. The process id can be found using the jps command. An example use of the jps com- mand is described in the previous section as part of the jstatd daemon configuration setup. The VisualGC plug-in for VisualVM can be found in VisualVM’s Plug-in Center. The Plug-in Center is accessed in VisualVM via the Tools > Plugins menu. The Visu- alGC plug-in can be found on the Available Plug-ins tab. The standalone VisualGC GUI can be downloaded from http://java.sun.com/ performance/jvmstat/#Download. Regardless of whether you are using the VisualGC plug-in for VisualVM or the standalone VisualGC program, to monitor an application locally, both VisualGC and the application to be monitored must share the same user credentials. When moni- toring a remote application, the jstatd daemon must be configured and running with the same user credentials as the Java application to be monitored. How to configure and run the jstatd daemon is described in the previous section. Using the VisualGC plug-in for VisualVM is covered in this section since it is easier to use than the standalone GUI and VisualVM also offers other integrated monitoring capabilities. After the VisualGC plug-in has been added to VisualVM, when you monitor an application listed in the Applications panel, an additional window tab is displayed in the right panel labeled VisualGC (see Figure 4-14). VisualGC displays two or three panels depending on the garbage collector being used. When the throughput garbage collector is used, VisualGC shows two panels: the Spaces and Graphs panels. When the concurrent or serial garbage collector is Figure 4-14 Additional VisualGC window tab ptg6882136 142 Chapter 4 JVM Performance Monitoring used a third panel is shown below the Spaces and Graphs panels called Histogram. Figure 4-15 shows VisualGC with all panel spaces. Any of the three panels can be added or removed from the VisualGC window by selecting the appropriate check boxes in the upper-right corner. The Spaces panel provides a graphical view of the garbage collection spaces and their space utilization. This panel is divided into three vertical sections, one for each of the garbage collection spaces: Perm (Permanent) space, Old (or Tenured) space, and the young generation space consisting of eden, and two survivor spaces, S0 and S1. The screen areas representing these garbage collection spaces are sized proportionately to the maximum capacities of the spaces as they are allocated by the JVM. Each space is filled with a unique color indicating the current utilization of the space relative to its maximum capacity. The unique color is also consistently used for each of the garbage collection spaces where they exist in both the Graphs and Histogram panels. Figure 4-15 VisualGC ptg6882136 Garbage Collection 143 The memory management system within the HotSpot VM is capable of expand- ing and shrinking the garbage collected heap if the values set for -Xmx and -Xms differ. This is accomplished by reserving memory for the requested maximum Java heap size but committing real memory to only the amount currently needed. The relationship between committed and reserved memory is represented by the color of the background grid in each space. Uncommitted memory is represented by a lighter gray colored portion of the grid, whereas committed memory is represented by a darker gray colored portion. In many cases, the utilization of a space may be nearly identical to the committed amount of memory making it difficult to determine the exact transition point between committed and uncommitted space in the grid. The relationship between the sizes of the eden and survivor spaces in the young generation portion of the Spaces panel is usually fixed in size. The two survivor spaces are usually identical in size and its memory space fully committed. The eden space may be only partially committed, especially earlier in an application’s life cycle. When the throughput garbage collector, enabled via -XX:+UseParallelGC or -XX:+UseParallelOldGC, is used along with the adaptive size policy feature, which is enabled by default, the relationship or ratio between the sizes of the young gen- eration spaces can vary over time. When adaptive size policy is enabled, the sizes of the survivor spaces may not be identical and the space in young generation can be dynamically redistributed among the three spaces. In this configuration, the screen areas representing the survivor spaces and the colored region representing the uti- lization of the space are sized relative to the current size of the space, not the maxi- mum size of the space. When the JVM adaptively resizes the young generation space, the screen area associated with the young generation spaces updates accordingly. There are several things to watch for in the Spaces panel. For example, you should watch how quickly the eden space fills. Every fill and drop in the eden space repre- sents a minor garbage collection. The rate of the fill and drop represents the minor garbage collection frequency. By watching the survivor spaces you can see how on each minor garbage collection one of the survivor spaces is occupied and the other is empty. This observation provides an understanding of how the garbage collector cop- ies live objects from one survivor space to another at each minor garbage collection. More importantly, though, you should watch for survivor spaces overflowing. Survivor spaces overflowing can be identified by observing their occupancies at minor garbage collections. If you observe full or nearly full survivor spaces after each minor garbage collection and a growth in the space utilization in the Old generation space, survivor spaces may be overflowing. Generally, though, this observation is an indication that objects are being promoted from the young generation space to the old generation space. If they are promoted too early, or too quickly, it may result in an eventual full garbage collection. When a full garbage collection occurs, you observe the old generation space utilization drop. The frequency at which you observe a drop in the old generation space utilization is an indication of full garbage collection frequency. ptg6882136 144 Chapter 4 JVM Performance Monitoring The Graphs panel, shown previously in Figure 4-15, is the right panel of the Visu- alGC window. It plots performance statistics as a function of time for a historical view. This panel displays garbage collection statistics along with JIT compiler and class loader statistics. The latter two statistics are discussed later in this chapter. The resolution of the horizontal axis in each display is determined by the selected Refresh Rate, found just above the Spaces panel. Each sample in the Graphs panel historical view occupies 2 pixels of screen real estate. The height of each display depends on the statistic being plotted. The Graphs panel has the following displays: Compile Time. Discussed later in this chapter. Class Loader Time. Discussed later in this chapter. GC Time. Displays the amount of time spent in garbage collection activities. The height of this display is not scaled to any particular value. A nonzero value in this graph indicates that garbage collection activity occurred during the last interval. A narrow pulse indicates a relatively short duration, and a wide pulse indicates a long duration. The title bar indicates the total number of garbage collections and the accumulated garbage collection time since the start of the application. If the monitored JVM maintains the garbage collection cause and the last cause statistics, the cause of the most recent garbage collection is also displayed in the title bar. Eden Space. Displays the utilization of the eden space over time. The height of this display is fixed, and by default the data is scaled according to the current capacity of the space. The current capacity of the space can change depending on the garbage collector being used as the space shrinks and grows over time. The title bar displays the name of the space and its maximum and current capacity in parentheses followed by the current utilization of the space. In addi- tion, the title also contains the number and accumulated time of minor garbage collections. Survivor 0 and Survivor 1. Displays the utilization of the two survivor spaces over time. The height of each of these two displays is fixed, and by default the data is scaled according to the current capacity of the corresponding space. The current capacity of these spaces can change over time depending on the garbage collector. The title bar displays the name of the space and its maximum and current capacity in parentheses followed by the current utilization of the space. Old Gen. Displays the utilization of the old generation space over time. The height of the display is fixed, and by default the data is scaled according to the current capacity of the space. The current capacity of this space can change depending on the garbage collector. The title bar displays the name of the space ptg6882136 Garbage Collection 145 and its maximum and current capacity in parentheses followed by the current utilization of the space. In addition, the title also contains the number and accumulated time of full garbage collections. Perm Gen. Displays the utilization of the permanent generation space over time. The height of the display is fixed, and by default the data is scaled accord- ing to the current capacity of the space. The current capacity of this space can change depending on the garbage collector. The title bar displays the name of the space and its maximum and current capacity in parentheses followed by the current utilization of the space. The Histogram panel, shown previously in Figure 4-15, is displayed below the Spaces and Graphs panels when the concurrent or serial garbage collector is used. The throughput garbage collector does not maintain a survivor age since it uses a different mechanism for maintaining objects in the survivor spaces. As a result, the Histogram panel is not displayed when monitoring a JVM that is using the through- put collector. The Histogram panel displays surviving object and object aging statistics. The Histogram panel contains a Parameters subpanel and a Histogram subpanel. The Parameters subpanel displays the current size of the survivor spaces and the param- eters that control the promotion of objects from young to old generation space. After each minor garbage collection, if an object is still live, its age is incremented. If its age exceeds a tenuring threshold age, which is calculated by the JVM at each minor garbage collection, it is promoted to the old generation space. The tenuring threshold calculated by the JVM is displayed as the Tenuring Threshold in the Parameters panel. The maximum tenuring threshold displayed in the Parameters panel is the maximum age at which an object is held in survivor spaces. An object is promoted from young to old generation based on the tenuring threshold, not the maximum tenuring threshold. Observing a frequent tenuring threshold less than maximum tenuring threshold is an indication objects are being promoted from young to old generation space too quickly. This is usually caused by survivor spaces overflowing. If a survivor space overflows, then objects with the highest ages are promoted to the old generation space until the utilization of the survivor space does not exceed the value displayed as the Desired Survivor Size in the Parameters panel. As mentioned earlier, survi- vor space overflow can cause old generation space to fill and result in a full garbage collection. The Histogram subpanel displays a snapshot of the age distribution of objects in the active survivor space after the last minor garbage collection. If the monitored JVM is Java 5 Update 6 or later, this panel contains 16 identically sized regions, one for each possible object age. If the monitored JVM is earlier than Java 5 Update 6, there are 32 identically sized regions. Each region represents 100% of the active ptg6882136 146 Chapter 4 JVM Performance Monitoring survivor space and is filled with a colored area that indicates the percentage of the survivor space occupied by objects of a given age. As an application runs, you can observe long-lived objects traverse through each of the age regions. The larger the space occupied by long-lived objects, the larger the blip you will observe migrate through the age regions. When the tenuring threshold is less than the maximum tenuring threshold you see no utilization in the regions representing ages greater than the tenuring threshold since those objects have been promoted to the old generation space. JIT Compiler There are several ways to monitor HotSpot JIT compilation activity. Although the result of JIT compilation results in a faster running application, JIT compilation requires computing resources such as CPU cycles and memory to do its work. Hence, it is useful to observe JIT compiler behavior. Monitoring JIT compilation is also useful when you want to identify methods that are being optimized or in some cases deopti- mized and reoptimized. A method can be deoptimized and reoptimized when the JIT compiler has made some initial assumptions in an optimization that later turned out to be incorrect. To address this scenario, the JIT compiler discards the previous opti- mization and reoptimizes the method based on the new information it has obtained. To monitor the HotSpot JIT compiler, you can use the command line option -XX:+PrintCompilation. The -XX:+PrintCompilation command line option generates a line of output for every compilation performed. An example of this output is shown here: 7 java.lang.String::indexOf (151 bytes) 8% ! sun.awt.image.PNGImageDecoder::produceImage @ 960 (1920 bytes) 9 ! sun.awt.image.PNGImageDecoder::produceImage (1920 bytes) 10 java.lang.AbstractStringBuilder::append (40 bytes) 11 n java.lang.System::arraycopy (static) 12 s java.util.Hashtable::get (69 bytes) 13 b java.util.HashMap::indexFor (6 bytes) 14 made zombie java.awt.geom.Path2D$Iterator::isDone (20 bytes) See Appendix A, “HotSpot VM Command Line Options of Interest,” for a detailed description of the output from -XX:+PrintCompilation. There are graphical tools that can monitor JIT compilation activity. However, they do not provide as much detail as the -XX:+PrintCompilation. At the time of this writing, JConsole, VisualVM, or the VisualGC plug-in for VisualVM do not provide information on which methods are being compiled by the JIT compiler. They only provide information that JIT compilation is taking place. Of the graphical tools, ptg6882136 Class Loading 147 VisualGC’s Graph window’s Compile Time panel, an example shown in Figure 4-16, may be the most useful since it shows pulses as JIT compilation activity occurs. It is easy to spot JIT compilation activity in VisualGC’s Graphs panel. The Compile Time display of VisualGC’s Graphs panel shows the amount of time spent compiling. The height of the panel is not scaled to any particular value. A pulse in the display indicates JIT compilation activity. A narrow pulse implies a relatively short duration of activity, and a wide pulse implies a long duration of activity. Areas of the display where no pulse exists indicates no JIT compilation activity. The title bar of the display shows the total number of JIT compilation tasks and the accumu- lated amount of time spent performing compilation activity. Class Loading Many applications utilize user-defined class loaders, sometimes called custom class loaders. A JVM loads classes from class loaders and may also decide to unload classes. When classes are loaded or unloaded depends on the JVM runtime environment and the usage of class loaders. Monitoring class loading activity can be useful, espe- cially with applications that utilize user-defined class loaders. As of this writing, the HotSpot VM loads all class metadata information in the permanent generation space. The permanent generation space is subject to garbage collection as its space becomes full. Hence, monitoring both class loading activity and permanent genera- tion space utilization can be important to an application realizing its performance requirements. Garbage collection statistics indicate when classes are unloaded from the permanent generation space. Unused classes are unloaded from the permanent generation space when addi- tional space is required as a result of other classes needing to be loaded. To unload classes from permanent generation, a full garbage collection is required. Therefore, an application may suffer performance issues as a result of full garbage collections trying to make space available for additional classes to be loaded. The following out- put shows a full garbage collection where classes are unloaded. Figure 4-16 VisualGC’s Graph window’s compile time panel [Full GC[Unloading class sun.reflect.GeneratedConstructorAccessor3] [Unloading class sun.reflect.GeneratedConstructorAccessor8] [Unloading class sun.reflect.GeneratedConstructorAccessor11] [Unloading class sun.reflect.GeneratedConstructorAccessor6] 8566K->5871K(193856K), 0.0989123 secs] ptg6882136 148 Chapter 4 JVM Performance Monitoring The garbage collection output indicates four classes were unloaded; sun. reflect.GeneratedConstructorAccessor3, sun.reflect.Generated ConstructorAccessor8, sun.reflect.GeneratedConstructorAccessor11, and sun.reflect.GeneratedConstructorAccessor6. The reporting of classes being unloaded during the full garbage collection provides evidence the permanent generation space may need to be sized larger, or its initial size may need to be larger. If you observe classes being unloaded during full garbage collections, you should use -XX:PermSize and -XX:MaxPermSize command line options to size the perma- nent generation space. To avoid full garbage collections that may expand or shrink the committed size of the permanent generation space, set -XX:PermSize and -XX:MaxPermSize to the same value. Note that if concurrent garbage collection of the permanent generation space is enabled, you may see classes being unloaded during a concurrent permanent generation garbage collection cycle. Since a concur- rent permanent generation garbage collection cycle is not a stop-the-world garbage collection, the application does not realize the impact of a garbage collection induced pause. Concurrent permanent generation garbage collection can only be used with the mostly concurrent garbage collector, CMS. Tip Additional guidelines and tips for tuning the permanent generation space including how to enable concurrent garbage collection of the permanent generation space can be found in Chapter 7. The graphical tools JConsole, VisualVM, and the VisualGC plug-in for VisualVM can monitor class loading. However, at the time of this writing, none of them display the class names of the classes being loaded or unloaded. The JConsole Classes tab, as shown in Figure 4-17, shows the number of classes currently loaded, number of classes unloaded, and total number of classes that have been loaded. VisualVM can also monitor class loading activity in the Monitor tab via the Classes display. It shows total number of classes loaded and the number of shared classes loaded. Observing whether class data sharing is enabled on a monitored JVM can be confirmed by looking at this view. Class data sharing is a feature where classes are shared across JVMs running on the same system to reduce their memory footprint. If class sharing is being utilized by the monitored JVM, there will be a horizontal line in the graph showing the number of shared classes loaded in addition to a hori- zontal line showing the total number of classes loaded similar to what is shown in Figure 4-18. You can also monitor class loading activity in the VisualGC Graph window by observing the Class Loader panel, as shown in Figure 4-19. ptg6882136 Class Loading 149 Figure 4-17 Total loaded classes and current loaded classes Figure 4-18 Observing class sharing in VisualVM Figure 4-19 Observing class loading activity in VisualGC ptg6882136 150 Chapter 4 JVM Performance Monitoring In the VisualGC Graphs window, a pulse in the Class Loader panel indicates class loading or unloading activity. A narrow pulse indicates a short duration of class load- ing activity, and a wide pulse indicates a long duration of class loading activity. No pulse indicates no class loading activity. The title bar of the Class Loader panel shows the number of classes loaded, the number of classes unloaded, and the accumulated class loading time since the start of the application. Observing pulses in the class loader panel and directly vertically below in the GC Time panel can be an indication the garbage collection activity that is occurring at the same time could be the result of the JVM’s permanent generation space being garbage collected. Java Application Monitoring Monitoring at the application level usually involves observing application logs that contain events of interest or instrumentation that provides some level of informa- tion about the application’s performance. Some applications also build-in monitoring and management capabilities using MBeans via Java SE’s monitoring and manage- ment APIs. These MBeans can be viewed and monitored using JMX compliant tools such as JConsole or using the VisualVM-MBeans plug-in within VisualVM. The VisualVM-MBeans plug-in can be found in the VisualVM plug-in center, via the Tools > Plugins menu. The GlassFish Server Open Source Edition (also referred to as GlassFish here- after) has a large number of attributes that can be monitored via MBeans. Using JConsole or VisualVM to monitor a GlassFish application server instance allows you to view the MBeans including their attributes and operations. Figure 4-20 shows a portion of the many GlassFish MBeans in the MBeans window in VisualVM using the VisualVM-MBeans plug-in. You can see on the left the expanded list of GlassFish MBeans in the com.sun. appserv folder. VisualVM can also be extended to monitor Java applications since it is built on the NetBeans Platform plug-in architecture. Plug-ins for VisualVM can be created as if they are NetBeans plug-ins. For example, a custom VisualVM plug- in to monitor a Java application can take advantage of the many rich features of NetBeans including its Visual Graph Library. Java applications that want to make available performance monitoring information can do so by developing a VisualVM plug-in. Several existing VisualVM plug-ins are available in Visual- VM’s plug-in center. Applications that have built JConsole plug-ins can use the VisualVM- JConsole plug-in to automatically integrate their custom JConsole plug-ins into VisualVM. ptg6882136 Java Application Monitoring 151 Quick Lock Contention Monitoring A trick often used by the authors to get a quick sense of where lock contention is occurring in a Java application is to capture several thread dumps using the JDK’s jstack command. This approach works well when operating in more of a monitoring role where the objective is to quickly capture some data rather than spending time to set up and configure a profiler where a more detailed analysis can be done. The following jstack output is from an application that has a set of reader threads and writer threads that share a single queue. Work is placed on the queue by writer threads, and reader threads pull work from the queue. Only the relevant stack traces are included to illustrate the usefulness of using jstack to rapidly find contended locks. In the jstack output the thread, Read Thread-33, has successfully acquired the shared queue lock, which is identified as a Queue object at address 0x22e88b10. This is highlighted in the output in bold as locked <0x22e88b10> (a Queue). Figure 4-20 GlassFish MBeans ptg6882136 152 Chapter 4 JVM Performance Monitoring All the other thread stack traces shown are waiting to lock the same lock held by the thread, Read Thread-33. This is highlighted in the other stack traces in bold as waiting to lock <0x22e88b10> (a Queue). ”Read Thread-33” prio=6 tid=0x02b1d400 nid=0x5c0 runnable [0x0424f000..0x0424fd94] java.lang.Thread.State: RUNNABLE at Queue.dequeue(Queue.java:69) - locked <0x22e88b10> (a Queue) at ReadThread.getWorkItemFromQueue(ReadThread.java:32) at ReadThread.run(ReadThread.java:23) ”Writer Thread-29” prio=6 tid=0x02b13c00 nid=0x3cc waiting for monitor entry [0x03f7f000..0x03f7fd94] java.lang.Thread.State: BLOCKED (on object monitor) at Queue.enqueue(Queue.java:31) - waiting to lock <0x22e88b10> (a Queue) at WriteThread.putWorkItemOnQueue(WriteThread.java:54) at WriteThread.run(WriteThread.java:47) ”Writer Thread-26” prio=6 tid=0x02b0d400 nid=0x194 waiting for monitor entry [0x03d9f000..0x03d9fc94] java.lang.Thread.State: BLOCKED (on object monitor) at Queue.enqueue(Queue.java:31) - waiting to lock <0x22e88b10> (a Queue) at WriteThread.putWorkItemOnQueue(WriteThread.java:54) at WriteThread.run(WriteThread.java:47) ”Read Thread-23” prio=6 tid=0x02b08000 nid=0xbf0 waiting for monitor entry [0x03c0f000..0x03c0fb14] java.lang.Thread.State: BLOCKED (on object monitor) at Queue.dequeue(Queue.java:55) - waiting to lock <0x22e88b10> (a Queue) at ReadThread.getWorkItemFromQueue(ReadThread.java:32) at ReadThread.run(ReadThread.java:23) ”Writer Thread-24” prio=6 tid=0x02b09000 nid=0xef8 waiting for monitor entry [0x03c5f000..0x03c5fa94] java.lang.Thread.State: BLOCKED (on object monitor) at Queue.enqueue(Queue.java:31) - waiting to lock <0x22e88b10> (a Queue) at WriteThread.putWorkItemOnQueue(WriteThread.java:54) at WriteThread.run(WriteThread.java:47) ”Writer Thread-20” prio=6 tid=0x02b00400 nid=0x19c waiting for monitor entry [0x039df000..0x039dfa14] java.lang.Thread.State: BLOCKED (on object monitor) at Queue.enqueue(Queue.java:31) - waiting to lock <0x22e88b10> (a Queue) at WriteThread.putWorkItemOnQueue(WriteThread.java:54) at WriteThread.run(WriteThread.java:47) ”Read Thread-13” prio=6 tid=0x02af2400 nid=0x9ac waiting for monitor entry [0x035cf000..0x035cfd14] java.lang.Thread.State: BLOCKED (on object monitor) ptg6882136 Bibliography 153 It is important to note the lock addresses, the hex number surrounded by < and >, are the same address. This is how locks are uniquely identified in jstack output. If lock addresses in the stack traces are different, they represent different locks. In other words, thread stack traces that have different lock addresses are threads that are not contending on the same lock. The key to finding contended locks in jstack output is searching for the same lock address across multiple stack traces and finding threads that are waiting to acquire the same lock address. Observing multiple thread stack traces trying to lock the same lock address is an indication the application is experiencing lock contention. If captur- ing multiple jstack outputs yields similar results of observing lock contention on the same lock, it is stronger evidence of a highly contended lock in the application. Also notice that the stack trace provides the source code location of the contended lock. Being able to find the location in the source code of a highly contended lock in a Java application has historically been a difficult task. Using jstack in the manner described here can help significantly in tracking down contended locks in applications. Bibliography Monitoring and Management for the Java Platform. http://download.oracle.com/ javase/1.5.0/docs/guide/management/. Java SE Monitoring and Management Guide. http://download.oracle.com/javase/6/ docs/technotes/guides/management/toc.html. Connecting to JMX Agents Explicitly. http://download.oracle.com/javase/6/docs/tech- notes/guides/visualvm/jmx_connections.html. VisualVM Features. https://visualvm.dev.java.net/features.html. jvmstat 3.0 Web site. http://java.sun.com/performance/jvmstat. at Queue.dequeue(Queue.java:55) - waiting to lock <0x22e88b10> (a Queue) at ReadThread.getWorkItemFromQueue(ReadThread.java:32) at ReadThread.run(ReadThread.java:23) ”Read Thread-96” prio=6 tid=0x047c4400 nid=0xaa4 waiting for monitor entry [0x06baf000..0x06bafa94] java.lang.Thread.State: BLOCKED (on object monitor) at Queue.dequeue(Queue.java:55) - waiting to lock <0x22e88b10> (a Queue) at ReadThread.getWorkItemFromQueue(ReadThread.java:32) at ReadThread.run(ReadThread.java:23) ptg6882136 This page intentionally left blank ptg6882136 155 5 Java Application Profiling Chapter 2, “Operating System Performance Monitoring,” made a clear distinction between the activities of performance monitoring, performance profiling, and per- formance tuning. Before jumping into the details of what is involved in performance profiling a Java application, it is worthy to revisit the performance profiling defini- tion. Performance profiling is an activity of collecting performance data from an operating or running application that may be intrusive on application performance responsiveness or throughput. Performance profiling tends to be a reactive type of activity, or an activity in response to a stakeholder reporting a performance issue, and usually has a narrower focus than performance monitoring. Profiling is rarely done in production environments. Rather it is typically done in qualification, testing, or development environments and is often an act that follows a monitoring activity that indicates some kind of performance issue. As suggested in Chapter 1, “Strategies, Approaches, and Methodologies,” perfor- mance testing, including profiling, should be an integral part of the software devel- opment process. When performance testing is not an integral part of the software development process, profiling activities are usually performed as the result of a stakeholder complaining that performance of the application is not as he or she desires. For applications having a strong emphasis on meeting performance and scalability requirements, constructing prototypes of areas identified as being at per- formance risk and profiling them are ideally done early in the software development process to mitigate risk. This activity offers the opportunity to entertain alternative architectures, designs, or implementations at a stage where it is much less costly to make changes than later in the software development process. ptg6882136 156 Chapter 5 Java Application Profiling In this chapter, the basic concepts of how to profile a Java application using a modern profiler are presented. Both method profiling and memory profiling, also known as heap profiling, are presented. Method profiling provides information about the execution time for Java methods in a Java application. The Oracle Solaris Studio Performance Analyzer (formerly known as the Sun Studio Performance Analyzer) is one of two method profilers presented in this chapter that can provide both Java and native method profile information. The Oracle Solaris Studio Performance Analyzer, often called the Performance Analyzer can also provide execution information about the internals of the Java Virtual Machine, which can help isolate potential issues observed in a Java Virtual Machine. In contrast to method profiling, memory profiling provides information about a Java application’s memory usage, that is, the number of object allocations, the size of object allocations, and which object allocations live, along with stack traces showing the method where the object allocation occurred. Many capable profilers are available, both free and commercial, that can perform method profiling or memory profiling. This chapter shows how to use the free Oracle Solaris Studio Performance Analyzer and the free NetBeans Profiler. Performance Analyzer offers several advanced capabilities. For example it profiles at the native level, which means it has the capability to collect accurate profiles. It also has the capability to distinguish the difference between a running thread and a paused or blocked thread. For example, it can tell the difference between when a thread is blocking on a read() system call versus blocking in a system call to wait(). As a result, Performance Analyzer reports read() operations as the amount of time actually spent doing a read operation and reports separately the amount time it spends blocking on a read(), in a call to wait(), waiting for more data to arrive. If it could not differentiate between those two operations and lumped both the blocked and waiting time together with the time spent in a read() operation it could lead to misleading information about how much time is really spent in a read() operation and how much time is spent blocking and waiting for more data to arrive. The Performance Analyzer also has the capability to collect and report on Java monitor, or lock information. Monitor contention, or lock contention, is a scalability blocker for Java applications. Traditionally, tracking down and isolating hot Java monitor contention has been a difficult problem. As shown later in this chapter, the Performance Analyzer makes this task much easier. There is also an example use case shown in Chapter 6, “Java Application Profiling Tips and Tricks.” The Performance Analyzer is also easy to set up and use, as described in the next section, and can provide an enormous level of detailed information. How- ever, one of the challenges with the Performance Analyzer is it is available on the Oracle Solaris (also referred to as Solaris hereafter) and Linux platforms only. It is not available on the Windows platform. Tools such as AMD’s CodeAnalyst ptg6882136 Terminology 157 Performance Analyzer and Intel’s VTune could be used as alternatives on the Windows platform. They are similar tools with similar functionality as Perfor- mance Analyzer. The concepts with using a profiler such as Performance Analyzer apply to AMD’s CodeAnalyst and Intel’s VTune. Another good alternative to use on Windows is the NetBeans Profiler. NetBeans Profiler is also available for Solaris, Linux, and Mac OS X platforms. The NetBeans Profiler’s method profiling capabilities are also covered in this chapter. In addition, memory profiling with the NetBeans Profiler is covered. Also included in this chapter is how to use the NetBeans Profiler to identify memory leaks in a Java application. This chapter begins by presenting some profiling terminology, which should make the understanding of the tasks involved in profiling easier. The profiling terminology is followed by two major sections. The first major section describes how to use the Performance Analyzer for method profiling and isolating monitor or lock contention profiling. The second major section is followed by how to use the NetBeans Profiler for both method profiling and memory profiling along with how to use it to identify memory leaks. Chapter 6 illustrates some of the more commonly observed perfor- mance issues the book’s authors have seen in Java applications. Terminology This section describes terms that are used throughout this chapter. Terms that are common to both the Performance Analyzer and NetBeans Profiler are described first, followed by terms specific to the Performance Analyzer and then terms specific to the NetBeans Profiler. Common Profiling Terms Common profiling terms include the following: Profiler. A tool that shows its users the behavior of an application as it exe- cutes. It may include both the behavior of the Java Virtual Machine and the application including both Java code and any native code. Profile. A file that contains information collected by a profiler while execut- ing an application. Overhead. The amount of time spent by the profiler collecting the profile information instead of executing the application. Call Tree. A listing of methods in a call stack form illustrating the dynamic call stack of the program as it was run. When method profiling, looking at a call tree ptg6882136 158 Chapter 5 Java Application Profiling can be useful when determining the hot use cases. When memory profiling, looking at a call tree can be useful to understand context of a Java object allocation. Filter. An artifact that can be applied to either the collected profile or to the collecting of a profile that narrows the scope of information collected and/or presented. Oracle Solaris Studio Performance Analyzer Terms Oracle Solaris Studio Performance Analyzer terms include the following: Experiment. An experiment or experiment file is the artifact produced by collecting a profile of an application using the Performance Analyzer. The Per- formance Analyzer uses the term experiment where many other profilers use the term profile. collect. A command line tool used to collect an experiment or profile by profil- ing and tracing function usage. The data collected can include call stack infor- mation, microstate accounting information, Java monitor information, and hardware counter information. Analyzer. A GUI used to view a collected experiment or experiment file. er_print. A command line utility that can be used to view the collected experi- ment or experiment file. It can also be scripted to automate the processing of a collected experiment or experiment file. Inclusive time. The amount of time taken to execute a method and all the methods it calls. Exclusive time. The amount of time taken to execute a specific method. It does not include any time consumed by methods called by the specific method. Attributed time. The amount of time attributed to a given method by a method that calls it or is a callee of. Caller-Callee. A relationship of a method either being called by some method (a caller), or a method being called by some other method (a callee). The Analyzer GUI has a view that shows the Caller-Callee relationship of a given method. System CPU. The amount of time, or percentage of elapsed time, a method listed in a collected experiment spends executing within the operating system kernel. User CPU. The amount of time, or percentage of experiment elapsed time, a method listed in a collected experiment spends executing outside the operating system kernel. ptg6882136 Oracle Solaris Studio Performance Analyzer 159 NetBeans Profiler Terms NetBeans Profiler terms include the following: Instrumentation. The insertion of counters, timers, and so on into the Java bytecode of an application to be profiled. The insertion of these counters, timers, and so on do not change the logic of an application and are removed once the profiling is terminated. Heap. The memory pool used by the Java Virtual Machine for all objects allocated in a Java application using the Java keyword new. Garbage collection. The operation responsible for the removal or cleaning of Java objects from the Heap that are no longer in use by the Java application. The Java Virtual Machine is responsible for the scheduling and executing of garbage collection. Memory leak. An object that is no longer in use by an application but cannot be garbage collected due to one or more Java objects holding a reference to it. Self time. The amount of time needed to execute the instructions in a method. This does not include the time spent in any other methods called by the method. Self time is analogous to the exclusive time in Oracle Solaris Studio Perfor- mance Analyzer terminology. Hot spot. A method that has a relatively large Self Time. Root method. A method selected for performance profiling. Oracle Solaris Studio Performance Analyzer This section covers how to use the Performance Analyzer to profile a Java application, in particular how to do method profiling and monitor profiling. The Performance Analyzer is a powerful tool. Its capabilities go well beyond profiling Java applications. It can also be used to profile C, C++, and Fortran based applications too. As mentioned earlier in this chapter, the Performance Analyzer can profile both Java code and native code. In addition, since it profiles at the native level, it can collect more accurate profiles. As a Java profiler, it is most useful as a method profiler and Java monitor/lock profiler. Tip The features of the Performance Analyzer that are most useful for method profiling Java applications are covered in this chapter. Additional features and capabilities of the Performance Analyzer can be found at the Performance Analyzer’s product Web page: http:// www.oracle.com/us/products/tools/050872.html. ptg6882136 160 Chapter 5 Java Application Profiling As a method profiler, the Performance Analyzer can show the amount of time spent in user CPU, system CPU, contending for locks, and several others. However, the three categories of user CPU, system CPU, and lock contention are usually of most interest to Java applications. In addition, within each of those major categories, the collected data can be presented in either inclusive time or exclusive time. Inclusive time says that all the time reported includes not only the time the application spent in the selected method, but also all the methods it calls. In other words, inclusive time includes all the methods a selected method calls. In contrast, exclusive time includes only the amount of time it takes to execute the selected method. In other words, it excludes the time spent in any methods that the selected method calls. The steps to profile a Java application with the Performance Analyzer are a little different from traditional Java profilers. When using the Performance Analyzer, there are two distinct steps to profiling. The first is collecting an experiment using the Performance Analyzer’s collect command and executing the Java application. The second step, analysis, is viewing the collected experiment and analyzing its results with either the Performance Analyzer’s Analyzer GUI tool, or using the Performance Analyzer’s command line tool er_print. Supported Platforms The Performance Analyzer can profile Java applications running a Java Virtual Machine that supports the JVMTI (JVM Tool Interface). Java 5 Update 4 and later, including all Java 6 updates, support JVMTI. Tip Java 6 Update 18 and later JDKs include enhancements that provide additional information to the Performance Analyzer, which further enhances the view of the collected data. Since the Performance Analyzer contains native code and can also profile native code, it is platform specific. The supported platforms are Solaris SPARC. Performance Analyzer 12.2 version is supported on Solaris 10 1/06 and later updates of Solaris 10 along with Solaris 11 Express. It also supports all UltraSPARC based systems and Fujitsu SPARC 64 based systems. Solaris x86/x64. Performance Analyzer 12.2 version is supported on Solaris 10 1/06 and later updates of Solaris 10 along with Solaris 11 Express. Linux x86/x64. Performance Analyzer 12.2 version is supported on SuSE Linux Enterprise Server 11, Red Hat Enterprise Linux 5, and Oracle Enterprise Linux 5. ptg6882136 Oracle Solaris Studio Performance Analyzer 161 Before using the Performance Analyzer, you should also check the Performance Analyzer documentation system requirements for the operating system versions supported and required patches. These may vary depending on the version of the Performance Analyzer. Running the Performance Analyzer’s collect command with no arguments checks whether the platform you are running on is supported and has the required patches. Any missing required patches are reported in the output. In addition, if a valid JDK distribution is not found on the system, it too is reported. If the Java application you want to method profile runs on a platform not listed as supported for the Performance Analyzer, you may consider running the Java application on one of the supported operating systems platforms as an alternative. Another alternative is using the NetBeans Profiler as a method profiler. One of the advantages of using the Performance Analyzer is that its intrusiveness on the Java application’s performance tends be less than other Java profilers. Downloading and Installing Oracle Solaris Studio Performance Analyzer There are several ways to download and install the Performance Analyzer. As of this writing, the home page for the Performance Analyzer is http://www.oracle.com/ technetwork/server-storage/solarisstudio/overview/index.html. At the preceding URL, the latest version of the Performance Analyzer can be downloaded and installed for any of the supported platforms. The Performance Ana- lyzer is a free download. There are two different types of installation bundles for the Performance Analyzer: a package installer or tar file installer. The package installer installs the Performance Analyzer as either Solaris or Linux packages. This type of installation requires root access to the system where the Performance Analyzer is to be installed. The package installer also provides the ability for the installation to be patched. There is also a support contract option available with this type of installer. In contrast, the tar file installer does not require root access to install the Performance Analyzer. However, it is not eligible for patches or a support contract. But there is community support available through the Performance Analyzer forums, which at the time of this writ- ing can be found at http://www.oracle.com/technetwork/server-storage/solarisstudio/ community/index.html. Tip A large amount of detailed information is available at the Oracle Solaris Studio home page (http://www.oracle.com/technetwork/server-storage/solarisstudio/overview/index.html), including many demos, tutorials, screencasts, detailed documentation, and FAQs. ptg6882136 162 Chapter 5 Java Application Profiling The version of the Performance Analyzer covered in this chapter is Oracle Solaris Studio Performance Analyzer 12.2 (also known as Oracle Solaris Studio 12.2). Earlier versions may have slightly different screenshots and menus than those illustrated in this chapter. Many of these differences are identified. In addi- tion, many of the general concepts of how to use the Performance Analyzer apply to other profilers. Capturing a Profile Experiment with Oracle Solaris Studio Performance Analyzer As mentioned earlier, profiling with the Performance Analyzer is a two-step process. The first step is collecting the experiment, which is described in this section. The second step is viewing and analyzing the collected experiment, which is described in the following section. One of the advantages of using the Performance Analyzer is the ease at which it can collect an experiment profile. In its simplest form, collecting an experiment pro- file is as easy as prefixing collect -j on to the java command line used to launch a Java application. The collect command is the Performance Analyzer command that collects the experiment profile. Here are the general steps to collect a profile experiment. 1. Update the PATH environment variable to include the Performance Analyzer tools, that is, include the bin subdirectory of the directory where the Performance Analyzer is installed. 2. Prefix the text collect -j on to the Java command line used to launch the Java application you want to profile. If the Java application is launched from a script, you can update or modify the script. 3. Run the Java application and allow the Performance Analyzer collect command to profile the application and produce an experiment profile. By default, the experiment profile is placed in a file called test.1.er in the directory where the Java application is launched. The number in test.1.er name is incremented with each subsequent invocation of the collect command. Hence, if the collect command finds a test.1.er experiment file already exists, the collect command creates a new experiment file with the name test.2.er. Collect Command Line Options Many additional options may be of interest to pass to the Performance Analyzer collect command. Here are some that may be of particular interest when profiling Java applications: ptg6882136 Oracle Solaris Studio Performance Analyzer 163 -o When used, this option creates an experiment file name with the name specified after the -o. -d When used, this option puts the experiment file in the directory specified as experi- ment directory path. If -d is not used, the direc- tory where the collect command is launched is the default directory used as the location where the experiment file is placed. Note, in a networked file system environ- ment, it is advantageous to use a local directory/file system as the location to place the experiment to avoid unnecessary high network file system activity. -p