JakartaCommons ONLINE BOOKSHELF Vikram Goyal MANNING © Copyright 2005 Manning Publications Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu i preface Welcome to the wonderfully eclectic world of Jakarta Commons, where everything is always changing and the only constant is change itself. If you’ve never heard of Jakarta Commons, you’re in for a treat; and if you’ve heard of it, then you’re in for some surprises. Jakarta Commons, or Commons for short, bypasses technologies and architectures, provides solutions to common problems, and helps make the work of technical architects and software developers easier. It’s a unique collection of components that are dissimilar in what they do but are united by the common goal of providing solutions for everyday tasks. This preface provides an insight into the Commons world. It describes the history of this suite of components and discusses its organization, rationale, and charter. This preface also gives you an idea of what to expect in this book’s modules. Let’s start at the beginning. In the beginning In the beginning, there was chaos. This chaos resulted from disparate systems creating similar utility modules to solve common problems. The result was multiple copies of multiple utilities thriving in several systems, performing basically the same functions. The problem was compounded when these diverse systems began modifying shared code, resulting in out-of-sync utilities that resembled new utilities rather than a common shared code base. On March 19, 2001, the first attempt was made to create order out of this chaos. That day, the Jakarta Project Management Committee (PMC) officially adapted a charter1 that mandated the creation of a subproject under the Apache/Jakarta umbrella called Commons. The PMC wanted to kill two birds with one stone by creating this subproject: Promote package-driven development for individual server-side utility components. Provide a playing ground for all Apache Jakarta committers (developers who have permissions to modify Jakarta code base) to test new ideas, experiment with frameworks, and gauge the feasibility of moving existing utility components to Commons. Toward this end, the charter divided the Commons project into two logical and physical entities: the Sandbox and the Commons Proper. The Sandbox is a test pit that serves the charter’s second purpose, and the Commons Proper is the repository of reusable package-driven components. The Sandbox is usually full of components that are in a high state of flux, but it doesn’t contain any actual releases. These components stay in the Sandbox until they’re ready to be released in the Commons Proper, usually via a voting mechanism among the Commons committers. But we’re getting ahead of ourselves. The initial code base for the Commons subproject came from existing projects in Jakarta and some donated packages. These included Database Pools (now in Pool and DBCP; see module 9), Collections (module 7), Cactus (no longer a Commons project, but a separate Jakarta subproject), HttpClient (module 1), Digester (module 4), Logging (module 14), and BeanUtils (module 8). 1 This Charter was revised on May 19, 2004. Licensed to Tricia Fu ii JAKARTA COMMONS ONLINE BOOKSHELF The PMC, which was only trying to bring together a few common utility components with this charter, was in for a big surprise: Commons became hugely popular with the open-source Java community and soon developed into the central point for companies to donate package-based Java components and for committers to share ideas. A lot has changed since that initial charter. Jakarta Commons is now host to 33 (as of February 2005) released components in the Commons Proper. The Sandbox is host to more than 20 components in various stages of development. Several components, like Cactus and HttpClient2 have grown in stature and are now recognized as separate subprojects within the Jakarta framework. A good measure of the popularity of a technology is the number of books written about it. At last count, there were six books, including this one, in various stages of development on Jakarta Commons. Clearly, Jakarta Commons has arrived and is important enough that all software professionals should understand it. What’s in it for me? Are you a technical architect? a software developer/programmer/analyst? a software project manager? If you answered “Yes” to any of these questions, Jakarta Commons means a lot to you. I’m a software programmer/developer/analyst Well, good for you! In addition to being in a role that puts you in direct contact with everything the software world has to offer, you’re also directly responsible for making or breaking the future of your company. (No pressure!) If only you could lay your hands on something that would take the grunt work out of everyday tasks. Suppose you’re writing a routine in Java to parse a configuration file to create an application’s domain objects. Based on the configuration file architecture, you decide to compose the routine so that it handles additions to or deletions from the array of objects you expect (based on those pesky marketing folks’ suggestions). You’re happy with yourself for having written a brilliant routine that is future-proof, and you look forward to your next task. And then you’re fired. Too bad! Being the brilliant programmer you are, you have no trouble finding another job. But what’s this? A new day, a new job, but you find yourself doing the same thing as before! Your first task is to write a configuration parser for the new company’s applications. If only you could have sneaked away with the copyrighted code from your previous company. Ethical behavior prevented you from doing so, right? The point of this scenario is to illustrate the fact that as software developers, we encounter tasks in our programming life cycles that are repetitive and nonunique. Around the world, software developers encounter the same problems and do exactly what each of us does. We repeat solutions instead of converging on one shared and proven technology. Sharing saves time and alleviates common mistakes. Of course, some of us (actually, a lot of us, considering the last count of the Commons mailing list) have started to see the light—the light at the end of the Jakarta Commons tunnel. The tunnel is filled with solutions to everyday, repetitive, proven tasks, with a worldwide community of developers as a support base. Can you afford not to look at the light? I’m a technical architect That means you already sort of know about Jakarta Commons. At least, you already know about the open- source solutions provided for common architectural problems from the Apache stable. Jakarta Commons must already be in your periphery. 2 HttpClient was only recently granted separate status; for all practical purposes, it’s still a Commons component. Licensed to Tricia Fu PREFACE iii Even if it isn’t, it’s time that you looked at it seriously. You’re responsible for creating the blueprints of software applications and creating robust, maintainable architectures; this requires knowledge of the best possible solutions to common architectural problems. Jakarta Commons offers a smorgasbord of technologies that are relevant and indispensable once you realize their benefits. For example, consider that as part of your architecting duties, you’re supposed to provide solutions to consistently and effectively write detailed logs from an application. As an architect, you want a generalized approach to such a problem rather than narrowing the focus by suggesting exact logging techniques and software components (and earning the wrath of software developers for encroaching on their territory). But how do you do that? Do you suggest the creation of an in-house Logging API and expand previous resources? Do you take a leap of faith and suggest exact external solutions—for example, by espousing the use of one product over another? Maybe. But there is a better way. You could, instead, suggest the use of the Jakarta Commons Logging component (see module 14). This component doesn’t tie your application to a single logging architecture; instead, it lets you use multiple implementations via a single interface. It’s a wrapper around other logging components, using a simple, intuitive, well-defined access mechanism. Not only does this solution endear you to your clients (perhaps because you used an open-source solution, thus saving money; or because you saved time in delivering a solution), it also causes you to be well liked by the software developers, because they’re free to choose the right logging algorithm. Commons is full of such components. Happy exploring in this book’s table of contents! I’m a project manager I’ve been meaning to talk to you about the raise I was promised! On a more serious note, you’re responsible for delivering cost-effective and timely solutions to your clients (whoever they may be, internal or external to your organization) and managing the day-to-day activities of application development. One of today’s hot topics, open-source solutions, is your gateway to cutting costs and still delivering effective solutions. Although this isn’t a book about the virtues of open-source, this is a book about open-source technologies. Specifically, this book is about the Jakarta Commons suite of open-source components. If you’ve already started using open-source in your organization, you already know the benefits it provides. If you haven’t, then this book can serve as an introduction to several of these components at once. Each of the Jakarta Commons components can be a gentle introduction to the world of open-source for you, and this book is the best medium to learning about them. Licensed to Tricia Fu iv JAKARTA COMMONS ONLINE BOOKSHELF acknowledgments Writing a book isn’t an easy task. When I first proposed the schedule for this book to Ben Sullins, my acquisitions editor, he wisely pointed out that my estimates for completing the manuscripts were grossly underestimated. I replied with false bravado that I was going to be very focused on writing this book, and I wanted to stick to my schedule. Yeah, right! One and a half years later (more than a year over my initial estimate), I’m writing these acknowledgements. During this period, I have changed houses three times, changed my day job once, survived two computer crashes, gone on an overseas trip, and remembered innumerable birthdays and anniversaries. Throughout, my wife—my friend—stood by me. She encouraged me to continue when I felt I couldn’t finish the book, and she threatened me with dire consequences when I said I would quit writing. She complained less about the time we couldn’t spend together, and she tried to make most of the time that we did spend together. She stood by me all this time and didn’t carry out the threat to divorce me. I proudly dedicate this book to her. Several other people helped me with the book-writing process. First, Marjan Bace had the vision to agree to publish a book on a not-so-hot topic from a not-so-known author. Ben Sullins, my acquisitions editor, gets a special nod for helping me settle into the writing process. Bruce Byfield was my initial development editor and provided some valuable insight into my first review chapter. Jackie Carter took the initial work done by Bruce and guided me through the rest of the chapters with considerable ease. She provided lots of input, and I hope I was able to apply most of them in my writing. David Roberson started the review process for the book, and Karen Tegtmeyer took it to completion; both of them did a fine job of organizing the best possible group of people to review the book. Karen did an exceptional job in the last few weeks of the book’s production by organizing and collating all the information from the reviewers. She further helped by providing essential copy for the book’s web site. Many thanks to all the reviewers: Jack Herrington, Oliver Zeigermann, Yoav Shapira, Glen Smith, Doug Warren, John Keyes, Dirk Verbeeck, Robert Burrell Donkin, Henri Yandell, Cos DiFazio, and Wahid Sadik. This book has benefited many times over because of your input. Doug Warren also agreed to be the technical proofreader for the book and was exceptionally thorough. In the production stage, Mary Piergies, my project editor, made sure every little detail was on track and that all the necessary people were on board. Leslie Haimes provided the cover design and spoilt me for choice. At the moment, I’m working with Helen Trimes in promoting this book, and I’m confident in her ability to provide the maximum possible coverage. Finally, I would like to thank Tiffany Taylor, who, as a copy editor, made an infinite number of corrections to my manuscript. I didn’t realize that in spite of my best efforts to say things the correct way, there was always a better way, and I’m thankful to Tiffany for making me realize this. I will endeavor to make fewer mistakes in my next book. Licensed to Tricia Fu ABOUT THIS BOOK v about this book Jakarta Commons Online Bookshelf provides detailed technical information about 18 components from Jakarta Commons Proper and 1 component from the Commons Sandbox (Chain, module 13). Modules, which can be purchased individually from the Manning web site at www.manning.com/goyal, are organized roughly according to related concepts, as follows: Web related—HttpClient (module 1); FileUpload (module 2); Net (module 3) XML related—Digester (module 4); JXPath and Betwixt (module 5) Packages—Validator (module 6); Collections (module 7); BeanUtils and Lang (module 8) Utilities—Pool and DBCP (module 9); Codec (module 10); Modeler (module 11); CLI (module 12) Frameworks—Chain (module 13); Logging and Discovery (module 14) To cater to this book’s various audiences, each module is organized in a similar format; each module is divided into sections with their own area of relevance. You can skip sections, depending on whether you’re reading from a software developer’s point of view or that of a technical architect. However, the boundaries between the sections aren’t hard and fast, and there is nothing stopping you from reading a module from start to finish. The structure of each module is as follows: 1. Introduce a component. 2. Develop the background for the component’s technology. 3. Explain the technology with examples, if necessary. 4. Explain the component’s structure. 5. Show beginning to intermediate (to advanced, in some cases) examples of using the component. If you’re a software developer, and you aren’t interested in the background technology of a component, you can skip the beginning of each module and jump to later sections. On the other hand, if you want detailed information about the technology behind each component—for example, if you want to know about a component’s background protocols and rationale—you may want to read the first part. To make the best use of this book, I suggest that you do read the information about the background technologies before you try the examples. This is especially important for components that involve varied and obscure technologies, such as Modeler (module 11). Knowing how Dynamic MBeans work makes the task of understanding the Modeler component much easier. Licensed to Tricia Fu vi JAKARTA COMMONS ONLINE BOOKSHELF What’s covered? As I said before, this book provides detailed examples and thorough background information about 18 Commons components. This section gives you a bird’s-eye view of what is covered in each module: Module 1, HttpClient—A component that helps you deal with the client side of HTTP. Module 2, FileUpload—The server end of the processing required for uploading files. Module 3, Net—Provides implementations for a diverse range of Internet protocols like FTP, SMTP, NNTP, and so on. Module 4, Digester—An XML-to-Java object mapping technology to help parse XML configuration files. Module 5, JXPath and Betwixt—JXPath traverses complex objects using the XPath syntax, whereas Betwixt is a Java-XML mapping tool. Module 6, Validator—Provides a guideline and API for validating user data from any source. Module 7, Collections—Leverages the Java Collection API by providing several new collection classes and enhancements for existing ones. Module 8, BeanUtils and Lang—BeanUtils works with the Java Reflection and Introspection API to provide access to JavaBeans properties, whereas Lang enhances the classes of the java.lang API. Module 9, Pool and DBCP—Pool is an object pooling architecture, whereas DBCP is a precise implementation that uses it to manage database connections. Module 10, Codec—Provides several common (and some not-so-common) encoding and decoding routines. Module 11, Modeler—Provides an easy way to manage Model MBeans, which are used in Java Management Extensions. Module 12, CLI—Provides an easy-to-use interface to deal with application startup options. Module 13, Chain—An implementation of the Chain of Responsibility pattern. Module 14, Logging and Discovery—Logging is used as a wrapper around existing logging implementations. Discovery is an attempt to provide common resource-finding services. Each module is independent of the others, so you can jump straight to the component of your choice. There are no dependencies on prior learning; for example, knowing about Logging isn’t a prerequisite to learning about any other component. However, it’s expected that you know about Java, know how to download each component, know how to set Java CLASSPATH, and know how to code and compile. However, if you’re reading this book without doing any coding, you don’t need to know how to download or compile the components. The background reference material doesn’t assume any coding experience. You can access each component’s web page by going to http://jakarta.apache.org/commons and following the individual links. Licensed to Tricia Fu ABOUT THIS BOOK vii How to holler for help Sometimes you’ll get stuck. In spite of the best intentions and hours of trying to solve problems, there will be times when you want to ask someone else for help. A fresh insight into a problem can bring rich dividends. I’ll be available for help in the Author’s Online forum at Manning’s web site as soon as the book is released. You can log onto the forum by going to http://www.manning.com and then navigating to the Author Online link. The book has its own devoted section, and I can help on any Commons-related topic. You can also access the Author Online forum from the Jakarta Commons Online Bookshelf web page (as well as purchase individual modules) at www.manning.com/goyal. In addition to the Author Online forum, the Commons mailing lists are excellent resources to ask for help. The Commons committers who develop and maintain the Commons packages monitor these lists and help out with most problems. To subscribe to the Commons User mailing list, send an email to commons- user-subscribe@jakarta.apache.org. You can also browse the archives at http://www.mail- archive.com/commons-user@jakarta.apache.org. The Commons HttpClient has its own mailing list; to subscribe, send an email to httpclient-user-subscribe@jakarta.apache.org. You can browse the archives at http://nagoya.apache.org/eyebrowse/SummarizeList?listId=280. Source code downloads Source code for the examples in each module can be downloaded from the publisher’s web site at www.manning.com/goyal. The purchase of an individual module entitles the reader to a free download of the source code for that module. About the author Vikram Goyal is a Java Developer by day and a technical writer by night. Vikram has worked with Java since its inception and remembers the time when Tomcat meant an animal and not a servlet engine. Vikram regularly writes how-to articles on open source projects and his series of articles on Jakarta Commons was the first such effort to make some sense out of the chaotic world that Jakarta Commons is. This article series is still reflected in the official main entry page of Jakarta Commons as the only online series covering these components. Vikram enjoys working with Open Source technologies, Team Leading and Civilization 3. Vikram lives in Brisbane, Australia, with his wife. Licensed to Tricia Fu viii JAKARTA COMMONS ONLINE BOOKSHELF about the cover illustration The figure on the cover of the Jakarta Commons Online Bookshelf is a “Sultana, or Kaddin,” which means “wife” in Turkish. The illustration is taken from a collection of costumes of the Ottoman Empire published on January 1, 1802, by William Miller of Old Bond Street, London. The title page is missing from the collection and we have been unable to track it down to date. The book's table of contents identifies the figures in both English and French, and each illustration bears the names of two artists who worked on it, both of whom would no doubt be surprised to find their art gracing the front cover of a computer programming book...two hundred years later. The collection was purchased by a Manning editor at an antiquarian flea market in the "Garage" on West 26th Street in Manhattan. The seller was an American based in Ankara, Turkey, and the transaction took place just as he was packing up his stand for the day. The Manning editor did not have on his person the substantial amount of cash that was required for the purchase and a credit card and check were both politely turned down. With the seller flying back to Ankara that evening the situation was getting hopeless. What was the solution? It turned out to be nothing more than an old-fashioned verbal agreement sealed with a handshake. The seller simply proposed that the money be transferred to him by wire and the editor walked out with the bank information on a piece of paper and the portfolio of images under his arm. Needless to say, we transferred the funds the next day, and we remain grateful and impressed by this unknown person’s trust in one of us. It recalls something that might have happened a long time ago. The pictures from the Ottoman collection, like the other illustrations that appear on our covers, bring to life the richness and variety of dress customs of two centuries ago. They recall the sense of isolation and distance of that period—and of every other historic period except our own hyperkinetic present. Dress codes have changed since then and the diversity by region, so rich at the time, has faded away. It is now often hard to tell the inhabitant of one continent from another. Perhaps, trying to view it optimistically, we have traded a cultural and visual diversity for a more varied personal life. Or a more varied and interesting intellectual and technical life. We at Manning celebrate the inventiveness, the initiative, and, yes, the fun of the computer business with book covers based on the rich diversity of regional life of two centuries ago‚ brought back to life by the pictures from this collection. Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Browsing with HttpClient Vikram Goyal MODULE MANNING 1 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu Module 1 Browsing with HttpClient 1.1 A taste of HttpClient: downloading a web page....................................................................................................... 1 1.2 Reviewing the HttpClient RFCs ............................................................................................................................... 3 1.3 The HttpClient API................................................................................................................................................. 14 1.4 HttpClient in action................................................................................................................................................. 19 1.5 Summary................................................................................................................................................................. 40 Index ............................................................................................................................................................................. 41 Ah! HTTP. I can hear the sighs of relief and the shakes of head in an affirmative action, as you contemplate the title of this module. Yes, it’s about HTTP—something, as developers, most of us are intimately aware of: the Hypertext Transfer Protocol that drives the thing we lovingly call the Internet. Specifically, it’s about the Commons component HttpClient. When you think about it, this is the best possible name for the component; it exactly demonstrates what it does. HTTP + Client = HttpClient—a component to help deal with the client side of HTTP. But wait a minute. I can hear you ask, “Isn’t my browser an HTTP client?” Of course, it is. In fact, with the help of HttpClient, you can build your own browser, or embed it into your distributed application, or write a web services client. With the help of HttpClient, you can do anything that you want to do over HTTP (on the client side, anyway). To start this module, we’ll look at the simplest case of using HttpClient, by making it act as a browser and letting it download a web page. We’ll then take a step back and take a brief look at the basics of HTTP: specifically, the Requests for Comments (RFCs) that HttpClient implements.1 This will help you understand the fundamentals of HTTP and better prepare you for learning about HttpClient. We’ll then take a whirlwind tour of the HttpClient API, to see its various packages and how they relate to each other. This will get you ready for the examples of using HttpClient that will follow. 1.1 A taste of HttpClient: downloading a web page In this section, we’ll look at how to use HttpClient in the simplest possible case of downloading a static web page. This is just a teaser section. More detailed examples will follow after we’ve discussed the HttpClient API. Listing 1.1 shows the code required to download a web page using HttpClient. In this listing, HttpClient acts as a browser by requesting Google’s home page. Listing 1.1 A taste of HttpClient: Downloading Google’s home page package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; 1 See module 2, section 2.1.1, for a definition of an RFC. Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF public class HttpClientTeaser { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); GetMethod method = new GetMethod("http://www.google.com"); int returnCode = client.executeMethod(method); System.err.println(method.getResponseBodyAsString()); method.releaseConnection(); } } HttpClient relies on three other Commons components for it to work. Before you run listing 1.1, make sure that you have the following libraries in your CLASSPATH, in addition to commons-httpclient.jar: commons- lang.jar (discussed in module 7, “Enhancing Java core libraries with collections”), commons-logging.jar, and commons-codec.jar (discussed in module 10, “Codec: encoders and decoders”). If you run this Java application on a command line, the HTML for Google’s default page will be printed (provided you’re connected to the Internet). A browser would interpret the tags in this HTML and format its display accordingly, but here we were only interested in getting the HTML. We could just as easily have asked for a text document, image, or results of scripts—anything available on the Internet via HTTP—and HttpClient would have fetched it for us. Using HttpClient starts with creating an instance of the client, as in the listing. In this case, the client that is created uses default settings. These settings include the connection manager used to create underlying connections, the attributes of these connections, the timeouts for giving up delayed connections, and so on. Next, the GetMethod class is used to specify the document to retrieve using the instance of HttpClient that was just created. The GetMethod class implements the HTTP GET method for retrieving documents of the Internet. (Not surprisingly, you would use the PostMethod class for the HTTP POST method.) The document to retrieve is specified as the sole argument to the GetMethod constructor. Notice that in this case, the argument specifies an absolute web address, as opposed to a relative web address, to Google’s default page (the argument must be interpreted as http://www.google.com/index.html or http://www.google.com/index.htm or whatever Google uses as its default page). The HttpClient instance is used next to execute the method request to the default page of Google. HttpClient’s executeMethod expects an HttpMethod interface implementation as its argument and returns an integer, which represents an HTTP response code for the execution of this request. The GetMethod class extends the abstract class HttpMethodBase, which implements the HttpMethod interface. Finally, the response from Google is printed on the command line by using the method getResponseBodyAsString on the GetMethod instance created earlier. This method evaluates the response received from the execution of a request and returns it by converting the bytes received into their String representation based on the character set specified in the response’s Content-Type header. You could choose to receive this response as a stream by using the getResponseBodyAsStream method, or as raw bytes by using the method getResponseBody. As you can see, you need only four lines of code to retrieve a web page from the Internet using HttpClient. However, there is much more to HttpClient than simply retrieving web pages. Before we get into these extras, let’s take a brief look at the RFCs that HttpClient implements. These RFCs define HTTP, and you need to understand them before you can begin to understand HttpClient itself. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 3 1.2 Reviewing the HttpClient RFCs It’s a fair bet that most of you would like to skip this section, fearing an overdose of theoretical interpretations and lengthy, boring text about HTTP RFCs. Fear not! In this section, we’ll review the RFCs in a relaxed manner, presenting only text that is relevant and highlighting key points. As you may be aware, HTTP was first formally introduced in rfc1945 in early 1996. This version of HTTP was called HTTP 1.0. Even while this version was being documented, it was clear that it wouldn’t serve the purposes of future browser-server communications. Therefore, HTTP 1.1 went into specification mode almost immediately and was first documented in rfc2068; the final version was documented in rfc2616. HttpClient supports both versions of this protocol. In addition to these two RFCs, HttpClient supports several related specifications. For example, HttpClient implements cookies, which in HTTP are used to maintain user information and state; they’re defined in rfc2109. Table 1.1 lists all the RFCs supported by HttpClient. Table 1.1 RFCs supported in HttpClient RFC number Protocol implemented 1945 HTTP 1.0 2616 HTTP 1.1 2109/2965 Cookies 2617 HTTP authentication (Basic and Digest) 2396 Uniform Resource Identifier (URI) definition 1867 Form-based file upload in HTML In addition to implementing rfc2617 for HTTP authentication, HttpClient also supports authentication using proprietary Microsoft technology. This technology is called Windows NT LAN Manager (NTLM) and we’ll review it in this section as well. Let’s begin our review of these RFCs with HTTP 1.1, defined in rfc2616. We won’t review rfc1867 in this module, because it’s discussed in detail in module 2, “Uploading files with FileUpload.” Note: To view any of these RFCs in full, visit www.faqs.org/rfcs/. 1.2.1 Understanding rfc2616 for HTTP 1.1 The RFC that finalized HTTP 1.1, rfc2616, ensured that clients and servers could talk to each other correctly even if they supported different versions of the HTTP protocol. Because HTTP 1.1 didn’t change the basic operation of this protocol from HTTP 1.0, it’s safe to assume that when we’re talking about HTTP 1.1, we also mean HTTP 1.0. There are several key differences, mainly enhancements, which we’ll list at the end of this section. This RFC defines a protocol for transferring hypermedia information using request-response architecture. Hypermedia is nothing but hypertext (text that is a link) expanded to include multimedia content as well. Therefore, when you click a picture in a browser to go to another web site, you’re clicking on hypermedia; the link to the second page is in form of an image, a multimedia component. In a nutshell, rfc2616 defines a way for clients and servers to transfer information between each other using hypermedia-enabled documents. In the next few sections, we’ll explore the key features of HTTP 1.1 as defined by this RFC. Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF Request-response architecture A request initiates from the client and is in form of a message. This message contains the request method (GET, POST, and so on), the URI (the location on the server of the requested information), the protocol (HTTP 1.1 or 1.0), and other request-specific information encapsulated in a Multipurpose Internet Mail Extension (MIME) format. This information may contain details about the client (is it Internet Explorer, Netscape, or HttpClient? does it accept images or multimedia files? and so on), extra information about the request in the form of request modifiers, and, finally, any data that the client may be sending to the server to complete the request. A typical request message is shown in figure 1.1. Figure 1.1 A typical request message and its parts The server responds in kind to this request. It sends a response message, which is similar in structure to the request message. This message starts with a response code on a status line, which indicates the success or failure of the request, followed by response headers. The message finishes with the requested document (if found) encapsulated as a MIME message. A typical response message is shown in figure 1.2. Figure 1.2 A typical response message and its parts Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 5 Since HTTP is driven by this request-response architecture, a response dies after it has completed a given request. This means that following requests have no recollection of previous requests and their responses. Thus HTTP is called a stateless protocol. Stateless protocol By not providing any means to maintain information between different requests, HTTP leaves open the provision of managing state to external factors. One of these factors is cookie management, which is defined in rfc2109. (We’ll cover this RFC a little later, in section 1.2.2.) But why did the writers of rfc2616 want this protocol to be stateless? Well, one of the binding reasons to create this protocol was to provide a fast responsive system for accessing distributed data. When you think about it, clicking a link on a web page accesses data across distributed systems. By requiring these systems to maintain state and defining a protocol that would mandate that, the protocol designers would have introduced several layers of complexity that would take more time than was necessary to accomplish the task of getting data. The HTTP protocol is straightforward enough that state management factors can be imposed on it, but these factors are outside the purview of this protocol. Hence, simple systems that don’t require state management can stay simple, whereas complex systems can use cookies, among other factors, to introduce state. Methods Methods tell servers serving requests what operation is to be performed with the rest of the information coming in from the client. In the case of the GET method, this means retrieving the resource requested in the header. In the case of HEAD, it means retrieving meta-information about a resource, not the resource itself, thus preserving bandwidth when the resource isn’t required by the client. Table 1.2 lists all the methods supported by HTTP 1.1. These method names must always be uppercase. Table 1.2 Methods supported in rfc2616 for HTTP Method name Purpose GET Retrieves a resource/document following this method HEAD Retrieves a resource’s meta-information: everything returned by the GET request except the resource itself POST Posts data to the server in the form of a message body preceded by request headers PUT Allows the client to create or modify an existing resource identified by the request URI DELETE Requests that the resource following this method be deleted TRACE Used for diagnostic purposes: Returns the request message in its entirety to the client as part of the response body CONNECT Used for HTTP tunneling via a proxy OPTIONS Used to interrogate the abilities of a server (by using * for the request URI) or a particular resource identified by the request URI You’ll see examples of all these methods in the code listings for HttpClient later in this module. Proxies and tunnelling A request on its way to the destination server could conceivably pass through a series of proxy servers, through which the response would also be expected. A proxy server is an intermediate machine that channels request and response messages; it acts like a client to the destination server and like a server to the originating client. Proxy servers reduce the time for oft-requested messages because they cache the response to identical requests, but they introduce the complexity of stale messages. (Caching is covered in the next section.) Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF Proxies may modify request and response messages to add extra header information that may help in either request processing or response caching by the client. A proxy that doesn’t modify any part of the request or response is called a transparent proxy. As opposed to a proxy, a request or response may pass through a series of servers, which may not understand the semantics of the message. A common example of such a server is a firewall, commonly employed in organizations to thwart unwarranted connections from either side of their access. In such cases, a tunnel for an HTTP connection is established that acts as a blind relay point through the server that it’s passing through. This tunnel, via the server on which it’s acting, can’t change the message, because it doesn’t understand the message. However, nothing stops the server from keeping a copy of the message, which brings us to the case of caching in HTTP. Caching HTTP allows systems through which request and response messages pass to cache them, to provide faster response times for the identical request. In HTTP 1.0, the only way to avoid getting cached messages was to set the Pragma general header as no-cache. A request message that contained this header would be passed through to the designated server (proxies included) without retrieving a cached copy, even if a copy was available on any servers through which the request had to pass. The Pragma header has given way to the Cache-Control header. HTTP 1.1 defines several parameters via the Cache-Control header to override the caching of request and response messages. A value of no-cache in a request message means that the request must be passed as is to the requested server without returning a cached copy for the response from any servers that it may pass through. A value of no-cache in a response message, however, means that any proxy or gateway along the way must not cache this response for identical requests. You can restrict this so it’s applicable for only certain fields by specifying them explicitly. For example: Cache-Control: no-cache = "field1" Table 1.3 lists all the possible values for the request Cache-Control header. Table 1.3 Possible values for the request Cache-Control header Value Comment no-cache Response to this request must come from the targeted server and not from any cache. no-store Request and its response must not be stored on backup systems. max-age = "" Indicates to the server (or any proxy or gateway in between) that the client is willing to accept a cached response to this request that isn’t older than the age specified in seconds. max-stale [= ""] Indicates to the server (or any proxy or gateway in between) that the client is willing to accept a cached response to this request that has gone stale. If max-stale has a value after the = sign, it indicates that the response shouldn’t be stale by more than the given number of seconds. min-fresh = "" Client will accept a cached response that will be fresh for at least the specified number of seconds. no- transform A request must not be transformed by a proxy or gateway on its way to the server. only-if- cached Only a cached response will be returned. If no cached response is found for this request, the proxy or gateway must return a 504 error, indicating Gateway timeout. Similarly, table 1.4 lists the values for the response Cache-Control header. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 7 Table 1.4 Possible values for the response Cache-Control header Value Comment no-cache = "field1" Any proxy or gateway must not cache this response. If a field is specified after the = sign, only this field must not be cached; others may be. public Response may be cached by any proxy or gateway on its way to the client. private = "field1" Response is cacheable only for one user. Other users’ requests must not get the same request. This may be applied to either the whole response or only parts of it, by specifying these parts after the = sign. no-store Response and its associated request must not be stored on backup systems. no- transform Response must not be transformed by any proxy or gateway on its way to the client. must- revalidate A proxy or gateway must request the server for fresh copy of the response, once it becomes stale. proxy- revalidate Same as must-revalidate, but applicable only on a per-user basis. max-age = "" Response is cacheable and is valid only for the given number of seconds. s-maxage = "" Same as max-age, except this entry isn’t valid in cases where a cache (a proxy or a gateway) is stored on a per-user basis. This covers the most basic features of HTTP. Let’s finish this section by highlighting the key differences between HTTP 1.0 and HTTP 1.1. What is the difference between HTTP 1.1 and HTTP 1.0? There are many differences between these two versions of HTTP; but as we said earlier, the basic process of HTTP remains the same. In this section, we’ll cover some of the most important differences so that you can better utilize HttpClient to deal with servers that may still conform only to HTTP 1.0. Following are the most important HTTP 1.1–specific features, with a brief explanation of each change: Explicit domain information in each request—This information can be provided using either the HOST header or using the absolute URI. HTTP 1.1 mandated the use of the HOST header to allow a single IP address to be used for multiple domains. The HOST header determines the target domain for the request. This allowed efficient use of the existing IP addresses in the World by supporting virtual hosting. If the HOST header is missing, then each request must contain an absolute URI to the requested resource. Here are two examples: GET /index.html HTTP/1.1 HOST: www.manning.com GET www.manning.com/index.html HTTP/1.1— Persistent connections—In HTTP 1.0, each resource had to be requested on its own connection. This was inefficient and led to the concept of persistent connections in HTTP 1.1. By default, all connections in HTTP 1.1 are persistent and must be explicitly closed by sending the Connection: close header as part of the request. Since all connections are persistent, multiple resources can be requested on a single connection by a browser or User-Agent such as HttpClient. This is typical for a web page, which may contain several images, scripts, and other media files in-lined within the page. Chunked transfers—To allow servers to start sending responses to requests before the total length of the response has been determined, HTTP 1.1 introduced chunked transfer encoding for message bodies. Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF This encoding is useful in cases where the response is driven by a server-side script that produces a lot of data, which takes time to be produced. The server, instead of storing this data in memory, can begin sending it to the client; the server lets the client know that more data is expected by setting the content-transfer-encoding header to chunked. The full response then consists of separate chunks of information; each chunk is sent when it becomes available. The Date header—Caching by proxy servers and gateways can’t be implemented efficiently if the original response to a request didn’t contain the date on which it was generated. This is the case because caches must know how to evaluate a response as stale or fresh, as the case might be, when serving an identical request. HTTP 1.1 mandates that each response must contain a Date header set to the date on which the response was generated, to enable caches to determine the appropriate action. The new concepts in HTTP 1.1 allow for faster response times and flexibility. But all these differences are backward compatible, so communication between disparate systems is allowed. This section completes the discussion of the main HTTP RFC, rfc2616. However, HTTP has several support RFCs that enhance its basic functionality. These RFCs deal with several factors, including state management and authentication. In the next few sections, we’ll examine these support RFCs. 1.2.2 Maintaining state with cookies: rfc2965 State management with cookies was first introduced in rfc2109. By including only two headers, one for request and one for response, this RFC added the concept of state management to HTTP—which is essentially a stateless protocol. These headers are Cookie for request and Set-Cookie for response. However, rfc2109 was made obsolete by the introduction of rfc2965. It defines three headers, Cookie, Cookie2, and Set-Cookie2; the first two are for request messages and the third for response messages. The Cookie2 header is used by a client and indicates that the client understands rfc2965, so a server can use Set-Cookie2 instead of the Set-Cookie header. HttpClient can handle headers from both RFCs. The process of state management via cookies is fairly simple. State is managed via a logical concept known as a session. A session is a series of connections between a client and a server, which continues until either one of them closes it explicitly or a timeout happens. Only the server can initiate a session, but a client can reinitiate an old session if the server so permits. To start a session, rfc2965 mandates that the server must send a Set-Cookie2 header as part of a response; after that, the client and the server are said to be participating in a session. The syntax of this header is shown here as an example: Set-Cookie2: cookie1="cookie1Value"; max-age="0"; Domain=".manning.com"; Version="1", cookie2="cookie2Value"; max-age="3600"; path="/forum"; Comment="Cookie for forum"; Version="1" Cookies are required to have a name. In the example, two cookies are defined: cookie1 and cookie2, separated by a comma. Both cookies have a value after the equal (=) sign and some attributes with values separated by semicolons (;). Except for the Version attribute, all other attributes are optional. These attributes, which are case-insensitive, are listed in table 1.5. For backward compatibility, using the Set-Cookie header is identical to using Set-Cookie2, except that it doesn’t support the CommentURL, Discard, and Port attributes. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 9 Table 1.5 Optional attributes of a cookie in a Set-Cookie2 header Attribute Description Default action (if this attribute is absent in each cookie header) Comment Short text describing the purpose of the cookie. None CommentURL Location on the Internet where more information about this cookie can be found. None Discard Indicates that when the client shuts down the User- Agent used to make the connection to the server, this cookie must be discarded and not used again. This attribute doesn’t require a value. Depends on the Max-Age attribute value, if present Domain Domain with which this cookie is associated. If no domain, the value should begin with a period (.); but this isn’t necessary because the User-Agent must supply it. The host to which the request was made Max-Age Number of seconds after which the cookie is discarded. A value of 0 indicates that the cookie shouldn’t be stored. Discard the cookie when the User-Agent exists; same as Discard Path URL, relative to the domain to which this cookie applies. The request URL Secure Indicates that the client must make an effort to transfer this cookie information to the server via a secure channel. The details of this secure channel aren’t defined. This attribute also doesn’t require a value. Send over any channel even if insecure Port Port(s) on which the cookie value may be returned to the server. Any request port The session continues when a client, having received a cookie before, sends it back as part of the request headers—provided the cookie hasn’t been discarded and hasn’t expired, as mandated in the original Set- Cookie2 header. To send the value of the cookie back, the User-Agent uses the Cookie header (not the Cookie2 header, which is used to indicate the acceptance of rfc2965). Here’s an example: Cookie: $Version="1"; cookie2="cookie2Value"; $Path="/forum" Note that the Version and Path attributes have a $ in front of them when they’re sent back to the server. Only the Version, Path, and Domain attributes need to be sent back, along with the value of the cookie; the other attributes are for cookie management on the client side. The server receives the value of the cookie, uses it to identify the current session of the client, and acts accordingly. It can send a fresh value for the cookie as part of the response or tell the client to discard the cookie, as it sees fit. If a fresh value for the cookie is sent, it overwrites the existing value on the client; if a Max-Age of 0 is sent back, the older cookie and the newer cookie are both discarded. Although cookies can manage state, another RFC is required to manage control over resources on the server and authenticate users. Rfc2617 details authentication and is discussed next. 1.2.3 Authenticating users: rfc2617 Rfc2617 defines two ways of authenticating users over HTTP: Basic and Digest. The two methods are almost identical in the process that they follow, but one doesn’t send the authentication details in clear text. Of course, Basic sends authentication details in clear text, and Digest uses a hashing algorithm to send hashes of authentication details (rather than the details themselves). The Basic authentication scheme was introduced in the original RFC for HTTP; that is, it has been part of HTTP from its inception. The process is simple and is applicable to the Digest authentication scheme as well. Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF When a client asks the server for a resource that has been marked as protected, and it doesn’t send any authentication details as part of the request, the server sends a header called WWW-Authenticate as part of the response. The response carries a status code of 401, which means the server is telling the client (or the User-Agent) to provide its credentials for accessing the given resource. An example response is shown here: HTTP/1.1 401 Unauthorized WWW-Authenticate: Basic realm="protectedSpace" The WWW-Authenticate header should first list the scheme being used, which in this case is Basic. This is followed by a logical name for the protected resources on the server to which the user’s credentials are applicable. This logical name is called a realm; in the previous example, its value is protectedSpace. This is displayed to the user by the User-Agent on the client. As far as Basic authentication scheme goes, this is all the server sends. For the Digest authentication scheme, the server sends several other parameters: domain—A space-separated list of URLs applicable to this logical realm. nonce—A unique string that is generated by the server for insertion into each response. opaque—A string of data that should be returned by the client unchanged in each request for this realm and domain. stale—A value of true or false. A true value indicates that the previous request wasn’t acceptable because it had expired. A false value indicates that the request was rejected because of an invalid or absent username/password. The default is false. algorithm—An optional parameter. The algorithm(s) the client should use to create hash values. The default is MD5. qop—An optional parameter. The value indicates the quality of protection offered by the server and affects the way the client sends its request. The value can be either auth or auth-int. An example response using Digest authentication is shown here: HTTP/1.1 401 Unauthorized WWW-Authenticate: Digest realm="protectedSpace", nonce="457dhedfshjir39f00df03f0sdf0f0df3450dfieo" opaque="dontchangemeiaminclear" qop="auth" Once the client receives a 401-status response from the server, it gathers the user’s credentials and puts them in a request body with the Authorization header. For Basic authorization, this amounts to Base64- encoding the user’s username and password, separated by the colon (:) character.2 For example, if the username is Vikram and the password is My Password, the User-Agent on the client puts it together in a string as Vikram:My Password, Base64-encodes it to get the string VmlrcmFtOk15IFBhc3N3b3Jk, and puts it in the header: Authorization: Basic VmlrcmFtOk15IFBhc3N3b3Jk 2 See module 9, “Pool and DBCP: Creating and using object pools,” for an explanation of Base64. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 11 Finally this is sent to the server, along with the rest of the request and any subsequent requests. For Digest authorization, the request body contains several other parameters; this is to be expected, considering that the response message contained several parameters. These parameters are listed here: username—The clear-text username of the user making the request. uri—The URL to the resource that is being requested. qop—A value that is the same as the qop value specified in the response header. If the response didn’t contain a value for this parameter, it shouldn’t be present in the request, either. cnonce—A required parameter, if a qop parameter has been set. However, if the server didn’t send a qop parameter, this parameter must not be set in the response. The value for this parameter is used in creating the hash of credentials that is sent to the server. It’s sent to the server in a request message so that the server can use it to match the credentials. nc—A count of the number of times a request has been sent, which includes the nonce value received in a response. Recall that the response message contains the value of a nonce parameter, and the corresponding requests are supposed to send it back. As with cnonce, this value must not be set if there was no qop parameter in the response message and must be set if there was a qop parameter. response—The parameter whose value contains the hash for user’s credential information. The process to calculate this value is shown in figure 1.3. In addition to these parameters, the Authorization header of a request message must also return the nonce, realm, and opaque parameter values received from the server. An example request Authorization header using Digest authentication is shown here: Authorization: Digest realm="protectedSpace", nonce="457dhedfshjir39f00df03f0sdf0f0df3450dfieo", uri="/index.html", qop="auth", nc=00000001, cnonce="somecleartext", response="rrt56879393348fjs9785dfd307c4ef1" opaque="dontchangemeiaminclear" The process to calculate the value of the response parameter is shown in figure 1.3. You shouldn’t need to use this process to calculate the response value yourself. HttpClient supports almost all branches in this process shown, except when the value of the qop parameter is set to auth-int. This setting for qop indicates that the server wishes to check the integrity of the data transfer as opposed to simply hashing the user credentials. When the server receives a request message with the Authorization header as shown in figure 1.3, in either Digest or Basic mode, it responds accordingly with a response header. The cycle continues until one party breaks it off. Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF Figure 1.3 Process to calculate the value of the response parameter based on the user’s credentials and other parameters Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 13 Basic and Digest authentication aren’t the only way of authenticating requests between client and server. Microsoft-specific NTLM authentication is another way of establishing trust. Let’s discuss this proprietary technology, which works only with Internet Explorer as the client and Internet Information Server (IIS) as the server. 1.2.4 Authenticating users: using NTLM NTLM differs slightly from the Basic and Digest authentication processes. The key difference relates to the fact that NTLM authenticates a connection, not a request; therefore, each time a connection is made, the authentication process has to be reinitiated. This works well with HTTP 1.1, because it automatically creates persistent connections. An NTLM authentication session starts with the client making a request to the server for a protected resource. The server responds, as in rfc2617, with a WWW-Authenticate header in a 401 response. The value of the header in this case is equal to NTLM. The server also closes the connection after sending this response. An example is shown here: HTTP/1.1 401 Unauthorized WWW-Authenticate: NTLM Connection: Close Clients that understand and support the NTLM protocol (mostly IE, but increasingly other browsers as well) respond to this response with what is known as the Type 1 request message in an Authorization header. This is called the negotiation stage of the authentication process. The Type 1 message contains the features that are supported by the client and are requested of the server. For example, the client may include in a Type 1 message a flag to indicate that it supports Unicode strings for data transfer. In addition to flags, the Type 1 message may also contain the computer name and domain of the client, which helps the server determine if there is any point in continuing (because the domain or the computer name may be restricted from accessing the requested document). The server, on receipt of a Type 1 message in a request Authorization header, responds with a Type 2 message in a WWW-Authenticate header. The Type 2 message serves two purposes. First, it agrees (or disagrees) with the flags requested by the client in its Type 1 message. Second, it contains a challenge to the client to authenticate itself based on the challenge. The challenge is nothing but a random string of data, 8 bytes in length, which the client uses to respond with a Type 3 message. The Type 3 message finalizes the client’s side of the authentication process. The client constructs this Type 3 message on receipt of the Type 2 message from the server by creating the password hash for the requested resource along with the challenge received in the Type 2 message, using a complex series of protocols. (These protocols are discussed in detail at: http://davenport.sourceforge.net/ntlm.html.) The Type 3 message is Base64 encoded, and the original request is resubmitted to the server, along with this Type 3 message in an Authenticate header. The server checks the validity of the header and allows access to the original resource. Further requests on the same connection are allowed without any checks, because the process is connection-oriented; therefore, the connection is considered sanitized for all future requests. NTLM authentication is widely popular in the Microsoft world, and HttpClient does well by supporting it fully. Next, we’ll discuss the RFC for locating resources on the Internet using the URI syntax. 1.2.5 Identifying resources by URI: rfc2396 Rfc2396 defines a way to consistently identify resources on the Internet. Regardless of whether resources exist physically, this RFC makes the process of referencing them a matter of making sure the general grammar of Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF accessing them remains the same across systems and processes. This allows URIs developed in one system to consistently reference the same resource, even if they’re ported to another system or process. Uniform Resource Identifiers (URIs) are strings of characters that provide location information for a resource. A resource, in this case, can be a file, an image, a server process, or anything else that can be located. URIs aren’t tied to a specific protocol. Thus URIs can be used to express the location of resources in various protocols. However, each URI must conform to this specific format: scheme:scheme-specific-part The scheme part identifies the protocol that is being used to locate the resource in question. Examples include, http, ftp, mailto, and file. The scheme-specific-part further qualifies the resource and differs to a large degree between schemes. However, a generic format for the scheme-specific-part, which is followed by most schemes, is identified as follows: ://? Note that this format isn’t mandated by rfc2396—it’s only a suggestion, but it’s the format followed by most schemes. Most of us are familiar with this format, because HTTP follows it for the location of documents on the Internet. For example, in the URI http://www.google.com/search?q=fishing, http stands for the scheme, www.google.com stands for the authority, search stands for the path of the document being referenced (in this case, a CGI program) and q=fishing is the query. This short refresher on rfc2396 concludes the list of RFCs that we’ll go through before looking at the HttpClient API. Note that we haven’t discussed rfc1867, which covers form-based file upload, because it’s covered in detail in module 2. 1.3 The HttpClient API In this section, we’ll discuss the key classes and libraries of the HttpClient API. It’s important to do this before we begin looking at code samples, because this discussion will help you decide which classes are relevant in which situations. Version note: This discussion and the code samples that follow are based on HttpClient 3.0 alpha2. 1.3.1 The org.apache.commons.httpclient package The main classes in HttpClient are located in the parent org.apache.commons.httpclient package. This package contains the HttpClient class, which is the starting point for every transaction. This class depends on two other classes, the HttpConnectionManager interface and the HttpClientParams class (which is in the org.apache.commons.httpclient.params package and is discussed in section 1.3.5 later). HttpConnectionManager is an interface in this package, and it has two implementations: SimpleHttpConnectionManager and MultiThreadedHttpConnectionManager. As the name suggests, this interface is responsible for managing the underlying network connections between client and servers, a bit like a connection pool. SimpleHttpConnectionManager is the default connection manager, and it provides access to a single shared connection. HttpClient instances are expected to borrow this single Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 15 connection from this manager for executing a request and return it when they’re finished. It’s re-borrowed when the next execution needs to be performed. The expected sequence of executions is sequential. If you need to execute requests in a multithreaded environment, where you may simultaneously execute several requests, you need to use the MultiThreadedHttpConnectionManager class. This class uses a HashMap to internally store a mapping between connections and hosts to which connections are made. The default number of connections per host is 2, and the default value for the maximum connections overall is 20. Once a request is executed using a connection from any one of the connection managers, the state of the request/connection can be gauged from the HttpState class. Recall, however, that HTTP is a stateless protocol. So, what state are we talking about? The state here refers to the values of cookies and authentication details, which may persist between requests/connections. The HttpState class provides methods to gauge the state of these attributes and to modify them if required for future requests. The HttpState class doesn’t keep the cookie and authentication details (the username, password, and domain, if applicable) on its own. It uses the Cookie class and the Credentials interface, respectively, to hold these values. The Cookie class is used with the classes from the org.apache.commons.httpclient.cookie package to provide the complete cookie-handling routine. The Credentials interface is implemented by the UsernamePasswordCredentials class, which in turn is extended by the NTCredentials class. The first class holds the username and password for a requesting user in a color-separated String object. The second class holds extra information, in addition to a colon-separated username password String, about the host and domain for which the username/password are a valid combination; these extra values are required in the NTLM authentication scheme. Note that the classes in the package org.apache.commons.httpclient.auth are required to provide a complete authentication process. Requests in HttpClient are executed using one of the instances of the HttpMethod interface. The HttpMethod interface represents a complete request and response cycle. This interface has an abstract base implementation in the HttpMethodBase class, but the actual implementations are in the org.apache.commons.httpclient.methods package. We’ll cover these implementations in the next section. In addition to the classes mentioned, the org.apache.commons.httpclient package contains several utility classes. For example, the Header class and its associated HeaderElement and HeaderGroup classes are used to represent an HTTP request or response header. Similarly, the StatusLine class is used to represent the first line received from the server for a request as part of the response. It breaks this line into its constituent parts (HTTP version, response code, and response reason) and provides methods to retrieve these parts. There are several exception classes, as well, most notable among them being ProtocolException; it indicates a protocol error, be it in a status line, a cookie, or an authorization. Finally, this package contains the implementation class for rfc2396 in the URI class. This generic class implements this RFC and is extended by the more specific schemes of HttpURL and HttpsURL. 1.3.2 The org.apache.commons.httpclient.methods package The previous package contained the interface definition and a base implementation for the various HTTP methods in the interface HttpMethod and the abstract class HttpMethodBase, respectively. However, the package org.apache.commons.httpclient.methods contains concrete implementations for the various HTTP methods. Table 1.6 lists the methods and the corresponding implementation classes. Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF Table 1.6 HTTP methods and their implementation classes HTTP method Corresponding class GET GetMethod PUT PutMethod HEAD HeadMethod POST PostMethod OPTIONS OptionsMethod TRACE TraceMethod DELETE DeleteMethod Missing a method in this table? If you compare it with table 1.2, you’ll notice that there doesn’t seem to be an implementation class for the HTTP CONNECT method. For some unknown reason, the CONNECT method implementation class is in the main org.apache.commons.httpclient package and is called ConnectMethod. All these classes (including ConnectMethod, but excluding PostMethod and PutMethod) extend the HttpMethodBase class. However, the POST and PUT method implementations have been conceptually grouped by extending them from the ExpectContinueMethod class, which in turn extends the HttpMethodBase class. The ExpectContinueMethod class is an abstract class; it’s used for handshakes during the POST and PUT requests because these requests require the server to send an intermediate 100 Continue status before sending the request. Another HTTP method class is implemented in this package, by extending the ExpectContinueMethod abstract class: the MultipartPostMethod class. This class is used to send multipart requests to the server, which is another form of the POST request, allowing request data to be sent in multiple parts. Module 2 describes this in more detail. Figure 1.4 shows the relationship between the various classes in this package. Note that this figure shows only the primary classes of this package and some classes from the main org.apache. commons.httpclient package. 1.3.3 The org.apache.commons.httpclient.auth package The package org.apache.commons.httpclient.auth contains the implementation of various authentication schemes supported by HttpClient for HTTP authentication. The three authentication schemes we’ve discussed (Basic, Digest, and NTLM) are all implemented in this package, along with support classes. The base definition for all schemes is specified in the AuthScheme interface. This interface defines methods to create authenticating requests and process responses from the server. The implementation classes implement this interface and provide their own mechanisms for creating these requests and processing server responses. These implementation classes are listed in Table 1.7. Table 1.7 Supported authenticating schemes in HttpClient Scheme Corresponding implementation class Basic BasicScheme Digest DigestScheme NTLM NTLMScheme BasicScheme and DigestScheme both extend from the RFC2617Scheme class (which implements the AuthScheme interface) because they’re similar in terms of the process they follow for authentication. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 17 Figure 1.4 UML diagram showing relationships in the org.apache.commons.httpclient.methods package This package contains several utility classes, which help in the authentication process. These classes are as follows: AuthState—Provides information about the state of the authentication process and allows you to change the authentication scheme being used. It can also be used to invalidate the whole process. AuthPolicy—Provides a one-stop shop for getting the proper scheme for authentication. Authentication schemes are registered with this class at startup and can be looked up as required. This allows custom authentication schemes to be plugged in. CredentialsProvider—Used to request credentials from the user, if the authentication process can’t find any credentials for the current process. AuthChallengeParser, AuthChallengeProcessor—Together, help to first parse the response headers in an authentication process and then process the challenge presented in the headers. This package contains several exception classes as well, which are used to indicate problems at various stages of the authentication process. For example, AuthChallengeException and Malformed- ChallengeException are used to indicate that a challenge from the server couldn’t be processed properly. Similarly, InvalidCredentialsException and CredentialsNotAvailableException are used to Licensed to Tricia Fu 18 JAKARTA COMMONS ONLINE BOOKSHELF indicate invalid and unavailable credentials from the user making the request. Note that all these exception classes, except MalformedChallengeException, derive from the general AuthenticationException class, which is a catchall exception for this package. 1.3.4 The org.apache.commons.httpclient.cookie package The org.apache.commons.httpclient.cookie package provides classes that handle cookies in HttpClient. However, as specified in section 1.3.1, this package relies on the Cookie class in the org.apache.commons.httpclient main package for the actual creation of cookies. The classes in this package are responsible for parsing response headers for Set-Cookie header values and creating Cookie instances out of it, for creating Cookie instances to be sent out with the request header Cookie, and for validating cookies according to the specifications. Toward this end, the interface CookieSpec in this package defines the methods that allow these functions to be performed. Two classes, CookieSpecBase and IgnoreCookiesSpec, implement this interface. As the name suggests, IgnoreCookiesSpec does nothing with cookies and headers that it gets; it returns empty values and performs no actions. CookieSpecBase is the base class for handling cookie functions. Two classes, RFC2109Spec and NetscapeDraftSpec, extend it and provide specific functionality based on rfc2109 (discussed earlier) and Netscape-developed specifications for cookie handling, respectively. The only exception class in this package is called MalformedCookieException (derived from ProtocolException). It indicates that a cookie is invalid according to the specification for which it’s being handled. 1.3.5 The org.apache.commons.httpclient.params package Managing HttpClient preferences in a separate package is a new concept in HttpClient version 3.0. The idea is to move the configuration of various instances from the instances themselves to a separate package. This allows centralized management of the configuration values and lets new configuration architectures be plugged in without causing too much disruption. The package contains classes that represent all the possible parameters that can be configured for instances of HttpClient, for instances of HTTP connection managers used in HttpClient, for instances of HTTP connections, and for instances of HTTP methods. These classes are listed in Table 1.8 along with the instances they represent. Table 1.8 Parameter classes in the org.apache.commons.httpclient.params package Instance Relevant class HttpClient HttpClientParams HostConfiguration HostParams HttpConnectionManager HttpConnectionManagerParams HttpConnection HttpConnectionParams HttpMethod HttpMethodParams For example, while creating an instance of an HttpClient, you would first create the HttpClientParams class and set the connection manager parameter on this class to a value you want. This HttpClientParams instance is then used to create the actual HttpClient instance. The DefaultHttpParams class in this package represents the default values of the parameters. Along with the DefaultHttpParamsFactory class, it lets you fall back to default values in case a parameter isn’t set. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 19 In addition to the packages we’ve discussed so far, the HttpClient API contains two other packages, org.apache.commons.httpclient.protocol and org.apache.commons.httpclient.util. The first package is used to create sockets using the protocol that is currently requested (SSL/non-SSL), and the second package contains utility functions. We won’t discuss these packages separately, but you’ll encounter their usage in the examples that follow. 1.4 HttpClient in action Section 1.1 gave you a taste of how HttpClient works. However, the example in that section was trivial, and you can do much more with HttpClient. In this section, you’ll see examples of HttpClient in various scenarios. These scenarios are divided into their own sections to give credible and comprehensive examples for each one. We’ll start with examples showing how to change the configuration parameters in HttpClient using the new org.apache.commons.httpclient.params package. 1.4.1 Managing HttpClient configuration using the Preferences Architecture There is a reason why the new API for managing configurations in HttpClient 3.0 is called the Preferences Architecture. The reason stems from the fact that configuration parameters follow a hierarchy where the value of the parameter at the lowest level is given preference over its predecessor’s value. This is best explained with an example. One of the headers sent as part of a request message in HTTP is called User-Agent. Its value identifies the process on the client machine that is making the request. For example, when you use IE to fetch pages, it identifies itself using this header as follows: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0). HttpClient, by default, uses the value Jakarta Commons-HttpClient/3.0-alpha2 or something similar, based on the version you’re using. What happens if you want to change this value? It’s very easy to change this default value in HttpClient. To do so, all you need to do is change the value of the global parameter httpclient.useragent to the value you want. The following code snippet shows how to change this value: client.getParams().setParameter("http.useragent", "My Own Browser at Client level"); Part of the snippet, client.getParams(), returns the collection of parameters applicable to the instance of HttpClient that is being worked with. This collection is encapsulated in the class HttpClientParams, which is part of the new org.apache.commons.httpclient.params package; that is, it’s part of the new Preferences Architecture. Similar to HttpClientParams, this architecture also includes HttpConnectionManagerParams for managing connection manager parameters, HttpConnectionParams for managing connection parameters, HttpMethodParams for managing method parameters, and HostParams for managing remote host parameters. To change the value of a parameter, you must access the parameters for that instance by calling the getParams() method. The new parameter value can then be changed by calling the setParameter(String name, String value) method, as in the previous snippet, and supplying the new value. Licensed to Tricia Fu 20 JAKARTA COMMONS ONLINE BOOKSHELF Although in the previous snippet we changed the value of the http.useragent parameter at the client level, it can also be changed at the host or method level. The following code snippet does exactly the same thing as the earlier one, except that the value is changed at the method level : method.getParams().setParameter("http.useragent", "My Own Browser at Method level"); What happens if the value is changed at both the client level and the method level? Which change is implemented? It depends which level you’re working on. Listing 1.2 shows this in action. Listing 1.2 Changing a parameter value at the client and method level package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.methods.GetMethod; public class HttpClientPreferencesV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); System.err.println("The User Agent before changing it is: " + client.getParams().getParameter("http.useragent")); client.getParams().setParameter( "http.useragent", "My Own Browser at Client level"); System.err.println("Client's User Agent is: " + client.getParams().getParameter("http.useragent")); GetMethod method = new GetMethod("http://www.google.com"); method.getParams().setParameter( "http.useragent", "My Own Browser at Method level"); try{ client.executeMethod(method); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } System.err.println("Method's User Agent is: " + method.getParams().getParameter("http.useragent")); } } Although http.useragent is a global parameter, its value depends on the level you’re working on. Therefore, at the client level, in listing 1.2, this value is set to "My Own Browser at Client level", but at the individual method level, this value is changed to "My Own Browser at Method level". Note that the value at the method level won’t be changed until the method is executed as in the listing. If you User-Agent changed at client level Prints client’s User-Agent User-Agent changed at method level Must execute method before method-level change is accepted Prints method’s User-Agent Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 21 were to now create a new method, it would inherit its parameters from the client level; therefore, its http.useragent value would be set to "My Own Browser at Client level". The idea is to make sure that components (HttpClient, methods, hosts and so on) at a lower level prefer the value of parameters at their own level, even though the same parameter may have been set to a different value at a higher level. Only if a parameter hasn’t been set at their level do the components go up a level until they find a component where the value has been defined. Of course, if the parameter doesn’t have a value at any level, the value is reverted to the highest level—called the global level—which may have a default value for that parameter. Figure 1.5 shows this hierarchy of components, along with the classes of parameters at each level in the org.apache.commons.httpclient.params package that implement the hierarchy. Although we have talked about the hierarchy, we haven’t discussed any of the parameters that can be changed. Listing all of them in this book would be too cumbersome, because there are too many. Please refer to the HttpClient documentation, especially the Preferences Architecture, for a complete list. Table 1.9 shows only those parameters that have a value at the global level. Table 1.9 A list of parameters that have a value at the global level Parameter Description Default value http.useragent Value used to identify the client using the User-Agent request header Jakarta Commons-HttpClient/3.0-alpha2 or similar, based on your version http.protocol.version Default protocol to use HTTP/1.1 http.protocol.element- charset Character set to use for encoding and decoding the HTTP status lines and headers US-ASCII http.protocol.content- charset Character set to be used for encoding the body of messages ISO-8859-1 http.protocol.cookie- policy Cookie policy to use RFC 2109 http.method.retry- handler Retry handler for retrying failed connections Default HttpClient retry handler called DefaultHttpMethodRetryHandler http.dateparser.patterns Date patterns to look for when parsing dates Several; refer to the documentation http.connection- manager.class Default connection manager to use for managing connections SimpleHttpConnectionManager class Now that you know how to get and set the various parameters of HttpClient, it’s time to look at how to use HttpClient with its various methods. The next section covers this with a variety of examples. 1.4.2 Using HttpClient methods HTTP method implementations are defined in the org.apache.commons.httpclient.methods package, except the CONNECT method, which is defined in the main org.apache.commons.httpclient package. In this section, you’ll learn how to use each of these methods. The basic setup is similar for each method; therefore, we’ll set the stage with GetMethod and then concentrate on the differences in method executions, rather than repeating ourselves. Downloading resources using GetMethod The GET method is the simplest possible HTTP method. It’s used to download resources from the Internet, and these resources can be the output of a server-side script, or a static document. The GetMethod class in the org.apache.commons.httpclient.methods package is used to execute these requests. Listing 1.3 shows a comprehensive example of how to use the GetMethod class. Licensed to Tricia Fu 22 JAKARTA COMMONS ONLINE BOOKSHELF Figure 1.5 Hierarchy of components for the Preferences Architecture Listing 1.3 Using GetMethod in a variety of ways package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.HttpException; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.protocol.Protocol; import java.io.File; import java.io.IOException; import java.io.FileOutputStream; public class GetMethodExampleV1 { public static void main(String args[]) { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); client.getParams().setParameter( "http.connection.timeout", Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 23 new Integer(5000)); GetMethod method = new GetMethod(); FileOutputStream fos = null; try { method.setURI(new URI("http://www.google.com", true)); int returnCode = client.executeMethod(method); if(returnCode != HttpStatus.SC_OK) { System.err.println( "Unable to fetch default page, status code: " + returnCode); } System.err.println(method.getResponseBodyAsString()); method.setURI( new URI("http://www.google.com/images/logo.gif", true)); returnCode = client.executeMethod(method); if(returnCode != HttpStatus.SC_OK) { System.err.println( "Unable to fetch image, status code: " + returnCode); } byte[] imageData = method.getResponseBody(); fos = new FileOutputStream(new File("google.gif")); fos.write(imageData); HostConfiguration hostConfig = new HostConfiguration(); hostConfig.setHost("www.yahoo.com", null, 80, Protocol.getProtocol("http")); method.setURI(new URI("/", true)); client.executeMethod(hostConfig, method); System.err.println(method.getResponseBodyAsString()); } catch (HttpException he) { System.err.println(he); } catch (IOException ie) { System.err.println(ie); } finally { method.releaseConnection(); if(fos != null) try { fos.close(); } catch (Exception fe) {} } } } Listing 1.3 shows a variety of ways in which the GetMethod can be executed. First, an instance of HttpClient is created, and two of its parameters are set. We have already covered the http.useragent Use GetMethod to get response body as String Get response body as byte array Execute GetMethod by specifying host HttpException indicates protocol error Always release connection Licensed to Tricia Fu 24 JAKARTA COMMONS ONLINE BOOKSHELF parameter. The second parameter, http.connection.timeout, is used to specify the time in milliseconds that HttpClient should wait for connections to be established. The first example is the simplest. The method is created and set with the document to fetch using the URI class. Note that this URI class isn’t java.net.URI but HttpClient’s own implementation, in the org.apache.commons.httpclient package. When a URI is set on a method using only the domain name or IP address and without an actual document, it defaults to fetching the index page of the domain, as in this case. When the client executes the method, it stores the response internally after separating it into its constituent parts; it returns only the response code, which is the status code returned by the server. To get the parts of the response, such as the body and headers, you need to call the relevant methods. For example, to get all the headers returned from the server, you would use method.getResponseHeaders(), which returns a collection of headers where each value is of type Header. The Header class in org.apache.commons.httpclient package is a simple name-value pair class. If you wanted the value of a specific header, you would instead use method.getResponseHeader(String headerName), passing in the specific header name. In this case, we wanted to get the body of the response, which is the index page of the domain requested. To get the body, we use getResponseBodyAsString(), which converts the received data to String format using the value of the Content-Type response header (or ISO-8859-1, if no such header value is available). If, instead of receiving the response body as String, we want to get the body in the form of a byte array, we can do so using the method getResponseBody(), as is done in the second example in listing 1.3. Here, we fetch an image of the Internet and store it on the local machine. Note that to fetch the image, we used the method from the previous fetch, but we changed the URI. To receive the response body as a Stream, you can use the getResponseBodyAsStream method, which returns an InputStream or null if no response body is available. It’s recommended that you use this method for large response bodies, where the size of the response body may not be known in advance. The third example in listing 1.3 shows how the GetMethod can be executed by specifying the host that the method should connect to in a separate configuration. The advantage of using the HostConfiguration class is that the remote host can be specified in a centralized place; therefore, changes to the host need to be made in only place. However, HttpClient only uses HostConfiguration set this way if the URI specified for the subsequent method(s) isn’t absolute. The methods derive HostConfiguration and override it, if an absolute URI is used, from the URI itself. The HostConfiguration is created by first specifying the host to connect to. Note that the URI of the document to fetch is specified on the method as a relative URI. Then the method is executed, passing in the host to connect to and the method to execute. Now that you know how to use HttpClient to execute methods using GetMethod, let’s see how to use the other methods in this API. Sending data using PostMethod The POST HTTP method lets you send data to the host machine by putting the data in the body of the request message. HttpClient makes sending this data very easy, by using PostMethod. To add data to the body of the response, the PostMethod class provides the addParameter(String name, String value) method. The PostMethod is created as the GetMethod in the previous section, and the data to be sent is combined using this method. You can also use the NameValuePair class to add these parameters. The NameValuePair class is a holder of names and values for parameters, and the PostMethod class provides a method each to add either individual parameter values or an array of these values. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 25 Listing 1.4 shows an example of running PostMethod. Listing 1.4 Using PostMethod to run a query on Yahoo package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.methods.PostMethod; import java.io.BufferedReader; import java.io.InputStreamReader; public class PostMethodExampleV1 { public static void main(String args[]) { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); BufferedReader br = null; PostMethod method = new PostMethod("http://search.yahoo.com/search"); method.addParameter("p", "\"Jakarta Commons in Action\""); try{ int returnCode = client.executeMethod(method); if(returnCode == HttpStatus.SC_NOT_IMPLEMENTED) { System.err.println("The Post method is not implemented by this URI"); // still consume the response body method.getResponseBodyAsString(); } else { br = new BufferedReader( new InputStreamReader(method.getResponseBodyAsStream())); String readLine; while(((readLine = br.readLine()) != null)) { System.err.println(readLine); } } } catch (Exception e) { System.err.println(e); } finally { method.releaseConnection(); if(br != null) try { br.close(); } catch (Exception fe) {} } } } Notice how the search term is set as the character p. This is what Yahoo’s search service expects it search term to be called, so that’s what it’s set as. It value is set to “Jakarta Commons in Action”, including the quotes. This search term is executed by calling the executeMethod. Add search term What if Yahoo doesn’t implement POST for search? If error occurs, response is consumed Read response as InputStream Licensed to Tricia Fu 26 JAKARTA COMMONS ONLINE BOOKSHELF Next, we check to see whether Yahoo accepted the query to search or rejected it by sending a Not Implemented error. This is possible, because Yahoo may only accept a search based on the GET protocol (at the time of this writing, it does accept a POST for search). Note that even if Yahoo rejects the search request, its response must still be consumed; otherwise the connection used for this request won’t be released by HttpClient. Finally, the response is read via an InputStream, because we expect the response body to be quite big, and we don’t know its size in advance. Sending multipart data using MultipartPostMethod When files are uploaded via HTTP, the file data must be sent in parts. This is commonly known as sending data in the form of multipart/form-data. PostMethod can’t be used to send the contents of files, because file data is likely to contain binary data. Using MultipartPostMethod is similar to using PostMethod. You use MultipartPostMethod’s overloaded addParameter methods to add parameters and their values to the method. If the parameter that is being added is a normal form element—that is, it isn’t file data—you use addParameter(String paramName, String paramValue), which is same as the PostMethod addParameter method. If it’s file data, you use addParameter(String parameterName, File parameterFile). Of course, the file to be uploaded must exist and must be readable; otherwise a FileNotFoundException is thrown. What happens, however, if the data that you want to send using this method is already loaded into memory and must be sent as a file? For example, suppose that in the act of sending data, a separate process in your application creates a report using a PDF file generator. The report data is already loaded into memory, so it seems futile to first save it to disk and then attach it using the addParameter method for files. MultipartPostMethod solves this problem by using the classes in org.apache. commons.httpclient.methods.multipart (specifically, in this case, the class ByteArrayPartSource) and by providing a method called addPart. The ByteArrayPartSource class is used by wrapping the in-memory bytes in a constructor. For example: ByteArrayPartSource source = new ByteArrayPartSource("report.pdf", bytes) This source is then used to create a FilePart object, to be added to the method using the addPart method, as shown here: method.addPart(new FilePart("report.pdf", source)); This adds the contents of the in-memory bytes to the request body as an independent part, which is identified as a file part and has the name report.pdf. Querying a server using OptionsMethod The OPTIONS HTTP method allows you to query a server to determine the list of methods it supports. This is handy in cases where you’re unsure of the capabilities of a remote server and want to find out your options before you request a particular method. Listing 1.5 shows a simple example of how to use OptionsMethod from HttpClient. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 27 Listing 1.5 Using OptionsMethod to determine a server’s capabilities package com.manning.commons.chapter01; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.methods.OptionsMethod; import java.util.Enumeration; public class OptionsMethodExampleV1 { public static void main(String args[]) { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); OptionsMethod method = new OptionsMethod("http://www.verisign.com"); try{ int returnCode = client.executeMethod(method); Enumeration list = method.getAllowedMethods(); while(list.hasMoreElements()) { System.err.println(list.nextElement()); } } catch (Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } This listing queries the server at www.verisign.com for a list of methods it supports and prints this list on the command line. OptionsMethod’s getAllowedMethods method is used to find these supported methods. OptionsMethod also contains a handy method to determine whether an HTTP method is supported. This method is called isAllowed(String methodName), and it returns a boolean true or false based on the methods support. Of course, it can only be called once the method has been executed. Proxy tunneling with ConnectMethod HttpClient provides a convenient way for applications to use the HTTP CONNECT method. The CONNECT method is provided so that connections can be made via proxy server/firewalls to remote servers through a tunneled socket connection over HTTP, even though the applications may themselves be talking to each other via non-HTTP protocols. The support for tunneling in HttpClient is provided by three classes: ConnectMethod derives from the HttpMethodBase class. ProxyClient is a handy class not only for making the connections and opening a tunneled socket via a proxy, but also for handling authentication. The ProxyClient class, in essence, replaces the Return enumerated list of allowed methods Licensed to Tricia Fu 28 JAKARTA COMMONS ONLINE BOOKSHELF HttpClient for tunneled proxy connections. ConnectResponse, which is a static inner class in ProxyClient class, provides access to the socket created via the ProxyClient class. Connecting to remote servers via a tunneled proxy in HttpClient comes down to implementing three steps: 1. Make an instance of ProxyClient (not HttpClient), and set the details of the proxy to be used and the remote server to which a connection is to be established. 2. Make the connection via this ProxyClient and, if successful, retrieve the socket that is created. 3. Write data to this socket using whichever protocol is required, and close the connection. A complete example of using ProxyClient is shown in listing 1.6. Listing 1.6 Using ProxyClient to make a tunneled socket connection via a proxy to a remote server package com.manning.commons.chapter01; import org.apache.commons.httpclient.ProxyClient; import org.apache.commons.httpclient.ConnectMethod; import org.apache.commons.httpclient.ProxyClient.ConnectResponse; import java.net.Socket; public class ConnectMethodExampleV1 { public static void main(String args[]) { ProxyClient client = new ProxyClient(); client.getParams().setParameter( "http.useragent", "Proxy Test Client"); client.getHostConfiguration().setHost("www.verisign.com"); client.getHostConfiguration().setProxy( "localproxyaddress", 80); Socket socket = null; try{ ConnectResponse response = client.connect(); socket = response.getSocket(); if(socket != null) { // write your raw data over this socket } else { // connection via the socket could not be established ConnectMethod method = response.getConnectMethod(); Give ConnectMethod that ProxyClient uses Create ProxyClient and set local and remote servers Write data over Socket here Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 29 System.err.println("Socket not created: " + method.getStatusLine()); } } catch (Exception e) { System.err.println(e); } finally { if(socket != null) try { socket.close(); } catch (Exception fe) {} } } } As you can see, the ConnectMethod that the ProxyClient uses internally to issue the CONNECT HTTP command can be retrieved from the response by calling the response.getConnectMethod() method. Because this ConnectMethod derives from HttpMethodBase class, it can be used like other HttpClient methods. Here, it’s used to determine the reason for the failed connection, by using the getStatusLine() method. Next, we’ll cover the remaining HttpClient methods, most of which are trivial enough to be bundled into one section. The remaining methods This section covers the rest of the HTTP methods supported by HttpClient. They’re bundled together in this section because they’re used very similarly, as is true for all the methods of HttpClient. The DeleteMethod class is used to delete a resource at a server by specifying the resource (as is done with the other methods) and then executing the method. However, support for this HTTP method isn’t widespread, so the method is of limited use. The same is true of PutMethod. This method is used to upload files to a server. However, it also isn’t widely supported due to security concerns. The difference between POST (multipart) and PUT lies in the fact that the resource identified by the PUT method is the exact place where the uploaded data is to be stored. This differs from the POST method, which only identifies the resource that will be used to handle the uploaded file. To use PutMethod, first create the PutMethod instance by identifying the resource on the remote server that the uploaded file will be saved as: PutMethod method = new PutMethod("http://someserver.com/uploads/logo.gif"); Next, set the request body of the method by attaching the file to be uploaded: method.setRequestBody(new FileInputStream("logo.gif")); Then, as with the other methods, execute this method using an HttpClient instance. The final HTTP method is TraceMethod. It’s used for debugging but also isn’t widely implemented. The idea is that on execution of this method, the remote server will return the unmodified original request body as it receives it. This helps detect whether the request body is being modified along the way to the server by a proxy or gateway, and it also helps trace errors. So far, we have only considered using HttpClient in a single-threaded environment, where methods are executed sequentially and there is no danger of data corruption due to simultaneous connection requests. In Reason connect failed Licensed to Tricia Fu 30 JAKARTA COMMONS ONLINE BOOKSHELF the next section, we’ll change this scenario a bit and show you how to use HttpClient in a multithreaded environment. 1.4.3 Using HttpClient in a multithreaded environment Using HttpClient in a commercial environment, it’s conceivable that an instance of HttpClient would be used to execute multiple requests simultaneously to cater to several requests at once. For example, consider your browser. At one time, an instance of your browser may be downloading the latest news from CNN.com, doing a search on Google, and posting your resume on several job sites. If you only had the option to perform these tasks one after the other, it would take a long, long time before you would net any jobs! You need the ability to simultaneously effect several requests using HttpClient. Let’s first consider what happens when you try to do this with HttpClient without any changes to the way it manages connections. Listing 1.7 shows an example of downloading two different files from a local server at the same time using two different threads. Listing 1.7 Simultaneously executing requests without proper connection management package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; public class ThreadingExampleV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "Test Client"); HostConfiguration host = new HostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); MethodThread bigDataThread = new MethodThread(client, host, "/big_movie.wmv"); bigDataThread.start(); MethodThread normalThread = new MethodThread(client, host, "/"); normalThread.start(); } } class MethodThread extends Thread { private HttpClient client; private HostConfiguration host; private GetMethod method; Start big movie download thread Start normal page download thread Thread class to manage method execution Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 31 public MethodThread( HttpClient client, HostConfiguration host, String resource) { this.client = client; this.host = host; this.method = new GetMethod(resource); } public void run() { System.err.println("Connecting to: " + host); try{ client.executeMethod(host, method); method.getResponseBodyAsStream(); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } If you run this listing, to be honest, I’m not sure what output you’re likely to get. The results are truly unpredictable. Why is that? On the surface, the listing should work as it’s intended. We start two threads: one to download a big file, the other to download the index page of the same local server. Each thread gets the same instance of HttpClient, the same HostConfiguration, and its own individual resource to get. Each thread then makes its own instance of GetMethod and executes it in its run method. Sounds simple enough, until you realize that each GetMethod tries to use the same connection instance to handle each request. This is because HttpClient, by default, uses a connection manager class that provides access to this same connection instance across all requests per HostConfiguration. This connection manager class is SimpleHttpConnectionManager. The SimpleHttpConnectionManager class ensures that if a new request is being made to the same host, then the last request to the host is forcibly closed before the connection for the next request is released. If a new request is made to a different host, it clears up the previous host details, sets the same connection with the new host details, and then releases the connection. As you can see, if two requests arrive at the same time for the same shared connection, unpredictability in the response is almost certain. One inelegant way to solve this problem is to have a separate client instance for each thread. Each client has its own connection instance, so there is no chance of unpredictable results. However, creating a new HttpClient for each request is fraught with resource-intensive operations. The best way to solve this problem is provided by HttpClient itself: Use the MultiThreadedHttpConnectionManager class. As the name suggests, this connection manager class is best suited for multithreaded applications, where simultaneous requests are handled by supplying a separate connection for each request. Connections are bundled together per host, which means that a finite number of connections are created per host, and requests to the same host are provided through this bundle via synchronized reuse. If a connection isn’t available at the time it’s requested, the connection manager waits a finite time before trying again; it times out if no connection is available. The default number of connections created per host is limited to two, but you can easily change that, as you’ll see in listing 1.8. Execute methods Read server responses Licensed to Tricia Fu 32 JAKARTA COMMONS ONLINE BOOKSHELF To make HttpClient use the MultiThreadedHttpConnectionManager, the HttpClient instance must be created with this manager or installed on it with the setHttpConnectionManager method before making any requests. Listing 1.8 shows a complete example of using this manager, borrowing from the situation created in listing 1.7. Listing 1.8 Simultaneously executing requests with proper connection management package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager; import org.apache.commons.httpclient.params.HttpConnectionManagerParams; public class ThreadingExampleV2 { public static void main(String args[]) throws Exception { MultiThreadedHttpConnectionManager manager = new MultiThreadedHttpConnectionManager(); HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams(); connectionManagerParams.setDefaultMaxConnectionsPerHost(5); manager.setParams(connectionManagerParams); HttpClient client = new HttpClient(manager); client.getParams().setParameter("http.useragent", "Test Client"); HostConfiguration host = new HostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); // first Get a big file MethodThread bigDataThread = new MethodThread(client, host, "/big_movie.wmv"); bigDataThread.start(); // next try and get a small file MethodThread normalThread = new MethodThread(client, host, "/"); normalThread.start(); } } If you run this listing, you’ll get the expected results. The big_movie file will be downloaded in its own thread and the index page in its own. One final thing about this example to note before we move on: An HttpConnectionManagerParams class is used to set the default maximum number of connections per host. This way of setting the parameters of a component in HttpClient is different from the way we discussed in section 1.4.1, but it’s valid. You can use each parameter class in HttpClient directly to change the value of parameters, instead of using the generic Set maximum connections for any host Set parameter class with manager Set client with new manager Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 33 setParameter method. In addition, the maximum connections are set as the default for all hosts. What if you wanted to set this value higher or lower per host? Easy: You can do so as shown here: connectionManagerParams.setDefaultMaxConnectionsPerHost(host, 10); This allows you to set the connection limit higher for highly requested sites (or lower for rare sites) and keep a bare minimum for all other hosts by using setDefaultMaxConnectionsPerHost. 1.4.4 Authenticating requests Accessing protected and secure resources over the Internet requires authentication between the client and the server. As you know, HttpClient supports three types of authentication, in this order: NTLM, Digest, and Basic. The order of authentication types is important because if multiple authentication types are available for a request/connection, HttpClient picks NTLM over Digest over Basic. You can change this order if required. We’ll begin this section with the simplest authentication scheme, Basic. The mechanics of using the Basic and Digest schemes are similar, both to the end user and to the developer, so we won’t discuss the Digest scheme. To set the background for the Basic authentication example, let’s first create a page that must be protected and deploy it in a web server protected by a username and password. We’ll use Tomcat (4.1.27) for the web server, and we’ll use its prebuilt roles for authentication. The role we’ll use is called tomcat, and the username and password for this role are tomcat and tomcat, respectively (set in the file tomcat-users.xml under the /conf directory). Listing 1.9 shows the entries to be made in a web.xml file to make a page protected. The page itself is simple; it shows the message Welcome to the protected page and is called protected.jsp. Listing 1.9 Modifying web.xml to make parts of a web application protected Security Constraint Protected Area /chapter01/* tomcat BASIC Chapter 1 Protected Area Note that this listing is only part of web.xml; the file must include other entries before it can be used. Deploy this web.xml on the server along with the protected page (protected.jsp). When you start Tomcat after this change and request the page via your web browser, Tomcat requests the authentication details as shown in figure 1.6. Correct authentication details will let you see the protected page. Protect everything under chapter01 directory Give protected area a realm name Licensed to Tricia Fu 34 JAKARTA COMMONS ONLINE BOOKSHELF Figure 1.6 Accessing the protected page from Tomcat via browser However, this isn’t about accessing a protected page via a browser, but about accessing a protected page via HttpClient. Listing 1.10 shows our first attempt at fetching this protected page via HttpClient. Listing 1.10 First attempt at fetching a protected page: will fail package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; public class BasicAuthenticationV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); HostConfiguration host = client.getHostConfiguration(); Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 35 host.setHost(new URI("http://localhost:8080", true)); GetMethod method = new GetMethod("/commons/chapter01/protected.jsp"); try{ client.executeMethod(host, method); System.err.println(method.getStatusLine()); System.err.println(method.getResponseBodyAsString()); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } If you run this code, it won’t fetch the protected page. Why (other than the fact that no authentication details have been supplied)? Because authentication is a two-step process. First the client makes a request for a protected page, and the server responds with a message saying it requires authentication details before it can send that page. In the second step, the client repeats the request with the authentication details attached, and the server responds with the page. We need to either request the page in two steps or tell HttpClient to perform only the second step. With HttpClient, you can use either of these two options. Listing 1.11 shows how to use the first, and listing 1.12 shows how to use the second. Listing 1.11 Two-step authentication package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpState; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.Credentials; import org.apache.commons.httpclient.auth.AuthScope; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.UsernamePasswordCredentials; public class BasicAuthenticationV2 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); HostConfiguration host = client.getHostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); GetMethod method = new GetMethod("/commons/chapter01/protected.jsp"); try{ int statusCode = client.executeMethod(host, method); if(statusCode == HttpStatus.SC_UNAUTHORIZED) { Check whether first attempt rejected due to lack of authorization Licensed to Tricia Fu 36 JAKARTA COMMONS ONLINE BOOKSHELF System.err.println("Authorization required by server"); Credentials credentials = new UsernamePasswordCredentials("tomcat", "tomcat"); AuthScope authScope = new AuthScope(host.getHost(), host.getPort()); HttpState state = client.getState(); state.setCredentials(authScope, credentials); client.executeMethod(host, method); } System.err.println(method.getStatusLine()); System.err.println(method.getResponseBodyAsString()); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } To authenticate an HttpClient request, we start with a Credentials object. The UserNamePasswordCredentials class stores the username and password independent of any host or realm. The AuthScope is created next; it narrows the scope (the host, the port, the realm on the host) to which the credentials will be applied (not just the credentials created earlier, but any credentials). These two, scope and credentials, are combined in the HttpState of the current client instance to provide authentication credentials for the current Host. When the client reexecutes the method, it uses these credentials to provide authentication details and download the protected page. The HttpState class is a storehouse for persistent information across HttpClient requests. Thus, the state can store not only the credentials, but cookie information as well (we’ll study this in the next section). Notice that we didn’t specify the authentication scheme anywhere in the previous code. This makes sense because the server requests the Basic authentication scheme (recall that it was done in the web.xml file in listing 1.9), and HttpClient automatically switches to using this scheme. Listing 1.12 shows a one-step authentication process in which HttpClient handles authentications on its own. Listing 1.12 One-step authentication package com.manning.commons.chapter01; import org.apache.commons.httpclient.URI; import org.apache.commons.httpclient.HttpState; import org.apache.commons.httpclient.HttpStatus; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.Credentials; import org.apache.commons.httpclient.auth.AuthScope; import org.apache.commons.httpclient.methods.GetMethod; import org.apache.commons.httpclient.HostConfiguration; import org.apache.commons.httpclient.UsernamePasswordCredentials; Create authorization credentials Retrieve client’s current state Resubmit original request Update state with credentials and scope Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 37 public class BasicAuthenticationV3 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); HostConfiguration host = client.getHostConfiguration(); host.setHost(new URI("http://localhost:8080", true)); Credentials credentials = new UsernamePasswordCredentials("tomcat", "tomcat"); AuthScope authScope = new AuthScope(host.getHost(), host.getPort()); HttpState state = client.getState(); state.setCredentials(authScope, credentials); GetMethod method = new GetMethod("/commons/chapter01/protected.jsp"); try{ client.executeMethod(host, method); System.err.println(method.getStatusLine()); System.err.println(method.getResponseBodyAsString()); } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } As you may notice, the state of the HttpClient is set before any method is executed. After this, any time a request is made to the specified host (and in this case, for any realm on that host, for a protected page), authentication is automatically handled when executeMethod is called. You can restrict the realm for which credentials are valid by specifying it as a parameter to the scope. For example, new AuthScope(host.getHost(), host.getPort(), "Chapter 1 Protected Realm") is a valid scope for the "Chapter 1 Protected Realm" realm only on Host localhost at port 8080. If you wanted the username/password combination of tomcat/tomcat to be valid for any realm on any host and any port, you would create the Scope as new AuthScope(null, -1, null). One niggling doubt remains about the way credentials are provided in code. Notice that both the username and the password in each of the listings in this section were provided as part of the source code. This is a highly undesirable way of specifying this information for several reasons, security of the authentication details being the most significant. It would be nice if an admin user running this application could provide these values at runtime or a separate process could get these details from secure storage. HttpClient provides the CredentialsProvider interface with a single method that has the signature Credentials getCredentials(AuthScheme scheme, String host, int port, boolean proxy) It can be implemented to provide the credentials from an external store at runtime, or from any other suitable medium. Thus, a class that implements this interface, called CredentialsProvider, returns the credentials based on the requested authorization scheme and host, and the implementation details are left for Credentials and scope are created before method is executed Method automatically handles authentication Licensed to Tricia Fu 38 JAKARTA COMMONS ONLINE BOOKSHELF the class to decide on. However, you must make HttpClient aware of this CredentialsProvider by calling client.getParams().setParameter(Credentials.PROVIDER, new CustomCredentialsProvider()) where CustomCredentialsProvider is the name of this provider. Before we finish this section, a note about the NTLM authentication scheme. As far as the developer is concerned, using NTLM authentication is exactly like using the Basic authentication as shown in the previous examples, except for one small detail. NTLM authentication requires the developer/user to provide the domain on the NT server for which authentication is being performed, in addition to the username and password. Therefore, instead of using UsernamePasswordCredentials in the previous examples, you must use the NTCredentials class, which extends UsernamePasswordCredentials. Supplying the domain and host, in addition to the username and password, as shown here, creates the NTCredentials class for use in authentication. The other details of the authenticating process remain the same: Credentials credentials = new NTCredentials("user", "password", "127.0.0.1", "mydomain.com") We mentioned earlier that the HttpState class contains persistent information between requests for an HttpClient instance. One such piece of persistent information related to authentication details, which we have covered in this section. The other relates to cookies. It’s time to take a deeper look at the cookie- handling mechanism in the next section. 1.4.5 Handling and using cookies HttpClient automatically handles cookies by accepting whatever cookies the server sends and then sending these cookies back if required. It also allows you to create your own cookies and send them back to the server; what the server will do with those cookies that it didn’t initially send is a different matter. In this section, we’ll demonstrate how to read the cookies that a server sends to HttpClient and how to modify a specific cookie before it’s re-sent to the server. Reading cookies sent by a server is fairly easy. Listing 1.13 shows an example of how to read cookies sent by the Tomcat server for accessing the index page. Listing 1.13 Printing a list of cookies sent by a server package com.manning.commons.chapter01; import org.apache.commons.httpclient.Cookie; import org.apache.commons.httpclient.HttpState; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.GetMethod; public class CookieExampleV1 { public static void main(String args[]) throws Exception { HttpClient client = new HttpClient(); client.getParams().setParameter("http.useragent", "My Browser"); GetMethod method = new GetMethod("http://localhost:8080/"); Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 39 try{ client.executeMethod(method); Cookie[] cookies = client.getState().getCookies(); for (int i = 0; i < cookies.length; i++) { Cookie cookie = cookies[i]; System.err.println( "Cookie: " + cookie.getName() + ", Value: " + cookie.getValue() + ", IsPersistent?: " + cookie.isPersistent() + ", Expiry Date: " + cookie.getExpiryDate() + ", Comment: " + cookie.getComment()); } } catch(Exception e) { System.err.println(e); } finally { method.releaseConnection(); } } } All the cookies associated with the current instance of HttpClient are retrieved, regardless of the host they came from. You can narrow down this array of cookies by using this method: client.getState().getCookies(String domain, int port, String path, boolean secure) method It gets all the cookies received from the specified domain by HttpClient. To modify a cookie before it’s automatically sent back to the server, start by retrieving the cookie, and then modify any of its configurable properties. However, note that servers have the final say in cookie management, and if the server thinks that the cookie it originally sent isn’t the same as the cookie it received back, it may reject that cookie. The following code snippet shows how to modify the value of a cookie once you’ve extracted this cookie from the current HttpState: cookie.setValue("My own value"); When the modified cookie is re-sent, its value is changed to “My own value”. Note that because HttpClient handles cookies automatically—that is, it sends cookies with requests only if the cookie hasn’t expired—the cookie may not be sent. However, you can change the expiration date on cookies and still send them back to servers. This isn’t guaranteed to work, and the result is mostly unpredictable. Because HttpClient doesn’t allow for permanent storage of cookies, there is no mechanism for storing permanent cookies—that is, cookies with no expiration date. As far as HttpClient is concerned, a session is the life of the HttpClient application. Once the application exits the Java Virtual Machine (JVM), these cookies are no longer valid, unless you take specific steps before the JVM exits to persist the cookies in long- term storage and read again the next time the application starts. In your application, if you don’t wish to deal with cookies, you can set your application’s cookie policy to ignore all cookie headers in requests and responses. Use the following code to do this: client.getParams().setParameter("http.protocol.cookie-policy", CookiePolicy.IGNORE_COOKIES); The default cookie policy is based on rfc2109. Retrieve all cookies for current client Print information about each cookie Licensed to Tricia Fu 40 JAKARTA COMMONS ONLINE BOOKSHELF 1.5 Summary From authenticating users to handling cookies to handling all HTTP methods, HttpClient provides a complete client-side implementation of the HTTP protocol. It’s one of the best-documented components of the Commons stable, and it has an active user community that keeps contributing and increasing its depth. In this module, we started with a simple teaser application that showed how to use HttpClient in the most basic manner. This gave you a taste of things to come and allowed us to digress a little to discuss the technologies behind HttpClient. These technologies form the basis of HttpClient; a discussion of HttpClient without HTTP, Basic and Digest authentication, and cookies would have left a void. We rounded the module off by showcasing the use of HttpClient. We began by using the new Preferences Architecture and continued with the functionality of HttpClient from basic to advanced. In the next module, we’ll continue with all things web-related and look at the FileUpload component. Licensed to Tricia Fu MODULE 1: BROWSING WITH HTTPCLIENT 41 Index AuthChallengeException, 17 AuthChallengeParser, 17 AuthChallengeProcessor, 17 authenticating users, 9 Basic and Digest authentication, 33–38 org.apache.commons.httpclient.auth package, 16 understanding NTLM, 13 understanding rfc2617, 9 using NTLM authentication, 38 AuthPolicy, 17 AuthScheme interface, 16 AuthState, 17 Basic authentication, 33. See authenticating users BasicScheme, 16 Cache-Control header, 6 possible request values, 6 possible response values, 6 ConnectMethod, 27 ConnectResponse, 28 cookie policy, 39 cookies ignoring headers, 39 modifying, 39 org.apache.commons.httpclient.cookie package, 18 reading, 38 understanding rfc2965, 8 using, 38–40 CookieSpec, 18 CookieSpecBase, 18 credentials. See also authenticating users CredentialsNotAvailableException, 18 CredentialsProvider, 17, 37 DeleteMethod, 29 Digest authentication. See authenticating users Digest authentication parameters calculating response, 11 DigestScheme, 16 ExpectContinueMethod, 16 firewall, 6 GetMethod, 21 HostParams, 18, 19 HTTP. See Hypertext Transfer Protocol differences between HTTP 1.1 and HTTP 1.0, 7 HttpClient, 1 API, 14–19 changing User-Agent header, 19 configuration. See HttpClient: Preferences Architecture: creating tunneled connections, 27 determining supported services, 26 downloading resources, 22–24 methods, 21 Preferences Architecture org.apache.commons.httpclient.params package, 18 using, 19 sending data, 24–26 simple example, 1–2 supported RFCs, 3 uploading files, 26 using ConnectMethod. See HttpClient: creating tunneled connections using GetMethod. See HttpClient:downloading resources using in a multithreaded environment, 30–33 using MultipartPostMethod. See HttpClient: uploading files using OptionsMethod. See HttpClient:determining supported services using PostMethod. See HttpClient: sending data HttpClientParams, 18, 19 HttpConnectionManagerParams, 18, 19, 32 HttpConnectionParams, 18, 19 HttpMethodBase, 16 HttpMethodParams, 18, 19 HttpState, 36 hypermedia, 3 hypertext, 3 Hypertext Transfer Protocol. See also rfc2616 IgnoreCookiesSpec, 18 InvalidCredentialsException, 18 maintaining state. See cookies MalformedChallengeException, 17 MalformedCookieException, 18 multipart/form-data, 26 MultipartPostMethod, 16, 26 multithreaded environment, 30 MultiThreadedHttpConnectionManager, 14, 31 NetscapeDraftSpec, 18 NT Lan Manager. See authenticating users NTCredentials, 38 NTLM authentication, 38 OptionsMethod, 26 org.apache.commons.httpclient package, 14 org.apache.commons.httpclient.auth package, 16 org.apache.commons.httpclient.methods package, 15, 21 org.apache.commons.httpclient.params package, 19, 21 org.apache.commons.httpclient.protocol package, 19 org.apache.commons.httpclient.util package, 19 PostMethod, 24 Pragma header. See Cache-Control header Preferences Architecture component hierarchy, 22 parameters, 21 Licensed to Tricia Fu 42 JAKARTA COMMONS ONLINE BOOKSHELF proxy tunneling, 27 ProxyClient, 28 PutMethod, 29 realm, 36. See also authenticating users rfc2109. See cookies RFC2109Spec, 18 rfc2396, 13 rfc2616 features, 3 caching, 6 methods, 5 methods supported, 5 proxies and tunnelling, 5 request-response architecture, 4 stateless protocol, 5 rfc2617. See authenticating users RFC2617Scheme, 16 scope, 37 SimpleHttpConnectionManager, 14, 31 TraceMethod, 29 transparent proxy, 6 Uniform Resource Identifiers. See understanding rfc2396 URI class, 15 UsernamePasswordCredentials, 38 © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu JakartaCommons ONLINE BOOKSHELF Uploading files with FileUpload Vikram Goyal MODULE MANNING 2 Licensed to Tricia Fu Jakarta Commons Online Bookshelf Module 1 Browsing with HttpClient Module 2 Uploading files with FileUpload Module 3 Handling Protocols with the Net Component Module 4 XML parsing with Digester Module 5 JXPath and Betwixt: working with XML Module 6 Validating data with Validator Module 7 Enhancing Java core libraries with Collections Module 8 Enhancing Java core libraries with BeanUtils and Lang Module 9 Pool and DBCP: Creating and using object pools Module 10 Codec: encoders and decoders Module 11 Managing components with Modeler Module 12 Command-line processing with the CLI Module 13 Understanding and using Chain Module 14 Working with the Logging and Discovery components © Copyright 2005 by Manning Publications Co. All rights reserved. To order more modules from the Jakarta Commons Online Bookshelf, go to www.manning.com/goyal Licensed to Tricia Fu 1 Module 2 Uploading files with FileUpload 2.1 HTTP and uploading files......................................................................................................................................... 1 2.2 FileUpload basics...................................................................................................................................................... 6 2.3 Uploading files without using FileUpload.............................................................................................................. 10 2.4 Uploading files using FileUpload........................................................................................................................... 13 2.5 Handling a complex input form.............................................................................................................................. 16 2.6 Uploading large files............................................................................................................................................... 20 2.7 Uploading files to a database.................................................................................................................................. 22 2.8 Summary................................................................................................................................................................. 27 Index ............................................................................................................................................................................. 29 The ability to transfer files from a user’s PC to a remote server over the Internet allows the user to share images, documents, and audio-video files with other users around the world. The FileUpload component is an integral part of this transfer and sits on the server end. It receives, validates, and processes upload requests in a seamless fashion. In this chapter, we’ll review the basics of transferring files from a user’s computer to a server, and you’ll learn about rfc1867, the standard on which FileUpload is based. You’ll then see an example of uploading files without using the FileUpload component. This will get you ready for the FileUpload component itself. We’ll discuss this component with some basic examples and advanced usage scenarios. 2.1 HTTP and uploading files File upload is the transfer of files from a user’s computer to a server via an Internet browser. It’s a subset of file transfer, which is the process by which files are transmitted between different machines. These machines must be interconnected by an underlying networking mechanism and must have software on either end that facilitates this transfer. A browser on a client machine, for example, is the high-level software that transfers files to a remote machine, which has server software running on it that handles the incoming files. To fully understand file transfer, it’s important to have a working knowledge of the underlying protocol that file transfer works with. Hypertext Transfer Protocol (HTTP) is this underlying protocol over which a browser transfers files. As opposed to HTTP, the File Transfer Protocol (FTP) is more specialized and geared toward the explicit transfer of files between machines. We’ll discuss FTP further in module 3, “Handling protocols with the Net component,” when we discuss the Commons Net component. HTTP works on the basis of a request/response architecture. The user’s browser sends a request message to a server for a particular resource, and the server responds with a response message to the original request. The format of these messages is standard. The browser sends the request to the server by encoding the user request in a simple fashion. This message contains the usual HTTP headers, and the user data is encoded using URL encoding. The server responds with a message that contains, in addition to the headers, the actual data in a Multipurpose Internet Mail Extension (MIME)–like format. File transfer is inherently bidirectional and implicit. Think about what happens when you request a web page over the Internet through your browser. The browser makes a request to retrieve the requested page from Licensed to Tricia Fu 2 JAKARTA COMMONS ONLINE BOOKSHELF the server. The browser then makes a separate request for each element within that page and transfers each of these elements to your machine. By transferring files from the server to your machine, the browser is acting as a file transfer agent. Each image, HTML file, or data file is retrieved from the server, transferred over the Internet, and stored on your machine. Thus file transfer is happening behind the scenes even though you may not realize it. When you request that a file be transferred from the server to your machine, you’re downloading files: This is explicit file transfer, as opposed to the implicit file transfer/download that the browser does on your behalf when you request a web page. But how is file transfer inherently bidirectional? The underlying protocol doesn’t put any restrictions on the order of transfer of files between machines. Files can be transferred in either direction and are transferred whenever a transaction takes place (it helps in this analogy to think about a file as blob of data). A browser can upload and download files; a server can receive and send files. Let’s look at how a simple file upload request is initiated and processed. A user requests a page that contains an HTML element of type and a form with an encoding of type multipart/form-data instead of the normal application/x-www-form-urlencoded. This page is rendered by the browser as part of a form element that allows the user to navigate his computer and select a file for transfer. The user then submits this form to the server. Now the browser takes over by converting this selected file into a MIME format for transfer to the server. At the server, this form is parsed, and the different parts of this form are processed. A Common Gateway Interface (CGI) program does this processing by interpreting the different parts according to their content type. The semantics of how file upload is handled are defined by rfc1867. 2.1.1 Understanding rfc1867 To understand how a browser handles a file-upload request, you need to first understand the rules that a browser follows to bundle the user’s request to upload files so that files can be transferred over the Internet using the HTTP protocol. These rules are defined by rfc1867. A Request for Comments (RFC) is an Internet Engineering Task Force (IETF) document that relates to creating standards for the Internet. RFCs define the many protocols, standards, and procedures that have become the guiding force of the Internet today. Protocols like HTTP (rfc1945), standards like MIME (rfc2045), and procedures like the Internet Standards Process (rfc2026) all began as RFCs published by the IETF. You can access these documents by logging on to www.faqs.org/rfcs. Rfc1867 defines how a browser handles file upload requests from page authors and how a browser is supposed to retrieve and parse this information from the client’s computer and send it to the server. It’s an extension of the original HTML and MIME formats and is the standard by which the server parses the files sent to it. This RFC enables file upload from HTML forms using two mechanisms. The first deals with what a page author needs to do to enable file upload and how such a request is represented by a browser to the end user. The second defines how a browser then formats the form for upload to the server. Both of these mechanisms are necessary for a fully functional and successful file upload. The program on the server that is responsible for parsing these formatted requests needs to be aware of the format and parse the request accordingly. The FileUpload component does this by adhering to this RFC. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 3 Understanding multipart messages Before we delve into the two mechanisms, it’s important for you to understand multipart messages. A multipart message is a message that contains one or more other message parts of different content types as defined by rfc1521 for MIME messages. So, for example, a multipart message may contain a message that is of Content-Type text/plain, and it may contain a message of Content-Type image/gif. Each part of a message is separated by a boundary, which is explicitly defined by the message composer. This boundary must be unique and must not occur within the data defined in the message. Each part of this multipart message can itself be multipart with its own separate boundary. A simple multipart message is shown here: Content-type: multipart/mixed; boundary="AABB09" This is a preamble and is ignored --AABB09 content-type:text/plain; charset=us-ascii This is a plain text message --AABB09 content-type: image/gif content-transfer-encoding: base64 .. base64 encoded message .. --AABB09-- This is an epilogue and is also ignored The first line clearly identifies the message as multipart and defines the boundary. A boundary must start by itself on a line and be preceded by a double hyphen (–); in this case, the boundary is -- AABB09. Next, the first part’s headers identify it as text/plain and of charset US-ASCII. The second part is encoded as Base64 so that binary image file can be transferred easily. The end of the message (the last part) is indicated by a double hyphen at both the beginning and the end of the boundary: -- AABB09--. Note that each header must be followed by a new line (CRLF) that doesn’t have anything else on it. Mechanism 1: creating and displaying a file upload request To implement the first mechanism to upload files from HTML forms, the RFC mandates the following: The page author must specify an ENCTYPE of multipart/form-data for the FORM element. In the absence of an ENCTYPE, the FORM element defaults to application/x-www-form-urlencoded. The RFC stipulates that if the ENCTYPE isn’t set to application/x-www-form-urlencoded and the enclosing form contains an INPUT type of FILE, the behavior is unspecified. The page author must specify an INPUT element of TYPE FILE, and this INPUT type must be nested within a FORM element. There is no restriction on the number of these elements within a page, and this Licensed to Tricia Fu 4 JAKARTA COMMONS ONLINE BOOKSHELF type can be intermixed with other INPUT elements of different types. The RFC gives the browsers a lot of freedom in the way they implement the FILE attribute. The RFC recommends the introduction of an ACCEPT attribute for the INPUT element, but this attribute has been ignored by almost all browser implementations. This ACCEPT attribute was recommended to limit the types of files a user could select for uploading. The page author must specify the METHOD attribute of the FORM element as POST. Since a GET request tags the values of a form at the end of the ACTION URL, this makes sense. You wouldn’t send file data appended to the end of a URL. On encountering an INPUT element with the TYPE set as FILE, the browser must display a text field with a file-selection button next to it. The user can directly type the location of the file to upload in the text box or use the file-selection button to select the file from her computer. Mechanism 2: packaging a file upload request The second mechanism that rfc1867 uses to enable file upload from HTML forms defines a format for the browser to send this data to the server by mandating the following: The browser must create a multipart message with a subtype of form-data, with a message boundary that doesn’t occur in any of the data that is being sent. So, the Content-Type of this message must be of the type multipart/form-data, and a boundary must be defined that separates each message. Each piece of the form’s original data must be sent as a part of this multipart message. Each part of this message must correspond to a single data entry in the original form. This part must contain the Content-Disposition header, which must be equal to form-data and must contain the original name of the data field in the form. For example, the content-disposition header for an element for file input may be content-disposition: form-data; name="fileToUpload". Similarly, a Content-Disposition header for a normal text input may be content- disposition: form-data; name="textField". Each part may have an optional Content-Type header. As with MIME messages, Content-Type specifies the type of the relevant part, which helps identify the type of application that can parse this particular part. If no Content-Type is specified, it defaults to text/plain. Each part may have a Content-Transfer encoding specified as part of the headers. This is useful in cases where the normal encoding of 7-bit US-ASCII may not be relevant or plausible for transfer of data. When formatting file data, a browser may choose to inform the server about the filename of the original file by putting this value in the Content-Disposition header of the part in which the data is represented. This is done by setting the filename attribute in this header: for example, Content- Disposition: form-data; filename="somefile.txt". The browser must send all the data, and it must send this data in the order in which it occurs in the original form. An example message composed according to these rules is shown in the following code. Assume for this listing that the end user filled out a form that asked her to type her name in a textbox called username and to attach a file called userfile. You’ll notice that the syntax is similar to that of the code shown earlier in the sidebar “Understanding multipart messages”; only minor changes in the required headers are needed: Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 5 Content-type: multipart/form-data, boundary="AOFDE00" --AOFDE00 content-disposition: form-data; name="username" Fred Somebody --AOFDE00 content-disposition: form-data; name="userfile"; filename="file1.txt" Content-Type: text/plain ... contents of file1.txt ... --AOFDE00-- Note: What happens when one of the form elements allows for multiple values to be selected—for example, using checkboxes or lists? How is this reflected in a multipart message? Each value the user selects is reflected as a part in itself. If the user selects two values in a group of five checkboxes, for example, there will be two separate parts in the resulting multipart message, each with its own boundary but the same name (if the checkboxes were given the same name by the page author). Similarly, each value a user selects in a multiple-selection list is represented with its own part that includes the selected value and the name of the list. Although rfc1867 mandates most of the requirements for successful file transmission, it also suggests several recommendations. These recommendations aren’t mandatory, which leads to incompatibilities in the way different browsers handle the same recommendations and causes these recommendations to sometimes be misinterpreted or ignored. In the next section, we’ll look at some of these omissions. 2.1.2 Rfc1867 omissions It’s useful to point out at this stage that rfc1867 includes several recommendations for the way a browser behaves for file uploads and what a page author is allowed to do. However, these recommendations aren’t mandatory and have largely been ignored by most browser implementations. In addition to specifying required behaviors, an RFC suggests or recommends other behaviors, which may or may not be implemented by the people implementing the RFC. Sadly, this is the case for rfc1867. In this section, we’ll look at some omissions from this RFC in today’s browser implementations. One of the most important omissions from current browser implementations is the ACCEPT attribute of the INPUT element. An ACCEPT attribute would have allowed page authors to specify the types of files that a user may upload. This would then restrict the end user from selecting files that aren’t of this particular type. Such an attribute would have saved a lot of parsing time at the server end, where the server program must determine whether the user selected the right type of file before processing it further. The VALUE attribute allows the INPUT element to be prefilled with a default value. However, with a TYPE attribute of FILE, it serves no purpose. The RFC recommends that, if specified, the browser may preselect a default filename in this field. Unfortunately, this isn’t implemented in any browser. It would have been useful in the case where a user wanted to replace a preselected file with another one from his computer. The browser could display the name of the preselected file as a reminder to the user. The SIZE attribute in the INPUT element lets the page author specify the size of the specified element’s display. The RFC recommends that for FILE input types, the SIZE attribute should accept width and height values, separated by a comma. The width would work as it works for other types, but the height would allow Licensed to Tricia Fu 6 JAKARTA COMMONS ONLINE BOOKSHELF multiple files to be uploaded using just one INPUT element. The browser would show a multiline input field, and the end user would be able to select multiple files to be uploaded simultaneously. However, this option hasn’t been implemented by any of the browsers, and the page author must create several INPUT elements to upload multiple files. Only the width attribute is honored. The MAXLENGTH attribute specifies the maximum input allowed for a particular INPUT element. The RFC suggests that page designers may use this attribute for the FILE type of INPUT elements to restrict the size of the file that the user may upload. This feature would have been a great way to restrict end users to uploading file of allowable lengths. Again, however, this attribute hasn’t been implemented. These omissions from the browser implementations mean that the server must work harder to parse data that contains a file. Not only must the server parse the headers to determine if the file uploaded is of the right type, it must read all the data before it can determine whether the size of the uploaded file is more than an allowed practical limit. Page authors also must create multiple input elements to request multiple file uploads, and they must preserve state between transactions in the absence of the recognition of the VALUE attribute. FileUpload helps to overcome some of these limitations by removing the drudgery of repetitive file parsing. In the next section, we’ll look at the basics of this component. 2.2 FileUpload basics Before we delve into examples of FileUpload in action, it’s instructive to peek at the classes that make up this component. We’ve divided these classes into blocks to separate their functionality, which provides better organization and will help you understand the component. A good understanding of the classes and the interactions between them will enable you to use FileUpload effectively. FileUpload is a library of classes that allows you to parse file-upload requests seamlessly. The library is built around the semantics of dividing the work of parsing file-upload requests into separate blocks. A set of high-level classes lets you parse the upload request, a low-level API processes the multipart data streams, and interfaces and implementations represent uploaded files. A general high-level exception class represents errors in parsing or represents file uploads and files, respectively; and low-level exceptions represent errors at a more specific level. Figure 2.1 shows these blocks. As shown in the figure, FileUpload can be divided into the following blocks: Parsing and processing API—This is a mixture of a high-level API and a low-level API. An abstract class called FileUploadBase and two implementations (DiskFileUpload and FileUpload) are part of the high-level API. The low-level API is represented by the MultipartStream class, which processes data streams of multipart data. Representation API—This API represents the uploaded files and form elements. The uploaded files and form elements are represented by the interface FileItem and its default implementation, DefaultFileItem. The items are generated using a factory interface called the FileItemFactory and its default implementation, DefaultFileItemFactory. Exception and utility classes—The general catchall exception class is called FileUploadException. The FileUploadBase class defines three specific exception classes to deal with various errors related to input request, and MultipartStream defines two exception classes. There is one utility class called ThresholdingOutputStream, which is extended by the DeferredFileOutputStream class. The following sections examine these APIs in more detail. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 7 Figure 2.1 FileUpload API conceptual groups 2.2.1 Parsing and processing API The parsing and processing API is composed of classes that parse and disassemble an HTTP multipart request. The data elements in the request, or the different parts of the multipart request, are dissected into individual FileItem objects. Note that a FileItem object is used to represent form data such as text fields and radio buttons as well as any uploaded files. The classes that make up this API are as follows: FileUploadBase—This is the base class for all the processing that is done on input requests. It’s an abstract class that provides an implemented parseRequest(HttpServletRequest req) method that dissects a multipart request and returns a List of separate FileItems, which are created using a factory class. Subclasses call this method before they implement specific functionality on the parsed items in the list. This class relies on using the MultipartStream class to parse the multipart stream within the request. It provides a static utility method called Licensed to Tricia Fu 8 JAKARTA COMMONS ONLINE BOOKSHELF isMultipartRequest(HttpServletRequest req) that application developers can use to check whether a request contains multipart content before the request is parsed. FileUpload—This class is a generic implementation of FileUploadBase. It serves as a template for other, more specific, implementations. DiskFileUpload—This is the only specialized implementation of the FileUploadBase class provided in the FileUpload API. It handles disk-based file operations: In a nutshell, it uses a factory class internally to create FileItems geared toward saving files to disk. For most usage scenarios, this is the ideal class to use. Its internal factory class, DefaultFileItemFactory (which should have been called DiskFileItemFactory) stores files to disk in a location called the repository. You can specify the maximum size of uploaded files, the size threshold of the files before they’re written from memory to disk, and the repository location in the parseRequest(HttpServletRequest req, int sizeThreshold, long maxSize, String path) method. This method calls the superclass parseRequest method after setting the class-specific values. MultipartStream—This low-level class handles the processing of data streams as per rfc1867. The FileUploadBase class delegates the processing of the input stream from the request to this class, which follows the semantics of rfc1867 in parsing the data. You’ll probably never need to call this class directly and will use the parseRequest method in FileUploadBase or its implementations. The MultipartStream class expects the input data stream to follow the rules defined in rfc1867 for composing messages that contain uploaded files. Before this class can be used, the calling class needs to identify the boundary of the multipart message and pass this boundary as a byte array to the MultipartStream class. This boundary is used to identify the various parts of the message and to identify when the message ends. Internally, the MultipartStream class uses a buffer with a default size of 4KB to process a request. You can override this size when the class is constructed by using the MultipartStream(InputStream stream, byte[] boundary, int bufSize) constructor. Think of the MultipartStream class as the workhorse of the FileUpload API. It doesn’t make any decisions about the data it’s reading; it simply reads the data as per rfc1867. It lets the calling class, FileUploadBase, make decisions about the data it reads. The FileUploadBase class points the MultipartStream class to the input stream and the boundary within that stream and then uses the part information that MultipartStream reads to decide what to do with that particular part. Consequently, MultipartStream has methods that enable such blind reading of data from within the stream—for example, skipPreamble(), readBodyPart(), readHeaders(), and so on. 2.2.2 Representation API The parsing and processing API helps you process the information coming in. The representation API helps to represent this information in an easy way that makes it possible for you to do something with the data. What you decide to do is up to you. The representation API includes the following classes: FileItem—This interface represents both an uploaded file and a form item. To distinguish between a normal form item, like a text field or checkbox, and an actual uploaded file, the interface provides a method called isFormField() that returns a boolean true if the item is a normal form item. The MultipartStream class writes the body of a file item part to its OutputStream. This interface defines several convenience methods that let you manipulate the data received in a way that is most convenient. For example, you can get the data received in its entirety by calling get(), which returns the contents of the data as a byte array; or you can open an InputStream on the data by Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 9 calling the getInputStream() method. For normal form items, you can get the value as filled by the user by calling getString(). For items that are represented in an encoding other than the default system encoding, you can use the getString(String encoding) method, supplying the encoding you want. For items that return multiple values for a single form item, as in a checkbox or a multiple- select list, multiple items will be created. You’ll need to call getString multiple times for each value selected by the user in order to get all the selected values. When called on an item that’s an uploaded file, this method returns the contents of the file as a string representation. Another useful method, write(File location), lets you save the contents of a file item to the location specified by the parameter location. This works for both normal form fields and uploaded files. DefaultFileItem—A DefaultFileItem is the default implementation of the FileItem interface. Internally, it uses a DeferredFileOutputStream as the OutputStream to which data is written by the MultipartStream class. (DeferredFileOutputStream is described in the next section.) DefaultFileItem is a concrete implementation of the FileItem interface and is geared toward storing files to disk. It holds the file data in memory until a size threshold is reached, after which it streams the data to disk in a temporary location. The default threshold size is 10KB, and the default temporary location is the system default directory as returned by the System.getProperty("java.io.tmpdir") method. FileItemFactory—This is a factory interface to create FileItems with a single method called createItem(String fieldName, String contentType, boolean isFormField, String fileName). This method returns the newly created FileItem with the specified parameters. DefaultFileItemFactory—This is the default implementation of the FileItemFactory; it creates instances of DefaultFileItem. If you want to change the size threshold of the items this factory creates or change the location where the files are stored temporarily, you can do so by invoking the setSizeThreshold(int sizeThreshold) and setRepository(File repository) methods, respectively. You can retrieve the current factory being used by calling the getFileItemFactory() method on the FileUploadBase class. 2.2.3 Exceptions and utility classes The overall exception class for FileUpload is FileUploadException. It has three direct subclasses that are used to indicate problems in parsing input data streams: InvalidContentTypeException—This exception is thrown when the incoming request isn’t a multipart request. This typically happens when the page author has forgotten to identify the ENCTYPE for a form with file-upload requests as multipart/form-data or has forgotten to specify that the METHOD for form submit is POST. SizeLimitExceededException—This exception is thrown when the size of the incoming data request is larger than the allowed value for file upload. Licensed to Tricia Fu 10 JAKARTA COMMONS ONLINE BOOKSHELF Caution: Is SizeLimitExceededException thrown when an individual file’s size limit is exceeded, or when the cumulative size of the request is exceeded? According to the FileUpload 1.0 release notes, the expected behavior of calling the function setSizeMax(long size) on FileUploadBase is to set the individual size limit. However, this isn’t the case—calling this function sets the size limit for the entire upload rather than for individual items. UnknownSizeException—This exception is thrown when the incoming request doesn’t have a Content-Length header or the content length can’t be determined. The MultipartStream class throws some exceptions of its own. They are as follows: IllegalBoundaryException—This exception is thrown when a browser has set a boundary on a mixed multipart message whose length is different from the length of the boundary of the parent message. This error occurs when multiple files are uploaded in the same page but the boundary length separating this mixed content is different than the boundary length of the overall multipart boundary. These boundaries themselves will be different; however, their lengths must be same. MalformedStreamException—This is the catchall exception class for the MultipartStream class. It indicates one of several errors resulting from streams ending unexpectedly (perhaps because the network connection was lost), invalid stream headers (perhaps because the browser didn’t format the message according to rfc1867), or IO errors (due to an IOException). The FileUpload package includes one abstract utility class, ThresholdingOutputStream, and one implementation, DeferredFileOutputStream: ThresholdingOutputStream—This is an abstract OutputStream that triggers an event when a specified number of bytes have been written to it. This functionality is very useful for implementing actions based on certain events during the writing of uploaded file data to a FileItem’s output stream. For example, if you wanted to limit the number of bytes of a file held in memory, you would use this class to set a threshold beyond which the underlying stream would be changed from ByteArrayOutputStream to FileOutputStream. DeferredFileOutputStream—The DeferredFileOutputStream implementation extends ThresholdingOutputStream. Internally, it initially builds a ByteArrayOutputStream. This stream is used to write data until the maximum size allowed for holding data in memory is reached. The stream then automatically switches to using a FileOutputStream with a file location that is specified at the creation time of this stream. The DefaultFileItem class uses this stream internally. 2.3 Uploading files without using FileUpload Before you learn how to upload files using the FileUpload component, let’s see how to upload files without using it. Why? Well, because this exercise will help you better understand and appreciate FileUpload by exposing you to the difficulties encountered in processing uploaded files without the use of a standard handling and parsing mechanism. Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 11 Listing 2.1 is an HTML file that contains a file-upload request we’ll use for this example. Only the relevant parts are shown. Listing 2.1 simpleform.html: HTML file that uploads files without using FileUpload There is nothing remarkable about this HTML file. It contains a simple file upload request; on submit, it sends its contents as a multipart request to badfileupload.jsp, which is shown in listing 2.2. Listing 2.2 badfileupload.jsp: an alternate way of handling file uploads <%@ page import="java.io.BufferedReader, java.util.Vector, java.util.Iterator, java.io.FileOutputStream, java.io.File" %> <% BufferedReader in = request.getReader(); Vector readData = new Vector(); String lineRead; Notice method, action, and enctype File that user is uploading c Read contents of request Licensed to Tricia Fu 12 JAKARTA COMMONS ONLINE BOOKSHELF while((lineRead = in.readLine()) != null){ readData.add(lineRead); } String DELIMITER = "--"; String CRLF = "\r\n"; StringBuffer fileDataBuffer = new StringBuffer(); Iterator itr = readData.iterator(); while(itr.hasNext()){ String data = (String)itr.next(); if(data.startsWith(DELIMITER) && !data.endsWith(DELIMITER)){ data = (String)itr.next(); int index; if((index = data.indexOf("filename")) != -1){ String fullFileName = data.substring(index + 10, data.length() - 1); itr.next(); itr.next(); String fileData; while(!((fileData = (String)itr.next()).startsWith(DELIMITER))){ fileDataBuffer.append(fileData + CRLF); } FileOutputStream fos = new FileOutputStream( getServletContext().getRealPath("/") + new File(fullFileName).getName()); fos.write(fileDataBuffer.toString().getBytes()); fos.close(); } } } %> c We begin processing the request by reading the contents of the request through a BufferedReader and holding the contents read in a vector line by line. The assumption made here is that the data is encoded in the default encoding of the system on which this JSP is being run. d These two constants define what we’ll use in the later version of the code for a delimiter and a carriage return and line feed (CRLF). A delimiter is used to find the start of an encapsulation boundary and the end of the data stream. Remember that different parts in a multipart message are separated by a boundary that begins with a double hyphen. Also remember that the message ends with --boundary--. We’ve used this in the program not only to find the beginning of each part but also to find the end of the entire multipart message. e We define a variable fileDataBuffer to hold the contents of our file data. f Iterating through the data that we’ve read from the request line by line, we look for lines that start with the delimiter and don’t end with the delimiter (which would indicate the end of the request data). The d Constants for delimiter and CRLF e Define fileDataBuffer variable f Iterate through g Look for filename variable in header h Read data until delimiter i Save data to local file c Read contents of request Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 13 delimiter at the start indicates that the encapsulation boundary of a part is beginning. Note that there will be headers before the data after this encapsulation boundary. g Remembering that for a file-upload request, the Content-Disposition header contains the name of the file as the value of the variable filename, we look for this variable in the header. If it’s found, then our current part will contain the contents of a file. We extract the filename from this header and ignore the next two lines (these contain the Content-Type header and then a blank line before the data begins). h We keep reading the data in the buffer we defined earlier. The data ends when we encounter a line that contains the delimiter. Notice that we need to introduce the CRLF in the buffer because while reading it through the request, the BufferedReader strips it out with each invocation of the readLine method. i After we’ve read the data, it’s a simple matter to save the data to a local file using the filename we extracted earlier. When you run this program in a servlet container and upload files, the problems with this approach become painfully obvious. Not only will this program not work for binary files, it also won’t work when multiple file uploads are requested in the same page. Relying on parsing the headers for file information is also incomplete and doesn’t give you any confidence when you run this program. In the next section, we’ll start exploring the FileUpload component with a basic example. 2.4 Uploading files using FileUpload To begin using FileUpload, we’ll use the same basic HTML file that we used in the previous section (see listing 2.1). Listing 2.3 shows how the FileUpload component can be used to process this file. If you contrast this with the way we handled uploading files in the previous section, you’ll immediately see that FileUpload is a consistent, reliable, and much easier method of handling multipart data. Note: To run this example and the later examples, you must have the commons-fileupload-1.0.jar file in your servlet container’s WEB-INF/lib directory. Listing 2.3 basicfileupload.jsp: basic example of using FileUpload <%@ page import="org.apache.commons.fileupload.*, java.util.List, java.io.File, java.util.Iterator" %> <% if(!FileUpload.isMultipartContent(request)){ request.setAttribute("message", "Request is not multipart!"); request.getRequestDispatcher("msg.jsp").forward(request, response); return; } DiskFileUpload fileUpload = new DiskFileUpload(); List items = fileUpload.parseRequest(request); Iterator itr = items.iterator(); FileItem item; c Is request multipart? d Create instance of DiskFileUpload e Extract FileItems Licensed to Tricia Fu 14 JAKARTA COMMONS ONLINE BOOKSHELF while(itr.hasNext()){ item = (FileItem) itr.next(); if(item.isFormField()){ String fieldName = item.getFieldName(); if(fieldName.equals("username")) request.setAttribute("message", "Thank You " + item.getString()); } else { if(item.getSize() > 0){ File fullFile = new File(item.getName()); File savedFile = new File(getServletContext().getRealPath("/"), fullFile.getName()); item.write(savedFile); } } } request.getRequestDispatcher("msg.jsp").forward(request, response); %> This example demonstrates the simplest way to run FileUpload. Although it shows the way FileUpload is used to handle a single file, extending it to handle multiple files and other form elements is relatively easy; we’ll cover that in the next section. c Before we start processing the request, it’s advisable to check whether the request contains multipart data. This part of the code checks for a valid multipart request and stops any further processing if it isn’t found. If we don’t do this at this stage (before processing the actual request), FileUpload will throw an InvalidContentTypeException when the request is parsed. So, it’s always a good idea to do this before you do anything else. However, there may be situations where the developer processing the upload request is different than the person who designed the upload request page. The page author is typically a web designer who may not realize that to make a successful upload request, the ENCTYPE of the form must be multipart/form-data or that the METHOD of form submission should be POST, not GET. d To begin processing the request, we need to create an instance of DiskFileUpload. DiskFileUpload is a concrete implementation of the FileUploadBase class; it’s geared toward holding the uploaded files in memory until they’re swapped to disk when the threshold size is passed. To specify this threshold, you can do either of two things: After creating an instance of DiskFileUpload and before parsing the request, specifically set this threshold: fileUpload.setSizeThreshold(10000); // set as 10Kbytes You can similarly set the path where the files will be temporarily stored, called the repository, by calling fileUpload.setRepositoryPath("C:\\data\\temp"); After creating an instance of DiskFileUpload, and while calling the parseRequest method, pass the sizeThreshold in to the method. You must also pass in the maximum size allowed for the upload request and the temporary location of the repository: f Check if Item is a normal Form field g Retrieve field name and value h Handle uploaded files e Extract FileItems Licensed to Tricia Fu MODULE 2: UPLOADING FILES WITH FILEUPLOAD 15 parseRequest(request, // the original request 1000, // 1KB for the size threshold 1000000, // 1MB for the max upload request System.getProperty("user.home")); //repository location Recall that DiskFileUpload uses an internal factory to create instances of FileItems. This DefaultFileItemFactory is created when you call the constructor for DiskFileUpload. If you want to use another factory, you can change this default factory by calling the setFileItemFactory(yourFactory) method. The parseRequest method returns a List of the items processed from the request. This list contains normal form items (text fields, checkboxes, and so on) and the uploaded files. e This code iterates over the FileItems and extracts them one by one for further processing. f To easily distinguish between normal form items and uploaded files, we call the method isFormField() on each FileItem. It returns true for normal form fields and false otherwise. g Having identified this as a form field, we use the method getFieldName to retrieve the form field’s original name as set using the NAME attribute by the page designer/author. This value is used to distinguish between the various other form fields that may be present in this form and to treat them accordingly. Having found that this is the username field, we retrieve the value set for this field using the getString method and set this value in the request scope for the next page to use. If this were an uploaded file, getString would return the contents of the file as a String using the default encoding of the platform on which this program was running. If you wanted to use a specific encoding, you would use the getString(String encoding) method instead. h This section deals with items that have been identified as uploaded files. Before attempting to save the file to disk, the code checks whether the user uploaded a file by calling item.getSize() and making sure it returns a value greater than 0. This method can also be used if the user types the name of the file to upload in the text box instead of browsing for it, and makes a mistake. The browser will still send a multipart request, but the size of this file will be 0. The item.getName() method returns the full name of the file as specified by the filename in the Content-Disposition header. In some cases, this represents the full path on the user’s machine for this file. If you use the getName() method on an item that isn’t an uploaded file, it returns null. To save this file to disk, we create a File object that points to the location where we want to save the file and then call the item.write(File file) method. In this case, we save the file to the root of the current web application in which this JSP is running. What happens if you call the write method on an item that isn’t an uploaded file? A file is created that contains the value of the form item! Try it by creating a File object for the items that aren’t uploaded files. Here’s an example: item.write(new File(getServletContext().getRealPath("/"), item.getFieldName())); You’ll see a file created with the field name specified here. It contains the value of this field. The form that you’ve just seen handled contains a single text field and one input file. This is a very simple case and doesn’t cover many real-world scenarios. In the next section, we’ll show you how to handle a more complex input form that includes all sorts of form elements and type-specific file uploads. Licensed to Tricia Fu 16 JAKARTA COMMONS ONLINE BOOKSHELF 2.5 Handling a complex input form Figure 2.2 shows an input form that contains several form inputs. These include form elements that can have more than one value associated with them. In addition, more than one file-upload request is handled in one form, and the file-upload requests are specific: This form requires the user to submit an image file and a Word file. Figure 2.2 A complex input form that we’ll process with the FileUpload API As you can see, we have one text field for submitting data, one radio button set, one checkbox set, one list of multiple items, and two requests for specific files to upload. The partial HTML for this page is shown in listing 2.4. Listing 2.4 Partial HTML for the complex form