Document Review Strategies for Success Best Practices and Processes for Efficient and Defensible Document Review
3
Introduction
4
Preliminary Considerations for Document Review
5
Selection and Training of the Document Review Team
7
Maintaining Privilege
8
Quality Control Tactics
9
Best Practices for Efficient and Defensible Document Review
10
Case Study
11
Conclusion
Copyright Š 2010 Kroll Ontrack Inc. All Rights Reserved. Kroll Ontrack, Ontrack and other Kroll Ontrack brand and product names referred to herein are trademarks or registered trademarks of Kroll Ontrack Inc. and/or its parent company, Kroll Inc., in the United States and/or other countries.
2
Introduction
Corporations create and store exponential volumes of disparate data types every year and managing e-discovery in a cost-effective and defensible manner is increasingly difficult. In previous years, litigants would try to create efficiencies through employing cost-effective “point” solutions to reduce expenses without negatively impacting quality. However, over time it has become clear that the high-stakes and the complexity involved in e-discovery necessitates an integrated approach to data collection, processing, filtering, review and production. Instead of one-off processes and technologies, corporations are demanding end-to-end, consultative solutions that reduce costs, strengthen defensibility and increase the likelihood of a favorable result. Document review is the most costly and time consuming part of the electronic discovery process. The hallmark of an effective document review is the ability to review documents efficiently, protect privileged and confidential data and meet production requirements in a timely fashion. The expense and variability that accompanies document review can be abated through partnering with an expert that can offer consultative, custom, end-to-end solutions that address the specific needs of your case. More specifically, the risk that accompanies document review can be reduced with adequate preparation, robust technology and state-of-the art processes. This white paper details best practices for efficient and defensible document review, including: ■P reliminary Considerations election and Training of the ■S Document Review Team ■M aintaining Privilege uality Control Tactics ■Q and Processes
3
Preliminary Considerations for Document Review
Before diving into document review, there are a few preliminary considerations to address. These include scoping the project, establishing deadlines, choosing a review tool and developing a workflow process.
Employ a process that supports the needs of your case. Assessing Scope Scoping the needs of a particular document review involves noting project deadlines and understanding the volume and data composition to be reviewed. In addition, it is also important to consider the location, type and format of the data, as these considerations will impact your project timeline. During project scoping consider the budget and complexity of the review to determine whether it should be conducted in-house or externally. Smaller, less complicated projects may be easily managed in-house, while more complex projects will likely require the assistance of an outside document review service provider. Establishing Deadlines Once the project is scoped, deadlines should be established for each stage of the review. Be sure to incorporate some
4
wiggle room as the unexpected always happens – whether it is the addition of custodians or the location of additional data. Important factors to consider in setting deadlines include: the review scope, hiring and training of the review team, data collection, data processing, data review and production. Tip: Start with the court deadline and work backwards to the production deadline, taking into account other important milestones such as deposition dates and any internal deadlines. Choosing a Review Tool After the deadlines are established, you need to determine the appropriate technology to review the data. In deciding which review tool to use, consider the size of the data and deadlines for review, along with the features and functionality offered by the review tool. If your case involves audio files or multiple languages, be sure your tool will support those file types. Additionally, be sure to consider how the technology supports productions. For example, some tools will allow the user a broad level of autonomy in managing the look and timing of productions whereas other tools require a higher level of third party involvement. It is important to select a document review process that supports the specific needs of your case. As such, litigants should utilize technology to help automate a pre-determined workflow and quality control plan. For example, a case may call for document review by custodian, while in other matters
the review is based on keywords or file type. If your case involves multiple languages or scientific data, it may involve various levels of specialized review. Relying on technology to manage document distribution, workflow and quality control automates the process, takes the burden off of review administrators and strengthens the defensibility of your process.
Utilize technology to help automate a pre-determined workflow and quality control plan.
Selection and Training of the Document Review Team
Selection When staffing the review team, consider the unique needs of the document review project, such as reviewer qualifications, areas of expertise and availability. If the data set involves multiple languages, you must consider your reviewers’ level of fluency in each language. Additionally, you must consider whether your review team should be made up of firm associates, contract review attorneys or a mix of both. The review team’s composition may
vary depending on the available budget, organization structure, project needs and data type. You will also need to determine how many reviewers to include on the team. Once an estimate of the number of pages contained within your dataset and the deadline for project completion has been determined, the number of reviewers can be finalized. Keep in mind the more complex the case, the longer it may take to review each document. The chart below
estimates the number of reviewers that will be needed based on the number of pages to be reviewed in one hour. (Please note these numbers can fluctuate depending on the specifics of the review project.) Additionally, you will need to determine the location for the document review. This should include a review of the space and available equipment and technology, such as computers and Internet bandwidth.
Pages
Rate Per Hour in Pages
Total Review Hours Needed
Weeks
Typical Hours Per Week
Review Hours Per Week
Suggested Total Reviewers
1,500,000
200
7500
8
37
937.5
26
1,500,000
300
5000
8
37
625.0
17
1,500,000
400
3750
8
37
468.8
13
When selecting your review team, be sure to consider reviewer experience, areas of expertise and availability.
5
Selection and Training of the Document Review Team
Training Training the review team is an essential part of the document review process, particularly if you are using contract review staff who are not familiar with your case. Properly training the review team can help prevent breakdowns in the process and can result in a more accurate and efficient document review. The training process involves three main components: orientation, substantive training and technical training. Orientation Review team orientation is the first step in establishing a sense of teamwork amongst the reviewers. During orientation you should: ■S et expectations regarding office hours
■ Distribute log-in information ■C over relevant facility information Substantive Training Substantive training involves teaching the review team about the case and substantive laws at issue. It is helpful to prepare a training manual ahead of time that the review team can later call upon for reference. The elements of the training manual will vary case by case, but generally it is prudent to include the following:
The length of the training manual will vary based on the complexity of the issues and the experience of the review team. It may also be helpful to have a third party review the training manual to confirm that it is thorough and comprehensive.
■C opies of pleadings filed with the court
Technical Training It will be essential to provide your review team with technical training to correctly utilize the review tool and understand the workflow process. Depending on the issues of your case and the level of detail the reviewers are expected to provide, you may want to consider additional training on how to manage the following items:
■ Review tool guidelines
■ Duplicate documents
■ Document coding guidelines
■ Family groups (or attachments)
■ An acronym list
■P rivilege log entries and redactions
■ A summary of the case
■ Review time sheet procedures ■D iscuss the process for asking questions and escalating problems
6
■ A privilege name list
Maintaining Privilege
Privilege raises much concern for attorneys during document review because it requires high attention to detail and holds great potential for mistakes. Careful planning and monitoring can significantly reduce time, costs and the likelihood of inadvertent production of privileged documents. Privilege Options Casting a Wide Net One option for privilege review is to cast a wide net, which involves instructing the review team to mark documents as privilged if they contain attorney names, law firm names or certain keywords, such as “privileged” and “confidential.” The category may be further defined with more strict instructions for the second pass privilege review team, consisting of members of the legal team or select reviewers with superior knowledge of the review and privilege issues during the initial pass. A drawback of casting a wide net is that it requires litigants to set aside significant time at the end of the project to narrow the set of privileged documents to those that are truly privileged. However, by taking an overly-inclusive approach, there is a decreased chance of missing privileged documents. An additional advantage is that it allows for refinement of the list of privileged parties, issues and documents as the case develops. Extensive Privilege Training The next option is to thoroughly train the review team on privilege, providing them with all the necessary information to make privilege determinations themselves. Attorneys can have
vastly different opinions as to what constitutes a privileged document and therefore privilege must be clearly defined from the outset. As compared with the wide net approach, this method requires less time at the end to winnow down the volume of documents. However, this method is more static and does not allow for refinement of the early decisions regarding privilege. There is also the possibility that privileged documents may be missed by the review team due to a lack of information or unclear guidelines. Hybrid Approach Another option is a hybrid approach – something short of classifying everything as privileged, but less than conducting a full privilege review. With this approach litigants can capitalize on the advantages and minimize the potential drawbacks of each option. For example, a hybrid approach may involve training the review team on coding for privilege, while asking them to mark questionable documents privileged. Privilege Training Once the review approach is selected, training materials must be created in order to properly train the review team. First, create a list of privileged people, entities and topics. The list of privileged people should include all attorneys, assistants and consulting experts, as well as e-mail addresses. The entities list should include relevant law firm(s), consulting firms and any other involved parties. The topics list should include any subject that is privileged. This can include external reports that have been compiled in anticipation of litigation. Be sure to create these
lists in a platform that is easy to search and understand. To develop the privilege training manual, assess the case data in the database. Spell out your chosen approach and provide specific examples of what should and should not be deemed privileged. For example, in a case involving the unlawful dumping of hazardous waste: ■P rivileged: E-mail exchange between the plaintiff and Joe Smith, insurance expert, discussing soil samples. ■N on-privileged: E-mail exchange between the plaintiff and Joe Smith regarding the football game. Where appropriate, include actual examples of documents with the details as noted above. It is also helpful to flag certain custodians that may have more privileged information than others. Conduct End Searches At the end of the review, before the review team is released and before the production sets are created, be sure to conduct clean-up searches for privileged documents that may have been missed. This will ensure the production set includes only the data you want to produce. One suitable method is to search for privileged terms. By creating search sets using the privileged terms and all the documents not categorized as privileged as your criteria, you are likely to find documents with the highest probability of mistaken categorizations. These documents should be sent back to the review team for further review.
7
Quality Control Tactics
Quality control (QC) is a critical aspect of all document review projects. With potentially millions or billions of pages of data being reviewed, mistakes are inevitable. With QC, changes in the review criteria, user miss-clicks and errors in coding judgment can be rectified before the documents are produced. Further, a defensible QC plan can demonstrate to opposing counsel and the judge that reasonable steps were taken to ensure a quality work product, which may help avoid a ruling of a waiver of privilege arising from the inadvertent production of privileged documents. A form of QC should start at the beginning of the project when the review tool is selected. It is critical to utilize a review tool that automates and incorporates sound quality control measures into the review. The reviewers’ work should also be checked by other supervisors to determine if the review team training was effective. As the review progresses, the work should be periodically evaluated to minimize the risk of repeated mistakes. Quality Control – When to Start The goal of QC is to catch coding mistakes, ensure coding consistency and spot database issues. Therefore, it is most effective to begin QC after ironing out gray areas and resolving outstanding issues. In an ideal situation, all first level review would be completed before beginning QC. Under this scenario, there would be no data surprises, since the team has already reviewed all the data. Unfortunately, this ideal situation
8
is not usually practical. For longer projects it may take several weeks or months before the project is stable enough to make QC productive.
"responsive" and "unresponsive"). In addition to technology, other "human" quality-control measures should be implemented based on the needs of your case, such as:
Selecting and Training the Quality Control Team Selection and training of the QC team is important to the overall quality of the QC process. The QC team could either be a subset of the main review team, comprised of reviewers who have demonstrated superior knowledge and coding skill on the project, or could include members of the legal team.
■1 00% QC of particular document categories ■ QC every tenth document ■Q C based on search terms, privilege names or key phrases C a certain percentage of ■Q documents ■ Random statistical sampling
Once selected, you must adequately train the QC team. Training should begin with a summary of the issues that arose during the first level review and changes that were made to the review criteria. Training should also define gray areas to the extent possible as well as the scope of privilege and should emphasize accuracy, consistency, communication and teamwork. Last, identify what coding errors should be corrected and determine when to alert the team lead if excessive errors occur in either a particular reviewer’s work or with a particular issue. If possible, it may be helpful for the QC team to work in close proximity to each other and open communication should be encouraged. The Process The process of document review should support and facilitate thorough quality control measures. Technology should help confirm that each document was reviewed and that documents were not coded inconsistently (e.g., both
■ Any combination of the above End Search and Clean Up As the last step in the QC process, final “clean up” searches should be conducted to spot any miscoded documents or miss-clicks. Searches must be tailored to the specific needs of your case, but can include: contradictory coding, potentially privileged terms and terminology typically associated with “junk” documents. The review team should be consulted to determine which additional searches may be helpful, based on what was learned through the entire review process.
Your document review process and technology should ensure that reliable quality control measures are in place.
Best Practices for Efficient & Defensible Document Review
Document review can be costly and difficult without proper planning and preparation. Planning ahead will save your organization time and money, and will decrease the risk of grave mistakes that place the case in jeopardy. The following are best practices for managing efficient and effective document review projects. Select the Right Partner and Tools Selecting the right third-party expert and technological tools can alleviate many concerns or potential issues. Seasoned document review service providers will appoint a document review manager to avoid inadequate productions, missed deadlines and higher costs. Having a leader – someone who continuously watches deadlines, work product and process – will prevent document review shortcomings. Additionally, a third-party expert will supply licensed, trained attorney reviewers who have significant document review experience. Special skill requests, such as fluency in foreign languages or experience in patent law, should be easily accommodated. Adequate facilities, including workstations and computers that are sufficient to handle even the largest reviews should also be readily available. In addition to choosing the right third-party expert, choosing the right document review tool is essential to the project’s success. The best document review tools automate the process, relieving the review administrators from distributing and collecting
documents, as well as running on-going quality control searches. In addition, the technology must be able to manage disparate types of data including foreign language documents and scientific data. Additionally, if your case involves audio files, be sure your tool can support those files as well. Furthermore, consider whether your legal team is interested in using the review tool for additional functionality. For example, some review tools provide data analytics and the ability to produce privilege logs. A final and important consideration is the tool’s compatibility with other software you may use in connection with your case. Effective Training As discussed previously, proper training of reviewers will lead to a more efficient and effective document review. Orientation is an essential part of the training process, and thorough substantive training on the case by the legal team will enable reviewers to effectively code documents to your expectations. The more thorough the training of the document review team, the more accurate the result. Document the Review Process In order to strengthen the defensibility of your review, it is important to document all the steps taken and decisions made. Failure to do so may result in an inability to determine them down the road. You may want to create a document review manual for documentation purposes.
9
Case Study: Leveraging One Expert Provider to Achieve Efficiency and Cost-Savings
Recently, a corporate client came to Kroll Ontrack as a third party to a lawsuit arising out of a multiparty contract dispute. As a growing and successful business, it did not have the time or resources to collect, process or review all of the materials in-house. Therefore, it wanted to outsource as much of the electronic discovery work as possible. Additionally, it sought a partner that was experienced in providing testimony in the event the document search or review protocol was called into question. Kroll Ontrack partnered with this client and provided a full array of services from early case assessment and consulting, to document review and after-production support – and everything in between. In five weeks from the start of the review project, Kroll Ontrack’s document
review team of experienced, licensed attorneys reviewed 405,833 pages of materials. The team conducted first pass review, second pass review and privilege log work. The documents reviewed returned a 12.2% privilege rate, a 15.5% responsive rate and a 34.7% non-responsive rate (leaving a 37.6% “other” rate). The Kroll Ontrack team reviewed approximately 99.85 documents per hour, which translated into approximately 370 pages per hour. This review rate allowed the client to accomplish its document review goals on-time and well-under budget.
99. Do 85 c /Hr
10
Conclusion
The most successful document reviews are those that are wellplanned, employ a team of properly trained reviewers and rely on the best technology. By following the tactics discussed earlier, litigation teams can reduce costs and strengthen the defensibility of document review. When faced with the challenge of rising costs and heightened judicial scrutiny, corporations choose experts that can provide end-toend solutions, from collection and filtering to document review and
Use Kroll Ontrack for Your Next Document Review Experience the power of having an expert on your side. Kroll Ontrack’s award-winning review tool, Ontrack® Inview™, and document review services can provide an invaluable advantage and mensurable results. Call 800 347 6105 or visit www.krollontrack.com/ ontrack-inview/ for more information.
production. This approach results in a defensible outcome, where price can be predicted and quality measured. 11
9023 Columbine Road Eden Prairie, MN 55347 800 347 6105 www.krollontrack.com Copyright Š 2010 Kroll Ontrack Inc. All Rights Reserved. All other brands and product names are trademarks or registered trademarks of their respective owners.