EMPIRICAL RESEARCH METHODS

Professor Vern R. Walker

Hofstra University School of Law
Hempstead, New York 11549   USA

EMPIRICAL RESEARCH METHODS OF THE LLT LAB

The LLT Research Laboratory is empirical in its orientation, and its logic-based methodology makes it possible to scale up resources, take a team approach to research, and conduct comparative research. The paradigm for the laboratory is the science research laboratory found in both the U.S. and Europe. The following lists, in a preliminary way, various components that are fundamental to an empirical method of this type.

A. Study Samples. Research projects in particular legal areas are based on identified samples of legal documents. Both target areas and samples are selected after considering factors that include the following:

• Whether the legal area has substantial social importance, in which increased accuracy of decision-making is desirable (e.g., various aspects of health law).
  
• Whether the legal area contains sufficiently complex rules, and decisions with documented evidence assessment and/or policy-based reasoning; examples include compensation for vaccine-related injuries and medical malpractice (in the U.S., documented evidence assessment and policy-based reasoning can be found in medical malpractice decisions under the Federal Tort Claims Act).
  
• Whether the legal area would benefit from increased efficiency in evidence production or decision-making (e.g., where there is a reasonably high volume of decisions, with multiple decision-makers, and limited time or other resources for decision-making).
  
• Whether the documents in the study sample are sufficiently representative of reasoning in the legal area with respect to particular issues of interest.
  
• Whether the study sample is large enough to create adequate sub-samples for (1) calibrating (developing) the reasoning models and (2) validating (testing) those models.
  

B. Verifiable Measurements. Research projects in particular legal areas have the descriptive goal of accurately modeling the reasoning found in the legal documents of the sample. Such modeling (as with any scientific measurement) must exhibit both reliability and validity. The methods required include the following:

• Methods for locating and identifying particular types of reasoning patterns in the legal documents, using natural language cues (e.g., rule formulation, evidence assessment, and policy-based reasoning).
  
• Methods for modeling the internal structure of various types of reasoning (e.g., the default-logic framework utilizes three-valued propositions and logical connectives to model legal rules, and seven-valued assertions and plausibility connectives to model evidence assessment).
  

These methods should be capable of achieving inter-researcher reliability (that is, multiple researchers should model the same natural-language reasoning in the same way). They should also be capable of validation for accuracy (e.g., through accurate prediction of outcomes in other cases, or through corroboration by actual decision-makers).

C. Schemas of Recurring Patterns. Research projects have the normative goal of critiquing past decisions and providing guidance for future decision-making. Schemas for correct reasoning are based on the rules and policies found in the particular legal area, as well as on past practice in actual decision-making in that area. Developing such schemas takes into account:

• An accurate description of a representative sample of legal decisions in the particular substantive area (the result of accurately modeling a representative sample of decisions, as discussed above).
  
• Research into analogs in logic, statistical theory, scientific method, etc. (e.g., best practices employed in scientific risk assessment).
  
• A theory of uncertainty about the potential sources of error in drawing inferences from incomplete and uncertain evidence.
  
• The residual major uncertainties, as well as the major legal policies that might make those uncertainties acceptable.
  

D. Systematic Improvement of the Methods. Research projects have the goal not only of describing and critiquing the reasoning in the particular legal area, but also of improving the empirical methodology of the research laboratory. Scientific research is characterized by the systematic effort to monitor and improve the use of scientific method. There should be methods in place for achieving the following:

• Documenting both the problems and the useful techniques discovered in the process of conducting individual research projects.
  
• Producing documentation and training materials for improving the product quality and efficiency of research teams generally.