Georgia State University
The Symposium organizers urged me to present this Article from two perspectives: one as the former Chair of the Committee on Legal Education and Admission to the Bar of the Association of the Bar of the City of New York (“ABCNY”) and one as a clinical law professor concerned with improving the methods by which we teach law students and assess their level of competence. While there clearly has been considerable overlap in my meeting these two sets of responsibilities, my work on each has proceeded independently for the most part, until now.
In my capacity as Chair, I recently oversaw our committee’s publication of the Joint Committee Report: Public Service Alternative Bar Examination (“Joint Proposal”). In lieu of the existing bar exam, this controversial Joint Proposal sets forth multiple methods of assessing applicants for admission to the bar. The core of the Joint Proposal involves the applicants’ provision of public services in the New York State court system. Both on-site supervisors and outside academic-type graders would evaluate the services for bar admission purposes. The Joint Proposal also includes additional methods of assessment such as the use of simulated lawyering encounters (for example, mock interviews of clients or witnesses or counseling sessions with clients). Outside clinical academics would evaluate these simulations.
In my law teacher role, I have been developing the concept of standardized clients as a more elaborate and more specific technique for assessing law student performance through simulation. I based this method of evaluating lawyering performance on a medical education model—standardized patients—that uses lay persons to role-play patients and then to provide written evaluations of the students’ interaction with the patient. The written feedback is on structured forms drafted by medical professors. Adapted for law schools, the actors portray clients and witnesses, and the law professors prepare the evaluation checklists. My primary goal for developing this model for law student use has been curricular improvement and not licensure revisions. Nevertheless, there is a logical and practical nexus between education and licensing. This was dramatically demonstrated when the medical profession recently incorporated standardized patient evaluations into the medical licensing exam. The use of this simulation technique both in the medical school curriculum and now in the medical licensing exam has encouraged me to connect my work on these legal profession projects for this Symposium.
My bar association committee work began as a follow-up to a report that exhaustively and critically looked at the bar admission process in New York. The goal of the current committee’s work was to move beyond a critique of what exists and even beyond generalized recommendations for improvement. The result is a proposed alternative screening process that would be implemented in a small pilot project. It would establish a fair and valid licensing procedure to assess a bar applicant’s qualifications and better reflect the array of skills and knowledge needed to be a competent lawyer. Whether it ultimately would supplement or replace the existing bar exam was among the many questions purposefully left unanswered in the Joint Proposal for a Public Service Alternative Bar Examination (“PSABE”). The Joint Proposal calls for very careful scrutinizing of the results of the pilot. That analysis would serve to guide decisions from that point.
While the comparison with the medical profession may be logical, there is no direct empirical evidence to support the law profession side of the equation. As this Article describes below, medical educators have been working with standardized patients for more than three decades and have accumulated an extraordinary amount of data that validates their use of this method. Much work remains before legal educators can realistically talk about using a similar technique as a component of law school grades, let alone as part of a high stakes test such as the bar exam. Thus, while not an immediate option, it is possible that at some future date the use of standardized clients might enhance the efficacy of a PSABE. None of this appropriate skepticism, however, should preclude serious consideration of the Joint Proposal for a PSABE, irrespective of the possible future use of standardized clients. Nor should this caution deter legal educators from considering the standardized client as a future component of the public service bar exam or of any other innovative bar admission proposal. Some people may feel that it is too much of a stretch to initiate discussion about changing bar exams simply on the basis of the experience of medical educators. I remain encouraged, however, by John Sexton’s call for us to “[think] outside the box” regarding the training of lawyers.
Part I describes the Joint Proposal for a PSABE, its genesis, its political development, and key unanswered questions. Part II discusses in some detail my recent work in further developing the standardized client in a law school curricular context. Part III discusses the similarities between some of the tasks of doctors and of lawyers. Then, building on the experience of medical educators, this Article sets forth proposed empirical analyses for the use of standardized clients. The legal profession might do well, once again, to follow the lead of the medical profession in better serving the public’s continuing need for competent and caring professionals. Stating it differently: Is it reasonable to think ahead to a time when the standardized client will be an integral part of the bar exam?
I. The Public Service Alternative Bar Exam
A. The Genesis of the PSABE Proposal
The Joint Proposal is the product of an intensive joint effort of two independent bar association committees from two similarly independent and highly respected bar associations. From the start, these collaborative energies were quite purposeful. The Chair of the New York State Bar Association (“NYSBA”) Committee, Anthony Davis, and I had worked together on a comprehensive critique of the New York bar admission process a decade earlier. A third central figure was Dean Kristin Booth Glen, whose views on the deleterious effects of the bar exam and the need for drastic changes were reasonably well known in New York. Recognizing that a radical departure regarding bar admission was appropriate and necessary, we concluded that a potential combined effort of these two committees (and, indirectly, the two bar associations) would be the most effective way to engender serious consideration by those in a position to actually make changes. While there has been very little official response to the Joint Proposal, there has been a considerable amount of public reaction and continued interest in the ideas reflected in the PSABE Joint Proposal. Indeed, this Symposium is just such an expression of that interest.
However laudatory or appropriate that collaborative political goal, it was difficult to implement. The two relatively large committees consisted of strong, experienced, and independent‑minded members representing a diverse group of practitioners, judges, and academics, as well as several very recent law graduates. While there was an early consensus in favor of trying to formulate a new and distinct approach to bar admission requirements, there was clearly no consensus as to what that approach might be. The committees recognized that the profession and the public wanted (or needed) some kind of screening or credentialing process. The committees also widely supported the notion that anything that it might propose would have to be attainable; it would have to pass the “laugh test” for officials to take it seriously. This latter point, for example, was the basis of making the Joint Proposal a very small pilot program (so that the bar could effectuate the program without any immediate revolutionary change in the legal profession). We circulated an earlier draft of Dean Glen’s public service proposal, and it became the vehicle for much of our discussion. Ultimately, it was the basis upon which we shaped the numerous compromises that the final Joint Proposal reflects.
B. The Key Provisions of the PSABE
The underlying premise of the PSABE is that the current bar admission requirements do not adequately address the full array of skills necessary for competent lawyering and that the almost exclusive reliance on the current bar exam is an expensive and wasteful rite of passage, which also presents an insuperable barrier to admission for a significant number of candidates. The bar also fails to acknowledge the different categories of intelligence required to be a competent lawyer that a written exam cannot test. At the same time, the committees accepted the reality that neither a more appropriate bar exam nor an increase in clinical education would fully prepare a successful applicant to practice law. The goal was to reform the bar exam to reflect lawyering competencies better, which might, in turn, have a positive influence on shaping law school curriculums. The more modest aim of the PSABE was to reduce the gap between what the bar tests and what it requires for minimally competent law practice.
The first pilot group would consist of 50 randomly selected applicants (chosen by application and lottery) for admission to the bar. The pilot program would require applicants to provide three months of public service in the New York state court system. That service could include drafting opinions for judges, conferencing cases (to set discovery schedules or to conduct settlement discussions), assisting unrepresented litigants, or mediating cases. On-site supervisors and outside evaluators would evaluate the work of applicants. The pilot would also require applicants to complete satisfactorily a performance-type written exam, the Multistate Professional Responsibility Exam (“MPRE”), and one or more videotaped simulation exercises. To be eligible to participate in the PSABE, an applicant would have to satisfy a relatively easy law school course requirement—a practical skills course and a New York State Civil Procedure course. After a short orientation period, the participating applicants would begin their placement in the courts. Toward the end of the three‑month placement period, the applicants would complete the written performance test and the videotaped simulation exercises. The applicants also would have to perform, pro bono, 150 hours of similar services in the New York courts during the first 18 months after admission.
Aside from the costs and administrative aspects of this alternative bar exam, ensuring fairness concerned the two committees (like all boards of law examiners). Any bar admission screening process must be a valid and reliable method of testing applicants for that knowledge and those skills that are relevant to being a minimally competent lawyer. Validity requires that the method of testing measure “what it purports to measure.” Direct observation, by qualified evaluators, of the performance of the task being assessed easily satisfies that requirement. The evaluation also needs to be reliable; it must be “[consistent] from one . . . test‑taker to another.” In light of the multiple reviewers that the PSABE anticipates using, satisfying reliability standards will be a challenge.
C. The Key Unanswered Questions
There are really two key unanswered questions. One is who will finance the pilot (let alone a larger application of the PSABE). For now, that remains unanswered. The second is the fairness question. How can the bar administer the PSABE in a fair and valid manner? For example, one of the methods of assessment in the PSABE calls on outside evaluators to grade an applicant’s performance either by participating in a simulated interviewing or counseling exercise or by observing an actual interview by an applicant (for example, interviewing a pro se litigant). Putting aside the expense issue, the expectation is that clinical law professors would perform these evaluations. Using numerous professors might lead to variations in grading criteria. Evaluating actual interviews of real litigants would likely result in substantial variation in the circumstances encountered by the applicant. It could be unfair to evaluate one applicant interviewing a mentally-disturbed litigant while a second clinical professor evaluates another applicant who is assisting an intelligent, calm litigant. This parallels earlier versions of the medical licensing exam. Doctors would evaluate interns seeing real patients regardless of the nature of the patients’ illnesses. The variation, inconsistencies, and lack of uniformity led to the conclusion that it was not a fair method. That conclusion, in turn, has led, in part, to the use of standardized patients, which is a much more reliable, consistent, and fair method. In law, these aspects of potential unfairness, as noted above, lead back to the standardized client as a possible solution. If further research establishes that the standardized client method is valid (like the standardized patient in medicine), this method of evaluation could replace the law professor assessments. It would be much less expensive and probably more consistent, making it more reliable and fair. Indeed, that proposition is the fundamental thesis of this Article.
Another means of ensuring fairness that the committees addressed in the Joint Proposal is the use of multiple methods of evaluation. In addition to an assessment of a simulated lawyering activity, the Joint Proposal calls for the successful completion of an extensive written performance test and the MPRE. It also requires a passing score from an evaluator who observes actual lawyering work (for example, conferencing or mediating cases). By using more than one or two methods, the proposal limits the impact of any single method. Each different score can act as a check on over-reliance on any single indicia of competence.
The PSABE Joint Proposal is a work in progress. The goal was to build on the extensive progress over the past 30 years in developing tools to assess lawyering performance in clinical education. Working with small numbers in a pilot, the objective is to closely examine how well this alternative works. The possible inclusion of standardized clients in a pilot simply adds to the potential value of careful empirical analysis of the results of a pilot. Simultaneous empirical studies of the use of standardized clients, as described below, could be reinforcing. The additional public service benefit is simply another potential plus from using the PSABE. Both the PSABE and the use of standardized clients reflect efforts to consider objectively new ways to make the bar admission screening process more reflective of what practicing lawyers actually do.
II. The Continued Development of the Standardized Client at New York Law School
A. Context for Refining the Standardized Client
From the first day of law school, legal educators communicate to students in many different ways about what is important and what is not so important in the legal education that is to follow, and in turn, what is important to being a good lawyer. In almost every law school in the country, the first-year curriculum is essentially the same. It typically consists of eight to ten required courses, including a writing and research course and perhaps a course like the Lawyering course discussed later in this Article. The other courses are typically Contracts, Torts, Civil Procedure, Property, Criminal Law, and perhaps Ethics or Constitutional Law. The vast majority of teachers of those courses use standard casebooks and some variation of the Socratic method. All teach doctrinal reasoning, legal reasoning, and legal analysis. While some first-year teachers use the problem method and a few even use simulation in one way or another, both are exceptions to the norm. None of this is news to anyone who has been in legal academia for any period of time.
Given these norms, students understandably view any efforts to teach basic interpersonal lawyering skills as diversions from their main task of learning doctrine and legal reasoning and analysis. That is what is important to success in the first year and to being a good lawyer; this is the message that we communicate to them. This is a distorted view of what competent lawyering entails. Nevertheless, for many decades, that has been and still is the reality of the first year of law school. The standardized client method presents an opportunity to make a substantial dent in the armor surrounding this long tradition of what happens in the first year of law school. This Article does not mean to overstate the potential significance of this technique, but it urges a closer look at the possibilities.
New York Law School has been working with the standardized client concept for more than five years. As already noted, it builds on medical education’s standardized patient that uses trained lay persons: first to portray patients who are then interviewed and examined by medical students and second to provide a written evaluation of the medical student’s performance. Our adaptation of this teaching method in law school is the standardized client, whose function is to portray a client or a witness and to provide each law student with a written critique on an evaluation form that the law professors’ school drafts.
Legal education has used simulation for some time in many different contexts. It has not used a structured, written, lay-person evaluation of a law student’s performance. Only the law professors grade the students. Unlike all other forms of graduate education, legal educators have generally not employed readers (either lawyers or law students) to grade exams or even quizzes. Thus, the notion of a non-lawyer participating in the process that determines the successful completion of law school would be an unusual, if not revolutionary, change in the American legal education. The typical person who has portrayed the standardized client and provided the written evaluation at New York Law School has not been a lawyer.
Medical educators addressed this issue by conducting repeated comprehensive empirical analyses of the use of the standardized patient over many years to validate this method of evaluation. As a result of these studies, medical educators have been using graded standardized patient exercises as part of the required medical school curriculum: first on a modest low-stakes level and, more recently, as a requirement for medical school graduation. As noted above, the medical profession has recently legitimized this method of gauging the capacity and competence of future physicians on a high-stakes level. Beginning in 2004, the standardized patient will be an integral part of the medical licensing exam.
Legal educators, on the other hand, have yet to do these kind of empirical evaluations of the use of the standardized client. Indeed, legal educators have not done any studies. The completion of these types of studies, therefore, is a major goal of our continuing work in this area. It is by no means a foregone conclusion that the legal profession will reach the same results as the medical profession regarding the validity of this method. There are numerous questions regarding the efficacy of this means of evaluating law students that remain to be answered. One major issue, for example, is how much training the standardized clients require and, in turn, how to monitor their performance.
B. The Lawyering Course
The central vehicle for New York Law School’s current use of the standardized client is the first-year required course—Lawyering. The goal of this two-credit course is to introduce students to fact analysis, interviewing, and counseling (basic clinical practice skills), while reinforcing basic legal reasoning and analysis skills. We also seek to use this contextualized approach to legal and factual analysis to teach humane and ethical lawyering. Thus, in addition to the use of the standardized client as a new and unusual method of providing individualized feedback to students, the Lawyering course represents an alteration of the nature of the traditional first-year law school curriculum, a daunting challenge in itself. We have sought to integrate the innovative methods of the Lawyering course with the traditional goal of teaching students how to analyze, to reason, and to articulate a synthesis of the law and facts. Putting it slightly differently, we have concluded that we can accommodate these changes more easily by not disturbing those traditional first-year objectives in any significant way. If the program can enhance those objectives, it would be a welcomed side effect. Indeed, we have sought to communicate that legal reasoning and analysis is a critical and necessary component of competent lawyering but that it is not sufficient to achieve competency.
Our development of the Lawyering course began about ten years ago and has proceeded with those guidelines in mind. This section briefly describes the genesis of the Lawyering course as well as its current expanded version, and then it goes on to set forth much more ambitious, but realistic, goals: namely, to incorporate the use of standardized clients in all of the first‑year courses.
In very loose terms, Lawyering represented an effort to build on Professor Anthony Amsterdam’s groundbreaking first-year Lawyering course at New York University School of Law (“NYU”). We also sought to demonstrate the integral connection between (1) fundamental legal reasoning and analysis skills and (2) applied lawyering skills. A critical difference was that we would teach our course only in large class sessions (100 to 120 students), whereas NYU teaches its course in sections of 15 to 20. At the outset, our course had no individual feedback and very little opportunity for students to apply the law in a simulated context. The NYU course had significant individualized instruction from full-time Lawyering faculty and many opportunities for students to conduct simulated lawyering exercises. Both the NYU course and our course have undergone numerous changes in the last decade. While the differences in class size remain, we now offer substantial individualized feedback through the use of standardized clients.
We first adapted a breach of contract case file used at NYU to fit the needs of our large lecture course. Students worked on the case from the initial client interview through the filing of a summary judgment motion. When they finished, they had worked through the applicable substantive law, the pleadings, and the transcripts of depositions and interviews. We supplemented their written immersion into the case with videotaped depictions of various interviews and depositions in the case. At least once, we had the class “interview” someone role-playing a client or witness in front of the class. Each student individually conducted a witness interview in which another student role-played the witness for the exercise. The videotapes, the simulated witness interview, and the assigned readings on lawyering skills provided a base for large class discussion of the challenges confronted by a lawyer developing a case.
Our first significant innovation in the Lawyering course was to have each student interview a standardized “client” (actually a witness) who would provide written feedback to the student interviewer, using a checklist form that the professors had drafted. The final exam in the course required the student to synthesize new law and new facts in the same case file, to view a videotaped depiction of a lawyer performing one or more of the skills studied in the course, and to prepare a written critique of that performance.
The current version of the course now includes three standardized client exercises arising out of case files in three different areas of law: torts, contracts, and real property. Whereas the earlier course was in the first semester of the first year, this revised course takes place in the second semester. In the first case, the students conduct an initial client interview of a standardized client with a tort claim. The second case calls on the students to interview a standardized client who is a witness in a contract case. Finally, in the third case, the students must counsel a client in an adverse possession case. As we had done previously, we show several videotaped demonstrations of the same kinds of skills the students will perform. For example, in the first unit, which involves a claim based on the negligent infliction of emotional distress, the students view a demonstration initial interview of a client with a similar tort claim before they interview the client in their case. Likewise, in the second case, which involves a claim of undue influence, the students first observe an interview of another witness in the case before they interview their witness. Before they counsel their adverse possession client, students observe a tape of a counseling session with a client in a comparable situation. After the interview, the standardized client completes a detailed evaluation checklist assessing the student performance. While the students do not receive a grade on their performances, they could have their grade for the course reduced by as much as a third of a letter grade if they do not prepare and conduct the interview or counseling session appropriately.
One question arises immediately: Why use lay persons instead of professors to evaluate the law students’ performance? The answer is that the use of standardized clients seems to be a fair, reliable, and much more cost efficient method of providing individual feedback. Providing enough professors to give individual feedback to 500 first‑year students on three separate exercises is simply cost prohibitive, at least for most law schools. NYU is an exception. While using standardized clients is not free of charge, it appears to be a manageable option. Our proposal to establish a metropolitan training facility for all local law schools would lower costs even more.
As in the earlier version of the Lawyering course, the videotaped demonstrations, the standardized client exercises, and the substantive law and skills readings continue to provide the basis for classroom discussions of the development of case theories in each of the cases and of the challenges faced by the lawyers in the three cases. With respect to the standardized client feedback, the professors comment on their observation of a limited number of videotaped student exercises and also provide some statistical findings on the exercise. By compiling the results of the structured feedback forms, the professors have often been able to show that the problems and challenges faced by the students were typically experienced by many others. Conversely, the statistics could show how nearly all of the students were doing something right, at least from the vantage point of the standardized clients performing the evaluations.
Finally, as it had done previously, the final exam calls on the students to analyze and incorporate into their case theory new facts and law and then to provide a written critique of a videotaped depiction of a lawyer’s performance. In the current version of the course, each student receives feedback from a standardized client on three different exercises. Thus, last year we completed more than 1500 standardized client exercises. The students have the opportunity to view and to analyze critically numerous other videotaped lawyer performances. The final exam then presents students with the challenge of putting those experiences together to synthesize their skills and case theory readings with substantive law and with the challenge of developing facts into viable case plans. Finally, students critically analyze a lawyer’s legal reasoning and oral lawyering skills.
The following briefly describes the process that we have been using in drafting the structured feedback forms and in training and preparing the standardized clients. The objective is to have all of the persons role-playing the client or witness portray the individual involved in a standardized fashion. This means that we train each person to assume the same profile, to know the same facts, and to respond appropriately to student questions and techniques so that the experience of each student is as close as possible to that of all other students. Similarly, we want each standardized client to evaluate each of the student’s performances in a like manner. This is a critical training goal. Hence, the standardized client’s performance is standardized both as to role-playing and as to assessing. To do this, each of the four professors teaching the course conducts three two-hour training sessions with roughly ten to twelve potential standardized clients. In our case, they are nearly all actors. They need not be. The four professors spend approximately two-thirds of the training time on evaluation and grading issues.
Using the medical educators’ experience with standardized patients as a model, we have drafted a written form that facilitates the completion of a structured evaluation by the standardized client. It contains three parts: one addressing the substantive content of what the interviewing or counseling law student does, one addressing the communication skills of the law student, and one giving the standardized client the opportunity to include subjective comments. There is a need to raise the level of sophistication of these forms regarding the collection of data. The following section discusses this objective.
C. Evaluation of Our Use of the Standardized Clients
Our assessment of the success of this continuing experiment has taken many forms. Throughout, we have had the benefit of student feedback from our standard course evaluation forms. In the prior Lawyering class format, we also solicited specific written evaluations from the students of the standardized client exercises. We have also had the benefit of a small group of law professors who have worked together closely on this course. Several of us have sat in on one or another of our colleagues’ classes, and we have scrutinized the efficacy of these clinical teaching techniques in a large class setting. Collaboration among law professors to design and implement a uniformly taught course is itself an unusual experience among law professors. In this instance, it has been a wonderfully productive enterprise. Finally, there has been no shortage of ad hoc student responses as to all aspects of their work with the standardized clients.
With respect to student response, the view is nearly all positive. While we have not yet surveyed the students specifically on the current version of Lawyering (with three standardized client exercises), the standard course evaluations and the ad hoc comments have indicated quite clearly that the students have found the standardized client experiences very useful. Many students indicated that this was the first time they had the chance to talk to another person from the vantage point of a lawyer; they were appreciative. There have been many favorable observations about the value of viewing videotaped demonstrations of a particular lawyering function before attempting to perform the function themselves. One recurring comment, however, is that the students regret having put so much time into preparation for the exercises without getting a grade or some kind of credit for their work. Earlier comments suggested that the experience might be more effective if it were not in the first semester; as noted above, students now take the course in the second semester. Some second-career students questioned the need for these exercises because they were quite experienced in dealing with people at different levels. Two other recurring student comments have been: (1) standardized clients are not always consistent and (2) standardized clients do not always disclose information they possess.
As set forth below, our goal is to work with psychometricians to design surveys and questionnaires that will provide the framework for empirical analyses regarding the appropriate use of standardized clients. Only after we can substantiate, as solidly as possible, the reliability and validity of these assessment methods, can we seriously consider them as significant components in the law school grading regime.
D. Expansion of the Use of Standardized Clients
In our Lawyering course, we spend a good deal of time drawing connections between legal analysis and factual analysis (usually in the context of developing a case theory for a client). We also focus on the connections between a case theory and how the lawyer applies and utilizes that case theory in the context of performing a lawyering task such as interviewing a witness or counseling a client. This is the crucial means-ends analysis that is central to competent lawyering. It becomes clear quite quickly to the student (or at least that is our hope) that lawyers cannot effectively perform such a lawyering task unless they have their legal analyses clear in their minds. By placing them in role, the students begin to gain an understanding of their duties and tasks as a professional and of the combination of skills necessary to carry out those responsibilities that come with that role.
When our first-year students sit down to counsel their client about the likelihood of success in asserting an adverse possession defense to a claim for an order evicting him from the plaintiff’s property, those students must know the doctrinal property law nuances and must have synthesized a workable case theory for their client. They must then be able to explain their legal opinions in clear and non-condescending language; that is transmitting their expertise to their clients in plain English. Students cannot, and from everything I have seen do not, sit down until they feel they are capable of performing this task competently. A critical component in their preparation is their analysis of the law as applied to the facts of their client’s situation. Placing the student in the role of the lawyer working with the client to solve the client’s problem is the experiential learning that is critical to good lawyering. The student must work with the client to integrate law with all of the non-legal variables that affect the client’s decision.
If first-year students had to conduct similar standardized client exercises in their other first-year courses, it is likely that their view of the importance of interpersonal lawyering skills would change. This would be especially true if legal educators graded and included the exercises as a component in the overall course grade. If a metropolitan training center for all local law schools existed, law schools could incorporate the exercises into these other first‑year courses quite easily and with almost no intrusion into the other professors’ class plans. A Torts professor could select a key opinion that students will be reading in the traditional casebook and could ask the center to draft a standardized client counseling problem based on that case and an accompanying checklist for the standardized client to use to evaluate the performance of the student. That form would include both a substantive law component (for example, the explanation of the applicable law and a legal opinion on what the likely result will be in the case) as well as a communication skills component (that is, how well the student interacted with the client). The students would interview or counsel the standardized client in the week before the discussion of the key opinion in class. During that class, the Torts professor would conduct the usual discussion of the doctrinal lessons from that case. In teaching Civil Procedure and in observing other first-year classes, my experience has been that a simulation exercise prior to the relevant class discussion considerably enhances the level of student engagement and the quality of the doctrinal discussion. Contextualizing doctrinal learning is no longer unusual. “Simulations . . . enhance student motivation to learn.” Experiential learning can even “infuse[e] passion” into the doctrinal learning environment. All of this can take place without spending a moment of traditional first-year courses’ class time on the lawyering skills related to the student experiences with the standardized client. Professors could repeat the same thing in each of the other first-year classes.
The cumulative effect of integrating these exercises into traditional first‑year courses could be significant. First, were each exercise a component of the grade, students would understand the overall importance of mastering these oral lawyering skills. For better or worse, grades remain a primary motivator for students. The relevance of the ability to interview and counsel would compare to the importance of basic legal reasoning and analysis skills that are otherwise focused on and reinforced in all of the traditional first-year courses. Rather than communicating to the students that these lawyering skills are relatively unimportant by relegating them to a couple of credits in a thirty-credit first year, the students would be receiving the clear message that these skills are integral to all of what happens in the first year. Second, if there were a course like our Lawyering course (perhaps as part of the typical first‑year writing course), various skills themes could provide the larger context in which the students could place their various standardized client experiences. Third, the potential impact of this required first-year experiential learning on the content and sophistication of upper-class simulation courses and clinics is enormous. At New York Law School, the students’ completion of three exercises in the Lawyering course has called for re-examination of our upper-class simulation and clinical courses. If the standardized client experiences were a part of other first-year courses, the second- and third-year classes would be substantially different.
How might this type of system be implemented? Very briefly, my proposal is based on the medical education model established at the Mount Sinai Medical School Morchand Center in New York City. A metropolitan training center would have ten to twelve interview rooms, and an educational director, who would hire and train the standardized clients to role-play and evaluate the students and would administer the program. That director would work with the law professors in the various subject matters to draft the problems and the evaluation forms and to schedule the exercises. In New York City, for example, the law schools in the metropolitan area could participate and could send their students to conduct sessions at the center in a manner similar to sending students to an externship field placement. The significant cost effectiveness of this system (based on the economy of scale) would benefit all of the cooperating law schools. The more often a lay person could reprise a particular standardized client role, the less cost to the center. This is, in fact, how the Morchand Center operates in New York City for all of the local medical schools. Any area with a number of local law schools could duplicate this process effectively.
An expansion of the use of standardized clients is dependent on the establishment of this type of center. We are in the process of seeking grants to do just that. Once established, this center could be self-sustaining, with the cooperating law schools sharing the costs. In addition to the intrinsic educational value of the exercises described above, a metropolitan center in New York or elsewhere would provide the vehicle for the completion of the kind of empirical studies that educators in medical schools have been conducting for decades.
III. The Gap Between Medical Licensors and Bar Examiners
Can the legal profession make changes in the bar exam based on the experience of medical educators? With respect to the potential analogous use of standardized clients on a bar exam, is the empirical data gap so wide as to suggest that the analogy is pure fantasy? Based on the foregoing, it would be easy to say yes and to put off any further consideration of changes to the bar exam until legal educators have conducted more experiments on the use of standardized clients. However, there is enough information to keep this discussion alive while continuing to pursue further empirical study. In the meantime, as suggested above, the vast experience with standardized patients would seem to provide enough support for the use of low-stakes educational grading in the form of student evaluations completed by standardized clients.
A. The Similarity Between Lawyer and Physician Communication Skills
There are striking similarities between the communication challenges facing a physician speaking with a patient and a lawyer talking to a client. As two of the leading clinical education law professors put it some time ago: “We have found [interviewing] analogies to medical diagnosis especially useful.” While the differences are also clear, a physician’s initial interview with a patient who seeks assistance for an ailment presents challenges quite similar to those facing a lawyer interviewing a client regarding the problem that brought the client to the lawyer’s office. In both cases, the professional must obtain enough information to make a diagnosis. Both professionals must relate to people who may be upset in varying degrees and who may be unable to communicate clearly. Beyond the task of obtaining facts, there is the challenge of applying the professional’s expertise to the specific facts of the case before them and then translating that expertise into clear, non‑condescending language so that the patient or client can digest the information and apply it in his or her decision making process. Once again, both the physician and the attorney confront this translation-communication challenge in remarkably similar circumstances. The process of deduction is the same for both doctor and lawyer. It is because of these similarities that I first became interested in medical educators’ work with standardized patients.
In an extensive and illuminating review of Donald Schön’s work, Professor Richard Neumann cataloged Schön’s insights and then suggested ways in which legal educators might act on those insights. Among other suggestions, Neumann urged legal educators to “[s]ubject our theories of action [regarding lawyering notions] to empirical testing.” Further, he suggested that legal educators “[d]ocument in convincing detail the ways in which legal education is far behind all the other professions in providing reflective practica.” Finally, after looking at architecture and medicine, Neumann concluded that legal educators ought to disseminate that documentation to the widest possible audience. Maybe then change will occur.
The medical establishment’s extensive research with standardized patients has enabled it to reach the point of including standardized patients as part of the licensing exam. That medical experience provides pragmatic and moral support for ongoing efforts by the legal profession to conduct similar research and for the use of standardized clients in low-stakes law school grading. However, that support is insufficient grounds upon which to base any decisions to use a standardized client for a high-stakes exam. As Neumann pointed out, more work is necessary.
B. Empirical Analyses of Standardized Client Use
The use of non-lawyers and non-teachers to evaluate the performance of law students is both counterintuitive and, as noted above, contrary to long-standing legal education traditions. This does not mean that change is impossible. Rather, it means that, like medical educators, legal educators have to construct studies carefully to determine whether legal educators can effectively and efficiently train lay persons to give valid evaluations.
An expansion of the use of standardized clients, as described above, could provide the data for this needed empirical analysis. There are, broadly, two directions in which further examination of the use of standardized clients may proceed. First, there are empirical assessments that seek to measure the reliability, validity, and efficacy of performance and evaluations by standardized clients. Second, there should be surveys assessing the degree of satisfaction of the students. We did this previously at New York Law School, and with minor modifications of the questionnaire that we used then, we intend to do this again. Legal educators should look for specific guidance from medical educators as to how and in what directions legal educators might proceed with respect to measuring the value of standardized clients. In addition, legal educators should focus on developing effective checklists—the questionnaires that the standardized clients complete in their evaluation of the law student performance. The goal here is to refine the formats as much as possible as to both the content and the methods of quantification for grading purposes. A related issue is setting the standard for a passing score.
It is worth noting again the astonishing difference between what medical educators have done in terms of the quality and quantity of empirical research on methods of assessing professional competence and what legal educators have done or, more specifically, not done. Exactly why this has been the case is an interesting issue, but it is beyond the scope of this Article. For decades, medical professors, often jointly with psychometricians, have examined in great detail which testing methods have worked. Using the pyramid of knowledge, they have looked at alternative means to test pure knowledge (at the bottom of the pyramid), comparing multiple choice questions, true-false, and single-best-answer methods. For problem-solving and ultimate-performance assessments involving judgment, these medical educators have compared computerized problem-solving exercises, as well as observation of performance with real patients, and of course, they have engaged in extensive analysis of the reliability and validity of the use of standardized patients. For example, while recognizing the value of standardized patients, one study has also clearly acknowledged that “[p]erformance on [a standardized patient evaluation] might not be the same as performance in real life.” Yet, it is the lack of consistency and, ultimately, reliability in evaluating medical students’ work with real patients (which is dependent on the happenstance of the illnesses of the patients who are available at the time of testing) that has been one of the driving forces for the use of standardized patients.
Another lesson from medical educators is the importance of objectively assessing the value of the use of standardized patients or, in our case, of standardized clients. While there are benefits, there are also costs. Refined analysis might very well show differences between medical education and legal education insofar as the value of this technique or by revealing significant limitations on legal educators’ potential use of standardized clients. A recent article, for example, goes to some lengths to point out serious limitations in the use of standardized patients (as opposed to real patients) and the importance of using a variety of assessment methods. Similarly, in another close analysis of the value of standardized patients, the author reached the clear conclusion that, while standardized patient encounters are excellent for assessing the ability to interact with a patient, medical educators should not use them to test clinical reasoning, which they could test much more effectively with written or computer-based exams.
Thus, looking to the vast experience of medical education contemporaries for guidance on empirical research gives legal educators a very wide range of options to pursue. For example, with the cooperation of a first-year Contracts professor, we might ask for volunteers to participate in a standardized client experience. Assume that 80 students volunteer. We would ask 40 to participate in counseling a standardized client in a case that parallels an opinion that is the subject of class discussion. At the end of the semester (or perhaps at the conclusion of the unit in which the opinion is read), all 80 students would conduct a counseling session with a standardized client in a different case involving the same basic contract concepts. We could then compare the scores and evaluations of the two groups to determine if the results differed. The results may not conclusively establish a link between the additional counseling experience and the students’ ease in applying substantive knowledge because other factors might have an impact, such as the greater immersion into the subject matter by the experimental group as opposed to the control group. However, the results would be useful in determining whether there was an impact on interpersonal communication skills.
One clear method of assessing the use of standardized clients is to compare the scores given by the standardized clients with those of professors who observe and evaluate videotapes of the same student exercises. I did this in an initial experiment involving a much smaller class of 45 students. The results were not, however, statistically significant. With respect to the Lawyering course, this would necessarily entail much larger numbers, perhaps a random sampling of 50 to 100 students on each of three or more standardized client exercises. This would allow inter-rating for multiple exercises and would approach what a typical Objective Structured Clinical Exam, which usually includes eight to ten standardized patient exercises, does for medical students. Some refer to this as comparing the standardized patient to a “gold standard”—the professorial evaluation. In the case of the Lawyering course, it would require the cooperation of several Lawyering professors viewing many videotapes and scoring the students observed.
On a less scientific level, we might ask the students in a Criminal Law course to conduct a counseling session with a standardized client who is a criminal defendant to whom the students must give very bad news about the prospects of success at trial. A pre-counseling survey might ask the students: (1) What are your primary concerns? and (2) What is your level of comfort in giving such counseling? After the exercise, we might ask all of the participants the same questions as well as a question asking them to rate the educational value of the exercise on a scale of one to five.
An interesting medical school project parallels this concept. Recognizing that first-year medical students were quite uncertain about their clinical skills, the Northwestern University faculty designed and implemented a program using five standardized patient exercises for first-year students. The objective was to provide the new students with a non-threatening opportunity to identify their strengths and weaknesses as they begin work on clinical skills. The faculty did not grade students and even provided them with the checklists that the standardized patients would use in advance. These early formative experiences proved to be valuable introductions to clinical medicine.
Changes in legal education, like the law itself, occur slowly and incrementally. At the same time, a longer view provides a framework—a structure—within which change might take place. Here, the goal is twofold. First, from the outset, legal education should fully integrate doctrine and applied lawyering skills in the law school curriculum. Likewise, an emphasis on the role of the lawyer, along with the challenge of knowing what it means to be a professional, should be a more integral part of the first‑year experience. One vehicle to assist in this venture is the standardized client method. The means used to evaluate law students’ progress should also reflect a synthesized notion of theory, doctrine, and skills. Visual aids, interactive electronic programs, and standardized clients would all assist in the law student assessment effort. The second long-term goal is to match the means of evaluation applied in any high-stakes examination for entry to the profession with the integrated educative and evaluative methods, which the law schools will have established.
Bar examiners (and the state courts and governments that oversee the bar examiners) should support the incremental approach necessary to produce a refined vision of how to ensure that lawyers are competent to practice law. There is implicit recognition that, for cost reasons, legal educators cannot replicate either the clinical model of medical education in law school or a meaningful apprenticeship system after law school. However, that does not mean that legal educators should do nothing toward pursuing a more well-rounded preparation for the practice of law. The objective here is to maintain and even speed up that movement.
Drawing the connection between initial efforts to use standardized clients in a legal education setting and improvements to the bar exam requires imagination or at least faith in long-term planning. However, that is exactly what is necessary if changes in bar admission screening are ever to take place. Inertia plays a large role in maintaining the status quo regarding bar exams. It is an especially powerful force in this area because most lawyers are so happy to put the bar exam experience behind them that they really do not want to address the more fundamental issues about the purposes of the exam and whether the current exams achieve those purposes. Thus, I welcomed the opportunity to tie my work with standardized clients into a far-reaching proposal to experiment with alternative ways to assess applicants for bar admission. It may very well be that the small pilot that the New York bar association committees have proposed could integrate the use of standardized clients. The collection of data from both the Committees’ PSABE and the law school programs could be valuable in assessing the overall worth of the standardized client method, which would benefit both programs. A bonus under the PSABE would be the provision of legal services by the bar applicants participating in the pilot.
* Professor of Law, New York Law School. I want to thank Rick Marsico and Susan Rosenthal for their helpful comments and suggestions; Dean Richard Matasar and New York Law School for their continuing support of my work with standardized clients; and Debra Ficarra for her excellent research assistance.
. The Committee on Legal Education and Admission to the Bar of the Association of the Bar of the City of New York & The Committee on Legal Education and Admission to the Bar of the New York State Bar Association, Joint Committee Report: Public Service Alternative Bar Examination (June 14, 2002), available at http://www. abcny.org/pdf/report/Joint Report on the State Bar Exam.pdf [hereinafter Joint Proposal]. This Report was a joint proposal of the Committees on Legal Education and Admission to the Bar of the Association of the Bar of the City of New York (“ABCNY”) and the New York State Bar Association (“NYSBA”).
. The current two-day New York bar consists of 5 essay questions (40%), 50 multiple choice questions on New York law (10%), 200 multi‑state multiple choice questions (40%), and 1 ninety-minute written performance test (10%).
. See, e.g., Thomas Adcock, Pilot Proposal Would Make Public Service Count on Bar Exam: City and State Bar Committees to Endorse Rare Joint Report, N.Y.L.J., June 28, 2002, at 16; Victoria Rivkin, Bye-Bye, Bar Exam?: New York Program Would Offer Some Grads Alternative to the Written Test, A.B.A. J., Feb. 2003, at 16; Diane F. Bosse & Lawrence M. Grosberg, The Bar Exam: Should the Test Continue in its Current Form or Are Alternatives Needed?, N.Y.L.J. Mag., Apr. 2003, at 12; Tom Stabile, Surprise! Bar Exam Changes, Nat’l Jurist, Jan. 2003, at 20.
. News Release, United States Medical Licensing Examination, Americans Overwhelmingly Support New Medical License Test; Field Tests Show Fairness Reliability of Test, available at http://www.usmle.org/news/cse/newsrelease2503.htm (Feb. 5, 2003) (on file with Author) (announcing implementation of a clinical skills exam in 2004 that will require applicants for a medical license to complete a series of standardized patient exercises successfully). The United States Medical Licensing Examination also announced that a Harris Poll of adult Americans found that “97[%] consider clinical skills very important or extremely important,” “87[%] want to see students pass a clinical skills exam before receiving their medical license,” and 67% believe that state medical boards should add this type of exam even if it costs applicants $1000. Id.
. The United States Medical Licensing Examination’s announcement of the imminent inclusion of standardized patient exercises included an explicit acknowledgment that “medical schools vary greatly in the emphasis they place on clinical skills.” Id. Indeed, they go on to say that “4[% of U.S. students] said they had never taken a history or conducted a physical examination under the watch of a faculty member, and 20[%] said they had been observed only two or fewer times.” Id. The use of the standardized patients “will establish [a national] standard [so that] physicians enter practice with core competencies.” Id.
. The Committee on Legal Education and Admission to the Bar of the Association of the Bar of City of New York, Report on Admission to the Bar in New York in the Twenty-First Century—A Blueprint of Reform, 47 Rec. Ass’n Bar City N.Y. 464 (1992).
. Jayne W. Barnard & Mark Greenspan, Incremental Bar Admission: Lessons from the Medical Profession, 53 J. Legal Educ. 340 (2003) (including the use of standardized clients among a series of proposals to improve bar admission screening mechanisms).
. The committees officially submitted the Joint Proposal to the New York State Board of Law Examiners; Judge Jonathan Lippman, the New York State Chief Administrative Judge; and Chief Judge Judith Kaye of the New York Court of Appeals, New York’s highest court. While members of the Board of Law Examiners met informally with members of the two committees to discuss the Joint Proposal (essentially dismissing it as practically impossible), there has been no other response. Coincidentally, within two or three months of the issuance of the Joint Proposal, the Board of Law Examiners issued an extensive report recommending that the passing score for the bar exam be increased by 15 points (which would result in an additional 600 to 1000 applicants who would fail the exam). That Board of Law Examiners’ Report substantially diverted the attention of the public, the Board of Law Examiners, and the two committees from a fuller consideration of the PSABE. The opposition to the Board’s recommendation was widespread. See, e.g., The Committee on Legal Education and Admission to the Bar of the Association of the Bar of City of New York, In Opposition to the Board of Law Examiners’ Proposal to Increase the Passing Score on the New York Bar Examination, 58 Rec. Ass’n Bar City N.Y. 98 (2003).
. See, e.g., Thomas Adcock, Debate Over Bar Exam Passing Score Heats Up, N.Y.L.J., Jan. 31, 2003, at 16; see also Adcock, supra note 3; Susan Saulny, Case for Tougher Bar Exam Prompts a Forceful Rebuttal, N.Y. Times, Apr. 13, 2003, at A32.
. The membership of New York City’s Association of the Bar Committee ranged from 17 to 23 members over the nearly three years of working on the Joint Proposal, and the New York State Bar Association Committee, I believe, had roughly 25 to 35 members.
. While there remains one state, Wisconsin, that adheres to a “diploma privilege,” which dispenses with a bar exam for graduates of any of the three accredited Wisconsin law schools, the members of both committees concluded that such an alternative was simply not feasible. See Beverly Moran, The Wisconsin Diploma Privilege: Try It, You’ll Like It, 2000 Wis. L. Rev. 645, 645.
. In this regard, we found consolation and emotional support in our knowledge that the State of New York had enacted legislation in the past that completely exempted an entire category of candidates for attorney licensing from taking the bar exam: graduates of accredited New York law schools whose potential law careers were interrupted by military service. New York adopted these exemptions for veterans of World War II, the Korean War, and the Vietnam War. See Adcock, supra note 3. The fact that New York’s admission of this group of lawyers to the bar without taking or passing a bar exam produced no observable adverse impact on the public suggests that New York might not view the small number of applicants who would participate in the pilot as a serious danger to the public’s confidence. Indeed, as a result of a continuing survey by the New York Law Journal, it is now clear that judges and other distinguished lawyers are among those admitted without having taken any kind of bar exam.
. Researchers generally assume public dissatisfaction with the quality of lawyering. However, the availability of data supporting this assumption is limited. See Linda F. Smith, Medical Paradigms for Counseling: Giving Clients Bad News, 4 Clinical L. Rev. 391, 398-99 (1998) (summarizing surveys that found more than a quarter of consumers of legal services were dissatisfied with the services rendered).
. See Ian Weinstein, Testing Multiple Intelligences: Comparing Evaluation by Simulation and Written Exam, 8 Clinical L. Rev. 247 (2001) (proceeding on the assumption that lawyers require multiple intelligences (for example, interpersonal as well as mathematical-logical), and concluding that the skills of taking a written exam and conducting a graded simulation are independent skills). Students who do well on the latter may not do as well on written exams. Id. Weinstein’s study supports the propriety of including tests of interpersonal skills such as interviewing and counseling. Id.
. Accord Richard A. Matasar, Skills and Values Education: Debate About the Continuum Continues, 46 N.Y.L. Sch. L. Rev. 395, 426-28 (2002-2003) (faulting the profession for not meeting its share of the responsibility for preparing practitioners and ensuring their continuing skillfulness).
. Current bar admission rules do not generally include mentoring or apprenticeship requirements prior to licensing a law graduate to practice law. Cf. Task Force on Law Schools and the Profession, A.B.A. Sec. Legal Educ. & Admissions to the Bar, Statement of Fundamental Lawyering Skills and Professional Values: Narrowing the Gap (1992). This report is commonly known as the “MacCrate Report.”
. See Glen, supra note 14, at 1724-29, for additional details in the Public Service Alternative Bar Examination (“PSABE”) Joint Proposal. Dean Glen, however, omitted reference to the Joint Proposal’s requirement of a written performance type exam. Id. at 1719 n.89.
. Cf. Avrom Sherr, The Value of Experience in Legal Competence, 7 Int’l J. Legal Prof. 95, 115 n.40 (2000). In Professor Sherr’s study, there was similar use of outside evaluators. Expert senior clinical professors evaluated videotaped interviews of clients to determine the impact of years of experience on the quality of the lawyering performance. Id.
. The committees devoted considerable debate to whether a written performance test was necessary or desirable, and if so, what it should look like. They made comparisons between the ninety‑minute, multi-state performance test and the three‑hour California version. They concluded that it was best to leave open the details, allowing for an even longer performance test if appropriate. Dean Glen opposed any version of the written performance test. See generally Stella L. Smetanka, The Multi‑State Performance Test: A Measure of Law Schools’ Competence to Prepare Lawyers, 62 U. Pitt. L. Rev. 747 (2001) (regarding the efficacy of the performance test).
. The Joint Proposal outlined a three‑tiered administrative structure. The committees acknowledged that this would be a new venture that would require some adaptability and trial and error, but they asserted that it was a viable program. See Joint Proposal, supra note 1, at 11-13.
. The standardized client approach might address this problem. Because of the absence of research as to the validity of the standardized client, however, the committees did not consider incorporating this method of evaluating applicants.
. The committees were hopeful that foundations, and perhaps the State, would finance the pilot. However, there was also strong support for the position that any ultimate proposal ought to reflect what the bar should do regardless of the costs. It is instructive, once again, to note that the National Medical Licensing Exam will be imposing, over the considerable opposition of many (most particularly, new medical school graduates), an additional charge of $1000 on each applicant to cover the costs of including standardized patient exercises in the licensing exam.
. This could also occur in grading essay question answers. The bar trains exam graders to standardize their grading, thus minimizing this problem. The bar could do the same for those grading live performances.
. Some have made the same observation about first‑year writing courses. See Kenneth D. Chestek, Reality Programming Meets LRW: The Moot Case Approach to Teaching in the First Year, 38 Gonz. L. Rev. 57, 58 (2002-2003). “Because the legal writing course is taught in such a different way than other first‑year classes, some students think of it as an anomaly, unworthy of the same level of attention as the other more ‘substantive’ courses.” Id.
. There have been significant changes in the upper-class curriculum over the last 30 years, primarily as a result of the growth of clinical education, but very few of those changes have found their way into the first year.
. See Bethany Rubin Henderson, Asking the Lost Question: What Is the Purpose of Law School?, 53 J. Legal Educ. 48, 72 (2003) (noting that tradition, “institutional inertia[,] and financial disincentives” are substantial obstacles to change).
. I have worked closely with four other professors on the development of the standardized client. As discussed below, the vehicle for developing the standardized client is the first-year Lawyering course. It is that course in which my four colleagues—Professors Carol Buckler, Stephen Ellmann, Richard Marsico, and Richard Sherwin—and I have collaborated.
. This Article uses the term “lay person” rather than actor because non-actors may also assume standardized client roles. Professionally trained actors are preferable, if available, but not critical.
. In any case, I have not discovered any published accounts of the use of these structured evaluative techniques. Nor did I receive any responses to my solicitation of information about the use of standardized clients or comparable evaluation methods on a clinical educators’ listserv.
. There are different ways to look at this long-standing grading tradition. One favorable explanation is that the professor who teaches the students is in the best position to evaluate their work in light of that teacher’s goals and criteria. Still a more positive perspective is that the students are paying a lot of money and are entitled to feedback at the same level and quality as the lecturing they receive. One less positive view is that law professors arrogantly conclude that no one else could possibly evaluate a student’s work adequately. This professional grading tradition does not carry over to the ultimate high-stakes test—the bar exam. Practicing lawyers, hired and trained to read and grade bar exam essay answers, grade those exam answers. See, e.g., Greg Sergienko, New Modes of Assessment, 38 San Diego L. Rev. 463, 475 (2001) (“The reliance of many respected colleges on noninstructor-based assessment should cause law schools to reexamine their practices.”).
. For a succinct summary of the basis on which the United States Medical Licensing Examination has concluded that using standardized patient exercises as part of the licensing exam is a “valid” and reliable form of testing, see United States Medical Licensing Examination, Validity of Step 2 Clinical Skills (Step 2 CS), at http://www.usmle.org/news/cse/csevalidity21903.htm (Feb. 19, 2003), and United States Medical Licensing Examination, An Analysis of U.S. Student Field Trial and International Medical Graduate Certification Testing Results for the Proposed USMLE Clinical Skills Examination, at http://www.usmle.org/news/cse/cseftresults2503.htm (Feb. 5, 2003).
. See United States Medical Licensing Examination, Step 2 Clinical Skills Examination: Frequently Asked Questions, at www.usmle.org/news/cse/step2csfaqs1103.htm (Feb. 5, 2003) (discussing comprehensively the rationale for the inclusion of standardized patient exercises in the licensing exam, as well as the history and logistics behind this inclusion).
. More generally, it is only recently that law professors have begun to appreciate fully the value of empirical research and to engage significantly in empirical scholarship of any sort. See Michael Heise, The Past, Present, and Future of Empirical Legal Scholarship: Judicial Decision Making and the New Empiricism, 2002 U. Ill. L. Rev. 819, 820; Richard K. Neumann, Jr. & Stefan H. Krieger, Empirical Inquiry Twenty-Five Years After The Lawyering Process, 10 Clinical L. Rev. 349, 349 (2003); Peter H. Schuck, Why Don’t Law Professors Do More Empirical Research?, 39 J. Legal Educ. 323, 323 (1989).
. In addition to Professor Amsterdam, many others have helped with the design and teaching of the New York University School of Law (“NYU”) course. Most recently, Professor Peggy Davis has assumed the directorship of the Lawyering program at NYU and has implemented many changes.
. I first observed this in a simulation exercise designed by Professor Philip Schrag. See Philip G. Schrag, The Serpent Strikes: Simulation in a Large First-Year Course, 39 J. Legal Educ. 555 (1989). While this approach is obviously not a substitute for an individually conducted session, it can be an effective way to engage a large class than to simply lecture. The professor can comment on the various contributions when the session is over.
. See Lawrence M. Grosberg, Should We Test for Interpersonal Lawyering Skills?, 2 Clinical L. Rev. 349, 374-78 (1996). The overarching question is how, not whether, to assess students’ ability to perform various interpersonal lawyering skills, ranging from cross examination of a trial witness to interviewing a client. The answer is that the most direct and valid way to do this is to observe the student performing the skill in a real case in real time. The next-best method of evaluation might be to observe the student in a simulated performance. Another method, we suggest, is to require the student to watch a videotaped lawyer performance and to complete a written critical analysis of the performance. Finally, one could present a written problem and ask for an analysis of how to proceed in performing the applied personal skill. Our use of a final exam in the Lawyering course, in which the student must analyze a videotaped performance, reflects, in part, our acceptance of the reality of limited resources.
. There were two reasons for the shift in the placement of the course. First, student feedback suggested it would be a more effective introduction to what lawyers do if they had completed a full semester of basic learning in legal reasoning and analysis. Second, we made some other curricular changes in the first year that made the shift to the spring semester a logical move. Originally, a goal of the Lawyering course was to communicate to new law students from the first day of law school that, despite the importance of learning how to read, analyze, and synthesize appellate case law, there is much more to being a competent lawyer. While it seems that we have successfully merged the new course into the second semester, the move has relegated that goal to a secondary concern.
. We are in the process of producing a CD-ROM or DVD containing all of the videotaped vignettes, as well as the written case-file materials and the standardized client evaluation forms that we use in the course. We will make these available for purchase.
. The issue of grading students more affirmatively on their performance in standardized client exercises goes to the heart of this discussion—namely, how to demonstrate the validity, reliability, and basic fairness of this method of evaluating law student performance. That is the project in which we are now engaged.
. Using adjunct professors or volunteer lawyers would also be impracticable for these large numbers of simulated sessions. Hiring, training, and retaining these persons does not seem achievable. Hiring lay persons as standardized clients also incurs costs but at a much lesser rate than the alternatives.
. Last year, we spent roughly $17,000 to provide more than 500 students (an unexpectedly large class; usually about 440) with a standardized client exercise, including the individualized written feedback. The administration at New York Law School—Deans Richard Matasar, Jethro Lieberman, Carol Buckler, and Stephen Ellmann, and, previously, Harry Wellington and Associate Deans Ellen Ryerson—has been incredibly supportive of this continuing experiment.
. An earlier paper describes how more than one law school could utilize a central standardized client facility, thereby increasing savings from the economy of scale. See Grosberg, supra note 5, at 230-33.
. A much more limited use of standardized clients or patients occurs where a student or series of students interviews or counsels an individual person in front of a large lecture class. See, e.g., Catherine A. Birndorf & Marsha E. Kaye, Teaching the Mental Status Examination to Medical Students by Using a Standardized Patient in a Large Group Setting, 26 Acad. Psychiatry 180 (2002).
. The training objective is to teach the lay people to role-play clients (or witnesses) carefully so that they respond as realistically as possible to novice efforts to elicit information. See, e.g., Fernando Colon-Navarro, Thinking Like a Lawyer: Expert-Novice Differences in Simulated Client Interviews, 21 J. Legal Prof. 107, 122-27 (1997). In the study discussed in this Article, the professors conducting the study, not the role-playing clients, scored videotaped simulated interviews done by persons of varying degrees of experience. Id. at 120. Such evaluations by teachers are the norm. Feedback by a trained standardized client is clearly the exception in legal education. See also Adrian Evans & Clark D. Cunningham, Specialty Certification as an Incentive for Increased Professionalism: Lessons from Other Disciplines and Countries, 54 S.C. L. Rev. 987, 996 (2002-2003) (describing the use of a simulated client interview as part of the specialist licensing process in Australia, where the examiners, not the person portraying the client, assess the sixty-minute videotaped interview on several levels).
. As noted above, although we do not require professional actors, we prefer them. Actors are certainly available in any of the large metropolitan areas such as New York or Los Angeles. However, even in less populated areas, the medical educators’ experience is that university drama schools and the general populace have provided more than enough candidates for standardized patient duties.
. As discussed below, this is not the most efficient way to utilize standardized clients. Rather, the most cost efficient and effective way would be to establish a centralized facility (with interview rooms) that more than one law school could share. This Article describes the centralized facility concept below.
. As noted earlier, the four persons who have worked with me on the current version of the Lawyering course are: Professors Carol Buckler, Stephen Ellmann, Richard Marsico, and Richard Sherwin. The prior version included those teachers as well as Professors Carlin Meyer and Donald Zeigler.
. The central question is: Can legal educators include standardized client grades as part of the students’ course grade before substantiating the validity of the evaluative techniques? Indeed, it goes to the immediate need for conducting research to establish the validity and reliability of this method of assessing law student performance. See Stacy L. Brustin & David F. Chavkin, Testing the Grades: Evaluating Grading Models in Clinical Legal Education, 3 Clinical L. Rev. 299, 306-08 (1997).
. This is also a response often heard from clinic and simulation course students: The law school does not need to teach these students how to talk to a client or a witness because they are adults who already possess these interpersonal skills. As in those courses, the teachers here try to present the view that honing one’s legal interviewing and counseling skills is a desirable pursuit even for the most experienced interviewer.
. Their legal analyses are not always correct, and their counseling skills performance is not always sparkling. However, they at least go through the necessary preparatory steps in formulating the legal opinions and in thinking about how they intend to covey those opinions to their client.
. Most first-year law students have an appellate moot court experience. That opportunity to use oral argument skills is useful, but it does not address the need for training in basic interpersonal lawyering skills, which is the subject of this Article.
. Grading the standardized client exercise is a critical element in this effort to alter the first‑year themes. Ideally, legal educators should not do this until empirical studies directly support this change, at least not for high-stakes grading. The difficult issue, however, is what legal educators might do prior to that point with respect to low-stakes grading. For low-stakes grading purposes (for example, a small percentage of the grade for one law school course), legal educators can reasonably infer, based on the experience of medical educators’ work with standardized patients, that we can accept the standardized clients’ graded evaluations of law students’ performances.
. If a professor assigned and required the completion of the standardized client exercise before the class discussion, the students would have to learn the law so that they could properly explain it (“translate it”) to their clients. Therefore, the subsequent class discussion would presumably reflect a much deeper appreciation of the substantive issues involved. Alternatively, if a professor assigned the student to conduct the standardized client exercise after the class discussion, it would better prepare the student to explain and apply the law. From the perspective of enhancing the doctrinal teaching, most first-year professors would likely prefer the first alternative.
. See Lawrence M. Grosberg, The Buffalo Creek Disaster: An Effective Supplement to a Conventional Civil Procedure Course, 37 J. Legal Educ. 378, 387 (1987); see also Alice M. Noble-Allgire, Desegregating the Law School Curriculum: How to Integrate More of the Skills and Values Identified by the MacCrate Report Into a Doctrinal Course, 3 Nev. L.J. 32, 39 (2002) (describing the “synergistic effect” of combining skills and doctrine).
. See, e.g., Owen M. Fiss & Charles D. Berger, The Center for Computer-Aided Legal Instruction, Woburn: A Game of Discovery (2003), at http://www2.cali.org/index.php?fuseaction=lessons.lessondetail&lid=394; Lewis A. Grossman & Robert G. Vaughn, A Documentary Companion to A Civil Action (1999).
. If legal educators did not coordinate the standardized client exercises with skills teaching in a Lawyering-type course, it could easily frustrate students in their untrained attempts to perform these lawyering skills. This is a real issue that legal educators would have to (and could) successfully address.
. Cf. David A. Binder & Paul Bergman, Taking Lawyering Skills Training Seriously, 10 Clinical L. Rev. 191, 208 (2003) (“[M]edical schools have substantially increased the amount of training time they devote to clinical skills, typically embedding clinical skills training in the entire medical school curriculum.” (emphasis added)).
. For a comprehensive discussion of bar exams and how to improve them, see Andrea A. Curcio, A Better Bar: Why and How the Existing Bar Exam Should Change, 81 Neb. L. Rev. 363, 394-98 (2002). Professor Curcio looked to architecture as well as medicine for models that might improve bar exams. Id. In particular, she focused on how legal educators might benefit from emulating the efforts in both of those professions in using computer‑based testing methods. Id.; see also Barnard & Greenspan, supra note 11 (suggesting that legal educators follow the medical licensing model of stepped exams, as well as more effective use of computerized testing methods).
. For example, lawyers often have to deal with clients who have committed moral or criminal wrongs of one sort or another, a situation not often encountered by doctors. This presents different and somewhat unique interviewing challenges for the lawyer.
. See, e.g., Evans & Cunningham, supra note 64, at 1004. (comparing the extensive empirical data assessing patient satisfaction with physician services to the dearth of information as to client satisfaction with lawyer services).
. See Colon‑Navarro, supra note 64, at 113 (comparing the need to learn “the process of clinical reasoning” in medicine and in law, and concluding that, without experiential learning, neither the law graduate nor the medical graduate could effectively practice their respective professions).
. However, even with all of that data, there is still opposition to the adoption of the standardized patient exercises as part of the licensing exam. See Bonnie Booth, Delegates Oppose Testing of Clinical Skills for Licensure, Am. Med. News, Jan. 6, 2003, available at http://www.ama-assn.org/amednews/2003/01/06/prsa0106.htm. Medical students objected to the additional $1000 cost. Id.
. Another difference between bar examiners and medical licensing examiners is worth noting. In New York, for example, full-time academics are not eligible for membership on the Board of Law Examiners (a part-time position). When I confronted the Board and the Clerk of the New York Court of Appeals (who administered a solicitation of applications for a recent vacancy on the Board) about the reason, they informed me that there was an unwritten concern that a conflict of interest would exist (causing the academic to take actions that would favor the academic’s institution). This policy of keeping academics off the Board of Law Examiners exacerbates the lack of constructive cooperation that is necessary to synchronize curricular and licensure concerns properly. In contrast, the National Board of Medical Examiners recently elected as its chair Dr. Laurence B. Gardner, a professor and Vice Dean at the University of Miami Medical School. See New NBME Chair, NBME Examiner (National Board of Medical Examiners), Spring/Summer 2003, http://www.nbme.org/examiner/SpringSummer2003/news.htm (last visited Apr. 1, 2004).
. Cf. Janet Fleetwood et al., Bringing Medical Ethics to Life: An Educational Programme Using Standardised Patients, 36 Med. Educ. 1100, 1100-01 (2002) (describing similar satisfaction assessments in a medical school setting).
. Determining that standard is a complex matter. Cf. Tim J. Wilkinson et al., Standard Setting in an Objective Structured Clinical Examination: Use of Global Ratings of Borderline Performance to Determine the Passing Score, 35 Med. Educ. 1043 (2001).
. One need only glance at a single database search showing the hundreds of research articles addressing these issues in leading medical research journals to demonstrate this point. A search “[using] the MEDLINE database for the period 1966 to 2001 . . . for articles that studied the reliability or validity of measures of clinical or professional competence . . . yielded 2266 references.” Ronald M. Epstein, M.D. & Edward M. Hundert, M.D., Defining and Assessing Professional Competence, 287 JAMA 226, 229 (Jan. 9, 2002). While legal educators have not conducted this kind of empirical research on evaluation methods, there certainly has been no shortage of scholarship in the clinical and skills area on the importance of educating future lawyers on a fuller range of applied lawyering skills.
. A recent article in The Lancet, a leading medical journal, points out that, in the full array of testing methods used to assess physician competence, there is an “increasing focus on the performance of doctors and on public demand for assurance that doctors are competent.” Val Wass et al., Assessment of Clinical Competence, 357 Lancet 945, 945 (Mar. 24, 2001). One might speculate as to why there has not been a similar response to the public’s demand for lawyer competence.
. To augment their reliance on the fortuitousness of real patient medical problems, medical educators sought a means to ensure that they presented students with a comprehensive array of problems for evaluation. This led to greater use of standardized patients. Using standardized clients in a law school setting could accomplish a similar objective. See Binder & Bergman, supra note 89, at 208-13.
. After a comprehensive discussion of the components of competence for a physician, the authors then went on to analyze the limitations of the various methods for assessing competence. In particular, the authors distinguished between a student’s ability to “show how” to do something in a standardized patient context and the ability to “do” something in a real case situation. Videotapes of work on real cases may be a better vehicle for measurement. Epstein & Hundert, supra note 107, at 229.
. Rachel Yudkowsky, Should We Use Standardized Patients Instead of Real Patients for High-Stakes Exams in Psychiatry?, 26 Acad. Psychiatry 187 (2002) (“The difficult logistics associated with real patients typically result in nonstandardized assessments consisting of one or two case[s] that are unsystematically selected from a nonrepresentative pool of patients . . . [resulting in] relatively low validity and reliability[,] and that can be perceived as unfair.”).
. This is based on a medical school project that involved the use of standardized patients to teach management of a disease (in this case diabetes) over a period of time. The experimental group had multiple encounters with the standardized patient, whereas the control group had none. At the end, researchers gave both groups the opportunity to see a diabetes standardized patient. The experimental group achieved much better results, both as to the substance and the level of interpersonal skills. Amanda Brown et al., Using Standardized Patients to Teach Disease Management Skills to Preclinical Students: A Pilot Project, 15 Teaching & Learning Med. 84 (2003).
. As far as the validity of the professors’ evaluation of the students is concerned, the professorial grading is the “gold standard” because they are best qualified to determine whether the student achieved what the test called on them to achieve. There could still be some reliability issues (for example, consistency), but as to the validity of standardized clients as evaluators of students, professorial comparisons may be critical. See, Arthur I. Rothman & Michael Cusimano, Assessment of English Proficiency in International Medical Graduates by Physician Examiners and Standardized Patients, 35 Med. Educ. 762 (2001) (concluding that the physicians were less prone than the standardized patients to negatively rate the candidates); cf. Mark H. Swartz, M.D. et al., Global Ratings of Videotaped Performance Versus Global Ratings of Actions Recorded on Checklists: A Criterion for Performance Assessment with Standardized Patients, 74 Acad. Med. 1028 (1999) (presenting research that suggests global ratings of checklists by faculty may be insufficient).
. This is based on a study done at the University of Iowa in which researchers gave the medical students the opportunity to conduct several standardized patient encounters in which they had to give bad news of one kind or another. At the conclusion, the researchers surveyed the students, and the results showed positive responses to the educational value of these encounters. See Marcy E. Rosenbaum & Clarence Kreiter, Teaching Delivery of Bad News Using Experiential Sessions with Standardized Patients, 14 Teaching & Learning Med. 144, 148 (2002).