Can ChatGPT help LLMs pass the bar exam?

The good news: Yes, it probably can!

The bad news: But it’s not the LLMs you’re probably thinking of.

I recently noticed in the abstract for the article “GPT Takes the Bar Exam” that the last line reads:

While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.

At first I did a double take and had to re-read the full abstract to understand how in the heck GPT’s relative success in answering bar exam questions could portend that one lucky future LLM student will pass the multiple choice section of the bar exam.

Then I remembered that LLM is different from LL.M. Because in the context of artificial intelligence, LLM means “Large Language Model” which is the term used to encapsulate what ChatGPT is and which is obviously very different than the Master in Laws or Legum Magister meaning which refers to a one-year degree at a law school and which is often associated with international students in US law schools.

This is clearly a distinction that those of us in the legal English field will have to get used to in order to avoid potential confusion in the future. It also suggests that the periods in “LL.M.” may need to come back in fashion for those out there (like me) who have been trying to get away with leaving them out in the name of efficiency.

Here’s the full abstract, in case of interest:

**********************************

GPT Takes the Bar Exam

13 Pages Posted: 31 Dec 2022

Michael James Bommarito

273 Ventures; Licensio, LLC; Bommarito Consulting, LLC; Michigan State College of Law; Stanford Center for Legal Informatics

Daniel Martin Katz

Illinois Tech – Chicago Kent College of Law; Bucerius Center for Legal Technology & Data Science; Stanford CodeX – The Center for Legal Informatics; 273 Ventures

Date Written: December 29, 2022

Abstract

Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as “the Bar Exam,” as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in “AI?” In this research, we document our experimental evaluation of the performance of OpenAI’s text-davinci-003 model, often-referred to as GPT-3.5, on the multistate multiple choice (MBE) section of the exam. While we find no benefit in fine-tuning over GPT-3.5’s zero-shot performance at the scale of our training data, we do find that hyperparameter optimization and prompt engineering positively impacted GPT-3.5’s zero-shot performance. For best prompt and parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam, significantly in excess of the 25% baseline guessing rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5’s ranking of responses is also highly correlated with correctness; its top two and top three choices are correct 71% and 88% of the time, respectively, indicating very strong non-entailment performance. While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.

Keywords: GPT, ChatGPT, Bar Exam, Legal Data, NLP, Legal NLP, Legal Analytics, natural language processing, natural language understanding, evaluation, machine learning, artificial intelligence, artificial intelligence and law

JEL Classification: C45, C55, K49, O33, O30

Suggested Citation:

Bommarito, Michael James and Katz, Daniel Martin, GPT Takes the Bar Exam (December 29, 2022). Available at SSRN: https://ssrn.com/abstract=4314839 or http://dx.doi.org/10.2139/ssrn.4314839

Christmas movie legal analysis!

Post by Prof. Stephen Horowitz, Professor of Legal English

This fun video titled “Wet Bandits v. McCallister? Law Professor Prosecutes the Case Against Home Alone’s Kevin McCallister” was originally published on the Georgetown Law School website’s News page on December 20, 2022.

In it, Georgetown Law’s Jonah Perlin (associate professor of law, legal practice and creator of the HowILawyer podcast) re-examines the classic Christmas comedy Home Alone (starring Macaulay Culkin) and delves into what legal case the house robbers Marv and Harry might have against their 8-year-old foe, who goes to very creative lengths to foil their robbery attempts.

A fun way to work on one’s legal English over the holiday break, especially since the video has captions and Jonah provides very clear yet simple explanations!

AI/ChatGPT as a tool for Legal English and LLM students


Post by Prof. Stephen Horowitz, Professor of Legal English

As we start to shift past the “wow” factor of AI and ChatGPT (see, e.g., this very cool post from the FCPA Blog posing questions to ChatGPT related to the Foreign Corrupt Practices Act, and also this academic article titled “GPT Takes the Bar Exam“), I’ve seen articles and social media posts and heard comments and commentary focused on the potential plagiaristic dangers of ChatGPT, the artificial intelligence-fueled chatbot that can produce complex, natural-sounding essays in a matter of seconds:

But my initial reaction was less of concern and more along the lines of, “What a great potential legal English tool! How can we use this to help our LLM students learn better?”

And this thinking feels connected to what I’ve read in articles like “AI and the Future of Undergraduate Writing” by Beth McMurtrie in The Chronicle of Higher Education which essentially says that the horse is out of the barn; how are we as teachers and educational institutions going to adapt our assessment methods and how can we use this as a teaching tool. (This is really the underlying point of “The End of High School English” as well.)

Some of my own tests of ChatGPT, by the way, have included:

1) To ask it to “write an essay comparing Marie Antoinette and Rachel Carson,” the idea being to see if it could find connections on two seemingly unrelated people. And it did this quite effectively, acknowledging the lack of connection but finding comparison and contrast in that they were women of different social status who had certain accomplishments. About as good as I could expect from any student given a similar question.

Continue reading “AI/ChatGPT as a tool for Legal English and LLM students”

Updates from Georgetown Legal English Faculty

Craig Hoffman

  • In November, Professor Hoffman traveled with Georgetown Law Dean William Treanor to visit Georgetown Law LLM alumni in Saudi Arabia, Qatar, Abu Dhabi and Dubai.
  • During the Spring 2023 semester Prof. Hoffman will teach a law and linguistics course in which students will examine originalism from a linguistic perspective.
Prof. Hoffman with Abdulaziz Altuwarijri (Georgetown 2-Yr LLM), Dean of Prince Sultan University

Yi Song

  • Professor Song’s essay Lawyering While Chinese will be published in the book Fostering First Gen Success and Inclusion: A Guide for Law School by Carolina Academic Press forthcoming February 2023.

John Dundon

Julie Lake & Heather Weger

Prof. Lake and Prof. Weger have been invited to co-present at the following conferences:

Paula Klammer

Stephen Horowitz

  • Co-presented with Prof. Daniel Edelson (Seton Hall Law) to the NY Bar Association’s Committee on Legal Education and Admission to the Bar (“NYCLEAB“) on the topic “Online education for foreign-educated LLM students” (Nov. 16, 2022)
  • Presented online webinar for Tashkent State University of Law on the topic “The Benefits of Extensive Reading & Listening in Studying Law in English”
  • Organized a “Legal English Book Club” discussion with guest Alissa Hartig, Professor of Linguistics at Portland State University, on her use of Jeffrey P. Kaplan’s book Linguistics and Law in her course on linguistics and law titled, You Have the Right to Remain Silent: Language and the Law. (Dec. 7, 2022)
  • Interviewed Georgetown Law professors of legal writing Eun Hee Han and Jonah Perlin for the Multilingual Lawyer series for the USLawEssentials Law & Language Podcast. The episode (to be published in January) focused on international students in legal writing courses.
  • Completed co-teaching (with Prof. Daniel Edelson, Seton Hall Law) a 10-week online legal English course titled “Reading US Cases” for Ukrainian graduate law students at Yuriy Fedkovych Chernivtsi National University. (The course was part of a larger initiative born by collaboration between USAID and the Global Legal Skills community through which a number of US law professors have been teaching courses, giving guest lectures, and supporting English language law publication for law schools in Ukraine.)

Thank you to Tashkent State University of Law!

I want to take a moment to thank Senior Teacher Munisa Mirgiyazova and her colleagues and students at Tashkent State University of Law for inviting me to give an online webinar earlier this week on “The Benefits of Extensive Reading and Listening in Studying Law in English.” I greatly enjoyed the discussion and getting to meet everyone, and I look forward to future collaborations.

Reflecting on the presentation afterwards, I also realized that the collaborative opportunity came about because I posted something on LinkedIn (a new podcast episode maybe? I can’t remember now.) And Munisa saw it and sent me a LinkedIn connection request. I accepted and asked her how teaching was going in Tashkent. She replied and asked me how my teaching was going at Georgetown Law, and the subsequent conversation led to a discussion of ways to collaborate. A reminder that it’s often a good idea to leave conversational doors open and ask people about themselves.

I plan to share a recorded version of the presentation on this site after I make some edits to the original presentation and create the recording. Stay tuned!

Here’s the flyer for the event created by Tashkent State University of Law:

A different podcast interview with Georgetown Law’s Paula Klammer, legal translator and legal English professor

Post by Prof. Stephen Horowitz, Professor of Legal English

I was fortunate to have the opportunity to interview my colleague Paula Klammer for the Multilingual Lawyer interview series for the USLawEssentials Law & Language podcast, a podcast aimed at helping foreign-educated lawyers and law students improve their legal English.

Here’s a summary of the episode from USLawEssentials:

The USLawEssentials Law & Language podcast continues its series of interviews with multilingual lawyers as Stephen Horowitz interviews Professor Paula Klammer

Paula is a legal English Lecturer & Research Fellow at the Georgetown Center for Legal English. Currently earning her Ph.D. in Law from Universidad de Palermo in Argentina, Paula is an experienced lawyer and translator and is bilingual in Spanish and English. She speaks a few other languages, too, including French and Brazilian Portuguese (but she’s modest and says she’s not proficient yet).

This is a really cool interview with a fascinating guest. Paula’s bilingual background and her work as a translator enable her to provide insights on the special challenges of translating legal English, especially when dealing with false cognates, different writing styles, and very different legal systems.

Link to the episode is in the comments.

And hey – – do you think Spanish people speak faster or slower than most English speakers? Not sure? Got a hunch? You’ll find out.

Paula also discusses her doctoral dissertation so you’re going to learn a lot from this podcast.

Enjoy and let us know what you liked most about this episode.

Want to hear more from Paula? Listen to this May 2022 interview with her on the American Translators Association (ATA) podcast.

Georgetown Legal English Faculty Update: What we did over the summer

We haven’t posted much on here over the last few months, and that’s in a large part because we’ve all had busy summers. Below are updates on what some of the Georgetown Legal English faculty has been up to over the summer:

Professor Craig Hoffman

Prof. Craig Hoffman

Professor Hoffman is back at Georgetown Law this fall after being on sabbatical in the spring.  While on sabbatical, he traveled to Iceland, Argentina, Uruguay, France, England, and Scotland.  In October, Professor Hoffman will accompany Dean Treanor on a trip to the Gulf.  The law school will be hosting alumni receptions in Riyadh and Dubai.  In addition to teaching his classes in the Legal English Program, Professor Hoffman will teach a seminar in the spring called Language and Law in the Linguistics Department at Georgetown University.

Professor Julie Lake & Professor Heather Weger

Prof. Julie Lake
Prof. Heather Weger

Professor Weger and Professor Lake spent the summer revising their innovative language-focused curriculum for Fundamentals of Legal Writing for the 2022-2023 academic year. In the fall semester, the students will learn about the lawyer-to-lawyer genre and gain language-based strategies to write a high-quality memo. In the spring semester, the students will learn about the scholarly writing genre and how to write a high-quality mini-scholarly legal research paper.

Professor John Dundon

Prof. John Dundon

This June and July, Professor Dundon was thrilled to return to IE Law School in Madrid, Spain, where he’s been teaching a class on contract drafting for the past three summers.  The course is designed to simulate real-life contract-drafting assignments, with a primary focus on contracts governed under U.S. law. After teaching in Spain, Prof. Dundon spent several weeks traveling through Eastern Europe with his son, including a stop in Budapest, where he took some Hungarian lessons.

In August, Prof. Dundon taught a section of U.S. Legal Research, Analysis & Writing as part of the Summer Experience at Georgetown Law, which is a program designed to allow entering LL.M. students get a head-start on their coursework. Work also continued all summer in Prof. Dundon’s Ph.D. program, and he finalized a couple of linguistics articles over the summer for submission to academic journals.

Professor Paula Klammer

Prof. Klammer and her husband Pablo at the opera

This was Prof. Klammer’s first summer at Georgetown! She spent most of her summer at Williams Library working on her PhD in Law dissertation on the intricate relationship between law and language and what that means for legal education.

But being on campus all summer paid off and she had the unique opportunity to meet Georgetown Law’s Summer Institute students thanks to Professor Michael Cedrone’s kind invitation to sit in on his Foundations of U.S. Law course and to attend the Legal Writing Institute’s Biennial Conference at Georgetown law. (You can read more about Paula’s experience at the conference here.) Following that course, she helped Professor Craig Hoffman teach US Legal Research & Writing (USLRAW) to the summer institute students in preparation to co-teach the course with him during the Fall semester. 

Prof. Klammer with her husband at the Washington Nationals game

But it wasn’t all work and no play. She also explored DC with her husband, going to everything from the Smithsonian Folklife Festival at the Mall to the Opera at Wolf Trap, while watching the Nats win a couple of games (and lose many others!) at Nationals Park.

Professor Patricia Dutra

In August, Prof. Dutra taught a section of the U.S. Legal Research, Analysis & Writing to the Summer Institute students. The rest of her summer was busy getting her youngest son ready to go off to college. She is now teaching a section of US Legal Research & Writing to students in the Two-Year LLM Program. 

You can also read a great interview with Prof. Dutra by the Georgetown Brazilian Law Association (aka BrazLA) published in March 2022.

Professor Stephen Horowitz

Prof. Stephen Horowitz

Starting in May, Professor Horowitz set up two Georgetown Law Online Legal English (OLE) courses available to all incoming Georgetown LLM students to help them prepare for the Fall 2022 semester–OLE: Orientation to the US Legal System and OLE: Reading Cases. In May, Professor Horowitz also co-created and co-taught an online legal English writing course to a cohort of female judges from Afghanistan, collaborating with Seton Hall Law School Professor Daniel Edelson who is founder of the USLawEssentials.com online legal English platform. Profs. Horowitz and Edelson also designed an online legal English course on reading US case law to be offered to Ukrainian law students during the fall 2022 semester.

In addition to his teaching, Prof. Horowitz also continued his series of multilingual lawyer interviews for the USLawEssentials Law & Language podcast, including interviews with Georgetown’s Tax LLM Director Ellis Duncan, University of Minnesota Law School legal English professor Karen Lundquist, immigration lawyer Nick Harling, and Italian legal English teacher and legal translator Claudia Amato. Professor Horowitz also enjoyed a Great American Roadtrip with his family this summer that took them through a total of 17 different states.

Multilingual lawyer perspectives on the LWI’s Biennial Conference at Georgetown Law School

Post by Prof. Paula Klammer, Adjunct Professor of Law

I recently attended the Legal Writing Institute’s Biennial Conference at Georgetown Law School (July 20-23, 2022). As a first-time attendee, I had no idea what to expect. In fact, I wasn’t even sure why I had insisted on going in the first place. I don’t teach legal writing to JD students. Most of my experience teaching has been in my home country, Argentina, to non-native English speaking lawyers either at a tier-one law firm or at a top rated, but relatively small, law school. I was, as we say in my country, “un sapo de otro pozo” (which literally translates as “a frog from another pond”).  

“un sapo de otro pozo”

Although I was happy to find out that I was not the only first-time, first-gen, or rookie attendee, when many a kind colleague warmly pointed out that I was not the only foreigner (there was, after all, one Canadian among us), the irony of that didn’t escape me. So here I am, the new girl in academia from a non-English speaking developing country exposed for the first time to American scholars, the kinds of scholars I’ve long admired from over 5000 miles away. Intimidated and out of place doesn’t begin to describe it… but then the sessions kicked off and things slowly started to change.

There were basically three kinds of sessions: best practices, scholarship, and organization-specific. By best practices I mean sessions in which professors from law schools all over the country talked about the different pedagogical techniques they use in the classroom. Many of them even had the audience experience the exercises themselves so we could get a feel for the technique beyond the purely theoretical. This is a kind of session Americans are famous for and with good reason. They focus on practice rather than theory and give valuable insights into an aspect of American education that I find particularly valuable. I call it teaching people how to fish

As a civil lawyer trained in Argentina, after five years of law school, the first time I saw a case file I didn’t know how to open it. And this is not because there’s anything cognitively wrong with me, but simply because no professor in five years had ever thought to show us a case file or alert us to the fact that the pages are sewn together and new pages go on top, so you don’t open it like an ordinary book. Of course it doesn’t take long to figure that out in practice, but practicing law is stressful enough without anyone ever teaching you how to fish, which is why I value Americans’ emphasis on practical matters in addition to legal theory.

Then there were scholarship sessions. In those sessions, different professors talked about their current research, where they are, their preliminary conclusions, and where they’re headed. I particularly enjoyed two sessions on rhetoric and argumentation but it didn’t take me long to see the limitations of a less multilingual academic culture. Some of the ideas that float around as relatively novel in the United States are not-so-new outside the U.S., particularly in the E.U. where multiculturalism and multilingualism have forced legal scholarship to think about the place of culture and language in the law. So while I loved the work my American colleagues are doing, I couldn’t help but think about how much they could benefit from a more outward-looking perspective. 

With those thoughts going through my mind, I approached a much more seasoned professor who had just given a very interesting talk on culture in legal argumentation. I worked up the courage to talk to this professor about the research coming out of Europe in comparative legal-linguistic studies, where definitions of culture and language are naturally less rigid than what we’re used to on this side of the pond. That led to an incredibly enriching discussion that could only have happened among two people interested in the same topic, where one was fortunate enough to be able to read our French and Italian colleagues and the other receptive enough to want to know more about it. This to me is academia at its best and highlights the importance, not just of dialog among scholars but of the need to translate and publish more foreign scholars in the United States. 

Lastly, there were sessions about the Legal Writing Institute itself, its inner workings and mission. I found those sessions quite interesting in that one can really see how a relatively young area of scholarship is strategically building its own sense of identity. 

All in all, it was a positive and enriching experience. It made me aware of my own strengths and knowledge gaps. But it also left me wondering what the place is for multilingual foreigners in American academia. I’m not aware of any schools outside of Georgetown with a similar program to ours in which linguists, JDs, and LLBs work together to address the specific needs of our international, multilingual, and multicultural LLM students. There’s an obvious gap in American legal education. And, although I was happy to see everything that colleagues in the Global Writing Committee and beyond are doing to address that gap, I wonder what else can be done. 

For one thing, we have a representation and inclusion problem in the global classroom. With rare exceptions, professors are monolingual and monocultural; and although there is nothing wrong with that per se, legal practice is neither monolingual nor restricted to American legal culture for LLMs or the growing constituency of foreign-educated JDs in the U.S. That is where I see a representation problem. International or foreign students aren’t seeing professors who are like them or can relate directly to them in very relevant and important ways. And representation matters. It mattered to me in my first year of law school in Argentina when only 10 of my upcoming 45 professors were women (which I’m proud to say my school reversed while I was a student). And it mattered more than once when I was the only woman in the classroom and was often singled out for it by my professor.  

Prof. Paula Klammer, Georgetown Law

Working as a professor with international Georgetown students over the summer, especially those who, like me, are from developing countries, I’ve seen more than one face light up when I told them I’m from Argentina, worked at a big law firm and found my way to Georgetown non-traditionally. Students need to be able to see themselves in at least some of their professors. More importantly, they need someone who they can see themselves in a few years down the line. Global students experience education differently not just because they come from different backgrounds but because they are exposing themselves to an entirely foreign world. And these differences translate into specific educational needs. 

Inclusion is another story. Inclusion was a big theme at the conference and I was absolutely delighted to meet colleagues doing fantastic work toward a more inclusive classroom. According to the Center for Immigration Studies, “67.3 million residents in the United States now speak a language other than English at home, a number equal to the entire population of France.” With rising numbers of non-native English speakers in JD programs and international LLMs, even more awareness needs to be raised in American legal academia about language inclusion in the classroom; and colleagues who are already making great strides in that direction need our support. 

As for me, the conference started a revolution in my mind. It helped me figure out who I want to be in academia, a question my department director had asked me just a few days earlier and I didn’t quite have a sophisticated answer to yet. As usual, and consistently with everything I ever do in life, I have more questions than answers now. I know for sure that whatever I do will involve global students and turning up the volume of multilingual voices in American legal education. 

Legal English podcast interview with Italian lawyer-linguist Claudia Amato

Post by Prof. Stephen Horowitz, Legal English Lecturer & Adjunct Professor of Law

Claudia Amato, Italian lawyer-linguist

I recently had the privilege of interviewing Italian lawyer-linguist-teacher Claudia Amato, founder of SpeechLex, for the USLawEssentials Law & Language podcast, a podcast intended to help foreign-educated lawyers and law students to improve their legal English.

From USLawEssentials:

“The USLawEssentials Law & Language podcast continues its series of interviews with multilingual lawyers as Stephen Horowitz interviews Claudia Amato.”

“Based in Italy, Claudia is a remarkable attorney, translator, and legal English instructor. Among other things, she is the founder of SpeechLex, where she provides courses in legal English to help prepare attorneys and judges for the TOLES examination. In this episode, she also shares how her experiences working as a translator and teacher inspire her to help others as she explores the “human side” of people’s interaction with the law.
Oh – and as a surprise bonus- you get some helpful travel tips for Japan!”

New sections in the Legal English Resources Page: Vocabulary Resources and Legal English Articles!

Post by Prof. Stephen Horowitz, Legal English Lecturer

I recently added two new sections to the Legal English Resources page: 1) Vocabulary Resources and 2) Legal English Articles. See below.

Continue reading “New sections in the Legal English Resources Page: Vocabulary Resources and Legal English Articles!”
css.php