Analyzing ChatGPT’s ability as a grammar fixer


Post by Stephen Horowitz, Professor of Legal English

I recently tried a simple yet potentially helpful ChatGPT activity with my LLM students to (a) build individual grammar awareness, (b) build a better understanding of the benefits and limitations of using ChatGPT to fix one’s grammar, and (c) gain a better understanding of what happens grammatically when ChatGPT is asked to fix grammar.

The Process:

  1. As part of the Legal English II course (which teaches US case reading and analysis via a series of Supreme Court decisions about Miranda rights to students in Georgetown Law’s 2-Year LLM program), my students were required to write an essentially IRAC-style answer in response to a fact pattern under timed conditions.
  2. Afterwards, as an assignment, I asked my students to input their essay into ChatGPT with the instruction to “Please fix any language issues in this essay:
  3. Students then had to compare the two versions of their essay and write a short analysis or commentary on what they noticed, what ChatGPT did/didn’t do well, how they felt about it, etc. I told students to either put the two versions in a table so they could compare the language side by side, or they could do a use the redline/track changes function to show the differences.
  4. I next reviewed the students’ submissions myself. And I then invited two Georgetown Legal English colleagues with PhDs in applied linguistics–Prof. Julie Lake and Prof. Heather Weger–to review the student submissions and then have a group discussion about what we noticed.
  5. Upon additional consideration (and inspired by a suggestion from Jack Kenigsberg, a former Hunter MA TESOL classmate), I took one paragraph from one student’s essay and fed it into ChatGPT with the instruction: “Fix any grammar errors in the quoted text. For each change you make, explain why you made the change.” And after it provided its answer, I clicked “Regenerate response” to create a second response to see what (if anything) came out different a second time.

The Takeaways:

The main takeaways by my students, my colleagues and myself were:

Continue reading “Analyzing ChatGPT’s ability as a grammar fixer”

Article: “Finding the right voice(s): An engagement analysis of L2 writers in hypothetical legal writing”

Post by Stephen Horowitz, Professor of Legal English

Prof. Yiran Xu

I’m very happy to share a link to an article titled “Finding the right voice(s): An engagement analysis of L2 writers in hypothetical legal writing” by Professor Yiran Xu of University of California Merced , who completed her PhD in Applied Linguistics at Georgetown in 2020. The article–which was Professor Xu’s dissertation project–is based on her analysis of the writing of several students who were in the Georgetown 2-Year LLM Program at the time.

Here are the highlights followed by the abstract from ScienceDirect. Click the link to see the full article, which is available for free. https://www.sciencedirect.com/science/article/pii/S0898589822001280

Highlights

  • L2 legal writers who spoke the same first language and had similar initial proficiency followed distinct developmental paths.
  • L2 writers who improved could maintain a consistent position, integrate supporting evidence, and engage with counterarguments.
  • L2 writers who did not improve had difficulty maintaining a consistent legal voice.
  • For some L2 legal writers, model essays helped them explore and expand the use of Engagement resources.
  • The system of Engagement is a useful tool to understand L2 legal writers’ linguistic choices as they learn a legal genre.

Abstract

This longitudinal case study tracks the development of four second language (L2) writers’ skills in hypothetical legal writing in a year-long legal language program. Drawing on the system of Engagement from systemic functional linguistics, the study analyzes how L2 writers engaged different legal voices and advanced their arguments via three discursive strategies: dialogic expansion, contraction, and justification. An examination of the Engagement resources the writers deployed in 32 essays illustrates their diverse developmental paths and highlights the linguistic choices that reflect the variation in their development. I discuss the influence of initial L2 proficiency and model essays on L2 writers’ trajectories and the distinct challenges these writers faced in maintaining a consistent argumentative position. I argue that the system of Engagement is a useful analytical framework for understanding the linguistic choices L2 legal writers make as they work toward the communicative goals of the target legal genre.

Can ChatGPT help LLMs pass the bar exam?

The good news: Yes, it probably can!

The bad news: But it’s not the LLMs you’re probably thinking of.

I recently noticed in the abstract for the article “GPT Takes the Bar Exam” that the last line reads:

While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.

At first I did a double take and had to re-read the full abstract to understand how in the heck GPT’s relative success in answering bar exam questions could portend that one lucky future LLM student will pass the multiple choice section of the bar exam.

Then I remembered that LLM is different from LL.M. Because in the context of artificial intelligence, LLM means “Large Language Model” which is the term used to encapsulate what ChatGPT is and which is obviously very different than the Master in Laws or Legum Magister meaning which refers to a one-year degree at a law school and which is often associated with international students in US law schools.

This is clearly a distinction that those of us in the legal English field will have to get used to in order to avoid potential confusion in the future. It also suggests that the periods in “LL.M.” may need to come back in fashion for those out there (like me) who have been trying to get away with leaving them out in the name of efficiency.

Here’s the full abstract, in case of interest:

**********************************

GPT Takes the Bar Exam

13 Pages Posted: 31 Dec 2022

Michael James Bommarito

273 Ventures; Licensio, LLC; Bommarito Consulting, LLC; Michigan State College of Law; Stanford Center for Legal Informatics

Daniel Martin Katz

Illinois Tech – Chicago Kent College of Law; Bucerius Center for Legal Technology & Data Science; Stanford CodeX – The Center for Legal Informatics; 273 Ventures

Date Written: December 29, 2022

Abstract

Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as “the Bar Exam,” as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in “AI?” In this research, we document our experimental evaluation of the performance of OpenAI’s text-davinci-003 model, often-referred to as GPT-3.5, on the multistate multiple choice (MBE) section of the exam. While we find no benefit in fine-tuning over GPT-3.5’s zero-shot performance at the scale of our training data, we do find that hyperparameter optimization and prompt engineering positively impacted GPT-3.5’s zero-shot performance. For best prompt and parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam, significantly in excess of the 25% baseline guessing rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5’s ranking of responses is also highly correlated with correctness; its top two and top three choices are correct 71% and 88% of the time, respectively, indicating very strong non-entailment performance. While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.

Keywords: GPT, ChatGPT, Bar Exam, Legal Data, NLP, Legal NLP, Legal Analytics, natural language processing, natural language understanding, evaluation, machine learning, artificial intelligence, artificial intelligence and law

JEL Classification: C45, C55, K49, O33, O30

Suggested Citation:

Bommarito, Michael James and Katz, Daniel Martin, GPT Takes the Bar Exam (December 29, 2022). Available at SSRN: https://ssrn.com/abstract=4314839 or http://dx.doi.org/10.2139/ssrn.4314839

AI/ChatGPT as a tool for Legal English and LLM students


Post by Prof. Stephen Horowitz, Professor of Legal English

As we start to shift past the “wow” factor of AI and ChatGPT (see, e.g., this very cool post from the FCPA Blog posing questions to ChatGPT related to the Foreign Corrupt Practices Act, and also this academic article titled “GPT Takes the Bar Exam“), I’ve seen articles and social media posts and heard comments and commentary focused on the potential plagiaristic dangers of ChatGPT, the artificial intelligence-fueled chatbot that can produce complex, natural-sounding essays in a matter of seconds:

But my initial reaction was less of concern and more along the lines of, “What a great potential legal English tool! How can we use this to help our LLM students learn better?”

And this thinking feels connected to what I’ve read in articles like “AI and the Future of Undergraduate Writing” by Beth McMurtrie in The Chronicle of Higher Education which essentially says that the horse is out of the barn; how are we as teachers and educational institutions going to adapt our assessment methods and how can we use this as a teaching tool. (This is really the underlying point of “The End of High School English” as well.)

Some of my own tests of ChatGPT, by the way, have included:

1) To ask it to “write an essay comparing Marie Antoinette and Rachel Carson,” the idea being to see if it could find connections on two seemingly unrelated people. And it did this quite effectively, acknowledging the lack of connection but finding comparison and contrast in that they were women of different social status who had certain accomplishments. About as good as I could expect from any student given a similar question.

Continue reading “AI/ChatGPT as a tool for Legal English and LLM students”

A different podcast interview with Georgetown Law’s Paula Klammer, legal translator and legal English professor

Post by Prof. Stephen Horowitz, Professor of Legal English

I was fortunate to have the opportunity to interview my colleague Paula Klammer for the Multilingual Lawyer interview series for the USLawEssentials Law & Language podcast, a podcast aimed at helping foreign-educated lawyers and law students improve their legal English.

Here’s a summary of the episode from USLawEssentials:

The USLawEssentials Law & Language podcast continues its series of interviews with multilingual lawyers as Stephen Horowitz interviews Professor Paula Klammer

Paula is a legal English Lecturer & Research Fellow at the Georgetown Center for Legal English. Currently earning her Ph.D. in Law from Universidad de Palermo in Argentina, Paula is an experienced lawyer and translator and is bilingual in Spanish and English. She speaks a few other languages, too, including French and Brazilian Portuguese (but she’s modest and says she’s not proficient yet).

This is a really cool interview with a fascinating guest. Paula’s bilingual background and her work as a translator enable her to provide insights on the special challenges of translating legal English, especially when dealing with false cognates, different writing styles, and very different legal systems.

Link to the episode is in the comments.

And hey – – do you think Spanish people speak faster or slower than most English speakers? Not sure? Got a hunch? You’ll find out.

Paula also discusses her doctoral dissertation so you’re going to learn a lot from this podcast.

Enjoy and let us know what you liked most about this episode.

Want to hear more from Paula? Listen to this May 2022 interview with her on the American Translators Association (ATA) podcast.

Multilingual lawyer perspectives on the LWI’s Biennial Conference at Georgetown Law School

Post by Prof. Paula Klammer, Adjunct Professor of Law

I recently attended the Legal Writing Institute’s Biennial Conference at Georgetown Law School (July 20-23, 2022). As a first-time attendee, I had no idea what to expect. In fact, I wasn’t even sure why I had insisted on going in the first place. I don’t teach legal writing to JD students. Most of my experience teaching has been in my home country, Argentina, to non-native English speaking lawyers either at a tier-one law firm or at a top rated, but relatively small, law school. I was, as we say in my country, “un sapo de otro pozo” (which literally translates as “a frog from another pond”).  

“un sapo de otro pozo”

Although I was happy to find out that I was not the only first-time, first-gen, or rookie attendee, when many a kind colleague warmly pointed out that I was not the only foreigner (there was, after all, one Canadian among us), the irony of that didn’t escape me. So here I am, the new girl in academia from a non-English speaking developing country exposed for the first time to American scholars, the kinds of scholars I’ve long admired from over 5000 miles away. Intimidated and out of place doesn’t begin to describe it… but then the sessions kicked off and things slowly started to change.

There were basically three kinds of sessions: best practices, scholarship, and organization-specific. By best practices I mean sessions in which professors from law schools all over the country talked about the different pedagogical techniques they use in the classroom. Many of them even had the audience experience the exercises themselves so we could get a feel for the technique beyond the purely theoretical. This is a kind of session Americans are famous for and with good reason. They focus on practice rather than theory and give valuable insights into an aspect of American education that I find particularly valuable. I call it teaching people how to fish

As a civil lawyer trained in Argentina, after five years of law school, the first time I saw a case file I didn’t know how to open it. And this is not because there’s anything cognitively wrong with me, but simply because no professor in five years had ever thought to show us a case file or alert us to the fact that the pages are sewn together and new pages go on top, so you don’t open it like an ordinary book. Of course it doesn’t take long to figure that out in practice, but practicing law is stressful enough without anyone ever teaching you how to fish, which is why I value Americans’ emphasis on practical matters in addition to legal theory.

Then there were scholarship sessions. In those sessions, different professors talked about their current research, where they are, their preliminary conclusions, and where they’re headed. I particularly enjoyed two sessions on rhetoric and argumentation but it didn’t take me long to see the limitations of a less multilingual academic culture. Some of the ideas that float around as relatively novel in the United States are not-so-new outside the U.S., particularly in the E.U. where multiculturalism and multilingualism have forced legal scholarship to think about the place of culture and language in the law. So while I loved the work my American colleagues are doing, I couldn’t help but think about how much they could benefit from a more outward-looking perspective. 

With those thoughts going through my mind, I approached a much more seasoned professor who had just given a very interesting talk on culture in legal argumentation. I worked up the courage to talk to this professor about the research coming out of Europe in comparative legal-linguistic studies, where definitions of culture and language are naturally less rigid than what we’re used to on this side of the pond. That led to an incredibly enriching discussion that could only have happened among two people interested in the same topic, where one was fortunate enough to be able to read our French and Italian colleagues and the other receptive enough to want to know more about it. This to me is academia at its best and highlights the importance, not just of dialog among scholars but of the need to translate and publish more foreign scholars in the United States. 

Lastly, there were sessions about the Legal Writing Institute itself, its inner workings and mission. I found those sessions quite interesting in that one can really see how a relatively young area of scholarship is strategically building its own sense of identity. 

All in all, it was a positive and enriching experience. It made me aware of my own strengths and knowledge gaps. But it also left me wondering what the place is for multilingual foreigners in American academia. I’m not aware of any schools outside of Georgetown with a similar program to ours in which linguists, JDs, and LLBs work together to address the specific needs of our international, multilingual, and multicultural LLM students. There’s an obvious gap in American legal education. And, although I was happy to see everything that colleagues in the Global Writing Committee and beyond are doing to address that gap, I wonder what else can be done. 

For one thing, we have a representation and inclusion problem in the global classroom. With rare exceptions, professors are monolingual and monocultural; and although there is nothing wrong with that per se, legal practice is neither monolingual nor restricted to American legal culture for LLMs or the growing constituency of foreign-educated JDs in the U.S. That is where I see a representation problem. International or foreign students aren’t seeing professors who are like them or can relate directly to them in very relevant and important ways. And representation matters. It mattered to me in my first year of law school in Argentina when only 10 of my upcoming 45 professors were women (which I’m proud to say my school reversed while I was a student). And it mattered more than once when I was the only woman in the classroom and was often singled out for it by my professor.  

Prof. Paula Klammer, Georgetown Law

Working as a professor with international Georgetown students over the summer, especially those who, like me, are from developing countries, I’ve seen more than one face light up when I told them I’m from Argentina, worked at a big law firm and found my way to Georgetown non-traditionally. Students need to be able to see themselves in at least some of their professors. More importantly, they need someone who they can see themselves in a few years down the line. Global students experience education differently not just because they come from different backgrounds but because they are exposing themselves to an entirely foreign world. And these differences translate into specific educational needs. 

Inclusion is another story. Inclusion was a big theme at the conference and I was absolutely delighted to meet colleagues doing fantastic work toward a more inclusive classroom. According to the Center for Immigration Studies, “67.3 million residents in the United States now speak a language other than English at home, a number equal to the entire population of France.” With rising numbers of non-native English speakers in JD programs and international LLMs, even more awareness needs to be raised in American legal academia about language inclusion in the classroom; and colleagues who are already making great strides in that direction need our support. 

As for me, the conference started a revolution in my mind. It helped me figure out who I want to be in academia, a question my department director had asked me just a few days earlier and I didn’t quite have a sophisticated answer to yet. As usual, and consistently with everything I ever do in life, I have more questions than answers now. I know for sure that whatever I do will involve global students and turning up the volume of multilingual voices in American legal education. 

Legal English podcast interview with Italian lawyer-linguist Claudia Amato

Post by Prof. Stephen Horowitz, Legal English Lecturer & Adjunct Professor of Law

Claudia Amato, Italian lawyer-linguist

I recently had the privilege of interviewing Italian lawyer-linguist-teacher Claudia Amato, founder of SpeechLex, for the USLawEssentials Law & Language podcast, a podcast intended to help foreign-educated lawyers and law students to improve their legal English.

From USLawEssentials:

“The USLawEssentials Law & Language podcast continues its series of interviews with multilingual lawyers as Stephen Horowitz interviews Claudia Amato.”

“Based in Italy, Claudia is a remarkable attorney, translator, and legal English instructor. Among other things, she is the founder of SpeechLex, where she provides courses in legal English to help prepare attorneys and judges for the TOLES examination. In this episode, she also shares how her experiences working as a translator and teacher inspire her to help others as she explores the “human side” of people’s interaction with the law.
Oh – and as a surprise bonus- you get some helpful travel tips for Japan!”

New sections in the Legal English Resources Page: Vocabulary Resources and Legal English Articles!

Post by Prof. Stephen Horowitz, Legal English Lecturer

I recently added two new sections to the Legal English Resources page: 1) Vocabulary Resources and 2) Legal English Articles. See below.

Continue reading “New sections in the Legal English Resources Page: Vocabulary Resources and Legal English Articles!”

Podcast interview with legal translator Paula Arturo

Post by Prof. Stephen Horowitz, Legal English Lecturer

I was very excited this morning to see that Daniel Sebesta of the American Translators Association (ATA) podcast had done a podcast interview with my Georgetown Legal English colleague, Professor Paula Arturo, about her work and career path as a legal translator. Episode and more info from the ATA website below:

From the ATA website:

“This is not another lawyer-turned-translator story which just goes to show you that there’s more one way to become a legal translator! In this episode of Inside Specialization, lawyer-linguist Paula Arturo tells ATA member Daniel Sebesta about the role passion plays in the decision to become a legal translator and why how much you’re willing to learn is key to becoming one of the best. You’ll also discover why “follow the money” is the secret to choosing a subspecialty, how you can compete against machine translation, and a surprising skill you’ll need to climb this career ladder.Comments? Email podcast@atanet.org.”

An interview with Georgetown Legal English Professor John Terry Dundon

In the latest episode of the USLawEssentials Law & Language podcast’s series on Multilingual Lawyers, I interview my colleague (and friend) Professor John Terry Dundon, an accomplished attorney and linguist, about his passion for language and teaching legal English. We learn about his career path from some of the most premier law firms in the world to teaching an immersive legal English transactional course at Georgetown Law.

Multilingual Lawyer: John Dundon
css.php