Post by Stephen Horowitz, Professor of Legal English. LE Journal is an opportunity to share some of the current goings-on of Georgetown Law’s Legal English Faculty.
Professors Julie Lake and Heather Weger met via Zoom this week with four Ukrainian philologists (i.e, historical linguists) to discuss pedagogical approaches and the use of Chat GPT in Legal English classrooms.
We enjoyed a fruitful conversation about our collective successes and challenges, and we plan to meet again in February to continue the conversation.
The relationship evolved from a larger effort initiated by the Global Legal Skills community back in 2022 to foster connections and collaboration among law and legal English faculty in Ukraine
The September 2023 issue of AL Forum (the applied linguistics newsletter for TESOL) is out, thanks in part to contributions from several members of the Georgetown Law faculty. And the theme is language, teaching, and generative AI.
Co-edited by Georgetown Legal English ProfessorHeather Weger and George Washington Teaching Associate Professor Natalia Dolgova, it leads with a letter from the editors and includes two articles by Georgetown Law colleagues.
“In this issue, you will find leadership updates summarizing past and future ALIS activities, and this issue provides a closer look at how educators are grappling with the impact of generative artificial intelligence (AI) technology, such as ChatGPT, on our field.”
“This article offers some resources and advice to consider as you make informed decisions about integrating AI into your workflows and prepare students to inhabit an AI-rich world.”
“A detailed case study of his experimental use of ChatGPT to design teaching materials for a vocabulary course, [including] examples of how to prompt ChatGPT to generate materials (e.g., quizzes and practice activities.)”
Prof. Weger with Georgetown Law alumni in front of Deoksugung Palace
Summer 2023 was packed with legal English endeavors! A highlight was meeting up with LL.M. and J.D. alumni during a visit to South Korea, generously hosted by Law Center alumnus, Chairman Seung-Hoon Lee .
The opportunity to connect with our multilingual community in a global setting affirms the legacy of the Georgetown experience. I was delighted to share insights about our Two-Year LL.M. program with these colleagues and welcome two of our incoming students.
Back in the U.S., I’ve been excitedly working on the release of the next issue of AL Forum, the Applied Linguistic Newsletter for TESOL, a publication I co-edit with Dr. Natalia Dolgova. This issue will explore the impacts of artificial intelligence (AI), such as the emergent use of ChatGPT, on educational practices; it includes articles from Georgetown colleagues Professor Stephen Horowitz and Technology Specialist Ellery Boatright.
Research also found a way into the summer! Professor Julie Lake and I are collaborating on upcoming conference presentations and publications that focus on integrating asset-based pedagogical practices into Legal English education. As this busy summer wraps up, I look forward to an even busier school year!
Professor Lake had a wonderful summer traveling around the U.S. with her husband and daughter. She spent a week at Cape May, NJ at the beach, a week in Chapel Hill, NC, and a week in Philadelphia, PA.
During her summer she made progress on her personal “language-based” summer project to learn Spanish. Language learning is a lifetime journey!
Professor Lake also spent time working with Professor Weger to revise the language-focused curriculum for Fundamentals of Legal Writing for the 2023-2024 academic year. In Fall 2023, incoming Two-Year students will learn how to use language-based strategies to craft a high-quality memo (i.e., a lawyer-to-lawyer document). In Spring 2024, incoming Two-Year students will learn about the scholarly writing genre and how to write a high-quality mini-scholarly legal research paper.
And finally, Professor Lake enjoyed researching productive ways to use ChatGPT as a learning tool for law and linguistic students.
Professor Dundon began his summer by presenting at the Sixth International Language & Law Conference at the University of Bialystok Faculty of Law in Bialystok, Poland in June (see prior post).
He then taught a summer class, Introduction to U.S. Contract Drafting and Interpretation at IE University Law School in Madrid, Spain, where he has taught for the past three summers.
In July, Professor Dundon presented a paper, When multilingual litigants encounter monolingual ideologies in U.S. judicial opinions, at the Twelfth Bonn Applied Linguistics Conference in Bonn, Germany; he’s been invited back to Bonn to appear as a discussant in a conference focused on legal discourse, taking place in September (more on that in a future post).
He also traveled in Morocco, Uzbekistan and Tajikistan with family, before returning to Georgetown to teach U.S. Legal Research, Analysis, and Writing as part of the LL.M. Summer Experience program.
Professor Dundon ended the summer with a presentation at Georgetown Law’s faculty summer research workshop, where he spoke about his paper “A shifting precipice of unsettled law?”: A survey of how U.S. courts treat expert testimony using forensic stylistics, which was published this month in the International Journal of Speech, Language and the Law.
Prof. Yi Song takes a selfie with Prof. Michael Cedrone and their Georgetown Summer Experience “Foundations for American Law” class.
1. Summer Teaching
Professor Yi Song taught two courses during the Summer Experience – “Foundations for American Law” (co-teaching with Professor Michael Cedrone) and her summer stable, “U.S. Legal Research, Analysis and Writing.”
She’s especially proud of being responsible for the idea of the group assignment for the Foundations class.
In July 2023, 102 international lawyers from around the world walked into McDonough 203 as strangers.
They were asked to form 10 groups to present before class “What I have learned from the Foundations so far.”
In the subsequent 10 days, we saw history reenacted as Marbury v. Madison came alive.
Like most historic events, it all began with a fateful night at the bar.
The foundations of the American legal system were reimagined in the multiverse with the prompt “what if the Founding Fathers were_____?”
An uncanny Professor Cedrone Impersonator? A jury trial, where a top international model found herself in the midst of legal dramas? A tort case that occurred on the premises of Georgetown Law, inspired by the Office-style-behind-the-scene footage?
When Dean Treanor came in one morning for a surprise visit, Prof. Song regretted that she forgot to take a group class selfie with him. But the one she got with Professor Cedrone still came out pretty good.
Master of Laws Interviews Project has come to the classroom this summer. Season 2 is being recorded now. Stay tuned for the Fascinating journeys such as how a lawyer got hired and became the first shareholder with international background in a firm’s 137 year history; How a former star student from legal research and writing class successfully turned her externship into the international associate position at BigLaw. And more!
Prof. Michelle Ueland (standing, far left) with fellow presenters and faculty at Universidad Nacional in Costa Rica.
I was an invited keynote speaker for this conference in Costa Rica (at a branch campus of my MA alma mater, la Universidad Nacional) on August 17 & 18. I gave two presentations (#1 and #2 below) and the closing plenary (#3).
1. “Advancing Listening and Speaking Skills in English for Specific Purposes (ESP) Classrooms”
2. “In Your Voice and In Your Shoes: Experiencing Sanaz Toossi’s Pulitzer-prize-winning play “English”
3. “What’s all the chatter about? Writing educators’ pedagogical responses to generative artificial intelligence (AI) products like ChatGPT-3.5”
I presented at the Global Legal Skills Conference at Nottingham Law School at Nottingham Trent University, England (July 30-Aug 1, 2023). My presentation was titled “Addressing international law students’ pronunciation needs: Best-practices informed by linguistics research and pedagogy.”
Abstract
Although professors notice issues in their international students’ speech, they may not feel equipped to address them. This presentation will cover four research-based, best practices for teaching second language (L2) pronunciation: orienting towards intelligibility, creating task-based lessons, increasing talk-time, and giving feedback.
Many L2 speakers express a desire to “eliminate their accents”, however, accents carry valuable information of our diverse identities and experiences. Teachers can instead help students reorient towards the crucial feature of communication called intelligibility, which asks if the listener received the message the speaker intended to convey. Oral skills can then be addressed through task-based teaching, which focuses on tasks students face (e.g., oral case briefs, negotiations) and guides them through the language necessary to complete them. Third, increasing the amount of productive (versus receptive) interactions in the target language will help students to see progress more rapidly. One suggestion is assigning a video reflection after observing courtroom proceedings. Finally, explicit pronunciation feedback can be a salient tool for progress. Feedback can focus on unintelligible speech, articulation of a sound, and spoken grammar.
These four approaches can be applied in any classroom around the world. Digital access to all teaching materials will be provided.
Shuguftah Quddoos, the first Asian woman to serve as the Sheriff of Nottingham
Also,….
A fun tidbit about the city was that it is the birth place of the Robin Hood lore, and there is actually a real Sheriff of Nottingham position (from what I learned, it’s apparently similar to Mayor).
The current Sheriff is the first Asian woman to hold the position, and we learned from her that all the city buses and trams are electric vehicles too.
GLS was a great small conference, and next year it will be in Bari, Italy!
Georgetown Legal English professors John Dundon and Stephen Horowitz together with National Chengchi University law professor Anna Yan
*Collaborated with a USAID Ukrainian representative to establish online legal English training programs for Ukrainian law faculty starting in Fall 2023. The effort has involved identifying interested US legal English instructors and matching their areas of expertise with the interests of Ukrainian law faculty.
*Set up assessments (pro bono) for female judges from Afghanistan preparing to enter LLM programs at US law schools. The project, which involved collaboration with Prof. Daniel Edelson (Seton Hall/USLawEssentials.com) and Prof. Lindsey Kurtz (Penn State Law), was at the request of an ABA initiative working to mentor and support female Afghan judges.
*Taught a 4-week online Bar Exam Essay Writing for LLM Students course during May/June, in collaboration with Prof. Daniel Edelson. The mission (and experiment) was to make the course accessible to any LLM or non-native English speaking law student, regardless of ability to pay, and it worked well. A second section of the course had to be created to accommodate excessive demand.
*Wrote a soon-to-be published article abou for AL Forum, the Applied Linguistic Newsletter for TESOL, about using ChatGPT to create tax vocabulary practice activities.
*Invited to guest lecture (via Zoom) for a legal English course at Keio University in Tokyo, Japan during the fall 2023 semester.
*Had the opportunity to meet visiting scholar Professor Anna Yan, who teaches law at National Chengchi University in Taiwan and show her the new Legal English faculty offices in McDonough 477.
*Sadly was unable to attend the Global Legal Skills Conference in Nottingham, UK, July 30-Aug 1, which some attendees have shared was really fantastic. But hoping to attend the 2024 GLS Conference in Barri, Italy.
*Fortunate to have had a family vacation in July in a small town (Puerto Morelos) on the Yucatan Peninsula in Mexico, which was a very positive linguistic experience and first time out of the US for my children.
Post by Stephen Horowitz, Professor of Legal English, with special thanks to Prof. Julie Lake and Prof. Heather Weger for their time and linguistics expertise in analyzing and discussing the texts and editing this post, which is far more cohesive because of them.
Hot on the heels of my recent experiment to try and better understand ChatGPT’s view of improving language and grammar (See “Analyzing ChatGPT’s ability as a grammar fixer,” 2/23/23), I was grading my students’ timed midterm exams and noticed a paragraph in one students’ answer that had all the right pieces but decidedly lacked cohesion.
“….the biggest takeaway of all for this experiment…..ChatGPT can help instructors identify the kinds of cohesive devices that a student is not using and then support the student in learning to use and become more comfortable and familiar with those cohesive devices.”
So I mentioned this in a comment and gave some suggestions as to how to improve the cohesion in the paragraph. And then I had a thought:
Joe noted that even though he hasn’t even added sources yet to the draft chapter, he wanted to share in case any of the ideas are helpful to folks exploring using GPT in their classes.
The bad news: But it’s not the LLMs you’re probably thinking of.
I recently noticed in the abstract for the article “GPT Takes the Bar Exam” that the last line reads:
“While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.“
At first I did a double take and had to re-read the full abstract to understand how in the heck GPT’s relative success in answering bar exam questions could portend that one lucky future LLM student will pass the multiple choice section of the bar exam.
Then I remembered that LLM is different from LL.M. Because in the context of artificial intelligence, LLM means “Large Language Model” which is the term used to encapsulate what ChatGPT is and which is obviously very different than the Master in Laws or Legum Magister meaning which refers to a one-year degree at a law school and which is often associated with international students in US law schools.
This is clearly a distinction that those of us in the legal English field will have to get used to in order to avoid potential confusion in the future. It also suggests that the periods in “LL.M.” may need to come back in fashion for those out there (like me) who have been trying to get away with leaving them out in the name of efficiency.
Illinois Tech – Chicago Kent College of Law; Bucerius Center for Legal Technology & Data Science; Stanford CodeX – The Center for Legal Informatics; 273 Ventures
Date Written: December 29, 2022
Abstract
Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as “the Bar Exam,” as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in “AI?” In this research, we document our experimental evaluation of the performance of OpenAI’s text-davinci-003 model, often-referred to as GPT-3.5, on the multistate multiple choice (MBE) section of the exam. While we find no benefit in fine-tuning over GPT-3.5’s zero-shot performance at the scale of our training data, we do find that hyperparameter optimization and prompt engineering positively impacted GPT-3.5’s zero-shot performance. For best prompt and parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam, significantly in excess of the 25% baseline guessing rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5’s ranking of responses is also highly correlated with correctness; its top two and top three choices are correct 71% and 88% of the time, respectively, indicating very strong non-entailment performance. While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.
Keywords: GPT, ChatGPT, Bar Exam, Legal Data, NLP, Legal NLP, Legal Analytics, natural language processing, natural language understanding, evaluation, machine learning, artificial intelligence, artificial intelligence and law
As we start to shift past the “wow” factor of AI and ChatGPT (see, e.g., this very cool post from the FCPA Blog posing questions to ChatGPT related to the Foreign Corrupt Practices Act, and also this academic article titled “GPT Takes the Bar Exam“), I’ve seen articles and social media posts and heard comments and commentary focused on the potential plagiaristic dangers of ChatGPT, the artificial intelligence-fueled chatbot that can produce complex, natural-sounding essays in a matter of seconds:
This post from the Language Log blog about being able to identify essays created by Large Language Models like ChatGPT titled “Detecting LLM-created essays?” (And yes, it does appear that unhelpfully for those of us in the legal English world there is now a new and confusing meaning of LLM! Maybe some of us need to start being more intentional about including those periods in LL.M. 🙂
But my initial reaction was less of concern and more along the lines of, “What a great potential legal English tool! How can we use this to help our LLM students learn better?”
And this thinking feels connected to what I’ve read in articles like “AI and the Future of Undergraduate Writing” by Beth McMurtrie in The Chronicle of Higher Education which essentially says that the horse is out of the barn; how are we as teachers and educational institutions going to adapt our assessment methods and how can we use this as a teaching tool. (This is really the underlying point of “The End of High School English” as well.)
Some of my own tests of ChatGPT, by the way, have included:
1) To ask it to “write an essay comparing Marie Antoinette and Rachel Carson,” the idea being to see if it could find connections on two seemingly unrelated people. And it did this quite effectively, acknowledging the lack of connection but finding comparison and contrast in that they were women of different social status who had certain accomplishments. About as good as I could expect from any student given a similar question.
