Tip of the day: How to use ChatGPT to figure out if a text was written by ChatGPT

Post by Prof. Stephen Horowitz, Professor of Legal English

I had a thought about a week ago that perhaps the best way to check and see if a text written by a student was in fact written by ChatGPT or other AI. And it turns out there is a way to do it, but it’s not the way I thought it might work.

I had already tested ChatGPT by giving it the same final exam fact pattern/question I had given to my LLM students. And ChatGPT was able to provide a pretty decent response.

So next, I took that response and fed it back to ChatGPT asking it if ChatGPT had written this text. But ChatGPT responded by explaining that it has no memory, etc. So I figured that was the end of it.

However, I learned today from a commenter on LinkedIn that, as in law school, it’s really about asking the right questions. And in this case, the right questions is:

“Is this text AI-generated?”

This led to a very definite response that, yes, this text was indeed AI-generated. Problem solved!

Update: I learned after publishing this post that this is not exactly “problem solved.” A colleague at another law school shared that he followed a process similar to mine, but ChatGPT believe that the text was in fact written by a human. So the “tip” is far from foolproof.

But I wasn’t satisfied, so I asked it how it knew. And it very quickly and confidently listed off several factors:

“As an AI language model, I can recognize patterns and characteristics in text that are typical of machine-generated text. In this particular case, the text is characterized by a high level of technical language, a formal tone, and a logical structure that is typical of texts written by AI. In addition, the text does not contain any errors or irregularities that would be expected if it had been written by a human. All of these features suggest that the text was generated by AI.”

Wonderful! But wait a second….Lots of people, including most law students at Georgetown and other law schools, are trained to write with these very same characteristics. So I asked, “Can you be more specific?”

And it essentially listed the same characteristics, but this time in a numbered format which made it appear much more specific and persuasive. Even though it wasn’t.

So then I asked it specifically, “What kinds of differences might distinguish a text written by a highly skilled writer and a text generated by AI?” But it listed qualities that might distinguish a human’s writing from AI, such as style, creativity, context and human touch. As a representative of the human race, I guess I’ll take those as compliments. But it still doesn’t provide any concrete examples as to how it can distinguish between a highly-skilled human writer and an AI app like ChatGPT.

In other words, ChatGPT was essentially borrowing from Supreme Court Justice Potter Stewart who famously said in his decision on obscenity, “I know it when I see it.” (Jacobellis v. Ohio, 378 U.S. 184 (1964))

Thank you to Tashkent State University of Law!

I want to take a moment to thank Senior Teacher Munisa Mirgiyazova and her colleagues and students at Tashkent State University of Law for inviting me to give an online webinar earlier this week on “The Benefits of Extensive Reading and Listening in Studying Law in English.” I greatly enjoyed the discussion and getting to meet everyone, and I look forward to future collaborations.

Reflecting on the presentation afterwards, I also realized that the collaborative opportunity came about because I posted something on LinkedIn (a new podcast episode maybe? I can’t remember now.) And Munisa saw it and sent me a LinkedIn connection request. I accepted and asked her how teaching was going in Tashkent. She replied and asked me how my teaching was going at Georgetown Law, and the subsequent conversation led to a discussion of ways to collaborate. A reminder that it’s often a good idea to leave conversational doors open and ask people about themselves.

I plan to share a recorded version of the presentation on this site after I make some edits to the original presentation and create the recording. Stay tuned!

Here’s the flyer for the event created by Tashkent State University of Law:

Teacher control vs student control in legal English

Stephen Horowitz is the Director of Online Legal English Programs at Georgetown Law.

Giving students more control and more voice makes teaching so much more effective. Yet figuring out how to do it is a process that takes time and effort and does not necessarily come naturally. (At least for me anyway.)

I’m thinking about this topic because of an excellent post on the EAP Essentials blog by Prof. Olwyn Alexander titled “They have to talk and you have to listen: The importance of collaborative conversations in online classrooms.”

The post explains that “Without student talk, the teacher has no immediate way of knowing whether the students understand the materials and tasks and therefore no opportunity to adapt to the in-the-moment needs of the students.”

I think as legal English teachers, most of us intrinsically understand that. We want our students to talk. We want to know what’s going on in their heads. But we’re not always sure how to make that happen. And when it doesn’t happen, it’s easy to shift responsibility to the students, particularly students from certain countries or cultures that teachers perceive as not as talkative in class.

Consequently, I think it’s helpful to be aware of the ways that we, as teachers, get in the way of ourselves and our students. And I think this may be particularly heightened in a legal environment where lawyers and law professors are expected to be sources of knowledge and much of one’s identity as a lawyer or law professor is connected with the ability to share knowledge that others seek or need. In my own experience, this is very true in law schools where the professors–even when they use Socratic method–still often maintain full control of the dialogue and shift frequently from questioner to explainer and knowledge-distributor.

Continue reading “Teacher control vs student control in legal English”

EAP Essentials: “Should we teach grammar? Yes but no but!”

Stephen Horowitz is the Director of Online Legal English Programs at Georgetown Law.

The below blog post from EAP Essentials–“Should we teach grammar? Yes but no but!” by Olwyn Alexander is a thoughtful and healthy reaction to the shift away from “teaching grammar,” which itself has been a reaction to the perceived flaws in the traditional ways of teaching grammar. However, there’s been a shift back towards the teaching of grammar–conditioned on the premise that it’s “done right”–as more thought and research has gone into better ways to help students acquire grammar. 

What the “right” or “best” way to teach grammar is is still up for debate. But overall there is a recognition that grammar is not sufficiently acquired just by exposure (e.g., Krashen and the “natural method”), particularly when it comes to academic English (or legal English for that matter.) Intentional effort and guidance is needed to help learners acquire the grammar they need to communicate effectively at the academic English level.

But from that starting point of recognition, there is still a wide divergence on understanding and belief as to what “done right” ultimately means. I definitely don’t have all the answers. But I do have a few beliefs on the topic:

1. Form should follow function: The grammar that is studied should hue as closely to the content being studied and the communicative needs associated with that content. In this regard, a field like legal English is ideal from a teaching perspective because we have ready-made content and communicative purposes. It’s just a matter of scaffolding the content and then mining it for the grammar needed.

2. Grammar Fluency: It’s not enough just to learn and practice an aspect of grammar. There need to be repeated, natural exposures. And ideally in the regular course of studying the content. It’s hard to contrive natural ways to encounter grammar structures. But it’s a lot easier if you start with the content, work backwards to identify the grammar needs associated with it, and then develop grammar focus and curriculum based on those materials. And that allows for repeated exposures. Additional thought on repeated exposures: One of the advantages kids have is that they like repetition. As evidence, I cite the number of times my kids have watched and sung the songs from “Frozen” and other Disney movies as well as the number of times children like to read the same book over and over. Adults, on the other hand, are prone to getting bored. And that’s significant because motivation is a significant component of language learning. So creativity is key in figuring out how to generate repeated exposures for adult learners.

3. Ear Training: I think this aspect of grammar learning is vastly underrepresented in discussions of how to teach grammar. Especially since so much of grammar comes down to having a sense of what “sounds right.”

As native speakers of English, not only do we spend very little time thinking about the rules of the grammar we use, for the most part we never thought about them when we learned the appropriate grammar. This is particularly true of articles, prepositions, and -s endings (e.g., 3rd person and plurals.)

These are grammar points that so many of our LLM students struggle with. And these also happen to be parts of speech that are harder to hear, especially if your ear is not used to hearing them. In other words, if you can train your ear to hear those sounds, then you’ll hear them more and you’ll develop a sense of what sounds right and start using them more accurately in your own speech and writing. 

There is of course much more to learning and teaching grammar than my above points. But Alexander’s blog post got me thinking about what drives much of my focus and decision-making in teaching grammar to my students, so I thought I would try to add to the conversation. Feel free to share your own thoughts. 

Here are the first few paragraphs of Alexander’s blog post from EAP Essentials along with a “Continue reading” link at the end.

Should we teach grammar? Yes but no but!

Students need grammar but they don’t need grammar classes.

Olwyn Alexander

I was asked recently by a head of pathways programmes at an international college whether we should teach grammar in EAP. This manager was under pressure from some teachers to introduce a more structured approach to teaching and testing grammar. Some years previously, prompted by feedback from an external moderator, they had developed a bespoke grammar workbook, which was ‘aligned with the topics taught in the course, [covering] the language features which are considered to be salient in scholarly English [and targeting] areas where students show weaknesses when it comes to academic writing’. The workbook covers language patterns, such as noun phrases, active and passive voice, conditionals and modal verbs. However, teachers on the programmes have a number of issues with the resource:

  1. There is little time to teach grammar in the course
  2. It feels artificial to teach grammar this way (grammar rules and explanations, followed by practice)
  3. It does not address all issues that students have when it comes to grammar
  4. It’s dry and students do not engage with it

Continue reading