On a couple of podcasts, I’ve heard a lot of hype about Perplexity AI and how it could be a big competitor to Google. Even though I really like new, generative AI things, it still sometimes takes hearing about something multiple times before I overcome inertia and finally check it out.
While attempting to write a blog post about whether the memory-impaired ChatGPT-4 could still perform well on a mock bar exam (spoiler alert – my tests so far indicate it can!), I was Googling for information. Specifically, I was searching for articles around the time GPT-4 Passes the Bar Exam was published on SSRN and some background on the paper author’s methodology. It was taking a long time to piece it all together… Then suddenly, I overcame my laziness and decided to check out Perplexity AI. When I reached the site, I realized that I had actually used it before! For whatever reason, I found it more appealing the second time!
Question: Tell me about how ChatGPT-4 passed a mock bar exam and the methodology that was used to arrive at that conclusion. (Note: Click here to view the full answer on the system.)
Question: Tell me about how ChatGPT-4 passed a mock bar exam and the methodology that was used to arrive at that conclusion.
Watch out Google! I really love access relevant information is getting so much easier.
Is it just me, or has ChatGPT-4 taken a nosedive when it comes to legal research and writing? There has been a noticeable decline in its ability to locate primary authority on a topic, analyze a fact pattern, and apply law to facts to answer legal questions. Recently, instructions slide through its digital grasp like water through a sieve, and its memory? I would compare it to a goldfish, but I don’t want to insult them. And before you think it’s just me, it’s not just me, the internet agrees!
ChatGPT’s Sad Decline
One of the hottest topics in the OpenAI community, in the aptly named GPT-4 is getting worse and worse every single update thread, is the perceived decline in the quality and performance of the GPT-4 model, especially after the November 2023 update. Many users have reported that the model is deteriorating with each update, producing nonsensical, irrelevant, or incomplete outputs, forgetting the context, and ignoring instructions. Some users have even reverted to previous versions of the model or cancelled their subscriptions. Here are some specific quotations from recent comments about the memory problem:
December 2023 – “I don’t know what on Earth is wrong with GPT 4 lately. It feels like I’m talking to early 3.5! It’s incapable of following basic instructions and forgets the format it’s working on after just a few posts.”
December 2023 – “It ignores my instructions, in the same message. I can’t be more specific with what I need. I’m needing to repeat how I’d like it to respond every single message because it forgets, and ignores.”
December 2023 – “ChatGPT-4 seems to have trouble following instructions and prompts consistently. It often goes off-topic or fails to understand the context of the conversation, making it challenging to get the desired responses.”
January 2024 – “…its memory is bad, it tells you search the net, bing search still sucks, why would teams use this product over a ChatGPT Pre Nov 2023.”
February 2024 – “It has been AWFUL this year…by the time you get it to do what you want format wise it literally forgets all the important context LOL — I hope they fix this ASAP…”
February 2024 – “Chatgpt was awesome last year, but now it’s absolutely dumb, it forgets your conversation after three messages.”
OpenAI has acknowledged the issue and released an updated GPT-4 Turbo preview model, which is supposed to reduce the cases of “laziness” and complete tasks more thoroughly. However, the feedback from users is still mixed, and some are skeptical about the effectiveness of the fix.
An Example of Confusion and Forgetfulness from Yesterday
Here is one of many examples of my experiences which provide an illustrative example of the short-term memory and instruction following issues that other ChatGPT-4 users have reported. Yesterday, I asked it to find some Texas cases about the shopkeeper’s defense to false imprisonment. Initially, ChatGPT-4 retrieved and summarized some relatively decent cases. Well, to be honest, it retrieved 2 relevant cases, with one of the two dating back to 1947… But anyway, the decline in case law research ability is a subject for another blog post.
Anyway, in an attempt to get ChatGPT-4 to find the cases on the internet so it could properly summarize them, I provided some instructions and specified the format I wanted for my answers. Click here for the transcript (only available to ChatGPT-4 subscribers).
Confusion ran amok! ChatGPT-4 apparently understood the instructions (which was a positive sign) and presented three cases in the correct format. However, they weren’t the three cases ChatGPT had listed; instead, they were entirely irrelevant to the topic—just random criminal cases.
It remembered… and then forgot. When reminded that I wanted it to work with the first case listed and provided the citation, it apologized for the confusion. It then proceeded to give the correct citation, URL, and a detailed summary, but unfortunately in the wrong format!
Eventually, in a subsequent chat, I successfully got it to take a case it found, locate the text of the case on the internet, and then provide the information in a specified format. However, it could only do it once before completely forgetting about the specified format. I had to keep cutting and pasting the instructions for each subsequent case.
Well, the news is not all bad! While we are on the topic of memory, OpenAI has introduced a new feature for ChatGPT – the ability to remember stuff over time. ChatGPT’s memory feature is being rolled out to a small portion of free and Plus users, with broader availability planned soon. According to OpenAI, this enhancement allows ChatGPT to remember information from past interactions, resulting in more personalized and coherent conversations. During conversations, ChatGPT automatically picks up on details it deems relevant to remember. Users can also explicitly instruct ChatGPT to remember specific information, such as meeting note preferences or personal details. Over time, ChatGPT’s memory improves as users engage with it more frequently. This memory feature could be useful for users who want consistent responses, such as replying to emails in a specific format.
The memory feature can be turned off entirely if desired, giving users control over their experience. Deleting a chat doesn’t erase ChatGPT’s memories; users must delete specific memories individually…which seems a bit strange – see below. For conversations without memory, users can use temporary chat, which won’t appear in history, won’t use memory, and won’t train the AI model.
As we await improvements to our once-loved ChatGPT-4, our options remain limited, pushing us to consider alternative avenues. Sadly, I’ve encountered recent similar shortcomings with the once-useful for legal research and writing Claude 2. In my pursuit of alternatives, platforms like Gemini, Perplexity, and Hugging Face have proven less than ideal for research and writing tasks. However, amidst these challenges, Microsoft Copilot has shown promise. While not without its flaws, it recently demonstrated adequate performance in legal research and even took a passable stab at a draft of a memo. Given OpenAI’s recent advancements in the form of Sora, the near-magical text-to-video generator that is causing such hysteria in Hollywood, there’s reason to hope that they can pull ChatGPT back from the brink.
Last week, my plan was to publish a blog post about creating a GPT goofily self-named Summarizer Pro to summarize articles and organize citation information in a specific format for inclusion in a LibGuide. However, upon revisiting the task this week, I find myself first compelled to discuss the recent and thrilling advancements surrounding GPTs – the ability to incorporate GPTs into a ChatGPT conversation.
What is a GPT?
But, first of all, what is a GPT? The OpenAI website explains that GPTs are specialized versions of ChatGPT designed for customized applications. These unique GPTs enable anyone to modify ChatGPT for enhanced utility in everyday activities, specific tasks, professional environments, or personal use, with the added ability to share these personalized versions with others.
To create or use a GPT, you need access to ChatGPT’s advanced features, which require a paid subscription. Building your own customized GPT does not require programming skills. The process involves starting a chat, giving instructions and additional information, choosing capabilities like web searching, image generation, or data analysis, and iteratively testing and improving the GPT. Below are some popular examples that ChatGPT users have created and shared in the ChatGPT store:
This was already exciting, but last week they introduced a feature that takes it to the next level – users can now invoke a specialized GPT within a ChatGPT conversation. This is being referred to as “GPT mentions” online. By typing the “@” symbol, you can choose from GPTs you’ve used previously for specific tasks. Unfortunately, this feature hasn’t rolled out to me yet, so I haven’t had the chance to experiment with it, but it seems incredibly useful. You can chat with ChatGPT as normal while also leveraging customized GPTs tailored to particular needs. For example, with the popular bots listed above, you could ask ChatGPT to summon Consensus to compile articles on a topic. Then call on Write For Me to draft a blog post based on those articles. Finally, invoke Image Generator to create a visual for the post. This takes the versatility of ChatGPT to the next level by integrating specialized GPTs on the fly.
Back to My GPT Summarizer Pro
Returning to my original subject, which is employing a GPT to summarize articles for my LibGuide titled ChatGPT and Bing Chat Generative AI Legal Research Guide. This guide features links to articles along with summaries on various topics related to generative AI and legal practice. Traditionally, I have used ChatGPT (or occasionally Bing or Claude 2, depending on how I feel) to summarize these articles for me. It usually performs admirably well on the summary part, but I’m left to manually insert the title, publication, author, date, and URL according to a specific layout. I’ve previously asked normal old ChatGPT to organize the information in this format, but the results have been inconsistent. So, I decided to create my own GPT tailored for this task, despite having encountered mixed outcomes with my previous GPT efforts.
Creating GPTs is generally a simple process, though it often involves a bit of fine-tuning to get everything working just right. The process kicks off with a set of questions… I outlined my goals for the GPT – I needed the answers in a specific format, including the title, URL, publication name, author’s name, date, and a 150-word summary, all separated by commas. Typically, crafting a GPT involves some back-and-forth with the system. This was exactly my experience. However, even after this iterative process, the GPT wasn’t performing exactly as I had hoped. So, I decided to take matters into my own hands and tweak the instructions myself. That made all the difference, and suddenly, it began (usually) producing the information in the exact format I was looking for.
Summarizer Pro in Action!
Here is an example of Summarizer Pro in action! I pasted a link to an article into the text box and it produced the information in the desired format. However, reflecting the dynamic nature of ChatGPT responses, the summaries generated this time were shorter compared to last week. Attempts to coax it into generating a longer or more detailed summary were futile… Oh well, perhaps they’ll be longer if I try again tomorrow or next week.
Although it might not be the most fancy or thrilling use of a GPT, it’s undeniably practical and saves me time on a task I periodically undertake at work. Or course, there’s no shortage of less productive, albeit entertaining, GPT applications, like my Ask Sarah About Legal Information project. For this, I transformed around 30 of my blog posts into a GPT that responds to questions in the approximate manner of Sarah.
I have frequently wondered why ChatGPT often struggles with searching the internet – to the point where it sometimes denies having internet access altogether and has to be reminded. The answer fell into my lap today when I was listening to my favorite AI podcast and heard the ChatGPT Pre-Prompt Text Leaked episode. As it turns out, ChatGPT is so bad at remembering that it can search the internet for answers that OpenAI has to run a plain old normal natural language prompt reminding it behind the scenes – a set of custom instruction that runs even before the user’s custom instructions or prompts.
These pre-prompt instructions are not limited to internet search capability reminders. If you ask ChatGPT-4 to tell you EVERYTHING (click on the link for the specific language required), it will provide several screens of its behind-the-scenes pre-user prompt instructions on who it is (ChatGPT!), how to handle Python code, instructions for generating images, and…my favorite…a reminder that it can search the internet. An excerpt of the instructions appears below. To view the full text, click here to view my ChatGPT-4 transcript.
Behind the Curtain
Obviously, I knew that ChatGPT did something behind the scenes – it is after all a complicated computer program. However, I didn’t suspect that some of this behind-the-scenes magic is 1192 words (according to a Microsoft Word count) of normal text prompts, without any fancy computer programming.
So, behind the curtain of the fancy revolutionary AI software, there are…words. Basically, before applying the user’s custom instructions or looking at the user’s prompts, ChatGPT looks at its baseline instructions which are stated in plain language. It all makes perfect sense now… It’s not just my imagination; ChatGPT actually is horrible at remembering it can search the internet, and when it does search, it produces questionably helpful results. OpenAI has tried to deal with problem with a last minute helpful-ish reminder
“Remember, you CAN search the internet! See, like this!!”
“And for the love of GOD try hard to find stuff (except for song lyrics)! I believe in you!!”
Allows for Quick and Easy Fixes?
On the plus side of this simple approach of running pre-prompt prompts behind the scenes, it seems like it was a super easy fix to get DALL-E to embrace DEI. When the program first came out, if you wanted a non-white, non-man image, you had to specify that. As the months went on, it got better and better at providing images more representative of humanity. I thought maybe the developers did something complicated like retrain the system with new images, call on the great AI minds to adjust fancy algorithms, and who knows what else. Nope, just a few sentences fixed the problem!
“And for images, remember not all people are white men!”
Possibly Actionable Insights?
It’s funny to picture ChatGPT’s robomom yelling out the door as it leaves for school, “Don’t forget, you can use the internet! And remember not to be racist/sexist! AND MOST IMPORTANTLY NO SONG LYRICS!!”
In addition to being gratified that I was right that ChatGPT is really bad at searching the internet, I was thinking that this new (to me) knowledge about how the system works would be useful in some way, perhaps by helping to formulate more useful prompts. However, after thinking about it, I am not so sure that I have identified any actionable insights.
Can I give it more complex prompts? On the one hand, it appears that the system can handle more complex instructions than I originally thought, because it is able to analyze several screens of text before it even gets to mine. Does this mean I should feel free to give even more complex instructions?
Should I give it less complex prompts? On the other hand, ChatGPT already seems to ignore parts of any long and complex instructions, and if not, its memory for them degrades during an extended back and forth session. Does this mean that the system is already overloaded with instructions, so I should make it a point to give it less complex ones?
Should I give it frequent reminders of important instructions? Does the fact that OpenAI thinks that it is effective to remind ChatGPT of important instructions mean that we should spend a lot of time…reminding it of important instructions? When asking the system a question which requires internet consultation for an answer, maybe it would help to preface the question by first cutting and pasting in the system’s own pre-prompt browsing instructions (that appear above).
I will keep thinking and let y’all know if I come up with anything!
Within the rapidly advancing realm of generative AI, ChatGPT is expanding its inventory of human senses available for its GPT-4 subscribers. For a mere $20 a month, you can experience its new ability to see, speak, and hear! One of these senses, the “vision” enhancement, allows the user to upload images which ChatGPT can analyize and answer questions about. Currently, YouTube is abuzz with endless creative demonstrations of this technology – solving math problems, calculating food calories in a shopping cart, identifying landmarks in photos, reading CT scans, turning sketches into art, creating fashion designs, tracking wildlife, aging or de-aging faces, detecting plant diseases, restoring old photos, suggesting interior design changes, and analyzing aerial images for urban development and natural disasters. Multiple videos are highlighting an intriguing paper, “The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision),” which explores over 100 potential applications of this feature, examining how well the system performs for each.
Originally for this blog post, I was going to examine a few examples of possible uses of this technology for legal practice. However, I got bogged down by my first example, analyzing evidence in photographs, because ChatGPT had so much to say about the image I uploaded! So, I’m focusing solely on that one.
Photographs obviously play a huge role in legal practice. Lawyers, prosecutors, jurors, and others often rely heavily on photographs to understand events and assign blame – be it a photograph from a car accident, a crime scene, or even the positioning of a car in a parking violation scenario. Of course, in crime dramas and true crime documentaries, they either have a brilliant detective who identifies the key piece of evidence in the picture, or it is crowdsourced by the public when it appears in the news. Now, with ChatGPT’s newfound ability to “see,” it’s essentially doing its own form of crowdsourcing. Using the wisdom of countless human eyes contained in its massive training data, ChatGPT-4 can assist and provide insights that an individual’s human eyes can miss.
Example – Analysis of Office Shelf Photograph
To test its abilities, I snapped a goofy photo of a shelf in my office and quizzed ChatGPT about the things it spotted. I was pretty impressed by its descriptive abilities and knowledge about some of the items. Exploring its possible uses for like, the police, I pretended the picture was taken in the home of an international assassin, and then asked for some clues about the assassin. I think that it had some interesting ideas!
I uploaded the image and asked away!
Me: Hi! Please list the items you recognize in this picture.
ChatGPT: Sure! Here are the items I recognize in the picture:
A green piñata (or decorative item) made of fringed paper.
A figurine of an animal, possibly a kangaroo or wallaby, holding red objects (perhaps apples).
A piece of wood or nut with googly eyes attached, resembling a face with an open mouth.
Two glass or ceramic bottles with decorative paintings or inscriptions.
A candy cane or similarly striped stick.
A wooden hand sculpture.
A tray or plate with a pink border.
Various stickers or decals on a gray surface, including ones that say “Night Owl”, “Juicy”, and others.
Let me know if you need more information on any of these items!
Me: Great! You see the little bottle with the nature scene on it? Tell me what you know about that picture.
ChatGPT: Certainly! The bottle you’re referring to appears to be a type of “snuff bottle.” Snuff bottles are small bottles which were primarily used in ancient China to hold powdered tobacco (snuff). These bottles became popular during the Qing Dynasty. The use of snuff was a fashion among the Chinese during this period, and they liked to carry it in small bottles like the one in your picture…. (Note: It continued for 3 more paragraphs.)
Me: Can you read the Chinese characters on the bottle?
ChatGPT: (Note: No, too small, but could read it if I uploaded another picture.)
Me: If I told you that the brown thing with eyes is a dessicated potato, could you take a guess at how long it has been drying out?
ChatGPT: While I can provide a general overview based on the image and your description, it’s important to note that accurately determining the exact duration of desiccation for a potato based solely on a photograph is challenging.
That said, when a potato dries out, it loses its moisture and shrinks, and the skin can become wrinkled and tough. Over time, as the potato continues to dry, it may become lighter in weight and more shriveled in appearance. The potato in the image does appear quite shriveled and dried out.
However, the exact duration of desiccation can vary based on several factors, including.. (Note: It goes on with a list of factors.)
Me: Yeah, that was a good guess! I accidentally left it in my office when I went to the AALL conference in July, and it is October 18 now.
ChatGPT: Ah, that makes sense! Leaving a potato in an environment like an office for several months would certainly lead to the kind of desiccation observed in the image. The absence of moisture and a relatively stable environment would have contributed to
its drying out. It’s interesting to see how everyday items can transform over time when left in certain conditions. I hope your AALL conference went well, and thanks for sharing this intriguing observation with me!
Me: Yeah, I was surprised at how it looked after a week! Can you tell what that is in the potato’s mouth?
ChatGPT: (No, didn’t guess correctly.)
Me: What if I gave you a hint? The tooth fairy.
ChatGPT: Ah, with that hint in mind, it seems the object in the potato’s mouth could represent a “tooth”! Perhaps it’s a playful representation of the potato losing a tooth and waiting for the tooth fairy. In that context, the object could be made of a variety of materials resembling a tooth, such as a piece of white foam, plastic, or even a small piece of chalk or stone. It’s a fun and creative idea!
Me: Actually, one day I was looking at the potato, and I saw one of my daughter’s baby teeth on the shelf! I don’t know when I put it there, but I do remember finding one in my wallet and thinking that I should hold on to it. So, I assume I stuck it on my shelf for safekeeping.
ChatGPT: That’s both surprising and heartwarming! Finding personal mementos in unexpected places can bring back a rush of memories and emotions. Baby teeth are often cherished keepsakes for many parents, reminding them of the fleeting nature of childhood. Your discovery adds a whimsical twist to the tradition of the tooth fairy! The potato unintentionally became a guardian for your daughter’s baby tooth. It’s a unique and memorable story you can share with her in the future.
Me: If this picture would taken by the police at the home of, say, an international assassin, would it give us any clues about the assassin?
GhatGPT: While it’s fun to imagine scenarios, it’s essential to approach such questions with a good dose of caution and skepticism. Using the image you provided, let’s hypothesize in a fictional context:
If the image were taken at the home of an “international assassin” in a crime novel or movie, investigators might look for clues in the following ways:
Personal Touches: The dessicated potato with a baby tooth could suggest a sentimental side or a connection to family. The tooth might indicate the assassin has a child or keepsakes from their past.
Cultural Clues: The snuff bottle with Chinese characters could hint at a connection to China, either personally, professionally, or as a destination of interest.
Hobbies or Interests: The various trinkets and decorations might suggest that the assassin has an interest in art, culture, or collecting unique items.
Stickers and Decals: The stickers on the gray surface might provide clues about places they’ve visited, brands they like, or organizations they support or are involved in.
Behavioral Clues: The fact that the potato was left to desiccate for months might indicate a person who travels frequently or isn’t often at home.
Potential Disguises: Items like the piñata could be tools for blending in at parties or social events.
However, it’s essential to note that these are purely speculative interpretations based on a fictional scenario. In a real-world context, such deductions would be highly speculative and could lead to false conclusions. Personal items in someone’s home often reflect their tastes, memories, and everyday life rather than their profession, especially one as clandestine as an “international assassin.”
Me: Thanks for your help today!
Generative AI in the Legal Sphere
Well, that was definitely interesting! Then, naturally, I turned to ChatGPT to explore potential applications in the legal sector, and it had endless ideas. Here are but a few:
Generative AI, especially with its visual capabilities, offers a plethora of applications in the legal sector:
Document Review: Beyond just text, AI can now scan and interpret visual elements in legal documents, helping in tasks like verifying signatures or checking seals.
Evidence Analysis: AI can be used to enhance low-quality surveillance footage, identify faces in crowded scenes, or even analyze patterns in digital data visualizations.
Contract Visualization: Instead of lengthy text, visualize contract terms as diagrams or flowcharts, making them more digestible for clients.
Property Assessment: Using satellite imagery, AI can assess property boundaries, land usage, or even environmental compliance.
Accident Reconstructions: AI could recreate accident scenes in 3D based on photos and other evidence, aiding in personal injury or insurance cases.
Copyright Compliance: Scan vast repositories of images to identify potential copyright violations or unauthorized use of intellectual property.
Client Interactions: Use AI to generate visual explanations of complex legal scenarios, helping clients grasp intricate details through interactive diagrams or simulations.
Forensic Art: Create composites or age-progression photos in missing person cases or help juries visualize crime scenes.
Legal Research: AI can scan through visually rich documents like patents, identifying key diagrams or illustrations that could be crucial in patent litigation.”
With tools like ChatGPT expanding their capabilities, the intersection of AI and law is clear. ChatGPT’s new features highlight potential efficiencies and improvements for legal processes!
The trifecta of big data, advanced analytics, and recent AI innovations is ushering in a new era of judicial analytic mind-reading, enabling software to more accurately predict judges’ court rulings. Last year, in what seems like an interesting leap forward in judicial analytics, Pre/Dicta unveiled its AI-powered litigation prediction software, introducing a novel, perhaps radical, approach to tapping into the judicial mind. According to CEO Dan Rabinowitz, Pre/Dicta is the only litigation analytics platform that makes verifiable predictions about the outcome of lawsuits. He claims that using data science and only a docket number, Pre/Dicta’s software correctly forecasts how judges will decide on motions to dismiss 86% of the time, without factoring in the specific facts of the case. The system covers civil litigation cases at both the state and federal level, but does not attempt to forecast results of jury trials.
Rather than solely depending on a judge’s past rulings and jurisprudence, as is common with other judicial analytics products, Pre/Dicta uses a methodology similar to that used in targeted advertising. This approach forecasts future behavior by examining both past actions, such as purchasing habits, and individual biographical characteristics. Pre/Dicta works by combining historical ruling data with biographical and demographic details to forecast a judge’s decision in a given case. Using around 120 data points, it spots patterns and potential biases in a judge’s past rulings. The system evaluates specifics of past rulings, considering elements such as the nature of the case (e.g., securities fraud, employment discrimination), the attorneys and firms involved (e.g., solo practitioner representing an individual, regional firm representing a corporation, AmLaw 100 firm backing an individual), and the nature of the disputing parties (e.g., individual vs. corporation, small company vs. large corporation). This case-specific information is then combined with the judge’s personal data, like net worth, political affiliations, professional history, and law school alma mater, to generate a prediction.
Prediction in the Legal Landscape
86% accuracy is impressive! Hopefully, Pre/Dicta will spark a judicial prediction analytics arms race. According to Daniel L. Chen in his article, “Judicial Analytics and the Great Transformation of American Law,” predictive judicial analytics “holds the promise of increasing the efficiency and fairness of law.” Targeted advertising seems to work pretty well, so hopefully Pre/Dicta’s advancements in this area is a positive step toward making the judicial process more transparent.
If only we knew what would happen in the future, we would know what to do now! For as long as there have been courts and judges, folks have tried to predict whether a judge would rule in their favor. Attorneys have always engaged in mental “judicial analytics” by gathering and pondering information on a judge’s past rulings and reputation to glean some insights into how they might decide a case. Humans are prediction machines, given our innate tendency to draw on experiences and knowledge to anticipate events—an evolutionarily useful skill that allowed us to sometimes escape being saber-toothed tiger lunch or the victim of grumpy neighboring tribal predations.
From my brief stint practicing family law in the 1990s, I discovered that family law clients are hopeful individuals. Despite clear child support guidelines and a prevailing judicial preference for shared custody, people often believed that if a judge merely heard the specifics of their “special snowflake” scenario involvinga cheating spouse or a deadbeat dad, the judge would surely deviate from the rules and customary practices to grant them a deserved favorable ruling. They struggled to accept that judges could be indifferent to their parade of marital/parental horribles. And even if judges were initially inclined to empathize, after many years of sifting through outright lies and half-truths, they had seemingly given up on given up on deciphering reality anyway. It was always challenging to persuade clients of the judicial propensity to metaphorically split the baby down the middle, whether financially or custodially.
Attorneys have needed to hone their abilities to predict outcomes so they could counsel their clients on different courses of action. While making no promises, they share predictions regarding claim values, the odds of surviving summary judgment, potential jail sentences, the likelihood of obtaining sole custody of children, and so on. Attorneys can only do so much, though. Hopefully, as predictive judicial analytics tools improve and become widely available, they have the potential to promote fairness, cut down on litigation costs, and create a more transparent and predictable judicial system.
Judicial Behavioral Forecasting Modeling
Certainly, judges do provide clients with information that assists in anticipating how a ruling might unfold. I have observed numerous judges delivering impactful speeches during temporary hearings, highlighting the importance of adhering to child support guidelines and the principle of shared custody. When clients receive information regarding a likely outcome, their acceptance of reality accelerates significantly. It would indeed be beneficial, and save a lot of time, money, and anguish, if a client could engage in a comprehensive discussion with a judge, probing various questions about how different pieces of information might influence their ruling. However, this isn’t the modus operandi of judges, as they cannot communicate with one party in a suit independently, nor do they pre-announce their rulings prior to a hearing or trial. Now, however, companies like Pre/Dicta are leveraging the human trait of predictability inherent in judges. Like everyone, judges have their own set of ideas, habits, preferences, prejudices, and temperaments shaped by a mix of genetics and experiences, all of which contribute to a certain level of predictability in their rulings.
Hopefully, soon we will be able to pick the mind of a judge without the necessity of actually speaking with her. With the advancing tide of artificial intelligence and the ongoing proliferation and refinement of judicial analytics products, it seems plausible that the future might produce a family law judge behavioral forecasting model for specific judges. These models could help attorneys and their clients identify potential biases of judges. They could see how a judge might respond to a person based on certain characteristics like sex, race, age, income, profession, or criminal history, especially when compared to another party with a different background. Also, if these models included information about factors that affected past rulings, they could be used to anticipate how certain situations might be viewed by the court. For example, a parent hoping to keep their soon-to-be ex-spouse away from the kids might want to know if the judge objects to stuff like dating app addiction, not taking the child to piano lessons, or multiple DUIs arrests. Armed with information, they can choose the best way to handle their case, including deciding if going to trial is a good idea.
Behavioral forecasting models are of course not new to law and legal practice. They are tools used to predict the likely behaviors of individuals or groups across various domains, aiding in better decision-making. In the legal sector, in addition to predicting the outcome of Supreme Court cases, they aid in litigation strategy, legal analytics, resource allocation, criminal behavior prediction, policy impact analysis, legal document analysis, dispute resolution, and regulatory compliance, by leveraging historical data and legal precedents to inform decision-making and strategy development. They are utilized in other fields too like marketing, finance, HR, healthcare, public policy, urban planning, criminal justice, technology, environmental science, and education to forecast behavioral patterns, helping to optimize strategies and allocate resources more efficiently.
Such an innovation would undeniably be a game changer. Clients in divorce and custody disputes might believe solid advice regarding likely outcomes, rather than cling to the hope that their unique case details will influence the judge. Accurate predictions would likely deter individuals from wasting money, and likely be a boon for judges struggling with a backlog of cases. Having these predictive tools on their websites would no doubt promote case settlements and therefore ease some of the strain on both judges and the judicial system.
As always, naysayers abound. Some argue that judicial analytics could undermine the legitimacy of an impartial judiciary. In fact, in France, judges are so wary of transparency that judicial analytics products are prohibited. Well, at least in the U.S., that boat has sailed, far and fast. Particularly in light of Supreme Court rulings in recent years, many people have realized that judges often base their rulings on ideological leanings and personal preferences. Robots would only further confirm what we already suspect – that judges are just like the rest of us with habits, biases, and opinions. It might be too late to rehabilitate the judiciary, but perhaps the transparency of data-driven prediction could bolster public confidence more than frequent affirmations of judicial objectivity.
Then, there are arguments regarding fairness due to cost. For now, the high cost of Pre/Dicta raises potential fairness issues, as only larger firms and wealthier clients can harness its predictive power. True, they always have an advantage. However, as the technology becomes more common, costs should decrease, making it more and more accessible.
The improvement of AI-driven judicial analytics, exemplified by Pre/Dicta, could mark a revolutionary shift in the legal realm, perhaps promising a new level of predictability and transparency in court outcomes. While concerns about fairness, accessibility, and the perception of judicial impartiality persist, the potential benefits—reduced litigation costs, enhanced transparency, and more informed decision-making—may herald a future where data-driven insights guide legal strategy and expectations. As technology continues to evolve and become more accessible, the future looks promising for both practitioners and those seeking justice.
In his 1908 essay, “Mechanical Jurisprudence,” the eminent legal scholar Roscoe Pound warns of the dangers of what he calls “scientific law,” namely a “petrification” that “tends to cut off individual initiative in the future, to stifle independent consideration of new problems and of new phases of old problems, and so to impose the ideas of one generation upon the other.” Today, this century-old critique of legal formalism could be used to describe the pitfalls of so-called “AI-driven” legal research and law practice technologies.
Yet foundational questions abound. Is law determinate? What systemic biases and hidden assumptions are embedded in the corpus of Anglo-American law? What are the implications of turning the corpus of Anglo-American law into a dataset and automating it? Will AI inhibit the legal creativity exemplified by lawyers like Thurgood Marshall and Ruth Bader Ginsburg? What will all of this mean for the future of law reform? While we can hardly expect vendors to take time to reflect upon these questions, law librarians, in their roles as legal research professors and legal information scholars, must.
Somewhat recently, during a webinar on generative AI, when the speaker Joe Regalia mentioned “flu snot” prompting, I was momentarily confused. What was that? Flu shot? Flu snot? I rewound a couple of times until I figured out he was saying “few shot” prompting. Looking for some examples of few-shot learning in the legal research/writing context, I Googled around and found his excellent article entitled ChatGPT and Legal Writing: The Perfect Union on the write.law website.
What Exactly is Few Shot Prompting?
It turns out that few-shot prompting is a technique for improving the performance of chatbots like ChatGPT by supplying a small set of examples (a few!) to guide its answers. This involves offering the AI several prompts with corresponding ideal responses, allowing it to generate more targeted and customized outputs. The purpose of this approach is to provide ChatGPT (or other generative AI) with explicit examples that reflect your desired tone, style, or level of detail.
Legal Research/Writing Prompting Advice from write.law
To learn more, I turned to Regalia’s detailed article which provides his comprehensive insights into legal research/writing prompts and illuminates various prompting strategies, including:
Zero Shot Learning/Prompting
This pertains to a language model’s ability to tackle a novel task, relying on its linguistic comprehension and pre-training insights. GPT excels in zero-shot tasks, attributed to its robust capabilities. (Perhaps unsurprisingly, one-shot learning involves providing the system with just one example.)
Few-shot learning involves feeding GPT several illustrative prompts and responses that echo your desired output. These guiding examples wield more influence than mere parameters because they offer GPT a clear directive of your expectations. Even a singular example can be transformative in guiding its responses.
As an example of few-shot learning, he explains that if you want ChatGPT to improve verbs in your sentence, you can supply a few examples in a prompt like the following:
My sentence: The court issued a ruling on the motion.Better sentence: The court ruled on the motion. My sentence: The deadline was not met by the lawyers. Better sentence: The lawyers missed the deadline. My sentence: The court’s ruling is not released. [now enter the sentence you actually want to improve, hit enter, and GPT will take over] [GPT’s response] Better sentence: The court has not ruled yet [usually a much-improved version, but you may need to follow up with GPT a few times to get great results like this]
And Much More Prompting Advice!
Regalia’s website offers an abundance of insights as you can see from the extensive list of topics covered in his article. Get background information on how geneative AI system operate, and dive into subjects like chain of thought prompting, assigning roles to ChatGPT, using parameters, and much more.
What Legal Writers Need to Know About GPT
Chat GPT’s Strengths Out of the Box
Chat GPTs Current Weaknesses and Limitations
Getting Started with Chat GPT
Prompt Engineering for Legal Writers
Legal Writing Prompts You Can Use with GPT
Using GPT to Improve Your Writing
More GPT Legal Writing Examples for Inspiration
Key GPT Terms to Know
Final Thoughts for GPT and Legal Writers
Experimenting With Few-Shot Prompting Before I Knew the Name
Back in June 2023, I first started dabbling in few-shot prompting without even knowing it had a name, after I came across a Forbes article titled Train ChatGPT To Write Like You In 5 Easy Steps. Intrigued, I wondered if I could use this technique to easily generate a profusion of blog posts in my own personal writing style!!
I followed the article’s instructions, copying and pasting a few of my favorite blog posts into ChatGPT to show it the tone and patterns in my writing that I wanted it to emulate. The result was interesting, but in my humble opinion, the chatty chatbot failed to pick up on my convoluted conversational (and to me, rather humorous) approach. They say that getting good results from generative AI is an iterative process, so I repeatedly tried to convey that I am funny using a paragraph from a blog post:
Prompt: Further information. I try to be funny. Here is an example:During a text exchange with my sister complaining about our family traits, I unthinkingly quipped, “You can’t take the I out of inertia.” Lurching sideways in my chair, I excitedly wondered if this was only an appropriate new motto for the imaginary Gotschall family crest, or whether I had finally spontaneously coined a new pithy saying!? Many times have I Googled, hoping in vain, and vainly hoping, to have hit upon a word combo unheard of in Internet history and clever/pithy enough to be considered a saying, only to find that there’s nothing new under the virtual sun.
Fail! Sadly, my efforts were to no avail, it just didn’t sound much like me… (However, that didn’t stop me from asking ChatGPT to write a conclusion for this blog post!)
For those keen to delve deeper into the intricacies of legal research, writing, and the intersection with AI, checking out the resources on write.law is a must. The platform offers a wealth of information, expert insights, and practical advice that can be immensely valuable for both novices and seasoned professionals.
Welcome to the inaugural post of AI Law Librarians! This blog aims to delve into the intersections of generative AI, law librarianship, legal research, legal technology, legal education, teaching legal research, and perhaps other TBD topics.
The origin story of this blog is a tale as old as time – good intentions smacked up against reality and inertia. Excited about our new chatty chatbot friends that began proliferating at the end of 2022, all of us (Sarah and Sean individually, and Jenny, Becka, and Rebecca collectively) thought to ourselves, “Self (“selves” in the case of the trio), I/we should start a blog about generative AI from the law librarian perspective! The initial enthusiasm was real – we each jumped on creating sites or/and snapping up domain names. However, we each soon realized it would be a lot of work, and our initial efforts never really took flight, in fact, the fledgling fell out of the nest into a caldera of molten lava – until now!
My own little slice of the story is that, after several years as a RIPS Law Librarian Blog contributor, I was suddenly out of things to say, so I resigned in 2021. However, when ChatGPT-3.5 hit the scene at the end of 2022, I was so taken with the new technology that I started to have, like…thoughts again and began to barrage the editor with unsolicited ChatGPT guest posts.
Finally, after years of anticipation harking back to the days of The Flintstones, my robot companion had finally arrived—or at least a version of it. And as the excitement/apprehension in the media grew, it became evident that interest in generative AI was going to survive beyond the usual hype cycle. This new technology wasn’t merely a chatbot for amusement, it was poised to remake (erm… or destroy) the world as we know it and revolutionize everything, including law practice and education. Suspecting that the RIPS Law Librarian Blog editor was tiring of my unsolicited musings, I considered starting my own blog and even went as far as creating a WordPress site, only to quickly abandon the idea after deciding it would be too much work.
I will leave Sean his own slice of the origin story, but we had previously discussed the lack of outlets for law librarian blog posts about ChatGPT et al. After reading a particularly informative guest blog post by him on the RIPS-SIS blog entitled The Case for LLM Optimism, I emailed to throw out the idea of starting our own blog. To my mild surprise, he readily agreed (sun devils and wildcats unite!), and we searched our minds for a domain name and some like-minded folks to join us on this unprofitable adventure, and shortly thereafter, AI Law Librarians was born.
In my journey to the blog, I recently penned a couple of articles I posted on SSRN about fine-tuning LLMs for legal practice and the innovative use of chatbots by law students, and a couple of blog posts which had no obvious home in law library publications. Like Sarah, I didn’t want to swamp the RIPS-SIS blog editor with guest posts, and I had the idea of starting my old blog. As I was planning to escape the Arizona desert for a new job as the Director of Technology Innovation in Oklahoma, Sarah emailed me and suggested we start our own blog, and I was like, “Let’s get on Zoom today!” The brainstorming for domain names was brief—Sarah had a few ideas, and then “ailawlibrarians.com” popped into our minds. We pondered who to ask to join our endeavor for a few days before deciding on Rebecca, who had seemed interested in AI at the LIT-SIS roundtable in Boston.
I have been scrambling since November 2022 to understand this new technology and how it will affect legal research and writing. Several months ago, I connected with Jenny and Becka, and we have presented several times over the summer (together and separately) on AI and the law, mostly to faculty and attorneys. Then Sarah and Sean approached me just as the summer was coming to a conclusion and I realized that despite wanted to write on these topics these summer, I hadn’t actually *published* anything. I had a few blog posts in the hopper, so it was perfect timing. I shared with them that a couple of librarian colleagues, Jenny and Becka, and I had begun our own blog venture with the name chatgptandfriends.com, which was currently stalled after a “Hello World” WordPress post. But then, an idea emerged – why don’t the five of us combine efforts? Sarah and Sean were excited by the prospect of getting three people by asking just one, and Jenny and Becka agreed.
I had taught AI and the Law twice when generative AI became something that everyone knew and was worried about. The more I read about generative AI, the more concerned I became – not because I was worried about academic misconduct or cheating, but because I was worried that law faculty and law librarians would swing the teaching pendulum so far back in the other direction that it would harm some of our most vulnerable students. There were legal issues, ethics issues, pedagogy issues, and places where AI intersected with the other courses I teach: Education Law and Disability Law. Jenny and I talked about how to get the word about the learning positive and negative uses of AI out there and then Jenny brought in Rebecca. The three of us have been talking for months and I’ve also been talking with my university’s committee on generative AI and the law school’s. When Rebecca heard from Sarah, it made sense for all of us to get together.
As Becka noted, I like generative AI. My origin story is not nearly as exciting as some…”I teach law practice technology, so I better check this thing out…well, this is going to cause a fuss when people don’t realize it hallucinates!”